mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-02 16:58:52 +01:00
74ac3ec39b168f534d2c24cef1c340eb208ca4fb
246
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dotta
|
d334f59c98 |
Merge remote-tracking branch 'origin/master' into codex/sleep-status-card-evaluation
* origin/master: fix(evals): support official TestOps job metadata (#18385) fix(config): require shared storage for managed cloud (#18350) feat(skills): make managed content independent of web-server disk (#18334) Add a portable agent question and answer resumption journey (#18346) fix(ci): isolate test selection and require current CI CODEOWNERS (#18376) fix(evals): capture TestOps registration after watch (#18375) fix(runner): allow session reset after confirmed task retirement (#18373) fix(evals): batch task search result readback across renders (#18370) fix(ui): keep task plugin views available for manual access (#18363) test: wait for the evidence expiry worker to settle (#18241) # Conflicts: # server/src/services/company-skills.ts # server/src/services/native-runtime/paperclip-runner-tool-authority.ts |
||
|
DottaandPaperclip
|
4af412eaea |
refactor: drive status-card evaluation from committed work
Move generation sequencing into a feature-owned deadline worker and retain persisted evaluation, eligibility and relative-search obligations. Add transactional producer hints and initial PostgreSQL acceptance coverage; follow-up completes the producer audit and remaining lifecycle gates. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
198218bf55 |
fix(config): require shared storage for managed cloud (#18350)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Managed Cloud instances must retain shared objects and secrets across server replacement. > - Implicit storage defaults can select local disk or an unintended bucket. > - Cloud runtime settings now come from a canonical database snapshot. > - This pull request validates bootstrap prerequisites before database setup and effective S3 settings after snapshot resolution. > - The benefit is an actionable refusal before local signing state or HTTP setup, while valid canonical Cloud settings work. ## Linked Issues or Issue Description **What happened?** A managed Cloud instance could start with local disk storage, an implicit bucket, or a local key file. After canonical Cloud configuration landed, checking only environment or file values would also reject a valid database bucket. Shared schema defaults could make an omitted raw bucket appear explicit. **Expected behavior** Managed Cloud requires external PostgreSQL, a stable environment master key, and effective S3 settings with a nonblank bucket. The canonical snapshot remains the configuration authority. Invalid configuration refuses startup before signing or HTTP effects. Self-hosted defaults remain compatible. **Steps to reproduce** 1. Select managed Cloud with its tenant token or managed configuration document. 2. Leave storage as local disk, or omit the S3 bucket from a raw canonical snapshot. 3. Start the server. Startup must refuse. An explicit valid bucket in the canonical snapshot must work without a file or environment bucket. **Paperclip version or commit** Current base: `4edd10804e2b42941aa2038e90c362c13a0b98dc`. **Deployment mode** Managed Cloud. Ordinary self-hosted defaults remain unchanged. Related PRs: #18335 owns canonical Cloud runtime configuration. #18327 owns S3 run logs. #18352 owns credential persistence. #18347 owns cutover checks. This PR adds configuration prerequisites to those existing boundaries. ## What Changed - Validate external PostgreSQL and a stable environment master key at the Cloud bootstrap boundary, before database setup. - Validate effective S3 settings after canonical snapshot publication and before Cloud signing, runtime consumers, or HTTP setup. - Require a nonblank bucket in the Cloud snapshot schema. Reject omitted or blank buckets in every legacy Cloud file before one-time import normalization, even when an environment bucket is supplied. Keep shared self-hosted schema defaults. - Preserve typed bucket provenance at the existing file-read boundary. Accept explicit canonical bucket names, including `paperclip`, and environment settings when the snapshot is null. - Validate effective legacy storage with the existing resolver before seeding. A rejected candidate leaves no canonical rows, so a corrected source file can be imported on retry. Environment overlays are checked but never persisted. - Add real PostgreSQL snapshot and migration-retry tests, startup refusal tests, and environment/file compatibility controls. Document that Cloud always requires its stable environment key, even with an external default secrets provider. ## Verification - Latest focused acceptance passed 199/199 in six files (54.29 seconds), using the command below plus `server/src/__tests__/cloud-configuration-migration-postgres.test.ts`. Real PostgreSQL proves rejected storage leaves all three migration authority rows absent, then a corrected legacy file imports on retry. An external-default-secrets control proves the deliberate unconditional Cloud master-key contract. - Before the peer correction, the frozen product diff passed 178 tests in five files: managed-cloud-prerequisites, cloud-runtime-config, server-startup-feedback-export, config-file, and cloud-configuration-readers. Command: `pnpm exec vitest run server/src/__tests__/{managed-cloud-prerequisites,cloud-runtime-config,server-startup-feedback-export,config-file,cloud-configuration-readers}.test.ts --maxWorkers=1 --no-file-parallelism`. Real PostgreSQL used a qualified POSIX test-only PG18.6 override because the host cannot supply the stock runtime's SysV semaphore allocation. - Coverage includes raw database and legacy omitted/blank bucket rejection, valid database and environment buckets, null snapshots, explicit default-named buckets, self-hosted compatibility, and refusal before local signing mutation. - The real AWS SDK configured-provider control uses a simulated loopback S3 service. It writes, reads from a fresh provider, and deletes an object. It also reads an encrypted credential with the stable environment key. This does not qualify production cloud permissions. - `git diff --check` passed. Final merges add unrelated master changes from #18241, #18363, and #18370; the focused storage product diff is unchanged. - Peer review reproduced a fresh legacy local-disk import that gained the default bucket before an S3 environment overlay. The importer now requires a raw explicit bucket for every legacy Cloud file. Six controls cover omission/blank values with and without an environment bucket; the full focused rerun passed 184/184 across all five files in 194.89 seconds. - Initial revised startup testing passed 167/168: one fixture omitted the database URL and failed before its intended key assertion. The corrected final run passed 178/178. An initial direct server typecheck failed because Runner declarations were not built; this is a setup failure, not a compile pass. Integration verification rebuilds dependencies before full checks. - Prior head `fae1e4aa` passed complete CI with 53 successes, two intended skips, and Greptile 5/5. `de71345f379cecb71dbfb219f35e14f942c10a3f` then passed complete CI (55 successes, three workflow skips, and a neutral external review), with Greptile 5/5 and zero unresolved threads. The external review identified the preseed rejection problem; its correction is committed at `75e04d0d77e19dcef0a8cb070b6b518f0f173b14`. The final merge head is `19b809ea2823a6a4e5ceb48ef2c2bac4721733d6`; [Fresh complete CI](https://github.com/paperclipai/paperclip/actions/runs/41127309762) passed on that head: 55 successful checks, three workflow skips, and one neutral external review. Greptile reviewed this exact head at 5/5 with no actionable issues. Both earlier external review threads are resolved with the tested fixes. Earlier infrastructure retries remain documented in [the first diagnosis](https://github.com/paperclipai/paperclip/pull/18350#issuecomment-6529964690) and [the runner shutdown evidence](https://github.com/paperclipai/paperclip/pull/18350#issuecomment-6530364769). Earlier missing SDK build and fixture metadata failures were repaired before those passes. - Current frozen-head CI first attempt failed one unchanged Antigravity authentication success case in Runner Vitest 1/2 (`outcome=failure`, 1994 tests passed). The same file passed 11/11 locally in 2.33 seconds, and identical Runner source passed the prior complete CI run. Attempt 2 retried only failed jobs (the Runner shard and dependent aggregate). Both passed without a source change. No second retry was used. The exact timing cause is not established. [Failed job](https://github.com/paperclipai/paperclip/actions/runs/41127309762/job/124309181778). - No production data, buckets, or live credentials were used. Complete local repository suites are not duplicated in this worktree; current-head required CI supplies the full suite gate. ## Risks - Managed deployments with local disk, omitted buckets, file-only keys, or missing external database URLs now refuse startup. Operators must supply valid prerequisites before rollout. - Required deployment environment overlays must remain stable across replacements. For example, a saved local-disk provider with an explicit bucket can use an environment S3 provider; removing that required overlay correctly refuses startup until the environment or canonical configuration is corrected. Environment settings are never silently copied into the canonical row. - Effective legacy storage must also pass before first seeding. A bad candidate does not create authority rows. Existing saved rows remain authoritative and can require environment overrides or a canonical configuration change if they are invalid. - A present legacy Cloud file needs an explicit raw bucket even if its saved provider is local disk or its environment supplies a bucket. Correct the file before one-time import, or use the existing absent-file/null-snapshot environment-only setup. Environment overlays are not persisted as operator choices. - A previously stored canonical nonblank `paperclip` bucket is authoritative. Prior normalization erased whether that value was originally explicit. This change cannot recover that provenance and does not reject existing nonblank canonical values. - Validation proves configuration shape, not cloud permissions, database reachability, or key compatibility with existing encrypted data. Keep the master key stable across replacements. - Cloud snapshot JSON parsing is stricter; omitted or blank raw S3 bucket values need correction. No database migration or provider operation changes are added. These prerequisites do not prove that every server persistence path is disposable. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`. Context window: unknown. The model used reasoning, repository tools, code execution, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
dbff3747b8 |
feat(server): persist run logs and provider traces in S3 (#18327)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators use run logs and restricted provider traces to inspect agent work. > - Local log and trace files depend on the web server's persistent volume. > - That dependency prevents overlapping volume-backed Railway deployments. > - This PR reuses the instance S3 store for batched log and trace payloads with conditional publication. > - Existing Postgres rows retain their contracts; native export recovery reuses events already committed before Runner acknowledgement. > - S3-configured servers automatically copy and verify retained history before a later deployment cutover. ## Linked Issues or Issue Description **Subsystem affected** Server run logs, restricted provider traces, storage providers, and migration tooling. **Problem or motivation** Replacing the web server and its disk must preserve committed log output and retained traces. The current S3 mirror can repeatedly upload a growing file. Moving those payloads into Postgres would add database writes, WAL, replication, and backup work. **Proposed solution** Reuse the existing instance storage configuration, S3-compatible provider and credential chain. Store new run logs under run-logs/ and opted-in raw traces under provider-traces/, below the existing prefix. Cloud keeps its existing bucket and bucket-scoped credentials per stack. Add no new configuration switches. Store immutable payload segments and bounded binary range indexes directly in S3. Publish each batch with an ETag-conditional head. Keep existing database ownership and lifecycle records. Recover canonical native exports from their existing validated control event source, checkpointing export progress in S3. Add no new payload writes to Postgres. Preserve local defaults and migrate settled history with verified byte counts and digests. **Alternatives considered** Postgres payload chunks add control-plane load. Repeated whole-file mirrors amplify transfer work. The first index prototype left superseded index nodes; the final binary index keeps all normally published nodes reachable. **Roadmap alignment** This is a storage prerequisite for replacing the main web/API server. It does not enable deployment overlap, detach a volume, or change native Runner protocol acknowledgements. The maintainer approved this implementation scope. ROADMAP.md has no duplicate implementation item. Related PR: #16770 improves the existing in-flight mirror. This PR preserves that legacy backend and uses the existing instance S3 profile for new journals. ## What Changed - Automatically select S3 segments for new logs and opted-in traces when instance file storage uses S3. Reuse bucket, endpoint, region, prefix, path style and credentials. Remove separate backend switches. Keep local-disk instances local. - Share the existing storage configuration resolver with product files. Preserve config-file settings, environment precedence and the existing root prefix. Keep pre-existing run-log mirror settings for legacy history. - Add bounded buffering, conditional ownership fencing, immutable range reads, stable byte offsets, and completion hashing. - Dispatch all existing log consumers through the recorded backend. Coalesce output progress writes and publish live log messages after storage receipts. - Recover interrupted canonical native exports from their existing durable source. Exclude separate credential, cleanup, and reprojection records. Replay missing callback ranges and deduplicate late delivery. Repair settled archives after controller lease expiry. Bound shared repair work and stop it when all readers cancel. Serve the published prefix when best-effort repair fails. - Add encrypted optional trace channels, replacement reads, durable deletion tombstones, and deadline-driven local/external expiry cleanup independently of heartbeat scheduling. Stream historical source and remote verification. Propagate worker cancellation through trace open, conditional-write probing, upload, restore and reads. Fence deletion even before the first object head exists; abort and drain owned captures. - Retry transient storage errors, distinguish 409 from failed preconditions, and backpressure bounded log admission. Track interrupted trace imports explicitly so ordinary local deletion stays independent of S3. - Automatically migrate retained history through the existing delivery coordinator. Process one log and one trace per batch. Retry unresolved history with backoff and stop scans after completion. Keep the inventory/import command as optional diagnosis and repair. Verify source and remote bytes before switching references. Verify finalized native size and digest before setting its export cursor. Require ordered source coverage when completeness metadata is missing. Preserve original trace expiry. Block active, missing, corrupt, and unproven sources. - Preserve identity and run-secret redaction in live and recovered native logs. Register runtime log redaction secrets once through the existing encrypted registry. Keep agent private keys in their existing owner. - Add storage, migration, retention, UTF-8, and route-privacy coverage, plus operating documentation and qualification tooling. - Add no schema, payload table, per-record SQL, production dependency, or lockfile change. ## Verification - Final fixes at `b3a44c898e` passed all 116 selected startup, retention, attribution and cancellation controls. The disabled-scheduler cleanup and concurrent-repair defects reproduced red before the fixes. A timed-out repair still prevents later storage reads. - The merged source passed 124 storage and real-Postgres Cloud configuration tests. These include automatic history migration, existing environment/file vault keys, source-directory removal, five real-SDK stalled-request cancellation cases, shared capability-probe cancellation and multi-chunk trace ordering. Saved Cloud S3 settings work after the legacy config file is gone. - Workspace typecheck and full build passed at the merged parent `19f702d780`. Server typecheck and build passed again at `b3a44c898e`. - Actual app acceptance: finish a process-agent run through the normal API, stage its retained local-file input, then restart. Automatic migration reports one migrated item and zero blockers. Remove the original web home and restart at `b3a44c898e` with an empty home, the existing S3 profile and the existing vault key file. The normal log API preserves all five producer lines and exactly 788 bytes, with SHA-256 `74d9ae469da28bfe89b9b50a0e132aec24fe9e6d686496ed5e4ffa6167151036`. No log-specific storage settings are present. - Actual expiry acceptance: with heartbeat scheduling disabled, a trace remains unexpired after startup and later reaches its fixture deadline. The worker marks the row deleted and tombstones both S3 channel heads. The retained run log stays byte-identical. The process fixture emits no provider protocol frames, so nonempty encrypted payloads are qualified separately through the real SDK round trips. - Isolation tests use identical run identities in two stack buckets. Cleanup in one existing prefix leaves another prefix, another company and product files intact. Read-only Cloud source inspection confirms a bucket and bucket-scoped credentials per stack. Four existing Cloud storage environment-contract tests passed. No Cloud provisioning changed and no new live vendor permission probe was performed. - Earlier qualification passed 1,014 affected storage/native tests and 27 privacy-route tests with isolated Docker Postgres. An abrupt process test commits 1,000 records, receives receipts, dies by SIGKILL and loses its directory; another process reads every byte and continues. The real Rust Runner experiment acknowledges 32 records while the export cursor is zero, then continues the same Runner PID after controller loss. All 64 records export once with exactly 64 existing control rows. Repeated snapshot/Stop commands completed in 162/81 ms. - Emulator workload: 1,000 burst records used two batch PUTs. Eight steady streams produced 240 records with 104 conditional PUTs and 72 GETs over 6.9 seconds. These are synthetic operation counts, not fleet sizing or provider prices. - Failure evidence is retained. Compact native events over 64 KiB, missing file-key support, disabled-scheduler retention and repair failure handling all reproduced before their fixes. Fixture repairs addressed cold module setup, localhost/IPC permissions, the S3 bucket-name limit and a mode-0700 crash-test directory. Full local `pnpm test:run` was stopped after host PostgreSQL semaphore exhaustion; affected suites used isolated Docker databases instead. Native package library symlinks were restored with the package's standard hydration helper. - The `19f702d780` browser session-recovery case timed out while saving a task title. All eight browser shards passed on `b3a44c898e`, including that case, with no UI behavior or assertion change. Fresh-head CI passed at `b3a44c898e`: 55 successful checks, three skipped checks and one neutral check. The branch is conflict-free. - Map and module-boundary checks passed. Current-head Greptile is 5/5 with zero unresolved review threads. Reproduce the opt-in storage suites with `PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated `PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. See `doc/run-evidence-storage.md` for operating limits, automatic migration, optional diagnostics and rollout qualification. ## Risks - Keep the existing per-stack bucket, prefix and vault key stable across replacements. Relocating that storage is a separate whole-instance operation. - This release automatically changes new log storage for existing S3-configured instances. Local-disk instances remain local. Canonical native source replay is qualified with a deterministic real Runner; actual remote execution and production-provider qualification remain rollout gates. Generic server-only diagnostics and optional raw trace tails without a surviving source can be incomplete after a crash. This does not establish full zero-downtime deployment. - Before deploying this release, qualify the actual S3 provider. It must enforce conditional writes and pass read, version, deletion, workload, and latency qualification. A capability probe rejects providers which ignore preconditions. Moto is emulator evidence only. - Dirty output is bounded and unconfirmed until S3 publication. Storage failure leaves persistence incomplete while task results and usage can settle. - Trace expiry is not yet represented by Cloud's external wake contract. Outstanding trace deadlines conservatively block sleep. - Automatic history copying requires the original source volume. Keep it until all retained resources pass verified import and clean-filesystem reads. Stop old binaries from creating new local-file runs. Missing or corrupt history remains blocked and visible in server logs. Rollback requires a compatible reader against the same external state. Ordinary object deletion does not erase retained historical object versions or backups. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, code execution, and browser verification. The runtime did not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8f822bfcf7 |
fix(cloud): persist configuration and load release-image packages (#18335)
Land the approved Cloud configuration and release-image package stack.
Includes the separately reviewed adapter settings (#18335), external adapters (#18333), bundled plugins (#18336), startup configuration (#18339), and migration/package checks (#18341).
Includes the final image-owned loader and linked-dependency inventory corrections.
Verified stack head:
|
||
|
DottaandPaperclip
|
61bd4ac691 |
feat(agents): set up Antigravity with saved Google subscriptions (#18189)
## Thinking Path > - Paperclip manages agents and their work. > - Users choose providers and managed accounts while adding agents. > - Antigravity was missing from the ordinary additional-agent picker and its saved Google path did not complete native readiness. > - A user needs a successful Test, an enabled task action, and a stable account binding. > - This pull request connects the ordinary setup flow to the native candidate. > - It also preserves retired credential refreshes across database failures and server restarts. ## Linked Issues or Issue Description Built on merged #18188 and #18187. Refs: #12503, #14938 for related legacy CLI proposals. **Agent or provider** Google Antigravity ACP 1.4.3. **Why this adapter is useful** Users need to set up an Antigravity agent and run real work through their managed Google subscription. **How the agent is invoked** Native Paperclip Runner uses the pinned Antigravity ACP runtime. Exact operator admission remains required; the local candidate result does not enable general release. ## What Changed - Show Google Antigravity in the ordinary adapter grid and preserve the draft choice when setup resumes. - Reuse the same user’s saved personal default Google subscription and discover the exact low model through the native account. - Run a prompt-free native Test and wait for authoritative readiness before enabling task assignment. - Carry the managed account through native runs and recover retired refreshes through the existing encrypted, fsynced journal with exact generation checks. - Retain the latest rotated credential through verification, storage, and partial home-removal failures. Retry the exact account’s retained cleanup before another provider use. - Reconcile cleanup before taking the credential mutation lock, fence owners arriving during lock acquisition, and support nested lock holders without a join cycle. - Bound sign-in retirement outside database transactions and preserve claims for the existing cleanup owner; one pending sign-in does not block unrelated accounts. - Cover Gemini and Pi API-key recovery, keep API-key bulk manual-model editing usable, and keep adapter discovery available after candidate expiry. - Skip taskless native timer work before run or credential admission. Preserve current master monitor and deadline scheduling. ## Verification - The registered local saved-Google-subscription journey passed all 16 matchers in 196,520 ms using eval source `12aba` against actual serving source `4a37ee7c`. Both turns stayed in one conversation, produced exact downloaded files of 61 and 119 bytes after reload, and used distinct native primary processes. All observed owners retired, leases released, and credential fences settled. Independent readback confirmed Done work, no active owned runs, a connected saved account, and the retained agent paused for inspection. - The measured runtime is production-equivalent to pre-restack `65ef31dc` except its test-only transport fixture correction. Current-master head `ff7c9249` preserves all 99 product paths: 98 exact blobs and only eight inherited plugin-panel lines in `agents.ts`. Its native binary was rebuilt for inherited Codex-only diagnostic changes; the changed Rust behavior is outside Antigravity's ACPX execution path. Shared native/UI changes have focused regression coverage. The recorded live result is not represented as a measured replay of `ff7c9249`. - Full workspace build and typecheck passed on `ff7c9249`. All 376 focused tests in 14 files passed, covering Google credential/retirement controls and the inherited native-tool, redaction, route, and UI recovery changes. Token gates, maps, and diff checks passed. The required-close fixture was separately demonstrated red, then passed all 317 selected provider-wiring cases. - [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40959387972) is green: 55 successful checks, two skipped workflow checks, and an optional neutral review check. Required `ci / verify` and `ci / e2e` passed. Greptile and Paperclip Review both report 5/5 on exact `ff7c9249`, with zero unresolved review threads. - This qualifies the selected operator-authorized local saved-subscription candidate. The original exploratory journey remains historical evidence; the repeatable fixed-driver eval is in the next PR. ## Risks - Google refresh recovery must preserve the exact grant, account, and credential generation. An uncertain native owner or failed encrypted handoff retains the private home and blocks cleanup. - A successful local saved-subscription candidate does not qualify fresh OAuth, API keys, Cloud, Linux, other models, or general release. - Timer filtering changes only taskless native Runner admissions. Existing task-bound monitors and legacy adapters retain their owners. ## Model Used OpenAI Codex (GPT-6). The host did not expose the exact model revision or context limit. The model used tools for code edits, tests, and review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f1911a4c53 |
perf(skills): move repository packages out of Postgres (#18082)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/205/changes/paperclip/205 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents receive selected company skills before each turn. > - Imported skills can need files from the whole source repository. > - The current path stores those files in Postgres and reads large JSON payloads during preparation. > - This pull request stores verified packages in the configured file or object store and keeps preparation queries small. > - It also converts old rows automatically and restores missing files from the exact saved commit. > - The result reduces database traffic while preserving skill versions and repository dependencies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Company skill import, version storage, and preparation before agent turns. **Current behavior** Repository snapshot bodies live in JSONB. Skill preparation can read those bodies again even when the files already exist on disk. Historical packages and native assets also need a bounded cleanup path. **Proposed behavior** Store package archives and manifests through the existing storage service. Keep version identities, receipts, hashes, and recovery metadata in Postgres. Convert legacy rows after the stored package passes read-back verification. A missing package can use legacy bytes or its exact Git commit under current repository authorization. No separate data-migration command is required. **Reason and benefit** The warm path sends no repository bodies through SQL and reads no object bytes. Cleanup runs through the existing deadline coordinator. Current versions, active runs, explicit pins, and authored content retain their protection. **Breaking changes** The normal application schema upgrade is required. Single-version and skill-file responses keep their existing shape. Version lists, current-version summaries, and test-run version summaries now contain file names and kinds without file bodies. Diff and restore fetch selected versions through the existing single-version endpoint. Older server builds must not write after conversion unless they support the new package format. The storage override and rollback limits are documented in `doc/skill-package-storage.md`. **Additional context** Related work: #15731 preserves repository dependencies; #17923 changes version-cache scheduling; #18080 changes delivery to remote Codex homes. This change owns package persistence, acquisition, and expiry. It does not replace those delivery or scheduler changes. ## What Changed - Add company-scoped archive receipts, exact-commit recovery recipes, migration deadlines, and a durable object-deletion queue. - Add bounded archive upload, read-back verification, extraction, concurrent request coalescing, and verified local caches. - Convert old snapshots and versions automatically. Keep legacy bytes when verification fails. - Recover missing files through current source authorization and the original commit. Validate repository identity, file hashes, modes, and the aggregate hash. - Expire unused recoverable Git packages and native assets after 30 days. Preserve active pins and the durable copy of authored or unproven content. - Keep warm preparation queries metadata-only. Bound file digest caches and avoid repeated body hashing when file identities are unchanged. - Extend the existing storage contract with cancellation. Register package work with the existing coordinator. Optional conversion retries retain the original bytes and may wait for ingress without blocking sleep. - Add regression tests, operating guidance, a deep trace, benchmark receipts, and an optional maintenance command. ## Verification ### Final master integration and review repair — October 29, 2026 Current head: `a6065424249e763dfeac37161f1b52a9f25bd9b7`. Commit `f8bbdd3dbd` resolves the conflicts with master `76a796e964`; the final two commits repair the Apex deletion findings below. - Keep both package-maintenance and version-retention notifications in their write transactions. - Keep the shared run-pin predicate and master's controller-lease deadline metadata. Package expiry reads the updated protected-version result. - Preserve every master migration entry. Generate migration **0415 (`0415_zippy_zemo.sql`)** after master's 0414. Its skill SQL is byte-for-byte unchanged from the previously tested idempotent migration. Retain only the five newest schema snapshots. Migration numbering and safety checks pass. - Repair failed file/folder deletion: preserve a complete temporary subtree until transaction commit, restore on upload or SQL failure, and restore callback failures before releasing the mutation lock. Both PostgreSQL regressions reproduced missing files before the fix and now pass. They cover binary data, executable permissions, empty folders, unlisted files, unchanged database/version rows, successful retry after upload-claim expiry, and historical reads. - Reconcile lost commit replies under the same advisory and row locks. Check the saved version and source identity before restoring. Keep committed deletions intact and preserve backups when the database outcome is unknown or a newer edit exists. Four real-PostgreSQL outcome tests pass; the prior code failed three of them. - Full workspace typecheck and production build pass on the final commit. All **204** focused PostgreSQL package/cache/retention/authoring and filesystem-boundary tests pass, with none skipped. - Apex review **18 passes 5/5**, with zero new comments on the exact final head. All **34** review conversations are resolved. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/40914116508) completes successfully on attempt 2. The head has **55 successful checks**, two neutral reports, two skipped checks, and a successful Snyk status. No checks are pending or failing. All eight browser shards pass. - Runner Vitest 2/2 timed out once in the unchanged ACPX controller-replacement fixture. The same job passed on the preceding commit. One unchanged CI retry passes all five ACPX recovery cases, **2,253** Runner package cases and **126** native server integration cases; their standard one and five skips remain. The initial timeout's root cause is not established. Local diagnosis also hit the fixture's eight-second session.open timeout with an authenticated Runner; no local Runner pass is claimed. - Both full local `pnpm test:run` attempts were stopped when Apex identified deletion defects and source repair became necessary; neither is claimed as a passing full local run. The final **204** focused tests, full workspace typecheck, and production build pass. Full-suite acceptance uses the final-head CI jobs. The superseded merge-head CI included runners terminated by shutdown signals; each source repair started a new full run. - Auto-merge is enabled for squash merge. The final head is conflict-free and all checks pass. GitHub's active master ruleset still requires a code-owner approval for the dependency files. The existing review requests are with `devinfoley`, `nickyleach`, and `forgottendev`; no policy bypass was used. - The earlier hosted stress and full native receipts below remain tied to `a2e597d800`; they are not relabeled as a deployment of this final integration. The earlier local PostgreSQL startup failure remains recorded below; PostgreSQL ran successfully for this integration verification. ### Full native skill execution — October 29, 2026 The full native-agent acceptance gap is closed on candidate `a2e597d800eb989d565780e7aa947d5368676444`. Two qualifying native turns used OpenAI API / `gpt-5.4-mini` in real Daytona sandboxes, on the same disposable hosted TEST tenant as the stress test. A follow-up on the same task consumed a newly saved skill version. The task text asked for the assigned skill workflow. It did not contain the proof nonce or expected receipt. The assigned authored skill contained a Python verifier and a random package-local nonce. The agent read the actual installed skill files and executed that verifier against the imported, automatically converted 1,360-file repository package. It checked the repository file count and reference/binary bytes, and directly executed the repository's executable probe. Independent verification matched the emitted JSON to the nonce and expected byte hashes. The agent persisted the same receipt as its task response; native finalization committed, workspace finalization succeeded, and task state became Done. | Qualifying turn | End-to-end run time | Recorded skills.prepare | Repository files | Binary hex | Shell receipt SHA-256 | |---|---:|---:|---:|---|---| | Initial task | 47.881 s | 1822 ms | 1360 | `00ff07800a` | `a1deb30dc095f2306c7aa9cc5ec396da12c68961cf5b809f4d7c20f7fc68493b` | | Corrected follow-up, new nonce/version | 37.974 s | 377 ms | 1360 | `00ff07800a` | `dbcdc33fa1376b4edc136feae1cdf770560a73292467e93619efdda20e4282a7` | Both successful shell receipts reported `probe: STRESS18082` and `assetSha256: e8ea7a2f15f7bb3b332a46adbef848465884c8996595aa234748befd544d1f3a`. Their proof nonces, selected version IDs and receipt hashes differed, and both ran on the same task. These are observed timings, not latency SLOs; they include real provider work. The service stress-test timings remain a separate measurement boundary. Earlier failures remain in the evidence: - The original five-minute Daytona allocation timeout never reached model execution. After normal owned-stack restore, its durable cleanup record was `expired` / `success`. The cause of that original allocation timeout remains unknown; a later success establishes recovery only. - Initial continuation setup used unsupported generic agent-update fields and returned HTTP 409 before dispatch. Setup was corrected to use the supported skill-assignment and versioned instruction commands. - The intervening completed native turn read repository bytes and ran the probe, but omitted the verifier JSON under the older test agent's overly-specific instruction file. Its strict receipt validator failed; it is not counted as a qualifying turn. Correcting the test agent's instruction file and sending an explicit task-scoped wake produced the qualifying follow-up above. - The pinned application's immutable rate card omits `gpt-5.4-mini`, so completed usage triggered the normal block-on-unpriced budget policy. Supported append-only accounting corrections estimated the three completed runs using the [official OpenAI model rate card](https://developers.openai.com/api/docs/models/gpt-5.4-mini): $0.75 input, $0.075 cached input, $4.50 output per million tokens, assuming standard processing. Estimates totaled 12.482655 cents. Original usage and missing-price records are preserved; the $1 hard stop and unpriced-usage guard stayed active. - Model shell attempts included a missing `xxd` command and incorrect guessed file paths. Those failed calls and subsequent successful actual verifier calls remain visible in complete event streams. No zero-tool-error claim is made. No product source changed during this continuation. The previously measured contention, cold-start and memory limits remain. The later merge resolution is covered by the separate integration checks above; these hosted receipts qualify their recorded candidate. ### Hosted stress test — October 29, 2026 **Hosted acceptance:** full native-agent execution and a changed-skill follow-up pass on the recorded hosted candidate. The final integration section above records the subsequent conflict resolution and its separate checks. Tested baseline `929e840595c8933ca485ac7a2e99413e1e11ca64` and this exact candidate sequentially on one disposable TEST tenant, with external Neon PostgreSQL and actual S3. The fixture contained **1,360 files / 22.4 MB / ten skills sharing one repository**. Measurements cover deployed skill-preparation and native-context services in an adjacent diagnostic process, not a complete agent turn. The container had **24 vCPU / 24 GB**. | Warm workload | Median before → after | p95 before → after | DB returned value bytes/preparation before → after | |---|---:|---:|---:| | One skill, serial | 1,060 → 125 ms | 1,303 → 155 ms | 22,610,389 → 1,771 | | One skill, four concurrent | 4,415 → 297 ms | 5,143 → 328 ms | 22,610,389 → 1,771 | | Ten skills, serial | 5,723 → 535 ms | 6,057 → 600 ms | 224,739,030 → 10,671 | | Ten skills, four concurrent | 24,410 → 1,986 ms | 24,833 → 2,221 ms | 224,739,030 → 10,671 | - Candidate warm preparation performed **zero S3 GETs**. Ordinary-pool queries fell from 16 to 9 for one skill and 61 to 9 for ten, excluding dedicated lock connections. DB numbers measure returned values before Drizzle decoding, not wire bytes. Baseline/candidate sample counts were 30/100, 32/100, 5/50 and 8/48 respectively; the small baseline ten-skill samples are comparative evidence, not SLO estimates. - **Application-UID soak:** 800 ten-skill preparations over 10.3 minutes, four workers paced by 2.5 seconds; **zero failures**, median **559 ms**, p95 **775 ms**, p99 **910 ms**, zero warm S3 reads. All 726 accompanying health/company-list requests returned HTTP 200. Their end-to-end p95 was 569/502 ms, including network latency. The probe overlaps most of the soak. Soak DB-response capture was limited to its first 60 seconds. - **Important remaining costs:** an unpaced 16-way ten-skill burst had p95 **8.32 seconds** because company-level preparation serializes. Cold preparation took approximately **1.1–2.5 seconds** for one skill and **7.4–10.0 seconds** for ten. Ten-skill diagnostic RSS remained about **1.17 GiB** (baseline 1.18 GiB), and the soak peaked at **1.35 GiB**. A concurrent whole-container sample was 3.45 GB. This does not establish suitability for a small-memory host or eliminate per-entrypoint native asset duplication. - **Upgrade without a manual data migration passed:** legacy snapshot JSON shrank from 22,465,691 bytes to `[]`, with a stored package and zero pending conversions. - **Recovery passed:** sixteen concurrent cold requests coalesced into two S3 GETs; full bytes/binary/executable modes matched; same-size corruption repaired; an injected storage failure preserved warm access and failed cold access safely. A disabled public-Git source denied recovery, and after the actual five-minute backoff its exact SHA reconstructed all 229 files. Cross-company descriptor access was denied. Private-Git credential revocation was not tested. - **Retention passed:** reclaimed 27,754,496 allocated cache bytes, deleted an unreferenced package row and both objects, preserved aged authored history, and expired/reacquired reconstructible Git content. A version pinned by the failed native run was correctly retained. Only owned fixture timestamps were aged. - **Browser UI passed:** authored skill creation, saved edit surviving reload, version history, and restoration of earlier content. - **Initial native attempt (superseded by the qualifying turns above):** one real run failed at Daytona allocation after five minutes (`setup_failed`), before any model execution. Provider cleanup also reported failure. That attempt does not count as a passing native turn. Its receipt and all evidence remain preserved; its cleanup later recorded success after normal restoration. - Initial A/B commands ran as SSH root. Diagnostic-owned lock/cache permissions caused one setup HTTP 500 and the first UID-1000 soak to fail; exact fixture paths were repaired. The final soak, retention and recovery ran as application UID 1000. A fresh UID-1000 one-skill batch also passed (30 samples, median 116 ms, p95 321 ms, 1,771 DB value bytes/preparation). Earlier harness-export, SSH, npm provenance, backoff and UI-wait failures remain recorded alongside their successful retries. - Private evidence includes raw timings, assertions, HTTP probes, screenshots and failure/cleanup receipts. Initial failed-campaign cleanup suspended and retained the owned TEST tenant and released the shared QA account lease at **15:22:41 UTC**. The tenant was subsequently resumed under its existing ownership for the native execution qualification above. Final cleanup completed at **15:53:09 UTC**: the account lease is released and the tenant is suspended/retained under the original October 30, 12:24:52 UTC retention deadline. Its separate stack lease remains held; no active or pending-cleanup environment lease remained. The original retention window is deliberately preserved even after successful continuation. - Earlier hosted-tested head: `a2e597d800eb989d565780e7aa947d5368676444`, integrated with master `929e840595`. The earlier merge conflicts were resolved at this head. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40884087629) passed; all checks are complete with 55 successes, two neutral reports, two skips, and no failures. [Apex review 15](https://github.com/paperclipai/paperclip/pull/18082#issuecomment-6480257258) passed 5/5 with zero findings on this exact head. All review conversations are resolved. - Ten real-filesystem boundary cases and server typecheck pass locally for the security fix. Coverage includes cache roots, descendants, individual files, symlink aliases, recursive ancestors, a future concurrent cache fill, and allowed sibling directories. [Current-head private-project integration tests](https://github.com/paperclipai/paperclip/actions/runs/40884087629/job/123483306405) passed all 29 cases, including authorized and unauthorized imports of a cached private package. Full typecheck, build, server suites, all eight browser shards, and Runner checks passed in current-head CI. Local database tests could not run because PostgreSQL could not initialize; this limitation is recorded below. - The earlier generated migration preserved the newest five snapshots. The final integration renumbers its unchanged SQL to 0415. Migration safety/numbering checks pass. Its SQL is byte-for-byte identical to the previously verified idempotent skill migration; repeated application on this latest schema was blocked by the same local database startup failure. - The broader verification before the final migration-only rebase passed 1,025 package, lifecycle, remote-launch, native-session, and asset-retention tests, 52 focused UI tests, full workspace typecheck/build, token gates, and map checks. The linked complete CI run passed before the latest focused fixes. - A disposable MinIO instance passed multipart upload, verified read-back, and cold restoration of an 18 MB archive. - Twenty warm samples with a synthetic 1,349-file, 22.8 MB repository retained five metadata queries and zero object reads. Median preparation was 256 ms for one selected skill and 691 ms for ten, with 5 ms emulated network delay each way. These are local measurements, not production latency guarantees. - No production instance was changed. Earlier verification failures and their disposition are retained below. <details> <summary>Earlier failures, review findings, fixes, and rerun evidence</summary> - The first CI pass found a stale inventory-span expectation and a missing repository-read span. Both are fixed; all three observability tests pass. One browser test copied page text into its YAML parser after reload; its first unchanged-head retry passed, but the race recurred after rebasing. The test now waits for the renamed URL and completed reload before copying the editor, retaining its original assertions. The earlier full local suite did not complete cleanly: it reported setup timeouts and assertions, while isolated relevant reruns passed. The complete current-head CI run closes that broad verification gap. No production instance was changed. - The first Apex review scored 2/5 and identified six issues. The fixes protect authored history even when it shares an in-flight Git upload, recheck upload expiry during orphan deletion, bound failed-upload cleanup, retain per-version kind/encoding metadata, add sanitized failure diagnostics, and correct the migration guide. The final focused run passes all 140 tests; the preceding eight-file run passed all 149 tests, including local storage and S3 diagnostics. Server typecheck and migration safety/numbering checks pass after the fixes. The second Apex review scored 1/5 and found unbounded maintenance storage calls and mismatched package/Git batch limits. Those are fixed with a 30-second cancellable cleanup deadline, shared 100 MiB/10,000-file scanner and storage limits, and bounded Git read batches. The proposed manual lockfile edit was rejected with evidence of the enforced CI-owned lockfile policy. All 19 storage/recovery unit tests and server typechecking pass. The new database-backed cleanup/conversion tests were locally skipped because the host exhausted its PostgreSQL semaphore capacity (confirmed directly by initdb). That gap is closed by current-head CI: [package-store tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109300889) pass all 18 cases and [package-worker tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109301157) pass all 7, with no skips in either suite. The corrected YAML browser shard passes. A separate planning-fixture test failed its cleanup DELETE with an activity-log foreign key violation after its behavior assertions passed; its one unchanged-head retry passed. The third Apex review scored 1/5 and identified a one-slot pool deadlock, projected path limits, cleanup holding the preparation lock during archive reads, and a publication/expiry race. These are fixed with transaction-local reads, prefix-aware bounds, isolated verification, and shared package row locks with a publication grace. The separate Paperclip review also found stale historical inspection, eager history downloads, and conversion retries blocking sleep. Those are fixed. Local PostgreSQL capacity recovered: the new database regressions ran, including the one-slot fork, paused storage during a warm turn, and a locked publication during expiry. The expanded run passed 144/145 tests; its remaining assertion still expected file bodies from the version list and was updated to fetch the selected version. The corrected company-skill suite passes all 81 tests, and the final package/projection run passes all 27. All 52 focused UI tests pass after updating their query-provider fixtures; the first UI run exposed a fixture remount, and the follow-up exposed an editing syntax error, both corrected. Full workspace typecheck and build, token gates, and map checks pass. The sandbox initially blocked the build tool’s local IPC socket; the authorized build rerun passed. Master then claimed migration 0401. The branch is rebased on |
||
|
|
07c6df260b |
fix(server): schedule archived runner evidence from saved deadlines (#17925)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/85/changes/paperclip/85 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Retired native sessions leave continuity and quarantine evidence on disk. > - The old hourly loop mixes evidence retention with backup expiry, which has different database eligibility rules. > - Evidence already has saved creation timestamps and local publication signals. > - This pull request gives evidence its own registered worker and actual next deadline. > - The worker stops filesystem and SQL activity when its evidence obligations settle. ## Linked Issues or Issue Description Stack slice 65. Parent: #17923 (`codex/sleep-64-skill-version-retention`). Related foundation: #15625. **What happened?** At parent `7b94aa185c29673372646bbf2e07a2fbd16bc394`, an hourly backup/evidence tick still inspects native state roots. Evidence's private due time is checked by that recurring tick instead of the shared scheduler. Empty evidence and future retention windows have no reason to be rediscovered every hour. **Expected behavior** Reconstruct evidence once. Wake from the existing publication signal. Register the earliest saved retention deadline. Keep busy, failed, or incomplete evidence recoverable. Preserve backup eligibility as a separate duty. **Steps to reproduce** Start retention with an empty evidence root or archives inside the seven-day window. The old timer still visits the combined owner. The new evidence worker disarms when empty, or registers the saved expiry with no evidence filesystem or SQL calls between relevant changes and that deadline. ## What Changed - Add an evidence worker through the delivery coordinator and shared scheduler. Track startup reconstruction without delaying HTTP readiness. Unsubscribe local publication before delivery shutdown. - Select `backups` for the existing backup timer and `evidence` for the new worker. Existing callers that omit the selection retain combined behavior. Backup expiration remains the next separate migration. - Use hourly retries for concrete busy or failed evidence, with earlier saved expiries and one-second continuations independent. Restore candidate/removal/byte summaries. Schedule publication at the first removable instant; preserve legacy manual-tick semantics. - Expose the existing earliest evidence deadline. Preserve one-second bounded continuation, concrete failure/busy retries, cleanup-source archive exclusions, symlink protection, late activity checks, atomic rename, and interrupted removal recovery. - Preserve non-ENOENT inspection errors as recovery obligations. Retry a failed session root in a new evidence pass instead of continuing past it and forgetting it. - Register the first removable instant after the inclusive retention boundary. Keep the existing manual dry-run publication semantics. - Add real filesystem/isolated PostgreSQL lifecycle tests and disabled/held-startup wiring cases. Add no table or migration. The archive and quarantine writers already call `noteNativeRunnerEvidence`; this PR connects that existing signal to the scheduler. Shared code owns scheduling, while evidence rules stay with the retention service. See [`doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md`](doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md). ## Verification - Adopted actual prerequisite squash `b06ea111a254c62e0a55199b010d82e0adb33bf0` from #17923 through normal merges. Qualified runtime/package/UI/map trees remain unchanged. The final nine-path diff includes one browser-spec readiness repair. - Runtime qualification remains **652 passing TypeScript cases across 22 complete files**, plus **29 passing Rust cases** in native-backend, runtime-notification and configured-environment targets. One actual closed Pi snapshot launch is intentionally skipped because its external distribution root is not provided. Accepted source is `b80eee32f344ae0218d8b791c14f2ea174da09cc`; all runtime and earlier sleep-test bytes are preserved. - The complete outage-recovery browser file passes **all five journeys** on a fresh isolated database after four additive rendered-history readiness assertions. Every original assertion, outage control and timeout remains byte-identical when those new lines are removed. The intentionally unresolved first-session scenario keeps its pre-auth interaction and delayed-owner coverage. The prior CI/local timeout evidence and one rejected broader precondition remain in the audit. - Adapter/Runner/database builds, server/Runner typechecking, Rust formatting/workspace checks, server build, module boundaries, migration order and generated TypeScript/Rust profiles pass. PostgreSQL 17 fixtures apply all 413 journal entries through index 414; source hashes and temporary fixture imports restore exactly. - **Inherited validation exception:** Copilot profile 81's separate source-identity check fails on the merged Rust configured-environment hash. Its inputs and attestations equal upstream and stay unchanged here. Grok's identity check passes. No release attestation is regenerated or new live-provider qualification claimed. Required exact-head CI/security and Greptile 5/5 still must pass. - [The evidence audit](doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md) retains historical failures, counts and limitations. No full local repository-suite, live provider, external-alarm, measured suspension or fleet-savings claim. Fresh final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - Live publication is the existing server-owned executor in one process. Independent filesystem writers need an explicit signal contract. Startup reconstructs restart state. - Busy sessions and concrete inspection/removal failures retain retries. Unknown cleanup-source archives remain protected. - The existing scan enumerates directory names before applying its entry/removal budget. This PR preserves that behavior and does not claim a bounded total directory-listing allocation. - An in-process deadline cannot wake stopped compute. Cloud sleep-safety and external-alarm coverage stay unchanged. Physical evidence reclamation can wait for a supported host wake. - Backup expiration still has an hourly database eligibility scan. This slice removes only the archived-evidence duty from that loop and does not claim full instance suspension. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs and complete PostgreSQL/filesystem/startup suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
b06ea111a2 |
fix(server): schedule skill cache retention from changes and saved deadlines (#17923)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/83/changes/paperclip/83 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server caches saved skill versions on disk so runs can restore their exact instructions. > - The old retention owner scans company rows every hour, even after all caches are protected or removed. > - Cache publication, version changes, and pin release already have local or transaction-owned boundaries. > - Grace windows and terminal controller leases already have saved timestamps. > - This pull request uses those changes and deadlines to drive the existing bounded prune service. > - Idle retention stops SQL and directory reads while pending cleanup remains recoverable. ## Linked Issues or Issue Description Stack slice 64. Parent: #17876 (`codex/sleep-63-agent-timer-deadlines`). Related: #15625 establishes transaction-owned work signals. #17907 independently keeps an active cache-lock transaction alive; this PR preserves that lock and does not duplicate that repair. **What happened?** At parent `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`, skill-version retention has an hourly company scan, a startup timeout, and separate continuation timers. Empty or permanently protected caches still cause recurring SQL. **Expected behavior** Reconstruct once, inspect changed companies, and schedule saved access/staging grace or terminal controller deadlines. Stop recurring work when no timed or uncertain obligation remains. Keep current/newest/agent/run protection, restorable-byte validation, cache locking, final pin revalidation, atomic rename, and trash recovery. **Steps to reproduce** Start an enabled server with no cached versions, or only current/pinned versions. After the initial scan, advance an hour. The old interval selects companies again. The new worker has no deadline and performs no retention SQL or directory scan in that state. ## What Changed - Signal the missing-local-source reconciliation deletion in its transaction. Limit fresh scans of a successfully observed company to one per thirty seconds. Keep one-second page continuations and actual saved deadlines independent. Empty or uncertain passes do not delay first publication. - Restart a company scan if drain interrupts inspection or post-rename trash cleanup. Restore bounded candidate/byte logging with dry-run mode. Three controlled cases failed before repair and the complete 34-case retention suite passes afterward. - Register retention through the delivery coordinator and shared scheduler. Remove the hourly, startup, and continuation timers from `index.ts`. Track startup work without delaying readiness. Stop delivery work before closing retained iterators. - Keep one active company cursor and bounded 50-entry passes. Preserve observed deadlines across pages and bound company discovery to 51 rows per page. Retain concrete company failures and incomplete pins for retry. - Report saved grace and terminal controller deadlines from the existing cache owner. PostgreSQL remains the controller-lease authority. Current/newest/agent pins do not arm unrelated lease checks. - Wake on cache publication, rejected materialization, and access markers. Signal version creation/deletion and agent configuration/hire changes in their owning transactions. Extend existing instruction, finalization/environment, and retained-session fanouts with company hints. - Route reused provider-task prior-owner retirement through `writeEnvironmentCleanup`, so that supported pin-release write also settles a notification. - Add an optional scope callback to `registerDeadlineWork`. Existing consumers omit it and keep their behavior. The feature owns persistence predicates; startup only registers it. Add no table or migration. Full producer audit, shared-contract explanation, lifecycle evidence, and limitations: [`doc/plans/2026-10-27-skill-version-retention-sleep-audit.md`](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md). ## Verification - Reconciled all existing sleep changes with actual master `7e27bc5007c8f9cb8cdce06c24f9813e27f7120f` through normal merges. All 24 retention-owned and 95 unrelated upstream paths are preserved; package, UI, E2E and map trees equal master. - Combined-source qualification: **1,009 TypeScript cases across 42 complete files**, plus **28 Rust cases** in two targets. One intentional Linux-inotify-only adapter case is skipped on this macOS host. Coverage includes actual retention/cache/producer/runtime behavior, startup, shared deadline consumers, changed OpenCode/observability and native execution boundaries. - After the subsequent task-index/task-read upstream merges, **85 cases across six complete files** passed on final source `d9277b1858a2230bdb726974c90a0c9bc5c5ebe6`: real retention/cache/work, task reads/query behavior and concurrent migration repair. Counts overlap earlier coverage. Fresh PostgreSQL 17 schemas apply all 413 journal entries through index 414. - Database/adapter/Runner builds, production server and Runner typechecking, module boundaries, target migration order and generated TypeScript/Rust release-profile consistency pass. Selected source hashes and temporary fixture imports restore exactly. - Reproduced and repaired Greptile's own-company due-cleanup P2 with identical old-worker-failing/new-worker-passing assertions. The 30-second ordinary scan floor remains; already-due cleanup receives its owed scan and disarms afterward. Earlier 799-case and fresh-blank-plugin acceptance remain in the audit. - **Inherited validation exception:** the separate Copilot source-identity check fails after upstream OpenCode reserved-environment changes; all 72 identity inputs equal current master. No provider release attestation is regenerated or new live qualification claimed here. Required exact-head CI/security and Greptile 5/5 still must pass. - Historical results, repair details and limitations remain in [the retention audit](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md). No full local repository-suite, live provider, production-load, external-alarm, measured suspend/wake or fleet-savings claim. - CI on the prior published head found four connection-review assertions racing the concealed conversation after reload. Its fresh Greptile/Ponytail reviews were both 5/5, but the failing browser gate was not waived or retried unchanged. The fixture now foregrounds the task tab and waits for visible history before retaining all existing saved-result assertions. The full file passes four journeys locally; the controlled-restart case is intentionally skipped by the default configuration. Original local journeys also passed unchanged, so no deterministic local red reproduction is claimed. See the audit for CI failure artifacts and scope. Runtime and prior sleep-test sources remain unchanged. ## Risks - Supported live writers are the existing service/API owners in one process. Independent SQL/FS writers need an ingress contract. Startup reconstruction covers restarts. - Notifications cause a bounded inspection; they do not authorize deletion. Durable pin predicates and the cache lock remain authoritative. - A database clock behind application time can require another one-second check of the concrete expired controller lease. Unknown pins, lock contention, and failed inspection retain retries until resolved. - The in-process scheduler cannot wake stopped compute. This PR preserves Cloud sleep-safety gates and external-alarm coverage. It does not prove deployed suspension or fleet savings. Cache reclamation can wait until a supported wake when compute is stopped. - Native-harness backup retention and the other remaining duties are separate stack slices. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs, a controlled red/green regression and complete PostgreSQL/filesystem integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
ba07f85cad |
perf: schedule agent timers from durable deadlines (#17876)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/58/changes/paperclip/58 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent timers must start due work and preserve queue admission rules. > - The server woke their scan on every heartbeat interval, even with no eligible timer. > - Idle scans keep the database and instance active. > - This pull request derives the next deadline from saved policy and work records. > - Supported producer transactions notify the shared coordinator when eligibility changes. > - An idle timer worker stops scanning and resumes when real work or a deadline requires it. ## Linked Issues or Issue Description Refs #17863. This is slice 63 of the recurring-work migration and follows the merged issue-monitor deadline PR. Related: #16632 changes timer issue scope; #16881 changes live-run eligibility. This PR preserves the current admission policy. Those policy changes are separate work. A future time-sensitive predicate must expose its next transition deadline. **What existing behavior does this improve?** Agent timer heartbeat scheduling in the server. **Current behavior** A recurring heartbeat wakes the timer scan even when companies are paused, timers are disabled, or no actionable task exists. **Proposed behavior** Register the earliest eligible interval or positive daily-cap reset deadline. Re-read on supported producer commits and startup. Leave no recurring database scan when no timer obligation remains. ## What Changed - Keep policy and deadline calculation in the heartbeat queue. Scheduling retains company, reporting-chain, worktree, and CAS gates. - Use the existing deadline coordinator. Remove only the timer wake from the server heartbeat loop. - Notify before existing transaction writes that change eligibility or deadlines. Cover configuration, lifecycle, company budgets, task ownership, recovery, conversations, accounting corrections, and completion. - Use the lifecycle status-change hook for saved hire rejection. Cover checkout's second adoption transaction after an initial failed CAS. - Preserve pending work through idle and maintenance drains. Check cancellation between due agents. Reconstruct on restart. - Document the supported writer model and producer inventory. Add no schema, table, or provider branch. ## Verification - **1,710 passing cases across 49 complete files**, no skips, on accepted source `32db86c1974dcbf794f01a8c5e42024e2ad071fc`. Coverage includes real timer policy/CAS/producers, monitor and shared deadline consumers, native reconstruction, Slack lifecycle, skills/settings/configuration, queue isolation and agent lifecycle. - Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. Twenty-nine fresh PostgreSQL 17 schemas plus per-file fixtures applied all 412 journal entries through index 413; all selected source hashes and temporary fixture imports were restored exactly. - Four new deadline composition cases verify future/overdue timing, minimum scan spacing and disarming. A controlled mutation dropping only the inherited continuation option fails; the composed source passes with identical assertions and timers. - Normal parent reconciliation preserves task-drain expiry and both deadline contracts. The final monitor-parent publication changes only its audit, preserving every qualified runtime and selected test source. - Historical failures and review corrections remain in [the timer audit](doc/idle-agent-timers.md). No full local repository-suite, live provider, measured suspend/wake, external-alarm or fleet-savings claim. Final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - Process-local notifications cover supported service/API writers in one server process. Direct SQL and independent replicas need an explicit ingress contract. Startup reconstruction is not live cross-process notification. - Notifications can batch for up to the configured heartbeat interval. Known deadlines retain their scheduler path; failed reconciliation retains bounded retry. - Enabled timers without an actionable-work requirement remain real recurring obligations. Zero caps require policy changes; positive reached caps use UTC midnight. - The timer fixture substitutes adapter enqueue. Existing admission tests also pass, but this is not deployed provider or sleep qualification. - Cloud does not gain legacy agent timer alarms in this PR. Existing sleep-safety blockers remain. No database/host suspension or fleet savings claim is made. - Parent-stack CI and review must pass before merge. This PR does not authorize merging its dependencies. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs, a controlled red/green regression and complete PostgreSQL integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
0f6f0a3fee |
fix(server): schedule issue monitors without heartbeat polling (#17863)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/56/changes/paperclip/56 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Issue monitors save a future check on an existing issue. > - The heartbeat timer still scans monitors on every tick, even when none are due. > - The saved deadline and claim already provide durable scheduling state. > - This pull request gives monitors their own deadline worker and connects state changes to it. > - Idle monitor work stops querying. Existing admission and recovery rules remain in force. ## Linked Issues or Issue Description Refs #17840, #16325. Related: #17171 addresses rearming a monitor after its admitted run fails. This PR changes when monitor inspection runs and follows the merged routine prerequisite. **What existing behavior does this improve?** Issue-monitor scheduling and idle database use. **Current behavior** Heartbeat ticks call the combined agent and monitor sweep. The existing next-deadline query only retains future monitors. Polling covers overdue leftovers, stale claims, and changes that make a saved monitor eligible. **Proposed behavior** Use the shared deadline worker. Retain overdue work with bounded retries. Wake after committed changes to monitor state, issue eligibility, company state, or native ownership. Keep legacy agent ticks separate for the next slice. ## What Changed - Add monitor-specific sweep and deadline methods to the existing scheduling owner. Keep the manual combined method. - Include overdue work and the strict five-minute claim fence. Exclude active native owners until their state changes. - Register monitors with the shared deadline worker. Heartbeat ticks wake only the legacy agent timer worker. - Connect direct issue-state writers and existing execution-control notifications to monitor work. Checkout uses a short transaction around its existing CAS. Provider work stays outside it. - Add real database lifecycle tests and retain routine, startup, admission, execution-control, and native-projection coverage. - Record the obligation, producer inventory, recovery behavior, and limits in `doc/plans/2026-10-27-issue-monitor-sleep-audit.md`. ## Verification - **1,233 passing cases in 35 complete files**, no skips, on accepted source `c24aadfc91e9394d73c799f9e05fc67202160557`. Coverage includes real monitor scheduling/admission, native settlement, committed producers, retained routines and all shared deadline consumer integration suites. - Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. Twenty-four fresh PostgreSQL 17 schemas plus per-file fixtures applied all 412 journal entries through index 413; every selected source hash and temporary fixture import was restored exactly. - Normal parent reconciliation preserves master's admitted-wake result and scoped issue deletion while keeping committed monitor notifications. All other paths match the automatic merge; packages remain byte-exact to the qualified parent. The final routine-parent publication changes only its audit. - Historical failures and corrections remain in [the monitor audit](doc/plans/2026-10-27-issue-monitor-sleep-audit.md). Old delegated-Slack rate and Runner drain CI failures are retained; the current inherited clock fixture and package source are byte-identical to merged #17835. Fresh final-head CI remains required. - No full local repository-suite, live provider, deployed suspend/wake, external-alarm, production-load or fleet-savings claim. Final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - A missing producer notification can strand a monitor after polling is removed. The audit covers ordinary writes, direct status/assignment writers, recovery, and native ownership changes. - The writer model is process-local. Independent SQL writers and other replicas need a supported notification or reconciliation contract. - Overdue work uses the configured heartbeat interval as a retry floor for the monitor worker. New notifications during that backoff wait for its next pass. Normal state-change bursts coalesce for one second. - Notification fanout can inspect monitors during other active work. An unchanged idle subsystem has no recurring monitor scan. - The legacy agent timer still polls. Cloud sleep proofs and external alarm coverage are unchanged. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, ordinary Git merges, composition proofs, PostgreSQL fixtures and complete local integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
929e840595 |
fix(server): schedule routines without heartbeat polling (#17840)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/47/changes/paperclip/47 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Routines create work at saved cron deadlines. > - The heartbeat still wakes the routine worker when nothing is due. > - Its old next-deadline query also drops overdue work left by capped catch-up. > - Removing the heartbeat alone can strand that work and miss company reactivation. > - This pull request uses saved deadlines and committed producer changes. > - Idle routine scheduling stops querying until a change, retry, or real deadline. ## Linked Issues or Issue Description Refs #16514. Merged prerequisite: #17836. Related Cloud wake contract: #16215. **What happened?** Every heartbeat wakes the routine scheduler. The next-deadline query excludes overdue triggers. A 27-tick hourly backlog loses its deadline after the first 25-run pass. Company and budget lifecycle changes also lack routine schedule signals. **Expected behavior** Use the actual next saved deadline. Retain overdue work. Refresh after supported producer transactions commit. Keep idle schedules quiet between obligations. **Steps to reproduce** 1. Create an hourly routine with capped catch-up and 27 overdue ticks. 2. Run the registered worker once. The base processes 25 ticks and reports no next wake. 3. Create a future schedule and pause or reactivate its company. The base does not refresh its registered deadline until another wake. ## What Changed - Remove only the routine worker wake from the heartbeat tick. - Put routine eligibility and the next-deadline read in `routineService`. Include overdue triggers. Keep trigger/routine company scope consistent and require an active company. - Reuse `registerDeadlineWork` for retries, request generations, and passive drain behavior. Narrow its coordinator type to the `register` method it uses. Existing consumers keep the same runtime behavior. - Signal routine work from manual schedule invocation, company pause/resume/archive/delete, and budget-induced company pause/resume. Existing routine and trigger producers retain their transaction signals. - Stop between trigger batches during drain/shutdown. Finish a claimed batch so its advanced timestamp does not discard admitted work. - Add real PostgreSQL lifecycle tests and extend startup/query fixtures. Add a validated external database option to the company suite, including migrations and client closure. - Record producer coverage and limits in `doc/plans/2026-10-27-routine-schedule-sleep.md`. ## Verification - **417 passing unique cases across 14 complete files**, no skips, on accepted source `4659ef76d398193b15e7812d4a97f3da3b743226`. Coverage includes the real deadline worker, database eligibility queries, routine services/HTTP/routes/plugins, budget/company producers, git/device fixtures, startup and shared coordinator. - The first combined run passed 416 cases and had one UUID HTTP authorization timeout at its unchanged 20-second limit. One fresh complete 13-case E2E diagnostic passed (the affected case took 425 ms). No assertions, timeouts or production source were changed; the initial timeout's cause is not established. - Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. Fresh PostgreSQL 17 fixtures applied all 412 journal entries through index 413; all selected test hashes and temporary fixture imports were restored exactly. - Normal parent merges preserve the tested production and test source. Historical acceptance and review corrections remain in [the routine audit](doc/plans/2026-10-27-routine-schedule-sleep.md). - No full local repository-suite, deployed suspend/wake, external alarm, live provider or fleet-cost claim. Final-head CI, security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - Company pause now prevents due routine claims, matching the prior deadline query. Rows remain saved for reactivation. Existing routine reactivation, project-pause skip records, cron, timezone, catch-up, and concurrency policies remain with the routine owner. - Signals cover supported service writers in one process. Independent SQL writers and other replicas need their own integration. - This proves worker quiet, not host/database suspension or fleet savings. Cloud idle proof and external wake coverage are unchanged. - Issue-monitor and legacy-agent-timer polling remain separate duties. This slice does not finish the full migration stack or its CI/review work. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, ordinary Git merges, source-composition proofs, PostgreSQL fixtures and local lifecycle tests. No subagents were used in the current reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
fa5e70faf4 |
fix(server): schedule automatic backups through shared lifecycle (#17836)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/39/changes/paperclip/39 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip manages agent work and its durable state. > - Automatic database backups protect that state on a configured schedule. > - Their independent interval is outside the shared deadline and shutdown lifecycle. > - An enabled backup schedule owes real work even when no queue changed. > - The existing coordinator can own this deadline without another table or notification protocol. > - This change preserves backup policy and moves scheduling into that shared lifecycle. ## Linked Issues or Issue Description Refs: #17835. Related work: #11756 (abortable backups), #3549 (stale-backup catch-up), #2120 (interval unref), #16364 (bounded shutdown steps). This slice changes scheduling; it does not implement those separate features. **What existing behavior does this improve?** Automatic database backup scheduling and shutdown ownership. **Current behavior** A standalone interval invokes backups, skips drain and warm standby, and is not registered with the shared work scheduler. Its future deadline is invisible there. The interval is not cleared or joined by the delivery coordinator during shutdown. **Proposed behavior** Register the configured backup deadline with the shared coordinator. Call the existing backup owner only when due. Keep the startup-relative phase, skip missed slots, retain the next interval after failure or a busy-owner skip, and join active scheduled work on shutdown. **Reason and benefit** One scheduling lifecycle owns overlap, admission, deadlines and shutdown. Waiting does not query the database. This is a true periodic obligation, so enabled backups still recur; it is not an empty-queue polling removal or a claim of measured savings. This existing slice now follows the merged import-spool parent #17835 and targets master. Merge authorization and exact-head CI/review gates are tracked separately. ## What Changed - Add a small backup scheduling owner using the existing coordinator and scheduler. - Replace the automatic-backup interval in startup. Register only when backups are enabled, independently of heartbeat scheduling. - Preserve the existing backup executor, manual serialization, retention reads, pre-sleep verification, restart marker and host-sleep gates. - Test deadlines, drain/standby, failures, busy-owner skips, slow attempts, restart and shutdown. Update startup-marker tests to exercise the registered deadline. - Record the audit and update inventory item 15. No schema or backup format changes. ## Verification - Current source acceptance: **130 passing cases in four complete files**, no skips: backup worker, startup checkpoints, scoped deadline work and delivery coordinator. Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. - Runtime and all four selected test sources are byte-identical to accepted source `11775eee7ed795b989cbb31052e69643c468c3cf` after normal parent reconciliation. The parent carries the tool-access test-order repair; it changes no backup behavior. - Historical broader checks and the real JavaScript backup/restore probe, including their intermediate failures and limits, remain in [the backup audit](doc/plans/2026-10-26-database-backup-sleep-schedule.md). They are not claimed as repeated current-head acceptance. - No full local repository-suite, deployed suspend/wake, pg_dump, live provider or cost-savings claim. Final-head CI, security, Greptile 5/5 and any actionable Ponytail findings remain separate merge gates. ## Risks - Enabled backups still require periodic database and disk work. This change centralizes the deadline; it does not reduce backup frequency or establish deployed cost savings. - The backup engine has no AbortSignal contract. Coordinator stop joins an active scheduled backup, so a stalled backup can delay graceful shutdown. Bounded backup cancellation remains separate work. - Restart still starts a new ordinary interval; it does not replay historical slots. The existing pre-sleep marker still requests an immediate verified restart backup and persists on failure. - A due backup waits during drain or standby and retries on the next coordinator attempt after admission reopens, within one configured interval. - The internal deadline is not a persisted Cloud alarm. Existing scheduled-backup and checkpoint sleep gates remain unchanged. The real probe qualifies the JavaScript engine only, not pg_dump or deployed shutdown latency. ## Model Used OpenAI Codex (GPT-6-based coding agent). Repository inspection, ordinary Git merges, source-composition proofs and local lifecycle tests were used for this reconciliation. No subagents were used in the current merge campaign. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
c8b7afee06 |
fix(server): schedule native environment lease settlement (#17801)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/24/changes/paperclip/24 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip manages AI agents and the resources used by their work. > - Native run completion has several independent settlement duties. > - The recurring run reaper repeated environment-release discovery for completed native history. > - Existing lease rows already record releases that still need work. > - This pull request wakes the existing release owners from committed changes and schedules only unfinished release debt. > - Completed history becomes quiet while stop evidence, workspace preservation, and retention policy keep their existing owners. ## Linked Issues or Issue Description Refs: #17793, #17717. **What problem does this solve?** Periodic native finalization replay calls environment release even when no lease needs work. This adds recurring database traffic to an idle instance. **What is the proposed solution?** Register one scoped worker on the existing delivery coordinator. Discover work from native finalizations and existing environment leases. Wake it from native, lease, and settings transactions. Keep the release owner's saved pending-cleanup deadlines. Keep retries only while a concrete lease still needs settlement. **What alternatives have you considered?** A slower unconditional scan still queries idle history. A new job table duplicates existing lease truth. Replacing the compound callback would also change runtime-service and agent-status duties; those remain separate slices. This is slice 58. Parent #17793 has merged. This publication targets master and preserves that landed prerequisite. Fresh current-head checks and review remain required. ## What Changed - Keep ordinary company-only finalization hints out of environment settlement. Prove zero candidate SQL after quiet and preserve real lease delivery. - Reuse the existing durable destroy claim for run-owned cleanup, excluding concurrent attempts and fencing late receipts while preserving cleanup recovery. - Scope lease cleanup notifications to their company/run so settlement does not rediscover the entire backlog after each release. - Read only `nativeWorkspaceSync` and `workspaceExportRetry` JSON fields when discovering environment cleanup. Preserve saved run data and existing release decisions. - Reuse the environment runtime's release predicate for discovery. Add no table or independent timer. - Wake from existing finalization, projection, lifecycle, recovery, lease, and settings transaction boundaries. Simple lease acquisition now uses a transaction too. - Preserve recorded retention policy, pending-cleanup deadlines, original lease/provider ownership, and current release guards. - Re-read durable lease state after a release. A logged or swallowed per-lease failure does not discharge the obligation. - Delegate only periodic environment release. Preserve default live/manual behavior, runtime-service release, and agent-status callbacks. - Add an owner/producer/lifecycle audit and real database tests. ## Verification Accepted production source: `9596fa022b57feea026c4728c80bdc36037b804f`. Publication head: `fb252c8b0e52ce355e7d98db2e392e4831f99c61`. The actual parent squash is adopted by a normal merge with the complete accepted child tree unchanged. Only audit text follows. - **4,172 distinct cases in 64 complete server files pass**, plus **20 selected chat cases** (4,192 selected total). The main batch reports 3,440 passes and 49 skips. Both complete Stop/read files then pass 51 + 6 cases on separate fresh databases through their supported override. Their eight pure cases are counted once; skipped cases are never counted as passes. Separate process recovery passes 410; review response passes 273. The other 1,794 chat cases were not run locally. - **Quiet producer proof:** the controlled unscoped baseline fails with three candidate SELECTs after three routine company-only hints. The scoped case performs zero, releases a real eligible committed-run lease, and returns to quiet. The same test hash spans red and green. - **Static gates pass:** database build; server typecheck/build; module boundaries; migration order; current Copilot profile identity. Packages equal the accepted parent byte for byte. - **Fresh fixtures:** 39 prepared PostgreSQL 17 databases plus fresh helper-owned databases. The schema has all 412 journal entries through index/tag 0413_tired_kronos. All 65 original test hashes restore exactly; the indirect Stop fixture is also hashed. No assertion or timeout changes. - **Preserved records:** original 49 skips and their completed coverage remain documented. Parent handoff timeout and prior chat-idle failure records remain in their owning audits. This does not claim full repository, full chat, live provider, or deployed sleep acceptance. - [Current integration and native lease audit](https://github.com/paperclipai/paperclip/blob/fb252c8b0e52ce355e7d98db2e392e4831f99c61/doc/plans/2026-10-26-native-environment-settlement-sleep.md#current-master-integration-and-native-lease-quiet-proof-2026-10-29). Fresh CI, security, and exact-head Greptile 5/5 must pass before merge. <details><summary>Earlier revision verification and failure history</summary> Receiver failure fairness: a controlled 1,100 ms batch reproduced a 30-second delay for healthy due receivers. The worker now supplies its existing per-item deadline through `nextRunAtOnError`. All 20 receiver lifecycle tests pass on the owner and combined tip `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production typechecking passes on the owner. The failed receiver retains its own backoff, and settled receivers return to zero recurring SQL. The five-second assertion budget is unchanged. See [the owning audit](https://github.com/paperclipai/paperclip/blob/6f3a0b8de149fb5620f15909473eb7c4455b9824/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch). Current-head CI and review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/ae9bff1c06e34702f64a399e58b45a657c133bd1/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Inherited voice-cleanup verification: the controlled slow-provider regression and all 13 lifecycle tests passed on its owner and combined tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only the fixture clock advancement and its audit note. Current head: `6f3a0b8de149fb5620f15909473eb7c4455b9824`. This remains a single-duty PR stacked on #17793. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/7e619453567e078a7d745f28bfe8e5ede571f282/doc/plans/2026-10-24-recurring-server-sleep-stack.md), [2026-10-26-native-environment-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/7e619453567e078a7d745f28bfe8e5ede571f282/doc/plans/2026-10-26-native-environment-settlement-sleep.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. </details> ## Risks - Existing lease owners retain destruction claims, provider identity, stop evidence, retention policy, and pending-cleanup deadlines. Incorrect changes here can release a live resource or lose a cleanup obligation. - Logged or swallowed release failures remain concrete debt. Real retries must retain their saved floor across unrelated hints. - In-process signals do not cover independent SQL or other replicas without ingress. - This delegates only periodic environment settlement. Runtime-service and agent-status duties remain separate. Existing Cloud blockers remain; no deployed sleep, database suspension, external alarm coverage, or cost savings is claimed. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, real PostgreSQL tests, and GitHub tools. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
829f9e7eaf |
fix(server): settle Muse terminal assignments from committed changes (#17793)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/12/changes/paperclip/12 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its external-agent module owns Muse assignments, operation receipts, and worker holds. > - Running assignments and retained terminal history must not cause an empty maintenance scan every five seconds. > - Terminal run changes and operation results already have durable database transactions. > - This pull request wakes the existing maintenance worker from those commits and reconstructs work at startup. > - The worker retries only concrete settlement debt. It retains qualification deadlines and authority checks. ## Linked Issues or Issue Description Refs: #17772. This is slice 57. Parent #17772 has merged. Related: #17000, #17525, #17752, #17758. The publication targets master and preserves the merged parent. **What existing behavior does this improve?** Muse terminal assignment settlement and the existing Muse maintenance worker. **Current behavior** A running assignment can keep maintenance scanning every five seconds. Terminal reconciliation rewrites unchanged history. Native finalization also calls that reconciliation during periodic replay. Discovery excludes a revoked original binding even when its assignment still needs settlement. **Proposed behavior** Committed run changes and operation receipts wake the existing worker with the original company, run or binding identity. Startup reconstructs actionable work from existing rows. Maintenance processes bounded batches and disarms after settlement. Default live and manual callbacks remain. The periodic callback delegates this one duty. **Reason and benefit** Configured but idle instances stop querying for this obligation. Late receipts, restarts, drain and historical bindings still have recovery paths. No table or independent timer is added. **Breaking changes** No schema, endpoint or provider protocol changes. Explicit manual reconciliation retains its unbounded default. This does not alter Cloud sleep admission or claim that all instance work can sleep. ## What Changed - Add an owner predicate shared by discovery and locked settlement. Fenced private-worker uncertainty remains held without an automatic retry loop. - Preserve original binding generation and company/agent/run identity, including revoked bindings. Fence only the terminal assignment during reconciliation. - Extend existing maintenance with scoped hints, bounded batches, concrete-debt retry, startup reconstruction and quiet drain behavior. - Register notifications inside existing terminal, projection, cancellation, operation-result and send-failure transactions. Keep qualification and OAuth signals explicitly global when coalesced with scoped changes. - Serialize only four domain IDs. Real native-finalizer tests caught and prevent a circular-object serialization failure at this boundary. - Use existing restore/worktree suppression and settings reactivation. Delegate both periodic finalizer branches while retaining resource callbacks and default live/manual settlement. - Add the producer audit and regression evidence in `doc/plans/2026-10-26-muse-terminal-settlement-sleep.md`. ## Verification Accepted production source: `584015018ed4a9d57d2b0f192b75847503b54e53`. Publication head: `305014b030d6c5eb891c70bc57036f4192d4fdad`. Normal adoption of the actual parent squash preserves the complete accepted child tree. Only audit text, a migration-count clarification, and the proved Photon test extension follow. - **3,715 distinct cases in 52 complete server files pass**, plus **20 selected chat cases** (3,735 selected total). The main 49-file batch passes 3,008; process recovery passes 410; review response passes 273; the full Photon file passes 24. The other 1,794 chat cases were not run locally. - **Quiet producer proof:** moving only the Muse fanout outside its parent guard produces six candidate SELECTs and fails the regression. Keeping the guard yields zero and still settles a real terminal run. The same test hash spans the controlled red/green pair. - **Photon review evidence:** repeated active callbacks produce zero settlement hints/candidate SQL. Three concurrent attention recoveries produce exactly one of each. The existing locked attention guard was correct; production needed no fix. The parent security check accepted the resolved finding before merge. This PR keeps the additional settlement queue in the existing regression. - **Static checks pass:** database build; server typecheck/build; module boundaries; migration order; current Copilot identity. Packages equal the accepted parent byte for byte. No new live provider qualification is claimed. - **Fixtures and failures:** 34 fresh PostgreSQL 17 fixtures have 412 applied migration receipts through tag 0413_tired_kronos, plus fresh per-call helper fixtures. All 52 original test hashes restore exactly. The additional Photon test hash matches its passing proof. Earlier remote chat passive-idle failure remains recorded; both passive/missing cases pass in the new selection. Parent handoff timeout/fresh-file-pass accounting remains in its audit. No assertion or timeout was weakened. - [Integration and quiet producer audit](https://github.com/paperclipai/paperclip/blob/305014b030d6c5eb891c70bc57036f4192d4fdad/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md#current-master-integration-and-quiet-producers-2026-10-29). Fresh CI, security, and exact-head Greptile 5/5 must pass before merge. <details><summary>Earlier revision verification and failure history</summary> Receiver failure fairness: a controlled 1,100 ms batch reproduced a 30-second delay for healthy due receivers. The worker now supplies its existing per-item deadline through `nextRunAtOnError`. All 20 receiver lifecycle tests pass on the owner and combined tip `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production typechecking passes on the owner. The failed receiver retains its own backoff, and settled receivers return to zero recurring SQL. The five-second assertion budget is unchanged. See [the owning audit](https://github.com/paperclipai/paperclip/blob/f7477933469fdc960f555425dce66eff25a94e54/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch). Current-head CI and review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/bbd31dea5ef3a4875950a87b4636af8b6bde858e/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Inherited voice-cleanup verification: the controlled slow-provider regression and all 13 lifecycle tests passed on its owner and combined tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only the fixture clock advancement and its audit note. Current head: `f7477933469fdc960f555425dce66eff25a94e54`. This remains a single-duty PR stacked on #17772. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-24-recurring-server-sleep-stack.md), [2026-10-26-muse-terminal-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. </details> ## Risks - In-process signals do not capture independent SQL or other replicas. Supported external events still need ingress. - Pending assignments and operation receipts retain their original binding identity. A new agent configuration cannot take their ownership. - Genuine qualification, OAuth, and recovery deadlines remain obligations. They must not be discarded merely to make the worker quiet. - The parent coarse-hint guard prevents recurring Muse discovery; real company/binding authority changes retain their signals. - No full local repository suite, full chat suite, live Muse or Copilot qualification, deployed sleep, database suspension, external alarm coverage, or cost savings is claimed. Existing Cloud blockers remain. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, real PostgreSQL tests, and GitHub tools. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
4d0847c4ca |
fix(server): settle external conversations from committed work (#17772)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/6/changes/paperclip/6 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip manages AI agents and the tasks that they execute. > - External conversations must become idle after an authorized response is published and all related work is complete. > - Native replay and stranded-task recovery still discover this settlement work through recurring scans. > - A new deadline worker must preserve response ordering, monitor handoffs, and unfinished work. > - This pull request gives settlement one shared readiness predicate and activates its worker from committed changes. > - Existing rows provide recovery truth. No new table or per-feature timer is added. > - The benefit is a settlement worker that becomes quiet and resumes when relevant work changes. ## Linked Issues or Issue Description Refs #17758. This is slice 56. Slice 55 has merged. Related: #17606 (chat milestones), #17663 (bounded run reads), #17665 (monitor proof), and #13809 (answered Slack conversation lifecycle). The provider applicability remains Slack and Speko-owned conversations. This does not extend Telegram settlement from #16559. **What happened?** Conversation settlement depended on periodic native replay and generic task recovery. An exhausted monitor also cleared its schedule before recovery admission. During that gap, an answered conversation could look finished. Failed or deferred admission lost the monitor schedule. **Expected behavior** Relevant committed changes activate settlement. The worker uses the owning service's evidence and becomes quiet when no settlement is owed. Monitor recovery stays pending until admission, and restart does not admit an already accepted recovery twice. **Steps to reproduce** 1. Create an active external conversation with a successful run and an authorized published response. 2. Keep execution, approval, a goal, or a monitor pending. Let the settlement worker become quiet. 3. Resolve that prerequisite through its owning service. Verify that the conversation settles without another periodic sweep. 4. Delay or fail exhausted-monitor recovery admission. Verify that the monitor remains pending and the conversation cannot settle during the handoff. **Paperclip version or commit** Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`. Parent #17758 merged as `8ac5a128e66fb672e88106412f76cf278599bfc4`. The publication adopts that actual squash through a normal merge with an unchanged complete child tree. ## What Changed - Preserve quiet after coarse company-only maintenance hints. The shared inbound-notice and response-availability helpers keep settlement enabled by default. Only the milestone dispatcher suppresses their duplicate settlement fanout. Its scoped issue-review signal and the explicit company-authority signal still wake real work. Four database cases prove both quiet and real company-wide producer delivery. - Preserve source-generation checks for chat handoff, task-before-comment locking, issue-scoped decision invalidation, and current conversation event columns from master. - Track initial external conversation settlement through shutdown without blocking HTTP readiness on endpoint or task locks. - Reuse the real queue's coalesced recovery receipt after restart, without admitting the same monitor recovery again. - Share the existing conversation readiness predicate between bounded discovery and locked settlement. Preserve current provider policy, run/publication proof, company boundaries, and endpoint-to-issue lock order. - Register a scoped deadline worker through the delivery coordinator. Preserve startup, restore/worktree suppression, drain/resume, failure recovery, and joined shutdown. - Extend existing review, milestone, wake-notice, and finalization signals. Add notifications at independent issue, comment, goal, session, and wake-receipt unblocks. - Keep monitor owner recovery pending until guarded admission. Reuse an accepted receipt for the same monitor schedule after restart, even when its limit reason changes; retain compatibility with legacy receipt keys. Commit board escalation and its monitor clear together. - Delegate periodic native settlement while preserving resource callbacks. Keep direct/manual defaults and preserve generic recovery ordering with a read-only readiness guard. - Add the producer and lifecycle audit in `doc/plans/2026-10-26-external-conversation-settlement-sleep.md` and update the stack ledger. ## Verification Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`. Publication head: `b256bb52647affd9f5279ea8c40f48bbe40f031f`. The complete accepted child tree stays unchanged when adopting merged parent #17758. This publication adds only the audit document. - **3,129 distinct cases in 37 complete server files pass**, plus **18 selected chat-response cases** (3,147 selected cases). The other 1,796 chat cases were not run locally. Full settlement, queue, scheduling, run-state, goals, interactions, task Retry, decision targets, status reassessment, workspace, process recovery, and review consumers are covered. - **Preserved failure:** the first 35-file command passed 2,445 cases and timed out one restored-work handoff at its existing 15-second limit. That aggregate command failed. The unchanged full 89-case actions file then passed on a fresh fixture; the affected case took 22 ms. Root cause was not established. No timeout or assertion changed. The 34 other files passed originally. Distinct counts exclude duplicate actions passes. Fresh CI remains a separate gate. - **Indirect wake regression:** the original unconditional baseline failed two quiet checks. Guards alone left the chat case failing with three candidate queries. The final controlled indirect baseline fails one of four cases; the repair passes all four with the same test hash. Positive controls preserve real company-only producers. - **Source and fixture controls:** all 38 selected test files restore to their exact committed hashes after fixture imports. Fresh PostgreSQL 17 fixtures use 413 migrations. All five manual merge resolutions preserve current authority, locking, and event-column contracts. - **Static checks pass:** database build; server typecheck and build; module boundaries; migration order; Copilot profile identity. Runner packages equal the accepted parent byte for byte. No new Runner/package qualification is claimed. - [Current integration and wake audit](https://github.com/paperclipai/paperclip/blob/b256bb52647affd9f5279ea8c40f48bbe40f031f/doc/plans/2026-10-26-external-conversation-settlement-sleep.md#current-master-integration-and-indirect-wake-audit-2026-10-29). Latest-head CI and exact-head Greptile 5/5 must pass before merge. <details><summary>Earlier revision verification and preserved failure history</summary> Receiver failure fairness: a controlled 1,100 ms batch reproduced a 30-second delay for healthy due receivers. The worker now supplies its existing per-item deadline through `nextRunAtOnError`. All 20 receiver lifecycle tests pass on the owner and combined tip `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production typechecking passes on the owner. The failed receiver retains its own backoff, and settled receivers return to zero recurring SQL. The five-second assertion budget is unchanged. See [the owning audit](https://github.com/paperclipai/paperclip/blob/83204a67880d980cfbea3771cee19a3cda0022b7/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch). Current-head CI and review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/ef9ae5767430d0093efc736ae0b303ba5afaae65/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Inherited voice-cleanup verification: the controlled slow-provider regression and all 13 lifecycle tests passed on its owner and combined tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only the fixture clock advancement and its audit note. Current head: `83204a67880d980cfbea3771cee19a3cda0022b7`. This remains a single-duty PR stacked on #17758. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-24-recurring-server-sleep-stack.md), [2026-10-26-external-conversation-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-26-external-conversation-settlement-sleep.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. </details> ## Risks - Signals cover supported in-process writers. They do not supply change capture for independent SQL or other replicas. - Real company-wide changes can inspect active candidates. Coarse maintenance hints do not activate settlement. Configured idle and retained history must remain quiet. - Monitor receipt binding and admission order must preserve company, agent, issue, and exact schedule identity. Current parent handoff locks and source-generation checks remain intact. - The local restored-work timeout is retained above. A fresh full-file pass does not establish its cause; latest-head CI must also pass. - Generic recovery and Muse/resource duties remain separate slices. This does not make the whole server idle. Existing Cloud sleep blockers remain. No deployed sleep, database suspension, external alarm coverage, fleet cost, or live provider qualification claim is made. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, PostgreSQL tests, and GitHub tools. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
8ac5a128e6 |
fix(server): resolve native finalization notices from committed wakes (#17758)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native finalization can commit before its recovery notice is resolved. > - The periodic orphan pass revisits committed runs to resolve those notices. > - Retained completed history should not require repeated discovery. > - This PR moves that one duty to transaction wakes and concrete failure retries. > - Existing notice rows reconstruct work after restart. The action service still owns resolution. > - Other completion and resource duties keep their current recovery paths. ## Linked Issues or Issue Description Refs #17752 (stack parent: committed run projection), #17748 (accepted-result commit), and #17734 (workspace finalization). These are distinct recovery obligations. This slice incorporates parent #17752 at `a49a2dac13b0e1c19da5d3d6b435c781703b1c0b`, now merged as `50fe32684d2d4e1db129e3c1bb455571344a0c09`. Related #17724 retires obsolete exact-session notices for terminal runs with no result and preserves their replay hold. This slice resolves notices after a result committed; it does not change that no-replay policy. **Current behavior** Committed native finalization recovery repeatedly inspects active watchdog notices. A notice can also be updated in place to describe a different run, so resolving a previously selected ID alone is unsafe. **Proposed behavior** Wake a scoped worker from relevant committed writes. Reconstruct active finalization notices at startup. Resolve only the current notice after the native owner rechecks its binding under lifecycle locks. Stop scheduled work when no matching obligation remains. **Reason and benefit** Notice recovery becomes quiet after settlement and reacts to real changes without a polling delay. Reused notices remain attached to their current incident. This does not claim that every remaining server duty or the host can sleep yet. ## What Changed - Track the initial recovery-notice sweep through shutdown without blocking HTTP readiness on task locks. - Require issue/run scope for ordinary finalization fanout. Routine company-only hints leave dormant notices quiet; dedicated owner and explicit settings signals retain their scopes. - Extract the native finalization notice predicate and resolution owner. Keep the existing recovery-action service responsible for writing the resolution. - Bind candidates by company, issue and run. Require a committed result and an active or escalated finalization notice. Exclude held ownership, pending status effects and unfinished run projection. - Recheck coordinator, issue, run and matching actions under locks. A newer incident cannot be resolved from a stale candidate snapshot. - Use the shared scoped worker, transaction signals, retry backoff and lifecycle handling. Add no table, per-feature timer or provider branch. - Signal action insertion, update, supersession and resolution; finalization and effect changes; run projection and ownership release; issue changes; settings activation; and the recovery route that reopens an action. - Preserve no-op resolution quiet. Serialize only the three supported notification IDs, even when a caller supplies a larger internal object. - Delegate only notice resolution from the periodic path. Keep manual/default finalization and other completion/resource work. - Add real PostgreSQL producer, concurrency and lifecycle coverage and a detailed audit. ## Verification - Current local source `730e00a7b007a36d188ca0a328ef050d3a25d1ee` incorporates the parent with a normal merge. All 1,209 parent-only, eight owned-only and 12 automatic shared paths are preserved. The two manual paths retain both chat import sets and scope the new notice fanout. The entire package tree matches parent profile 80 byte-for-byte. - Controlled real-database regression: original notice-only unscoped fanout produces three candidate SELECTs for routine company hints. The same zero-query assertion passes after restoring scoped fanout. The real notice writer then creates and resolves an action for a committed run and returns to quiet. Both phases restore production bytes exactly; the 22 cases outside this focused pattern are not counted as passing there. - The complete main command passes **1,873 cases in 21 files**, without skips, in 309.69 seconds. It includes all 23 notice-worker cases, full presentation/ownership read coverage, 88 startup cases, and the projection/result/physical-work consumers. Process recovery passes 410 in 163.54 seconds and external-chat response passes 273 in 53.48 seconds. Total complete-file acceptance: **2,556 cases in 23 complete files**, with no skips. The additional focused chat selection passes 18 cases in 13.44 seconds; 1,796 unselected chat cases do not count. Total selected passes: **2,574**. Normal merge `a0f2b3f4348b078219a53d8cc135c84202ba505e` adopts actual parent ancestry with this complete tested child tree unchanged. - The chat selection covers the real reply/notice row-lock regression, Slack and Telegram response recovery, current-authority denial cases, exact canonical response binding and internal GitHub attachment navigation. Other cases in the giant chat file remain unselected; this is not full chat-suite acceptance. - Production server typecheck, build, module boundaries, migration order and source identity checks pass. Twenty dedicated PostgreSQL 17 databases use the unchanged migration 413 schema. Embedded-only invocations use the existing fresh-per-call recovery helper. All temporary imports restore exactly and all 24 selected test hashes match after completion. No assertion or timeout is relaxed. - Parent package acceptance remains applicable: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One existing opt-in closed Pi snapshot case is skipped because its distribution root is unset. This does not qualify a live provider. - Failure history and scope are retained in `doc/plans/2026-10-26-native-recovery-notice-sleep-audit.md`. Current-head acceptance: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `2a8777bbed8b1e10810e3d6d99aae3f25b4854ef`, with no actionable findings. Prior scores do not substitute. No Ponytail verdict is present under the authorized missing-review policy. No full local repository suite, deployed sleep, expanded Cloud wake coverage or measured savings is claimed. ## Risks - Resolution takes coordinator, issue, run and action locks in the native recovery order. Contention must preserve the current incident; the real concurrent-upsert and competing-resolver cases exercise this boundary. - Process-local signals cover instrumented application writers. They do not add support for independent SQL writers or replicas without supported ingress. Startup reconstructs the existing durable rows. - Completed finalization is not proof that external conversation settlement, Muse reconciliation or physical cleanup finished. Those obligations retain their owners and recovery paths. - Discovery uses narrow keyset pages. No fleet benchmark, host suspension or cost reduction is claimed. Cloud wake coverage remains unchanged. - Parent #17752 has merged. This PR targets master at `50fe32684d2d4e1db129e3c1bb455571344a0c09`. Fresh current-head CI and review must pass before the authorized protected merge. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
50fe32684d |
fix(server): wake committed native run projection from durable changes (#17752)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run has separate result acceptance, task decision, run status, and resource settlement steps. > - A task decision can commit before its run status is saved. > - Recovery currently revisits committed decisions through the periodic orphan pass. > - This PR gives the existing run projection repair its own transaction-driven worker. > - Existing records reconstruct unfinished work after restart. Completed history becomes quiet. > - Other completion and resource obligations keep their existing recovery paths. ## Linked Issues or Issue Description Refs #17748 (stack parent: accepted-result commit), #17734 (workspace finalization), and #17717 (retained session cleanup). Related #17614 preserves terminal notifications. This slice is prepared on current parent #17748 at `b2efdb2dbbbef5b2227ffa2bd2a9a6cfe3867fe2`; the parent has merged as `28e38da7e6bdd6af4092dda56bc2be05bed47af5`. **Current behavior** The native recovery pass revisits committed decisions to repair the heartbeat-run status and metadata. A crash between decision and run writes creates real work, but periodic scanning also revisits retained history. **Proposed behavior** Wake projection repair after committed producer changes. Reconstruct mismatches on startup and settings activation. Retry concrete failures through the shared scheduler. Keep the run projection before resource settlement. **Reason and benefit** The repair worker performs no recurring SQL after its work settles. A real decision, effect settlement, or ownership release reactivates it. This is one worker migration; it does not establish host or database suspension. ## What Changed - Extract the existing projection predicate and write into their native-runtime owner. The finalizer keeps initial metadata enrichment and task decision authority. - Discover accepted, committed, mismatched run projections with company, issue and run bindings. Exclude pending status effects and held runner ownership. Read narrow fields instead of whole result payloads. - Use existing transaction hints, scoped delivery scheduling, failure backoff and lifecycle handling. Add no tables or per-feature timers. - Require issue/run scope for ordinary finalization fanout. Routine company-only hints leave dormant projection quiet; explicit settings and dedicated producer signals keep their existing contract. - Connect finalization, pending effects, lifecycle status writes, issue changes and worktree execution activation. Preserve outer transaction commit and rollback behavior. - Lock the current run before deciding which terminal event and completion time to save. Concurrent live and recovery projection must not announce the same transition twice. - Delegate only already-committed projection from the periodic pass. Its candidate scan uses the same pending predicate to defer remaining consequences and cleanup until projection finishes. - Add real PostgreSQL producer, race and lifecycle tests plus an owner audit. ## Verification - Current local source `a83b9449233a275176e637c765059f414305d7c1` incorporates the parent with a normal merge. All 1,208 parent-only, seven owned-only and ten automatic shared paths are preserved. The only manual conflict keeps new projection fanout inside the ordinary issue/run guard. The complete package tree matches parent profile 80 byte-for-byte. - Controlled real-database regression: the original projection-only unscoped fanout produces three candidate SELECTs for routine company hints. The same zero-query assertion passes with scoped fanout. A real committed result then repairs the run and emits one terminal event before returning to quiet. Production bytes restore exactly in both phases. Twenty cases outside that focused pattern are not counted as passing there. - The complete main command passes **1,830 cases in 19 files**, without skips, in 301.14 seconds. It includes all 21 projection cases, 85 startup cases, required quality review and heartbeat run lifecycle, plus physical-work and result-commit consumers. Separate process recovery passes 410 in 163.58 seconds and external-chat response passes 273 in 53.26 seconds. Total: **2,513 distinct passes in 21 complete files**, with no skips. Normal merge `d4b46fd29cb23126d556cc3aeb883e6b41f1a8a5` adopts the actual parent squash with this entire tested child tree unchanged. - Production server typecheck, build, module boundaries, migration order and source identity checks pass. Seventeen dedicated PostgreSQL 17 databases use the unchanged migration 413 schema. Embedded-only calls use the existing fresh-per-call recovery helper. All temporary imports restore byte-exact and all 21 selected test hashes match after completion. No assertion or timeout is relaxed. - Parent package acceptance remains applicable: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One pre-existing opt-in closed Pi snapshot case is skipped because its distribution root is not configured. No live provider qualification is claimed. - The owner audit at `doc/plans/2026-10-26-native-run-projection-sleep-audit.md` retains prior failures and their disposition. Current-head acceptance: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `a49a2dac13b0e1c19da5d3d6b435c781703b1c0b`, with no actionable findings. Old scores do not substitute. No Ponytail verdict is present under the authorized missing-review policy. - These checks do not claim a full local repository suite, deployed suspension, expanded Cloud wake coverage or measured cost savings. ## Risks - Process-local signals cover instrumented application writers. Independent SQL writers and replicas without supported ingress are not newly covered. Durable startup reconstruction remains enabled. - Projection must precede remaining committed resource settlement. The real reconciler test proves the periodic path defers cleanup while projection is pending and resumes it after repair. - Status decisions, external conversation settlement, watchdog resolution, Muse reconciliation and resource release keep their own authority and recovery. A committed result alone is not proof of completed cleanup. - Candidate discovery uses narrow bounded pages. There is no large-fleet performance or deployed sleep claim. Existing Cloud wake coverage and sleep blockers remain unchanged. - Parent #17748 has merged. This PR targets master at `28e38da7e6bdd6af4092dda56bc2be05bed47af5`. Current-head CI and review must pass before the authorized protected merge. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
28e38da7e6 |
fix(server): wake native result commits from durable work (#17748)
## Thinking Path > - Paperclip manages AI agents and preserves their results through failures. > - Native completion must save workspace output before it commits the accepted result. > - The orphan recovery pass repeatedly revisits this work even when the subsystem is idle. > - Accepted results, workspace receipts, retry dates and leases already provide durable truth. > - This PR wakes a scoped worker from those writes and schedules only real unfinished work. > - It preserves existing finalization authority and keeps later projection and resource recovery separate. ## Linked Issues or Issue Description Refs #17734 and #17717. This is slice 53 of the existing sleep stack, targeting master after parent #17734 merged at `a019d00a393b57f531afe8dd8ab0b8a1820d1c4c`. A current search found no duplicate accepted-result commit PR. **What problem does this solve?** Native accepted-result recovery currently depends on periodic orphan scanning. Quiet instances keep doing work to discover that no result needs a commit. **What improvement do you propose?** Use the shared delivery coordinator with transaction-bound producer signals. Reconstruct unfinished results at startup. Preserve saved retry and lease deadlines. Remove only uncommitted-result execution from the periodic pass. **Why is this useful?** The accepted-result worker becomes quiet with no work or retained completed history. Real results and completed workspace operations reactivate it without a polling delay. This does not claim that the whole server or database can sleep yet. ## What Changed - Track the initial result-commit sweep through shutdown without blocking HTTP readiness on recovery locks. - Keep ordinary company-only finalization hints from reconstructing dormant result work. Existing company settings and dedicated owner signals remain supported. - Add a native result-commit worker. Bind candidates by company, issue, run, accepted result and a successful workspace receipt. Exclude committed, terminal, held and pending-effect work. - Signal both accepted-result writers and workspace receipt writes before their transaction changes. Extend the existing finalization fanout for retry, lease, effect and completion transitions. - Add an optional atomic due-date check to the existing finalizer claim. A lease collision or later retry does not consume a failed attempt. - Signal native retry cancellation so it removes the saved wake after commit. - Register through the existing heartbeat suppression and shared delivery lifecycle. Preserve outer commit/rollback, drain/resume, settings activation and shutdown behavior. - Delegate only uncommitted-result execution from the periodic orphan pass. Keep committed projection and resource settlement recovery enabled. - Add real producer/lifecycle tests and a detailed audit. No new tables, provider branches or per-feature timers. ## Verification - Current local source `2af88fa55f961a74340880240019f6cd31e21bf1` incorporates the parent with a normal merge. All 1,203 parent-only, nine owned-only and 12 automatic shared paths are preserved. The only manual conflict keeps the new result fanout inside the existing ordinary issue/run scope guard. The complete package tree matches parent profile 80 byte-for-byte. - Controlled real-database regression: the original result-only unscoped fanout produces three candidate SELECTs for routine company hints. The same zero-query assertion passes after restoring the scoped fanout. A real workspace receipt and accepted result then commit exactly one decision and return to quiet. Red and green restore production bytes exactly. Twenty cases outside the focused pattern are not counted as passing there. - The complete main selection passes **1,746 cases in 16 files**, without skips, in 277.97 seconds. This includes all 21 result-worker lifecycle cases and 82 startup cases, including held result recovery, HTTP readiness and shutdown joining. Separate process recovery passes 410 cases in 162.42 seconds and external-chat response passes 273 in 53.41 seconds. Total: **2,429 distinct passes in 18 complete files**, with no skips. Normal merge `998239b019defe52e7b5d64f884de4e871ccd736` adopts the actual parent squash and preserves this complete tested child tree. - Production server typecheck, build, module boundaries, migration order and source identity checks pass. Fifteen dedicated PostgreSQL 17 databases use the unchanged migration 413 schema. Embedded-only calls use the existing fresh-per-call recovery helper. All temporary imports restore byte-exact, and every selected test hash matches after completion. No assertion or timeout is relaxed. - Runner validation belongs to the unchanged parent package tree: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One existing opt-in closed Pi snapshot case is skipped because its distribution root is not configured. This does not claim live provider qualification. - Earlier Greptile 4/5 on `656f6722be1aedf3d52b7f853956e15838204278` correctly identified startup blocking on result recovery. The prior branch `108cf0dc8296bf451b721ddac55ecbcd035d00f0` already tracks its ready promise through the existing lifecycle owner. Current full startup acceptance verifies the fix in the merged context. The later Greptile service error is not a pass; current-head acceptance now passes: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `b2efdb2dbbbef5b2227ffa2bd2a9a6cfe3867fe2`, with no actionable findings. No Ponytail verdict is present under the authorized missing-review policy. - The owner audit at `doc/plans/2026-10-26-native-result-commit-sleep-audit.md` retains earlier failures and their disposition. These checks do not claim a full local repository suite, deployed app/database suspension, expanded Cloud wake coverage or measured cost savings. ## Risks - Process-local signals cover instrumented application writes in the owning server process. They do not cover independent SQL writers or another replica without supported ingress. Durable startup reconstruction remains enabled. - The initial native decision and its run projection are separate transactions. The periodic reconciler still repairs already-committed results and releases resources. This PR must not be used to infer that committed means fully settled. - Candidate discovery adds bounded, narrow reads at startup and after signals. No large-fleet query benchmark is claimed. - Existing Cloud alarm coverage, host sleep safety and resource blockers are unchanged. No deployed sleep or savings claim is made. - Parent #17734 has merged. This PR targets master at `a019d00a393b57f531afe8dd8ab0b8a1820d1c4c`. Fresh current-head CI and review remain required before merge. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
a019d00a39 |
fix(server): wake native workspace finalization from committed work (#17734)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs can accept a result before workspace copy-back finishes. > - The periodic orphan pass repeatedly tries this recovery, including cases that require operator action. > - Copy-back needs exact ownership evidence. A generic timer cannot grant that authority. > - This pull request wakes workspace recovery from committed changes and local owner completion. > - Saved retries use the existing scheduler. Empty work, completed barriers, and operator holds become quiet. ## Linked Issues or Issue Description Refs #17717. This is slice 52 of the recurring-server sleep stack, targeting master after parent #17717 merged at `4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Related work: #17709 (status reassessment), #17687 (unsafe export recovery), and #16478 (workspace repair holds). This PR covers native workspace copy-back and validation. Result commit/projection and resource settlement remain separate recovery duties. It does not claim full-instance sleep. ## What Changed - Register a company/run/task-scoped workspace worker. Reconstruct accepted, unfinished work from existing records. Use saved retry and lease deadlines. - Notify from both accepted-result writers (control-plane and PRP coordinator), workspace receipts, coordinator changes, operator confirmation, retained workspace repair, and relevant task updates. Notifications follow the outer transaction. - Distinguish active local ownership, busy external ownership, pending Cloud evidence, operator-required evidence, and changed bindings. Only concrete external ownership/evidence waits receive a scoped retry. - Preserve advisory locks, signed stop proof, exact binding checks, physical callback joins, and existing failure policy. Join physical operations during shutdown. - Respect the existing heartbeat execution-suppression policy before discovery and execution; wake suppressed work on settings activation. - Skip physical workspace recovery in the enabled periodic orphan pass. Keep direct/startup reconciliation and the remaining result/cleanup recovery paths. - Keep routine company-only finalization hints quiet. Ordinary work requires issue/run scope; explicit settings and owner-release events keep their company-wide contract. - Preserve the parent's entire Runner/package tree, profile 80 and all historical identities. Add the dated obligation, producer, lifecycle and limitation audit. ## Verification - Current local source: `4059b24e3ea5c623be54ade18ec9b6ab78eb0289`, prepared by a normal merge of the current parent. The sole manual conflict keeps ordinary workspace fanout inside the existing issue/run guard. All 1,198 parent-only, 15 owned-only and 10 automatic shared paths are preserved. The complete package tree matches the parent byte-for-byte. - Controlled regression: the original workspace-only unscoped fanout performs three candidate SELECTs for routine company hints. The same zero-query assertion passes with scoped fanout. A real accepted result then produces one successful workspace settlement and returns to quiet. The failed baseline remains in the audit. - The first complete local group passes **1,693 cases in 14 files**, including all 19 workspace-worker cases. It covers both accepted-result producers, physical ownership, export retry/failure, finalization recovery, arbiter corpus, issue recovery actions, control-plane port, workspace recovery, heartbeat recovery, startup, retained cleanup and native execution. - Separate process-recovery and external-chat response files qualify the changed recovery/settlement paths without sharing their general test database with other files. Both pass in full: 410 process-recovery cases in 162.79 seconds and 273 external-chat cases in 53.79 seconds. The total is **2,376 distinct passes in 16 complete files**, with no skips. Normal merge `5c46c52873158529736b8349e4d44658865108ee` adopts the parent squash with this entire tested child tree unchanged. - Production server typechecking, build, module boundaries, migration order and source-profile checks pass. Thirteen dedicated PostgreSQL 17 fixtures use the unchanged migration 413 schema. Embedded-only paths use the existing recovery helper with fresh per-call fixtures. All temporary imports are restored byte-exact. No test assertion or timeout is relaxed. - Runner validation belongs to the unchanged parent package tree: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One existing opt-in closed Pi snapshot case is skipped because its distribution root is not configured. These are source-scoped checks, with no live provider qualification claim. - The owner audit at `doc/plans/2026-10-26-native-workspace-finalization-sleep-audit.md` preserves earlier failures and their disposition. Current-head acceptance: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `7df5c9c44ea9e171be3c61b4430d3727b10f3337` with no actionable findings. Earlier scores do not substitute for this result. No Ponytail verdict is present under the authorized missing-review policy. - No claim of a full local repository suite, deployed app/database sleep, expanded Cloud wake coverage or measured cost savings is made. ## Risks - Supported wake producers are in-process application services. This adds no cross-replica or independent SQL-writer transport. - A real unresolved export still needs recovery. Busy owners and pending Cloud stop evidence retain a 30-second scoped check. No timeout authorizes copy-back. - Result projection and resource release still use their existing recovery. Some existing cleanup failures remain best effort; this PR does not infer cleanup success from a workspace barrier. - Cloud alarm coverage and host-sleep safety remain unchanged. No deployed suspension or cost savings have been measured. - Parent #17717 is merged. Current base is master at `4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Fresh current-head CI and review must pass before this slice merges. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
4c56702e06 |
fix(server): schedule retained native-session cleanup (#17717)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs can retain a local session after an interrupted durable suspend. > - The orphan reaper repeatedly scans completed history for that cleanup duty. > - Idle scans keep the database active. Removing them requires complete wake and ownership paths. > - This pull request moves retained-session cleanup to committed signals, local release events and saved deadlines. > - Existing physical proof checks still decide whether cleanup is safe. The shared scheduler does not grant execution authority. > - The benefit is quiet retained history with recoverable cleanup and owned shutdown. ## Linked Issues or Issue Description Refs #17709 and #18135. This is slice 51 of the existing recurring-server sleep stack. Both referenced dependencies have merged. The target is `master`. **What happened?** The periodic orphan pass scans retained native sessions even after their cleanup is complete or no automatic action is safe. Some eligible sessions also depend on a local session, environment lease or original maintenance callback releasing ownership. An earlier decision notification can be consumed before the run's later terminal projection commits. **Expected behavior** Reconstruct existing work at startup. Wake from committed writes and actual owner release. Register the saved claim expiry or a probe for a specific live process. Stop recurring discovery when no obligation remains. Preserve original callback ownership and all existing cleanup proof checks. **Steps to reproduce** 1. Retain completed native runs with a durable-suspend close failure. 2. Let the periodic orphan pass run with no new work. 3. Observe repeated cleanup discovery. Separately, delay terminal projection after its decision notification is consumed and observe the missing cleanup wake. **Paperclip version or commit** Parent #17709 at reviewed head `3b75f4bc1da1370304f069fc287af3d693575716`, landed as `df6156c2e37f542a69f393dde31e357b82b7cac1`. Current integrated master: `31b927e3013a8a4b3266854b740cf40a24753788`. ## What Changed - Move scoped, paged discovery to the retained cleanup owner. Preserve explicit manual recovery. - Register the duty with the shared scheduler and remove it from the periodic orphan pass. Keep concrete lease expiry and live-process probe obligations. - Use typed outcomes for settled/refused work and process, session, environment, maintenance or lease waits. Keep physical proof and quarantine decisions in the executor. - Wake from committed finalization and terminal projection writes, durable process/lease release, and local session or callback settlement. Routine company-only finalization hints leave dormant cleanup quiet. - Join duplicate cleanup and serialize inspections on one database lane. Keep original Runner callbacks owned until settlement and shutdown. - Preserve all master profile fixtures and the branch's historical identities. Save the colliding sleep profile 79 separately. Generate combined Copilot profile 80 from its source and retain master's other provider profiles. - Add the owner audit and regression coverage. No new table, migration or independent maintenance timer is added. ## Verification - Server acceptance: **2,342 distinct passing cases in 18 complete files**. The 14-file unaffected group contributes 1,532 passes. Full startup contributes 76, process recovery 410, environment service 51, and external-chat response 273. The 29-case cleanup worker suite and 977-case executor suite run in full. No case in these files is skipped. - Controlled scope regression: removing only the ordinary finalization issue/run guard wakes dormant retained cleanup; the same assertion passes after restoring it. A real local owner release still resumes the durable candidate. - Preserve the first broad failure: 1,607 passed and one startup shutdown case failed because the coordinator mock lacked `stop()`. Add only that mock method. All 76 unchanged startup cases then pass. Do not treat the first broad command as green. - Dedicated PostgreSQL 17 databases use the existing migration 413 schema. Embedded-only fixtures use the existing recovery helper with fresh per-call databases. All temporary imports are restored byte-exact. No assertion or timeout is relaxed. - Current-master integration preserves both complete source histories. The server, UI and database trees are identical to the server-tested source. All master historical profile fixtures remain byte-exact. The prepared sleep profile 79 remains independently inspectable. - Current runtime validation at `d08f764e3efe75e96929173afe7ee8739121ad9f` passes all 12 commands (22 installed ACPX/Grok contract cases and 26 native-provider Rust cases): `pnpm install --frozen-lockfile --ignore-scripts`; Copilot/Grok identity and generated profile checks; Runner and server typechecking; server build; Copilot/Pi installation, profile compatibility and transport settlement files (130 passed; one opt-in closed Pi snapshot case skipped because its distribution root is not configured); installed ACPX model/reconnect package contract and Grok identity tests; Rust native-provider backend tests; module boundaries and migration order. - Earlier 94-case installation/settlement acceptance applies to the prepared profile 79 source. It is retained as history and does not substitute for the current profile 80 validation above. - Failure history and duty-specific evidence remain in `doc/plans/2026-10-26-retained-native-session-cleanup-sleep-audit.md`. Exact-current-head acceptance: all 60 checks are clean, including protected CI and security; Greptile gives 5/5 for `281d579de0d08e907194b8b790a0597b07d16b7b` with no actionable findings. No Ponytail review is present under the authorized missing-review policy. - These checks do not claim a full local repository suite, live Copilot or Daytona qualification, deployed host/database suspension, or measured cost savings. ## Risks - Missing a supported producer could strand cleanup. Tests exercise real finalizer, heartbeat, environment and process writers. Process-local signals cover in-process writers; no multi-replica or external SQL notification transport is added. - A timeout can leave an original callback active. Cleanup and shutdown must keep waiting for it. The existing global maintenance drain is conservative across roots. - A retained live process still has a scoped 30-second absence probe. A saved lease keeps its actual deadline. Operational errors retain retry work. Immutable proof refusals and operator-required histories do not gain automatic retry authority. - Cleanup keeps all existing artifact, identity, claim, no-launch and quarantine checks. Terminal status alone does not authorize physical cleanup. - This PR does not expand Cloud alarm coverage or relax host sleep eligibility. It proves worker behavior, not deployed app/database suspension. - The Copilot source identity has changed; these checks do not claim new live provider qualification. - The parent has merged. This PR is merged only after its current-head protected CI and review gates pass under the existing campaign authorization. ## Model Used OpenAI Codex, GPT-6 family, with code execution and repository tools. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
df6156c2e3 |
fix(server): wake native status reassessment from committed changes (#17709)
## Thinking Path > - Paperclip manages agents, tasks and their durable native results. > - Changed evidence, task authority or review retirement can require a new status assessment. > - Periodic orphan recovery repeatedly reads history to find these changes. > - This change gives reassessment an owner driven by committed changes and saved deadlines. > - Existing result, assessment, decision and task rows remain durable truth. > - The status committer still verifies current sources and authority before writing. > - Settled or insufficient evidence becomes quiet until a supported producer changes it. ## Linked Issues or Issue Description Refs #17702 (merged), #17679 and #17680. This is slice 50, now based on master. It owns native status reassessment; workspace settlement, retained sessions and other orphan duties remain separate. **What happened?** Periodic recovery repeatedly inspects completed native results. Removing that scan without complete writer coverage or current authority checks could strand new evidence or commit an old result over newer work. **Expected behavior** Reassess from existing durable facts after relevant outer transactions commit. Retain real retry/lease deadlines and restart/drain recovery. Preserve current authority, quality-review and supersession checks. Make no recurring SQL when work is settled or dormant. **Steps to reproduce** Complete a native result awaiting evidence or review, leave it idle, then approve a real work product or retire the selected automatic review. Its saved result should advance once when authorized, or record the existing nonterminal business gate, without a periodic scan. ## What Changed - Extract reassessment and its pure status policy from finalization recovery into their owning modules. Add a scoped worker using the existing delivery coordinator and scheduler; no queue table or migration is added. - Retain the three durable discovery triggers: changed work-product history, authoritative task status/version, or retirement of the selected automatic completion review. Preserve selected-source, ownership, effects, retry and lease fences under the existing committer. - Notify from actual task/work-product, claim release, approval, interaction, OAuth, event evidence and workspace-settlement owners before their transaction writes. Keep provider calls outside short transactions and nested writes behind outer commit. - Scope ordinary milestone/finalization/coalesced-input wakes to an issue/run. Keep genuine company authority restoration explicit. Real Photon and coalescer regressions cover all three response/status queues. - Move the current parent's required quality policy into the owner and apply it to every reassessment that could complete a task. An unmet review records the existing in-progress decision and next action, avoiding an operational retry loop. Preserve conversation/session reset locks and the workspace omission-barrier signal. - Skip reassessment in periodic recovery while preserving explicit/startup recovery and the historical supersession guard before independent workspace work. Document the source contract, producer inventory, failures and qualifications in the linked audit. ## Verification - Broad source `9fddb3f0b1157df32b64fb1f22e9d6e9624470e8` passes **2731 distinct cases in 28 complete files**, with no skips: 2458 producer/owner/authority/workspace/quality cases in 394.43 seconds and all 273 governed-response cases in 53.74 seconds. Nineteen dedicated fresh PostgreSQL 17 databases migrate through 0413; eighteen embedded-only invocations preserve fresh-per-call isolation through the existing recovery helper. All 22 temporary imports restore exactly and all 28 hashes match. - Final runtime source `5ea3b0fd016bf7e63ba82a7044825a46a0cd2082` passes **1211 cases in five complete affected files** in 108.23 seconds after the narrowly reproduced evidence-trigger fix: all 43 worker, 145 corpus, 32 finalization-recovery, 968 native-session and 23 quality-review cases. Four fresh databases plus the quality helper's own fresh fixture use PostgreSQL 17. Its one adapted import restores exactly; all five hashes match. The other 27 broad-run test files are unchanged. These qualifications are source-scoped and not cumulatively double-counted. - Required/optional real regressions reproduce retry loops on retired-review and evidence-approval paths before repair; both become SQL quiet afterward with unchanged assertions/time limits. Controlled removal of only ordinary scope guards reproduces two coarse-hint SQL failures; exact guard restoration passes. Real producer, outer transaction, claim, authority, 103-item backlog, retry and drain cases pass. Photon transport is mocked; callbacks, locks and observed SQL are real. - Normal merges preserve 12661 parent/unrelated entries, 35 owned entries and 21 automatic shared results, with five original conflicts reviewed explicitly. The parent's actual squash reconciliation preserves the entire held child tree. The final quality fix changes only its owner and tests. Fourteen added comments are removed with identical nontrivia tokens; no directives are removed and the final comment check is empty. - Database build, server types/build, boundaries and exact-head migration order pass. The database tree is identical to merged #17702: no new SQL, queue table or snapshot is generated. Earlier benchmarks, host limitations, failed assertions and their repairs remain in `doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md` and `doc/idle-native-status-reassessment.md`. - This qualifies targeted external PostgreSQL behavior, not embedded bootstrap on the semaphore-exhausted host, a full local repository suite, deployed sleep or measured fleet savings. Fresh exact-head CI and review remain required. ## Risks - Notifications retain the process-local supported-writer contract. Independent SQL writers and uncoordinated replicas are not covered by a new transport. - Reassessment competes with current authority changes. Existing row/source locks and retries remain required; arbitrary missing identities have no row to lock, so this is not a general predicate-lock guarantee. - Discovery pages 100 rows but retains outstanding candidates in memory. It remains proportional to backlog size, and cancellation occurs between queries. - Other orphan/workspace/session duties and Cloud sleep eligibility remain unchanged. Missing supported producer coverage could strand dormant work, so the producer audit and real callback/transaction tests remain part of acceptance. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution and PostgreSQL tests. The runtime does not expose a more specific serving snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted scope is stated above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
79ce86630a |
fix(server): schedule governed review response recovery (#17702)
## Thinking Path > - Paperclip manages agents, tasks and their external chat responses. > - A completed chat turn can owe a response while an earlier review gate remains pending. > - Periodic recovery revisits that response history even when no work can advance. > - This change uses the existing delivery coordinator and saved retry deadlines. > - Approval, interaction and authority changes wake the affected issue or run after commit. > - The existing presentation owner verifies the durable result and current authority. > - Settled and dormant history becomes quiet without losing future eligible responses. ## Linked Issues or Issue Description Refs #17696 (merged) and #15625. This is slice 49, now based on master. It owns governed-review response presentation. Accepted responses and pending decision effects keep their existing owners. **What happened?** Native finalization recovery repeatedly inspects committed review responses. Removing the scan without covering approval, interaction, task-policy and authority changes would strand responses that become eligible later. **Expected behavior** Recover an owed response once from existing durable facts. Enforce authority at publication, wake after relevant outer transactions commit, retain real retry deadlines and restart/drain recovery, and stop recurring SQL when no response can advance. **Steps to reproduce** Complete an external chat turn while an earlier completion review remains pending. Block its response behind an approval or interaction, then resolve that blocker. The response must appear once without a periodic scan or resolving the original review gate. ## What Changed - Add a scoped governed-review worker using the existing coordinator, finalizations, decisions and comment receipts. No queue table or transaction wrapper is added. - Keep recovery and publication policy in the native finalizer. Preserve binding, source/session authority, ownership, deadline and deleted-comment fences; match coordinator-before-issue lock ordering. - Notify through actual approval, interaction, OAuth, task-policy and lifecycle producers before their owned writes. Serialize approval resolution with task linking so a concurrent link retains its wake. Nested transactions still settle at outer commit. - Scope ordinary milestones, finalization and coalesced input to their issue/run. Use the explicit company authority boundary for genuine global restoration, including membership changes. Both response workers share this boundary. - Register under maintenance and drain gates, disable the two periodic repair entrances, and retain fresh finalization and explicit recovery. Add the concurrent 0413 comment-receipt index and migration registration; retain five snapshots. - Preserve the producer inventory, source qualifications and earlier failures in the owner audit. Retain a real Photon regression proving routine ownership assertions stay quiet and concurrent authority restoration wakes each worker once. ## Verification - Scoped repair source `2341714720b5c273f0ce011b48345af0ef39227f` passes **2053 distinct cases**: 1703 in 15 complete producer/authority/recovery files, including all 273 governed-response cases and 95 wake-queue adapter cases, plus 350 selected chat cases. Another 1463 chat cases are outside that selection. Thirteen fresh PostgreSQL 17 fixtures migrate through unchanged 0413; the one temporary queue-fixture import restores exactly. All sixteen captured hashes match. - Two actual repeated coarse-hint SQL regressions fail before repair and pass afterward. Actual membership restoration passes before and after. The same assertions and time limits prove dormant work stays quiet and real authority restoration still publishes once. Real coalesced input covers both queues after commit. - Combined source `586f5674cd9798b33cd22437255e0480b5726d20`, including independent master #18129, passes **1297 distinct cases**: 1093 in five complete server files, 185 in five runner files, 14 Node protocol cases and five UI registry cases. This includes the complete 22-case Photon file for both response workers. The external Photon transport is mocked; Paperclip callbacks, transaction fences and observed SQL are real. The earlier and combined case counts are separate, with no cumulative double-counting. - A normal merge retains both complete owned and independent master binary patches. All twelve new-context hashes and sixteen earlier repair hashes match. Three dedicated fresh 0413 fixtures plus the live-event helper's own fresh database use PostgreSQL 17 without source adaptations. Database build, server types/build, boundaries, full-SHA migration order and generated TS/Rust profiles pass. The new inherited Runner vendor inputs rebuild successfully. - Database bytes remain identical to the previously qualified 0413 source. The recorded populated 0412-to0413 upgrade retains all 411 prior receipts and 38 companies, appends one receipt and none on replay; the index is valid and ready. The earlier 2828-case acceptance, CLI journey, setup failures and their repairs retain their recorded source scope in `doc/plans/2026-10-26-governed-review-response-sleep-audit.md`. - Parent #17696 merged with 58 clean checks, Greptile 5/5 and Ponytail 5/5. Its Photon warning was disproved by real callback/SQL evidence and the security check became successful. This PR still requires its own fresh exact-head CI and review gates. No deployed sleep, external alarm or fleet savings claim is made. ## Risks - Notifications use the existing process-local writer contract. Independent SQL writers and uncoordinated replicas are outside its coverage. - Genuine startup and company authority reconstruction remain proportional to the owed backlog. Queries page 100 rows, but discovery accumulates the candidates; this is not a constant-cost global operation. - Current publication authorization, durable binding, ownership and lock ordering remain mandatory. The worker does not resolve the original review gate. - The concurrent index uses existing migration replay handling. Other recovery duties and Cloud sleep eligibility remain separate. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution and PostgreSQL tests. The runtime does not expose a more specific serving snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
ec3b558277 |
fix(server): schedule accepted native response recovery (#17696)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs store accepted results and their finalization decisions. > - A completed run can still owe its public answer after an interruption. > - Periodic recovery scans for that answer even when no new work exists. > - This pull request wakes recovery from committed changes and saved retry deadlines. > - The existing owner still verifies the result, source, and current publication authority. > - Settled or dormant response work no longer needs its own periodic scan. ## Linked Issues or Issue Description Refs #17690 (merged). This is slice 48 of the sleep migration stack, now based on master. Related: #17661 handles GitHub terminal publication; this change handles native accepted-response presentation. Governed-review presentation remains a separate duty. **What happened?** Native finalization recovery repeatedly checks already accepted results for a missing answer. Removing that check without complete notifications would lose responses when evidence arrives late, an agent resumes, or admission settles after execution. **Expected behavior** Recover the answer from its saved result when a supported change makes it eligible. Retain real retry deadlines. Stop querying when the answer is selected or cannot currently be published. Reconstruct work after restart. **Steps to reproduce** 1. Commit a native accepted result with no selected response comment. 2. Run finalization recovery while its source is unavailable or locked. 3. Restore that source or release the lock. The answer must appear once without repeating provider execution. ## What Changed - Honor recorded non-create decisions and skip-comment policy in both discovery and recovery. Apply durable conversation/result/continuation guards before recovery locks. Preserve genuine unanswered and legacy recovery. - Scope routine response wakes to the affected issue/run. Coalesced input names its active run; pending-effect settlement names its issue. Sixteen existing global authority/configuration hooks use one explicit company invalidation boundary. No new transaction wrappers or tables are added. - Add an accepted-response worker using the existing delivery coordinator and scoped deadline helper. Preserve saved retry deadlines and pending-effect fences under the owner's lock. - Share the finalization notification boundary. Wake on finalization settlement, exact accepted-result/final-message evidence, agent availability, retry admission, and existing chat authority changes. - Keep lifecycle policy in services. A narrow pre-write status hook lets lifecycle retain ownership of its transaction and status transitions. - Register the worker during enabled maintenance startup. Remove its authority subscription on shutdown. Periodic replay skips this duty on both committed and fresh-commit paths; ordinary finalization and explicit recovery retain their behavior. - Add a partial index and expression statistics for unpresented native runs. Migration 0412 supports concurrent replay and adds no queue table. - Add real PostgreSQL and lifecycle coverage. Preserve the implementation and producer audit in the plan document. ## Verification - Additional real Photon callback acceptance on the published production code disproves the security warning: ten active assertions cause zero response-recovery hints/SELECTs, while three concurrent assertions during an actual attention-to-active recovery cause exactly one wake/SELECT. The existing locked status guard already enforces this boundary. No production change was needed; the resolved finding now has a successful current-head security check. The regression will be retained with the next slice. - Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883` passes **1489 distinct server cases**: 1139 in six complete recovery/finalizer/monitor/summary/workspace suites (81.02 seconds), plus 350 selected chat cases (297.68s). Another 1463 chat cases are outside the pattern; the complete 1813-case file is not claimed. All eleven new regressions are included in that count. Five dedicated fresh PostgreSQL 17 databases migrate through unchanged 0412, with no fixture-source adaptations. Database build, server types/build, boundaries and full-SHA migration ordering pass. All seven test source hashes match. - The exact published baseline first fails eight behavioral assertions while the three genuine-answer cases pass. The same corrected test source passes all eleven after repair: recorded none/reuse remain suppressed; five ineligible history families take zero recovery row locks across repeated issue hints; missing/legacy/create-null answers still publish once; fresh periodic finalization defers presentation to its worker. Earlier fixture setup and variable/compiler failures remain explained in the audit, rather than erased by later success. - Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821 server cases**: 1496 in 13 complete producer/lifecycle/authority suites (110.63 seconds), plus 325 selected chat cases (278.99 seconds). Coverage includes all native-session cases, queue admission, status decisions, membership/board access, the complete GitHub workflow, accepted-response lifecycle and real authority restoration. The other 1477 chat cases are outside this selection; the complete 1802-case file is not claimed for this repair run. - Actual PostgreSQL regressions first reproduce a coalesced-input hint missing its run scope and two routine hint paths producing 12 response-history SELECTs each after three notifications. The same assertions pass with run-scoped coalescing and zero response-history SELECTs for unscoped routine hints. Actual membership restoration still publishes the waiting response once. Assertion time limits are unchanged; this measures real driver SQL and persisted producer work. - Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412. One embedded-only wake-queue fixture import temporarily uses its own external database and restores byte for byte. All 14 selected source files match pre-run hashes. This qualifies external-database behavior rather than reproducing embedded CI locally. - Master advanced independently with #18126's Pi archive retry fix. Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both complete binary patches. All accepted server/database/UI/test/CLI trees stay identical; the three independent runner installer/manifest files are retained. All 13 installer tests pass, including stalled header/body cancellation. Combined server types/build, boundaries and full-SHA migration order pass. A subsequent normal merge of #18128 at `e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only file changes with both complete patches preserved. The changed HTTPS backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178 outside the pattern), and combined types/build, boundaries and migration order pass again. All 14 repair source hashes remain unchanged. The independent browser fixture changes are preserved; no local browser run is claimed. - The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed the 4171-case full baseline before this scope repair. That scope is preserved in the audit rather than represented as a current full rerun. All 0412 database bytes remain unchanged: original concurrent index/statistics/ANALYZE, prior executable history, journal entries and retained snapshots. Five snapshots 0408 through 0412 remain. The populated 0411-to0412 upgrade preserved 410 prior receipts and 38 companies, appended one receipt and none on replay; index/statistic checks passed. - The earlier P2 security failure, Greptile's former conditional dependency on #17702, and the subsequent actual Ponytail eligibility findings are recorded in `doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This PR now includes the issue-scoped pending-effect notification fix itself, so it is independently correct before #17702. Fresh published-head CI, Greptile 5/5 and a clean security scan remain required. No deployed-sleep, external-wake or fleet-savings claim is made. ## Risks - The writer contract is process-local. Independent SQL writers and other replicas are not covered by these notifications. - Unpresented backlog reconstruction still has memory and time cost proportional to candidate count. Pages contain at most 100 rows; execution uses the shared fair batch. Selected history is excluded. Health-only signals do not trigger reconstruction. - Publication policy remains with its current owner. Digest, company/run binding, source authorization, selected-answer suppression, and ownership fences remain required. - The rest of native recovery and governed-review presentation still run separately. This PR does not relax Cloud sleep eligibility or add external wake coverage. - The stack parent has merged and its actual ancestry is reconciled. This publication still needs fresh current-head CI and review. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution, and PostgreSQL tests. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
a8a038e659 |
fix(server): schedule native pending effect recovery (#17690)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native status decisions store effect targets and a delivery ledger. > - Historical pending or failed effects still need recovery. > - The periodic finalizer queries that ledger even when work is complete. > - This pull request gives pending effects their own commit-driven worker and saved deadlines. > - Completed history becomes quiet while incomplete effects remain recoverable. ## Linked Issues or Issue Description Refs #17687 (merged). This is slice 47 of the sleep migration stack, now based on master. **What problem does this solve?** Periodic native finalization repeatedly checks effect delivery. A tenant with completed history should not perform recurring SQL for this duty. Historical unresolved effects must still recover without duplicating their targets. **Proposed solution** Reconstruct current pending effects at startup. Wake from committed coordinator changes. Use the saved retry and ownership deadlines. Keep the existing dispatcher as the only recovery mutation owner. **Alternatives considered** A slower unconditional scan still creates idle traffic. A new queue table duplicates existing durable state. Replaying materialization can duplicate effects. This change uses the existing ledger, transaction signals, and shared scheduler. ## What Changed - Add a scoped worker with 100-row discovery pages and fair 25-entry dispatch batches. Delivered, superseded, and terminal-failure history has no recurring scan. - Preserve exact company/issue/run/decision binding and validate materialized targets before settling effects. Recheck phase, saved result, retry, and lease under the coordinator lock. Missing targets retain recovery. - Notify from decision replay, accepted-result recovery, coordinator claim/retry/release, attention replay, and execution-control recovery. Only two DB-only producer writes need a new transaction wrapper. Nested transactions use the existing outer-commit boundary. - Exclude pending-effect candidates from periodic finalization so it cannot skip ahead of their owner. Keep explicit and startup recovery behavior compatible. Respect maintenance disablement, drain, standby, and shutdown. - Add concurrent, idempotent migration 0411 with partial indexes for pending effects and their current coordinator binding. Retain the five newest schema snapshots. - Document producers, query evidence, failure history, and scope in `doc/plans/2026-10-25-native-pending-effects-sleep-audit.md`. ## Verification - Source `ba38ee465df02b6026286b5d5e15ae49fe07a367` passes all 2284 distinct cases across 25 complete files, without skips. The main selection passes 2225 in 203.94 seconds; safe-replacement passes 59 in 26.03 seconds. Coverage includes all 968 native-session cases, the 94-case status-authority/pending-effect corpus, actual decision/lease producers, Stop, queue, interaction, reconstruction and Slack consumers. The audit gives per-file counts. - The complete 94-case corpus passes again on the same database in 16.03 seconds. Retained PSC/OUT rows and an unrelated sentinel remain unchanged. The repeat is not counted again as distinct coverage. - Twenty-one separate fresh PostgreSQL 17 databases migrate through 0411 before setup. Four embedded-only fixture imports temporarily use their own external database on this IPC-exhausted host and are restored byte for byte. Assertions and timing limits stay unchanged. This qualifies external-database behavior; it does not claim an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-head migration order pass. Original runtime/schema edits and unrelated upstream blobs are preserved. Both original concurrent index statements, all prior executable history, journal entries and retained snapshot bytes are unchanged; five snapshots 0407 through 0411 remain. - Populated 0410-to0411 upgrade preserves all 409 prior migration receipts and 38 companies, appends one receipt and adds none on replay. Both indexes are valid and ready on fresh and upgraded databases. - Adopting the latest prepared prerequisite changes ancestry and parent audits only. All accepted core trees remain identical. The audit retains earlier failures and qualifications. Actual landed-parent ancestry is now reconciled; fresh published-head CI and exact-head Greptile 5/5 remain required. No deployed sleep, Cloud-wake or fleet-cost claim is made. - Current parent context: integration `5f7cd7bc589ad0169dfb5f33df611ead56510ca3` changes only the parent's GitHub observation bound and independent cleanup assertion. All production code, packages and 25 accepted test files remain unchanged. Both focused GitHub idle cases pass in 15.26 seconds and the complete six-case cleanup suite passes in 8.47 seconds on two fresh 0411 PostgreSQL fixtures. This does not claim the full parent workflow was rerun here. ## Risks - A missing producer can delay work until restart. The audit covers current supported writers. Independent SQL and other processes remain outside the local notification contract. - Recovery validates existing targets; it cannot recreate a missing target or silently mark it delivered. Such work retains an error and retry. - The indexes add write/storage cost and use the existing restart-safe concurrent migration runner. - Other finalization duties still poll until their separate migrations. This PR does not prove host/database suspension, extend Cloud alarms, or claim fleet savings. - The actual prerequisite squash is adopted through a normal merge. Its complete tree equals the reviewed parent, and the complete accepted child tree is preserved. Fresh published-head CI and review remain required. ## Model Used OpenAI Codex, based on GPT-6, with reasoning and code-execution tools. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandPaperclip
|
4e715eb7ea |
fix(server): schedule historical unsafe export recovery (#17687)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs can retain accepted results after an unsafe workspace export fails. > - Historical recovery can retire an old notice or resume the accepted result without another provider turn. > - The periodic reaper repeatedly scans failures that cannot currently recover. > - Existing rows and committed changes can identify the actual recovery obligation. > - This pull request wakes recovery from those changes and retries only owed work. > - Dormant history becomes quiet while ownership and explicit operator decisions remain protected. ## Linked Issues or Issue Description **What happened?** The native finalization reaper repeatedly visits historical unsafe workspace-export failures. Cancelled tasks, newer runs, changed contracts, and explicit operator decisions can make those rows permanently ineligible. They still produce periodic database reads. **What did you expect?** Retire stale unsafe-export notices and recover eligible accepted results. Stop reading dormant history until a relevant committed change occurs. Keep a retry while a real obligation is blocked by ownership or a transient failure. **Steps to reproduce** Retain an unsafe-export terminal failure whose task is cancelled or superseded. Leave the instance idle after startup. The periodic reaper continues reading that history. Prerequisite #17680 is merged into master. Related: #14402 established safe export finalization; #14329 was an earlier historical recovery proposal. #17679 owns separate policy-review cleanup. This PR changes only historical unsafe-export scheduling and its producers. ## What Changed - Register a scoped worker through the existing coordinator and scheduler. Remove this duty from periodic reaping. Preserve the manual/startup recovery entry point. - Distinguish stale notices, eligible accepted results, dormant history, and concrete ownership contention. Reuse the existing mutation owner and lock order. Preserve explicit settlement vetoes and exact result/contract identity. - Signal relevant task, run, action, and deletion writes before they commit. Existing transaction boundaries handle nested commits and uncertain outcomes. Notifications contain only scope IDs. - Retain a retry only for owed work. Reconstruct work at startup, preserve drain/resume, and stop through the shared lifecycle. - Add concurrent, idempotent partial index migration 0410 for unsafe terminal failures. Keep the five newest generated schema snapshots. - Document producer coverage, query plans, supported writer boundaries, and test failures in the unsafe-export audit. ## Verification - Source `2a13dd08a6cc52f0adab09e3df9ce31e97b746b9` passes all 2188 cases across 24 complete files, without skips. The main selection passes 2129 cases in 192.58 seconds; safe-replacement passes 59 in 26.04 seconds. Coverage includes all 968 native-session cases, real export/action producers, explicit continuation, workspace reconstruction, interaction routing, Stop, queue and Slack consumers. The audit gives per-file counts. - Twenty separate fresh PostgreSQL 17 databases migrate through 0410 before setup. Four embedded-only fixture imports temporarily use their own external database on this IPC-exhausted host, then are restored byte for byte. Assertions and timing limits stay unchanged. This is qualified external-database acceptance, not an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-head migration order pass. Normal integration preserves the complete original runtime/schema patch and unrelated upstream blobs. Original concurrent index SQL, all prior executable history, journal entries and retained snapshot bytes are unchanged; five snapshots 0406 through 0410 remain. Added comment removal preserves the full non-comment TypeScript token streams. - A populated 0409-to0410 upgrade preserves all 408 prior migration receipts and 38 companies. It appends exactly one receipt; replay adds none. The index is valid and ready on fresh and upgraded databases. - The owning audit retains earlier fixture failures, skipped checks and notice-retirement regression history. Older combined stack checks retain their original scope in their owning audits. Actual parent `73d24aa5acfcf17b3f676b4da7cd05eb70b60b99` is reconciled through a normal merge. Its tree equals the reviewed ancestor and the complete accepted child tree is unchanged. Fresh published-head CI and Greptile 5/5 remain required. No deployed sleep, fleet-cost or live provider qualification claim is made. - Inherited prefix correction: current integration `8f43c9f81fac23b50997e7f0e1d0d9cf383a8109` changes only two values in a previously unselected corpus fixture. All production code, the complete package tree and all 24 accepted test files remain byte-identical to the 2188-case source above. The additional complete 72-case corpus passes in 14.00 seconds and again on the same database in 13.08 seconds. Retained PSC/OUT and sentinel rows remain unchanged. The repeat is not double-counted. The parent audit retains the original collision and rejected cleanup attempt. - Latest parent context: integration `4d51231abe8b68c21ce896d0d2ac16865978c8ab` carries only the parent's one-argument GitHub observation correction and an independent master cleanup assertion. All production code, packages, 24 accepted suite files and the additional corpus are unchanged. The two focused native-presentation cases pass in 15.14 seconds and the complete cleanup-removal suite passes all six cases in 8.31 seconds on two fresh 0410 PostgreSQL fixtures. This does not claim the full parent workflow was rerun here. Current head `b28399babac72ba9db1c4a4ef882c5e87aded38c` has passing CI and Greptile 5/5 with no actionable findings. The first CI attempt lost runners: fourteen jobs explicitly received simultaneous shutdown signals, two reported lost runner communication, and two aggregate gates failed downstream. One failed-job retry on unchanged source passes. The underlying host shutdown cause remains unconfirmed; no code fix or extra retry is claimed. [CI run and attempts](https://github.com/paperclipai/paperclip/actions/runs/40819408528). No Ponytail verdict is present. ## Risks - A missing future producer can delay dormant-history discovery until restart. The audit names current producers. Independent SQL writers and other server processes remain outside the process-local signal contract. - Physical ownership and uncertain operations retain retries. Time alone does not authorize taking workspace ownership. No provider turn or unsafe archive retry is added. - The recovery owner remains authoritative if state changes after discovery. Terminal UI status does not authorize resource destruction or accepted-result replay. - The partial index adds matching-row write/storage cost. It builds concurrently through the existing restart-safe migration runner. - This proves the changed worker becomes quiet. It does not prove host suspension, external wake coverage, or fleet cost savings. Other sleep migrations remain. This slice still requires fresh current-head CI and reviews. ## Model Used OpenAI Codex, based on GPT-6, with reasoning and code-execution tools. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
73d24aa5ac |
fix(server): schedule automatic completion review retirement (#17680)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native finalization preserves accepted work and resolves task status. > - A historical cleanup removes automatic completion-review cards that current policy no longer creates. > - The periodic finalization pass repeats that lookup even after all matching cards are gone. > - This PR gives this cleanup its own committed hints and pending-card retries. > - The existing owner retains withdrawal receipts and protects genuine reviews. > - Idle history stops causing recurring cleanup reads. ## Linked Issues or Issue Description Refs #17679. This is a separate stacked slice for automatic completion-review retirement. Obsolete version-change reviews belong to the parent PR. **What needs to be improved?** `reconcileNativeFinalizations` repeatedly discovers old automatic completion reviews. The discovery helper also converts SQL errors to an empty result, which would let a new event-driven worker forget failed reconstruction. **Why is this improvement important?** Idle tenants should not query for absent cleanup work. Failed discovery must preserve the retry, and removing a historical card must preserve the durable trigger for later evidence reassessment. **Proposed solution** Use the existing card, decision, effect, assessment and contract rows. Reconstruct candidates at startup, signal the existing decision commit, and retry exact pending cards. Keep the direct cleanup call inside active finalization. ROADMAP.md and the related native review work were inspected; this change owns the scheduling migration only. ## What Changed - Add automatic completion-review registration to the shared scoped deadline worker. Reconstruct 100-row keyset pages and dispatch fair batches of twenty. - Signal company/issue IDs before the native decision transaction writes. Keep outer-transaction settlement with the existing work-signal boundary. - Split raw discovery from the public helper's compatibility error handling. Worker lookup failures propagate so the shared scheduler retains dirty scope and retry. - Reuse the complete historical predicate and existing withdrawal owner. Recheck the selected card identity, payload and system author fields in the conditional update. - Preserve the withdrawal reason, chat publication, activity record, direct finalizer call, startup/manual recovery, and subsequent evidence reassessment. - Skip only this duty in periodic reaping. Retain admission, drain/standby, retry and joined shutdown. - Add producer/lifecycle documentation. Add no table, schema migration, provider branch or independent timer. ## Verification - Current source `3f4d35ac0154aedc76cc39486d3758c73b1e6ff4`: all 462 cases across seven complete files pass with no skips in 49.31 seconds. This covers both review workers through the real status-authority corpus, startup/timing, recovery, finalization, external chat and governed milestones. Nested corpus fixtures are not double-counted. - Five separate fresh PostgreSQL fixtures migrated through parent 0409 before setup. Database build, server typecheck/build, module boundaries and target migration order pass. The whole database source tree matches the parent; no migration or table is added. - Normal integration preserves substantive owned edits and unrelated upstream blobs. Actual landed-parent ancestry and fresh current-head Greptile 5/5 and clean CI remain required. This is worker and integration acceptance, not deployed sleep qualification. - The inherited P2 fix changes only two fixture prefix values. All production code, other fixtures, assertions and timing limits are unchanged. The full 72-case corpus also passes again on the same database in 12.95 seconds. Retained PSC/OUT rows and an unrelated sentinel remain unchanged after both runs. The repeat is not double-counted among the 462 distinct cases. The parent audit retains the original collision and rejected cleanup attempt. - CI correction: the 40f head fails the passive native-presentation initial idle observation in chat shard 1. An unchanged local pair passes. A controlled committed hint plus a 600 ms held run attempt makes the original one-second observation fail; the identical probe passes with the established five-second observation bound. This demonstrates the coalescing boundary, not the unrecorded exact CI token timeline. No unchanged-head CI retry was used. - The probe is removed. The only permanent code change is that initial poll bound. It still requires zero active work and no deadline, followed by two minutes with no SQL, transaction or provider writes. Production code, worker/business deadlines, rollback and committed-settlement assertions are unchanged. All 274 cases in the named GitHub agent-review workflow pass in 232.06 seconds. Other cases in the large file are outside that named selection. The earlier 462-case selection retains identical source. Both owning audits preserve the failure and scope limits. Fresh updated-head CI and reviews remain required. - Master integration: independent master `49fe8b87a49aa5a8985a08dd3d5ce57bdd24756c` adds one company-skill cleanup assertion. A normal merge preserves both complete binary patches. All production code and the accepted GitHub integration file are unchanged. The full six-case cleanup-removal suite passes in 8.16 seconds on fresh PostgreSQL. Database history is unchanged. ## Risks - A normal native decision can cause one scoped lookup even when it creates no historical card. It retains no periodic scan when settled. - Startup reconstruction cost scales with historical data and joins. No constant-cost or fleet-savings claim is made. - Process-local signals cover instrumented writers. Independent SQL writers and concurrent older controllers need explicit integration. - Retirement and status reassessment are distinct obligations. The existing durable withdrawal receipt and remaining finalization owner are preserved. - Failed withdrawals retain retries while exact pending cards remain. A database lookup failure must not be interpreted as completion. - Other finalization and recovery duties remain. Cloud wake coverage and host-sleep eligibility are unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Remote acceptance: head `be668fb975f119efeb4830405d4bb1ab65791f87` has 53 successful gates, verified with `gh pr checks` (no non-passing checks). Greptile reviewed that head at 5/5. There are no unresolved review threads. The PR remains draft and unmerged. Current published head `181bf800a79449961e09c995f2785dd7f8060bb3` has passing CI, fresh Greptile 5/5 and Ponytail 5/5 with no actionable findings. Earlier failures remain recorded in the audits. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
dfd5b1ac62 |
fix(server): wake obsolete policy review cleanup from commits (#17679)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native finalization repairs saved results and related task state. > - Its periodic pass also searches for obsolete policy-review cards. > - Those cards have a precise historical signature and existing durable rows. > - This PR gives that cleanup its own committed signals and pending-work retries. > - The existing owner still protects real reviews and later task changes. > - Settled review history no longer causes repeated cleanup queries. ## Linked Issues or Issue Description Refs #17677. This PR is stacked on cancelled workspace preservation. It owns only obsolete policy-review cleanup within the native finalization reconciler. **What needs to be improved?** Every periodic native finalization pass queries for old version-change review cards, even after all such cards are gone. The parent reaper combines this cleanup with unrelated finalization and recovery duties. **Why is this improvement important?** Empty queries prevent idle tenants from becoming quiet. Removing the query must preserve historical repair, real review requests, task status authority, and cleanup after a failed write. **Proposed solution** Reconstruct matching pending cards at startup. Signal the existing native decision transaction before its writes. Schedule bounded cleanup only while an exact card remains pending. Skip this duty in periodic reaping while keeping startup and manual entry points. ROADMAP.md and related PR search found no duplicate of this migration. ## What Changed - Register obsolete-review cleanup with the existing scoped deadline worker. Reconstruct in 100-row keyset pages and process fair batches of twenty. - Use company/issue hints from the existing status-decision commit boundary. Serialize only the two IDs; preserve outer commit and rollback. - Reuse the exact historical signature and existing cleanup owner. Keep status-version, decision-pointer, separate-review, activity and chat-publication guards. - Recheck selected payload and system-card identity in the conditional withdrawal so a concurrent edit cannot be cancelled from a stale snapshot. - Retain a 30-second retry only while a matching pending card survives. Preserve drain, standby, startup and joined shutdown. - Skip only this duty in the periodic finalization pass. Automatic completion reviews, unsafe exports, status effects, evidence reassessment and workspace finalization remain separate migrations. - Add the owner audit and validated external-test database support for the existing status-authority corpus. Add no schema, table, independent timer or provider branch. ## Verification - Current source `4cc0d03ae96bd92a4fdad44168c2a8647d42466c`: all 443 cases across seven complete files pass with no skips in 46.46 seconds. This covers the full status-authority corpus and cleanup worker, startup/timing, heartbeat recovery, native finalization, external chat and governed milestones. The audit gives per-file counts without double-counting fixtures inside the corpus matrix. - Five distinct fresh PostgreSQL fixtures migrated through parent 0409 before setup. Database build, server typecheck/build, module boundaries and target migration order pass. The whole database source tree matches the parent; no migration or table is added. - Normal integration preserves substantive owned edits and unrelated upstream blobs. The earlier circular notification-payload failure remains in the audit. Actual parent `51c3f8924cb0bd5fe7cbda15ad9b8f06a6a11055` is reconciled through a normal merge, with its full tree equal to the reviewed ancestor and the entire accepted child tree unchanged. Fresh current-head Greptile 5/5 and clean CI are still required. No deployed sleep qualification is claimed. - Review correction: the old head reproduces a fixed-prefix collision on a same-database corpus rerun. The initial deletion attempt passed assertions but failed teardown on non-cascading audit foreign keys; it was removed. The final two-value change uses each fixture company UUID in its prefix and changes no production behavior, assertions, timing limits or teardown. - The complete 56-case corpus passes a second time on the same database in 10.15 seconds. Retained PSC/OUT rows and an unrelated sentinel company remain unchanged after both runs. The repeat is not double-counted among the 443 distinct cases. The audit retains both failures and their causes. Fresh CI and Greptile are required for the updated head. ## Risks - Startup reconstruction still scans historical candidates and joins; its cost scales with history. Scoped commits use existing indexes. No recurring query remains after this worker settles. - The signal covers instrumented single-process writes. Independent SQL writers and concurrent older controllers need explicit integration. - A normal native decision can cause one scoped lookup, even when it creates no obsolete review. It does not retain a periodic scan. - Failed withdrawals retain retries while matching cards remain. Actual cleanup still uses the existing locking, authority and activity owner. - Other finalization duties still run. This PR does not establish whole-instance quiet, external alarm coverage, provider suspension or fleet savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Current published head `9aa67199da64d700fd6230829d38f2ef522a07fe` has fresh Greptile 5/5, Ponytail 5/5 with an unconditional ship verdict, and all current CI checks passing or skipped. The reproduced rerun P2 is fixed and its thread is resolved. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
51c3f8924c |
fix(server): schedule cancelled workspace preservation (#17677)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A cancelled native run may still owe preservation of prepared workspace files. > - The existing recovery receipt and source lease record that obligation. > - The orphan reaper still searches for these cancellations on every pass. > - This PR registers committed cancellations with the shared deadline worker. > - The existing retention owner keeps file pinning and ownership checks. > - Idle and completed history no longer causes this preservation scan. ## Linked Issues or Issue Description Refs #17674 and #17145. Stopped instruction collection is merged into master. It owns file-preservation discovery only. The existing environment cleanup worker continues to own stop-only teardown. **What needs to be improved?** `preserveCancelledNativeWorkspaces` runs from every orphan-reaper pass. It queries even when no cancelled native workspace needs a file-recovery hold. Current foreground Stop can also defer retention after a lock or database failure, so startup-only recovery would be insufficient. **Why is this improvement important?** Empty queries keep idle tenants active. Removing them must not lose user files, weaken Stop ownership, replay cancelled work, or turn a retained allocation into destructive cleanup. **Proposed solution** Register the existing durable obligation from committed run-status, cancellation-acknowledgement and workspace-reference writes. Reconstruct it at startup. Keep retries while actual preservation remains unresolved. Disarm after the existing owner records the hold. Related-PR search and ROADMAP.md review found no duplicate of this scheduling migration. ## What Changed - Add a company/run-scoped preservation worker with 100-row keyset reconstruction and fair batches of five. - Separate pending file debt from immediate coordinator/lease readiness. Retain a 30-second retry for known unfinished preservation and shared failure backoff for errors. - Signal the existing status, acknowledgement, workspace-reference, deletion and settings transactions. Add no table or independent timer. - Keep exact source checks, no-workspace contracts, task/coordinator/run lock order, retention receipt and no-replay recovery card with their owners. - Recheck cancellation acknowledgement, run status and coordinator phase under the retention locks. Preserve the existing foreground preterminal API. - Keep startup/manual recovery; skip only this duty in periodic reaping. Preserve drain/standby and shutdown behavior. - Add concurrent, replay-safe partial index migration 0409 and retain the five newest snapshots. - Add the producer and lifecycle audit in `doc/plans/2026-10-25-cancelled-workspace-preservation-sleep-audit.md`. ## Verification - Current source `21b46a8e4293ca670235117458b984493a181759`: all 1472 cases in 16 complete files pass without skips. The owner passes 84 cases in 14.93 seconds. Fifteen consumer files pass 1388 cases in 133.46 seconds, including all 968 native-session cases and Stop, dispatch, restart, workspace-sync and upgrade-readiness coverage. - Twelve separate fresh PostgreSQL 17 databases migrated through 0409 before setup. The embedded-only lifecycle fixture temporarily used its own external database on this IPC-exhausted host and was restored byte for byte. Assertions and timing limits stayed unchanged. This qualifies external-database behavior; it does not claim an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-head migration order pass. The complete original runtime/schema patch and unrelated current-parent blobs are preserved. Original concurrent index SQL, all earlier executable history and retained snapshot bytes are unchanged; five snapshots 0405 through 0409 remain. - A populated 0408-to0409 upgrade preserves all 407 prior receipts and 38 companies. Exactly one receipt is appended; replay adds none. The preservation index is valid and ready on fresh and upgraded databases. - Actual parent `82529741418e138d9d623bf13d2fa22c1ff79d36` is merged through a normal ancestry reconciliation. Its complete tree equals the reviewed parent. The entire accepted child tree and all source hashes are preserved. The audit retains earlier failures, qualifications and per-file counts. Published head `b34f0589847f33754ecde9413fd36bf007353d2c` has Greptile 5/5 and all 57 checks passing or skipped. No Ponytail review is present. These checks do not establish deployed host/database suspension or Cloud alarm delivery. ## Risks - Signals cover instrumented single-process writers. Independent SQL writers and additional replicas need integration. - Status signals may inspect a run without relevant file debt once. They do not create recurring work. Discovery and memory still scale with actual candidates. - Missing ownership or stop evidence remains pending. Active retries are intentional while a concrete file obligation is unresolved. - Retry timing is process-local; startup reconstructs durable debt. The worker does not overwrite another native-finalizer deadline. - Admitted database operations may finish after drain begins. Shutdown joins them. Provider teardown remains with the existing cleanup owner. - The partial index adds storage and write cost. Its concurrent migration uses the existing interrupted-index repair path. - Cloud alarm coverage, sleep eligibility and host suspension are unchanged. This proves worker behavior, not fleet savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8252974141 |
fix(server): schedule stopped instruction collection (#17674)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runs can stop while their private instruction edits still need collection. > - Existing copy rows and stop evidence describe that unfinished work. > - The orphan reaper still queries for it on every maintenance pass. > - This PR activates collection from committed copy and run transitions. > - It uses saved retry deadlines when stop proof or collection remains pending. > - Active copies and completed history no longer cause collection polling. ## Linked Issues or Issue Description Refs #17671 and #17670. Both prerequisites are merged into master. Captured saves and post-save release remain separate reasons with their own workers. **What needs to be improved?** `recoverStopped` queries instruction copies on every orphan-reaper pass, even when no terminal run owes collection. A run can become terminal without changing its copy row, so copy-only notifications would miss real work. **Why is this improvement important?** Repeated empty queries prevent idle tenant instances from becoming quiet. Removing them must preserve edits, stop proof, current ownership, and recovery after restart. **Proposed solution** Use the existing transaction signals, scoped deadline worker, and scheduler. Reconstruct eligible copies at startup, signal supported copy and terminal-run writers, and retain saved retry deadlines for unresolved work. Preserve the existing collection and directory owners. Related PR search found the warm agent-file ownership work (#14695), verified stop boundary (#14311), and task-workspace cleanup (#16464). This PR does not replace those contracts. ROADMAP.md has no duplicate migration. ## What Changed - Add scoped stopped-instruction collection with keyset reconstruction and fair batches of 20. - Signal before copy preparation, mutation and adoption writes, and before terminal run writes in their existing transactions. - Cover heartbeat, native finalization/restart/session recovery, execution-control, queue and dispatch producers. - Signal settings changes and company/agent deletion. Preserve drain, standby, restart and shutdown ownership. - Recheck terminal-run eligibility in the due-time claim. Keep terminal status separate from process-stop proof. - Retain existing stop checks, attempt limits, saved candidate bytes and remote-resource fencing. - Skip only this lane in periodic orphan recovery. Preserve manual/startup recovery. - Add concurrent, replay-safe partial index 0408 and retain the five newest snapshots. Add no table. - Record the owner and producer audit in `doc/plans/2026-10-25-stopped-instruction-collection-sleep-audit.md`. ## Verification - Source `cb2818a8f46ef961f744977a258f0e9042e31a42`: all 1469 cases across 14 complete files pass without skips in 141.69 seconds. This includes all 968 native-session cases, filesystem saves, startup, terminal-run producers, dispatch, queue, execution-control, finalization and lifecycle. - Eleven separate fresh PostgreSQL 17 databases migrated through 0408. The embedded-only lifecycle fixture temporarily used its own external database on this IPC-exhausted host and was restored byte for byte. Assertions and timing limits stayed unchanged. This qualifies external-database behavior; it does not claim an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-current migration order pass. The original concurrent index SQL is unchanged. All prior executable migration history, journal entries and retained snapshot bytes are preserved; the five newest snapshots are 0404 through 0408. - A populated 0407-to0408 upgrade preserves all 406 prior migration receipts and 38 companies. It appends exactly one receipt. Replay adds none. The collection index is valid and ready on fresh and upgraded databases. - The normal merge of actual parent `267842fc3c6e6542deed8a1bcc2c78bec8df7862` preserves both complete patches, all independent sidebar changes from #18117, and the full accepted server/package/test trees. The complete sidebar test file passes all 36 cases on the combined tree. React act warnings remain. - Prior failures and fixture qualifications remain in the audit. Published head `d176d731f083c7edd2ede688cb901d1e66d8f1d8` has Greptile 5/5, all 57 checks passing or skipped, and no unresolved review threads. No Ponytail review is present. These local tests do not establish deployed host/database suspension or Cloud alarm delivery. ## Risks - Process-local signals cover instrumented writers. Independent SQL writers and additional replicas still need integration. - Terminal-run signals can cause one scoped inspection even if no instruction copy exists. They do not create a recurring scan. - Reconstruction and memory scale with actual pending candidates; active candidates are filtered by the run join. - Admitted file/provider operations can outlive a later drain request. Shutdown joins them. Uncertain ownership remains pending rather than being discarded. - The partial index adds storage and write cost. Its concurrent migration uses the existing interrupted-index repair path. - Cloud alarms, sleep eligibility and provider suspension are unchanged. Worker quiet is not a claim of deployed host/database sleep or measured savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL/filesystem tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
267842fc3c |
fix(server): schedule owed agent directory cleanup (#17671)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent runs can save their directory before temporary-file cleanup finishes. > - The existing receipt records stop proof and unfinished cleanup. > - The orphan reaper still searches for this debt on every maintenance pass. > - This PR activates cleanup from committed changes and saved retry deadlines. > - The directory owner keeps its resource checks and file lock. > - Settled directory history no longer causes recurring cleanup queries. ## Linked Issues or Issue Description Refs #17670, the merged captured-instruction commit slice. This PR targets master. It does not include stopped instruction collection or change the saved-file contract from #14695. **What needs to be improved?** The post-save part of `recoverCaptured` queries directory copies on each orphan-reaper pass, even after all temporary directories have been released. **Why is this improvement important?** Empty database work prevents idle tenant instances from becoming quiet. Cleanup must still survive a crash after a save and preserve files when stop or resource ownership is uncertain. **Proposed solution** Use existing copy receipts as the source of truth. Notify the shared coordinator inside relevant copy-write transactions. Reconstruct pending work at startup and register each saved retry deadline. Keep physical cleanup in the directory owner. Add a partial index over unfinished cleanup rather than a new queue table. Related-PR search found workspace archive and SSH task-workspace cleanup changes (#17175 and #16464). They own different resources and do not replace this stopped agent-directory receipt recovery. ROADMAP.md has no duplicate migration. ## What Changed - Add a scoped directory cleanup worker with keyset discovery and batches of 20. - Signal relevant copy transitions, company/agent deletion, and worktree execution policy changes. - Preserve the due-time claim, current receipt read, directory lock, stop proof, and exact remote-resource checks. - Read the receipt after release; a returned promise does not establish cleanup completion. - Keep unresolved cleanup on its saved retry or shared failure backoff. Final receipts disarm the worker. - Register the worker during startup and skip only this lane in periodic recovery. Keep manual recovery compatible. - Add concurrent, replay-safe partial index migration `0407_greedy_star_brand.sql`. Retain the five newest schema snapshots. - Add the producer audit and lifecycle evidence in `doc/plans/2026-10-25-directory-cleanup-sleep-audit.md`. ## Verification Published head: `210fcb1215a3b5e5ed84e420ddb73ab5952d9365`. Source acceptance: `70a6800eea874bdb87685bf6eea66de79c48b577`. Parent #17670 landed as `b294de546bf9cd3a8578a7a66d7e8ec0eb939799`; normal squash-ancestry reconciliation preserves the entire accepted descendant tree. Final follow-ups change audits only. - All 284 cases in six complete filesystem, startup and heartbeat owner files pass, without skips, in 48.88 seconds. Four unused fresh PostgreSQL 17 databases migrated through 0407 isolate destructive fixtures. - Three fixtures use supported external database paths. The embedded-only lifecycle fixture temporarily uses its own external database on this IPC-exhausted host, then is restored byte for byte. All assertions and original timing limits remain unchanged. This qualifies the local result; CI must still pass with the original embedded fixture. - Actual database build, server typecheck/build, module boundaries and exact-current migration order pass. The original concurrent idempotent index SQL bytes are unchanged. All prior executable migration history, journal entries and retained snapshots stay intact. The five snapshots are 0403 through 0407. - Populated 0406-to0407 upgrade preserves all 405 old receipts and all 38 companies, adds exactly one receipt and adds none on replay. The cleanup index is valid and ready on fresh and upgraded databases. The entire current database tree equals the source used for this proof. - Both complete original and upstream patches are preserved. The actual parent squash tree equals the already-adopted reviewed parent; resolving eleven duplicate-ancestry text conflicts leaves the whole accepted tree unchanged, not just selected files. Complete server, package, UI and test trees still match the 284-case source acceptance. - The audit retains earlier ownership, stopped-resource, receipt, fixture and local-runner failures. The no-projects build filter was not counted as a build pass; corrected actual build and complete reruns passed before publication. Old draft review scores do not count for this new head. - Current-head Greptile gives 5/5 with no actionable findings, and all 57 checks pass. These checks establish worker behavior, not deployed host/database suspension or measured savings. ## Risks - Signals cover the existing single-process writer model. Independent SQL writers and replicas need separate integration. - The worker retains identifiers for actual pending work. Its index excludes completed history but does not make large pending backlogs free. - File/provider cleanup can outlive an admission change once started. Shutdown joins admitted operations. Unknown remote ownership remains pending and no replacement environment is acquired for cleanup. - The partial index adds storage and write cost. Its concurrent migration uses the existing interrupted-index repair path. - Cloud alarm coverage and host-sleep safety remain unchanged. This establishes worker quiet, not deployed host/database suspension or measured savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL/filesystem tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b294de546b |
fix(server): recover captured instruction saves from committed deadlines (#17670)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A stopped run can leave captured instruction edits waiting for a persistent save. > - Existing working-copy records retain the candidate bytes, attempt count, and retry deadline. > - The orphan reaper scans those records on every heartbeat, even when no save is due. > - This PR wakes the existing save owner after committed changes and at saved deadlines. > - The worker preserves permissions, conflicts, and bounded retries, then becomes quiet. ## Linked Issues or Issue Description Refs #17667, the merged preceding PR in this recurring-server-work stack. This PR targets master. Related warm-file ownership work: #14695. Its process-stop and retained-directory guarantees remain with their existing owner. The dedup search found no equivalent captured-save scheduling migration. **What happened?** The commit portion of `recoverCaptured` runs inside the periodic orphan sweep. Prepared copies, completed history, conflicts, and exhausted candidates do not need repeated save discovery. A saved candidate does need recovery after restart or a failed foreground save. **Expected behavior** Committed candidate changes reactivate recovery. The worker honors the saved deadline and existing three-attempt bound. It stops querying after settlement or deliberate conflict/exhaustion. Permission checks and canonical save semantics remain in the revision service. **Steps to reproduce** 1. Start a server with heartbeat scheduling enabled and no captured save pending. 2. Observe repeated pending-commit queries from the orphan reaper. 3. Capture edited instructions and fail their first canonical file replacement. The durable candidate must be saved after its retry deadline without starting a provider. **Paperclip version or commit** Reviewed parent `405e533394b26215e22d0452e9b0207d3f784a79`. Actual parent squash and current master baseline `27270e9bf8b7a294cf03f8b84f4857f1b1ff05c7`. **Deployment mode** Self-hosted server with the existing single-process transaction signal model. Local verification uses PostgreSQL 17 and actual managed instruction files. ## What Changed - Preserve foreground instruction-save ownership and its failure deadline before recovery spends another attempt. - Add a bounded captured-commit worker inside the instruction-copy owner. Existing rows and deadlines supply its state. No new table, migration, or external scheduler is added. - Signal writes entering or leaving pending commit from the existing copy mutation boundary. Only these writes gain a short transaction. File collection, file replacement, and provider operations stay outside that new transaction. - Signal company/agent deletion and worktree execution setting changes. Preserve drain, standby, startup reconstruction, shutdown join, and uncertain-transaction handling through the shared coordinator. - Guard attempt claims with current copy ownership and a still-due deadline. A concurrent writer that postpones the retry wins without consuming an attempt. Manual/startup recovery uses the same guarded owner. - Remove only captured commits from periodic reaping. Stopped collection and the separate post-save directory cleanup sweep remain for later slices. - Add real producer, file, transaction, concurrency, and lifecycle tests. Add guarded external test databases to the two filesystem suites so they can run where embedded PostgreSQL is unavailable. ## Verification Published head: `915ef6f20a68fffe89711cc97a6196521ea9885d`. Source acceptance: `9fb75e508f06b05b740099403ce021a61e7c6603`. The actual #17667 squash ancestry is recorded by a normal merge. Its entire result tree equals the prior accepted descendant tree; the final commit changes the audit only. - All 268 cases in six complete files pass, without skips, in 41.89 seconds on the updated parent. Coverage includes captured saves, directory consumers, startup, instruction-copy lifecycle boundaries, recovery and run lifecycle. - Four unused fresh PostgreSQL 17 databases migrated through parent 0406 isolate the destructive fixtures. Three use supported external URL paths. The embedded-only lifecycle fixture temporarily uses its own external database on this IPC-exhausted host, then is restored byte for byte. Assertions and original timing limits are unchanged. CI must still pass with the original embedded fixture. - Actual database build, server typecheck/build, module boundaries and exact-current migration order pass. Commands select each operation by name. The earlier local no-projects filter error remains recorded and is not treated as a build pass. - The normal parent merge preserves both complete added/removed patches and every unrelated upstream blob. The entire database subtree is parent-identical, with no owned schema or migration changes. New parent worker behavior has current consumer/startup acceptance here and separate owner acceptance in #17667. - Earlier foreground ownership, postponed deadline, attempt cap, drain hold, fixture and skipped-test findings remain in the audit. The prior head's Greptile 5/5, Ponytail 5/5 and green CI are historical after this new publication. - Current-head Greptile and Ponytail both give 5/5. All 58 checks pass. Prior stacked-head scores are historical; the new published SHA requires fresh gates. These checks do not establish deployed host/database suspension or external wake delivery. ## Risks - Signals cover supported in-process writers. Independent SQL writers and other replicas require a separate ingress contract. Startup reconstructs persisted obligations. - Exhausted candidates stay preserved under the existing policy; the worker does not increase retry limits or silently discard saved edits. Database reconstruction failures retain shared backoff. - Inventory pages scalar fields but retains candidate identifiers in memory. The query-plan result covers completed history, not a constant-cost promise for arbitrary exhausted-pending history. - File saves can outlive a stop request once admitted. Shutdown joins them. Exact-byte replay and current authorization remain with the revision owner. - Other instruction recovery duties and other stack blockers remain. This PR does not prove host suspension, database suspension, or external wake delivery. An ordinary protected merge is authorized only after current review and CI gates pass. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and local PostgreSQL/filesystem tests. The runtime does not expose a more precise model identifier or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Review follow-up: remove seven newly added comment lines to satisfy CONTRIBUTING.md, preserving the explanations in the owning audit. Both changed files retain the exact non-comment TypeScript token stream; other source bytes are unchanged. The original cffe server-shard-9 run ended after a runner shutdown signal and cancellation, without a final assertion verdict; that failure is retained and not counted as a pass. No old-head retry is requested. Fresh current-head CI and Greptile are required. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
27270e9bf8 |
fix(server): wake interrupted retry cancellation from committed intent (#17667)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A parked scheduled retry has no executor to finish an interrupted Stop request. > - Its saved cancellation fence already records the work that remains. > - The orphan reaper currently scans for that work on every heartbeat, including idle periods. > - This PR registers the existing records with the shared delivery scheduler and wakes recovery after committed writes. > - Recovery keeps the existing cancellation guards and settlement behavior, then becomes quiet. ## Linked Issues or Issue Description Refs #17666, the merged preceding PR in the recurring-server-work migration stack. This PR targets master. Related cancellation work found in the dedup search: #12301 and #7340. This PR changes the trigger for interrupted parked-retry cancellation; it does not replace their cancellation or retry policies. **What happened?** The periodic orphan reaper queries scheduled retries with cancellation fences even when no cancellation needs recovery. Its first-20-row scan can also stop behind malformed evidence. **Expected behavior** Committed cancellation intent wakes bounded recovery. Failed work retains a retry. Startup reconstructs durable work. Empty or settled work performs no recurring SQL for this duty. **Steps to reproduce** 1. Start a server with heartbeat scheduling enabled and no interrupted retry cancellations. 2. Observe the orphan reaper repeatedly query for cancellation fences. 3. Cancel a scheduled retry and fail the final status write after its fence commits. Recovery must complete the existing Stop request without running a provider or dropping its metadata. **Paperclip version or commit** Current parent squash `7afee8f0622e273d10506b0febfb68f143d168b1`. Source acceptance `66107ae0a7ad85fd0476f91c4f8f98ae74a7a048`. **Deployment mode** Self-hosted server with the existing single-process transaction signal model. Local evidence uses PostgreSQL 17. ## What Changed - Add one domain-owned recovery worker over existing heartbeat run rows. No new table. It delegates to run control and preserves original cancellation, budget, and operator metadata. - Signal cancellation fences, lifecycle transitions, dispatch deferral/promotion/suppression, relevant settings changes, and deletion within their owning transactions. Wrap only the ordinary database fence in a short transaction; provider calls stay outside it. - Keep startup/manual recovery compatible and remove this duty from the periodic orphan sweep. Other reaper duties remain for separate PRs. - Extend the shared scoped worker to retain pending work with a memory-only admission retry while its owner is paused. Without pending work, it schedules nothing. Existing GitHub consumers retain their behavior. - Add a concurrent idempotent partial index and lifecycle/producer tests. Enable guarded disposable PostgreSQL URLs in affected test harnesses. - Reconstruct one hundred candidates per scope per scheduler pass. Resume from the saved scope cursor and retain fair twenty-item execution batches. Manual recovery starts cancellation before reading the full backlog. - Require the recovered browser reply to be saved before testing reload. Keep all original assertions and time limits. - Record the full producer audit, lifecycle limits, and failed attempts in `doc/plans/2026-10-25-scheduled-retry-cancellation-sleep-audit.md`. ## Verification Published head: `405e533394b26215e22d0452e9b0207d3f784a79`. Source acceptance: `a35fa43b98e2bd378d4a920d2775ce243654838b`. The final commit changes the audit only. - All 363 backend cases in ten complete owner and consumer files pass without skips. This includes actual Stop/retry producers, transaction settlement, interrupted-write recovery, pause/resume, shutdown, fair backlog processing and quiet after settlement. Seven separate fresh PostgreSQL 17 databases prevent fixture interference. - The two embedded-only backend fixtures temporarily use separate external databases on this IPC-exhausted host. Their fixture imports and native client teardown are restored byte for byte afterward. Assertions and timing limits are unchanged. CI must still pass with the original embedded fixtures. - Two paging regressions fail on the prior worker and pass on the fix. They cover cancellation before full reconstruction, a malformed first page, fair batches, a newly committed intent behind the cursor, and no later idle SQL. Per-candidate admission reads remain deliberate safety checks. - The repeated CI reload failure is reproduced with a temporary twelve-second fixture settlement delay. The trace retains the same five-second missing-reply assertion, even though a cleanup race masked it in the top-level error. Requiring exactly one saved reply from the resumed run passes under the identical delay. The fault is removed. All six browser journeys pass on the final source in 40.5 seconds, with zero retries and the original 120-second case and five-second reload limits. - Browser acceptance uses the canonical CLI/server/static-UI path and a fresh external database. Only fixture database-URI setup is temporarily adapted and then restored. This is not an unmodified embedded CI-environment reproduction. The six cases include the controlled case; repeated runs are not added to the distinct count. - Actual database build, server typecheck/build, module boundaries and exact-current migration order pass. The database source tree is unchanged from the verified 0405-to0406 upgrade: all 404 prior receipts and 300 companies stay intact, exactly one receipt is added, replay adds none, and the index is valid on upgraded and fresh databases. The five snapshots remain 0402 through 0406. - All earlier failures remain recorded in the audit. Prior head `48b3fcb83f305616df02a4337a42ec0867bb4b80` had exact-head Greptile 5/5 but failed the repeated browser assertion. It was not retried blindly. Its completed Ponytail review's backlog concern and old migration paragraph are addressed here. Old scores do not satisfy new-head gates. - Current-head CI passes. Exact-head Greptile and Ponytail both give 5/5, with no remaining actionable findings. These checks do not prove deployed instance or database suspension. ## Risks - Only supported in-process writers produce local transaction signals. Independent SQL writers or other replicas need a separate ingress contract. Startup reconstructs saved obligations. - A real failure retains bounded retries. Unknown commit outcomes retain the shared intent hold. Neither is discarded to report quiet. - Per-owner drain uses a memory-only admission recheck. The shared helper change affects existing consumers, which have focused regression coverage. - The partial index is concurrent and idempotent; it adds index storage. Each scoped load reads at most one hundred rows. The shared helper retains discovered pending IDs between passes, so a large unresolved backlog still uses memory. - Cloud alarms, idle-safety exemptions, and other reaper duties remain separate. Merge requires passing current-head CI and review gates. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and local PostgreSQL tests. The runtime does not expose a more precise model identifier or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f52da40f08 |
fix: resume queued turns after maintenance (#17967)
Keep an accepted turn's starting input in task history, show its admission wait, and resume existing queued runs promptly when maintenance ends or expires. Verified with behavioral red/green regressions, real PostgreSQL and process execution, browser/native Runner acceptance with a deterministic provider, workspace typecheck, production build, and current-head CI/review. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
09ad9adce2 |
feat(server): adopt running remote agents through durable controller checkpoints (#17439)
## Thinking Path > - Paperclip must keep admitted remote work alive while its main server is replaced. > - Runner owns recovery mechanics and exposes opaque checkpoints through #17097. > - The server owns company scope, encryption, Stop, tool authority and execution ownership. > - A successor must prove current ownership before it adopts the original execution. > - Checkpoint bursts must also leave capacity for normal requests and lease renewal. > - This PR persists and adopts that authority through the existing runtime boundaries. ## Linked Issues or Issue Description **What existing behavior does this improve?** Remote agents retain the original execution during a bounded main-server outage. **Current behavior** A server restart can lose the controller's local state and connection. An unsafe replacement could repeat a turn or an uncertain mutation. Checkpoint writes in the request pool can also delay ordinary requests during an aligned burst. **Proposed behavior** Restore encrypted control records from Postgres and adopt the original runner under a fenced lease. Recover known tool receipts through current tool authority. Give control checkpoints a shared pool capped at two connections per database owner. Depends on #17097. Successors are #17455 and #17456. Qualification and presentation remain separate review slices. ## What Changed - Add company/run-bound encrypted checkpoint records and migration 0394. Keep the SQL idempotent after renumbering against current master. - Recover startup and expired ownership through native runtime and heartbeat. Preserve the sandbox lease, admitted runner artifact, harness configuration, native session and active turn. - Keep Stop, finalization, preparation ownership and stale-writer fencing transactional. Preserve graceful detach and natural crash takeover. - Recover committed tool receipts through the existing authority. Reauthorize reads and hold uncertain mutations. - Bound startup retention with the existing coordinator and preserve zero idle scans. - Share at most two checkpoint connections per database client. Keep application and liveness capacity separate. Close the checkpoint pool during shutdown; late writes cannot reopen it. - Count SQL across both application and checkpoint clients. Prove request/liveness progress, command recovery and failure after pool shutdown with real Postgres. ## Verification Project creation recovery now uses the existing project activity receipt owner. It retains authenticated input normalization, fingerprint checks and current run authority after the receipt lock. An absent receipt returns no result before mutation. The new lost-HTTP-response and no-effect regression uses real API routes and Postgres. The seven transport cases pass locally. This Mac skips all 20 database cases. On current source `5de803c2cd`, Linux [job 122593415890](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415890) executes all 20 project-tool cases successfully (9.480 seconds), including the committed-project lost-response regression. Its shard passes 1,627 cases with three unrelated skips. Server typecheck passes. The transport regression initially has two failures; a subsequent incorrect expectation about schema defaults was corrected before the final pass. Ordinary execution adds no checkpoint writes or recurring work. Current candidate `5de803c2cd` includes the project-creation receipt recovery fix and master `2cd747d1b6`. Migration 0394 retains the prior idempotent checkpoint DDL. Every master SQL file and journal entry is preserved. Snapshots 0390–0394, migration checks and server typecheck pass. The ordinary warm-turn regression fails on its second turn before `a219ab51ad`. After correction, three completed warm turns select the correct prior checkpoint and preserve the original session. All 288 provider-wiring cases pass. The control plane, external persistence callback, identity rotation and filesystem are real; the backend and database-load seam are scripted. The 13 warm admission/configuration variants also pass after final master integration. This does not qualify an underlying harness. At current source `5de803c2cd`, Linux [job 122593415907](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415907) passes all 940 native-session-executor cases (61.431 seconds), including the warm external-checkpoint behavior; the shard passes 2,358 cases with one skip. Current [Runner job 122593415475](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415475) passes the native recovery lane: 115 passes and five container-only skips, including the encrypted restore guard, fresh preparation (63.189 seconds), blocked claim lock (1.226 seconds) and recovery beyond 101 blocked owners (3.897 seconds). The first lane passes 2,215 cases with one skip. The SQL cost test reports zero idle and streaming SQL for both output volumes; this is component evidence, not a production capacity claim. At prior candidate `1ff7bb44f4`, Linux job 122549307215 passes all 938 native-executor cases, including the three-turn external-checkpoint regression. Job 122549306469 passes the native recovery lane with 115 passes and five container-only skips, including the encrypted-restore session guard, fresh preparation (63.664 seconds), claim-lock contention (3.961 seconds), and 101-owner recovery (3.824 seconds). This Mac skips the real-Postgres suite. The earlier full local executor attempt has 921 passes and 16 failures during embedded-Postgres initialization before assertions. The original fixture collisions and their corrections remain recorded. At earlier candidate `8fbe62b41d`, server shard job 122535673841 has 3,876 passes, one failure and one skip in an unchanged expiry-worker retry/drain timing test. The same server/database tree on qualification source `b64cac6951` passes all 20 expiry cases and all 3,877 shard cases. The single diagnostic retry, job 122541891915, passes all 20 expiry cases and all 3,877 shard cases (one skip). No assertion was weakened. The original failure remains recorded and its exact timing race is not proven. All Actions checks and legacy status passed on `8fbe62b41d`, with completed Greptile 5/5 and no open feedback. The next candidate `1ff7bb44f4` also passed all Actions checks and Greptile 5/5. Its Paperclip Review remained queued before the new master conflict required integration. The new candidate needs fresh exact-head CI and reviews; old checks do not qualify it. Migration generation is checkpoint-only, all master SQL and journal entries remain unchanged, and migration checks, server typecheck and all 13 warm-owner variants pass after integration. The first migration check was blocked by TSX sandbox IPC; its permitted repeat passes. The first warm-test command used an incorrect path and ran no tests; the corrected command supplies the 13-case receipt. All 16 database pool-lifetime/options/teardown cases pass after granting loopback permission; the first batch had two EPERM failures and 14 passes. Earlier semantic-tool/HTTP recovery, adoption, encryption-batch, zero-streaming-write and idle-SQL receipts remain source-attributed in the [qualification record](https://github.com/paperclipai/paperclip/blob/2ff66f57da0da35bac1c19a2366ea4b84d87a986/doc/plans/2026-10-22-hot-upgrade-qualification.md). That record retains earlier assertion, setup and cleanup failures and their corrections. No failed gate is waived. The historical 300-session component campaign passes four after phases: steady read p99 is 107–108 ms at 500 reads/s and 100 writes/s; aligned-burst read p99 is 103–105 ms after the bounded pool, versus 584–689 ms before. This comparison isolates the persistence corrections. It does not measure the final stack's hosted active-agent capacity. The hosted zero-agent API pair leaves SQL and CPU essentially unchanged, but both arms fail absolute latency gates and candidate peak cgroup memory fails its gate. The active-agent/viewer matrix, different-artifact upgrade and rollback remain open. Run `PAPERCLIP_TEST_DATABASE_URL=<owned-disposable-postgres> pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`. Previous head `f2db8a6568` had one failed workspace setup gate: [job 122555672441](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122555672441) passes all 4,169 UI tests and 285 CLI cases (one skip), but the company import/export suite reaches its 30-second health deadline during doctor startup. Its output does not reach the server-start step. The CLI, server, database, shared, adapter and plugin trees are identical on qualification source `a866619d7b`, whose [job 122555886663](https://github.com/paperclipai/paperclip/actions/runs/40609057937/job/122555886663) passes that fixture and all 286 CLI cases. The one diagnostic retry, [job 122561057225](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122561057225), passes the formerly failing fixture and all 286 CLI cases. All Actions checks and legacy status on `f2db8a6568` pass. Its exact-head Greptile completed at 5/5 with no open feedback. The current integrated head requires fresh checks and a completed Paperclip Review before merge. The original failure remains recorded; its exact timing cause is unproven. A repeated failure requires investigation and repair rather than another retry. All current-head CI, legacy status, exact-head completed Greptile 5/5, resolved feedback and ordinary GitHub merge eligibility remain required before merge. The latest integration preserves master’s published Copilot v70 profiles and every published migration. Checkpoint migration `0394_conscious_whizzer.sql` was generated with `pnpm db:generate` and retains the prior idempotent DDL. The Runner vendor rebuild and replay parity checks pass. Server typecheck and all 13 warm-owner variants pass after that rebuild. The imported report-lock patch is byte-identical to master; its intentional patch-context whitespace is retained, and the feature delta passes whitespace checks. No hosted experiment was run. ## Risks The pool adds up to two active connections per instance, shared across agents. Checkpoint delivery can wait behind that bound; it still requires its commit. Steady component-test Postgres CPU rises by about 1.7 percentage points of a two-CPU allocation. These corrections do not remove the stack's control and renewal writes. The migration and key must survive replacement. Fresh-database tests and local load do not establish different-artifact hosted upgrade or rollback continuity. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
94a780286f |
fix(server): schedule tool health from committed changes and deadlines (#17554)
## Thinking Path > - Paperclip manages AI-agent work and its connected tools. > - Idle hosted instances need their database and network activity to stop. > - The maintenance tick queries tool connections even when no health check is due. > - Existing connection rows already contain eligibility and the last health observation. > - This pull request reconstructs health deadlines and refreshes them after committed changes. > - The shared coordinator preserves retries, drain, and shutdown without another private timer. ## Linked Issues or Issue Description **What happened?** The aggregate maintenance tick scans connection health every 30 seconds. Empty instances and connections with fresh health observations still cause SQL activity. **Expected behavior** No eligible connection means no recurring health SQL. Eligible connections retain the existing 15-minute proactive check. Configuration changes and health observations update the schedule after commit. **Steps to reproduce** Start a server with heartbeat scheduling enabled and no tool connections. Let startup finish and observe two maintenance ticks. The old code repeats the health inventory query. The new startup regression verifies that those ticks do not invoke the health inventory. **Paperclip version or commit** This branch includes merged parent #17518 and master `96727b3cfefb61f85ffe1fbc29a9b6dd6837efe8`. The 47 changed paths belong to the single tool-connection health duty. No database file differs from master. Related: #14638 covers runtime-auth exclusions. #15448, #15325, and #11910 concern MCP failure classification. This PR preserves those provider policies and does not replace their work. ## What Changed - Keep the GitHub error-projection unit fixture at the health-write boundary and verify the real transaction path in the PostgreSQL health-worker suite. - Add a health schedule owned by tool access. Read lean eligibility fields and saved observation timestamps. - Register health work with the existing coordinator. Refresh only dirty connection/application/company scopes using bounded inventory pages, check due connections in fair company batches, continue concrete due backlog after one second, and retain failures with backoff. - Notify before supported eligibility and observation writes in their owning transactions. Cover API/gallery setup, OAuth, catalog/gateway observations, email, voice, access removal, deletion, and settings. - Commit connection creation with grant, binding, and runtime-slot SQL before its wake. Keep provider calls outside the new transactions. - Keep provider-to-transport mapping in the chat catalog. SDK-only chat changes do not wake health work. - Replace aggregate health scans, retain the public manual sweep, and document the producer audit and remaining sleep limits. - Keep metadata-only example and Smoke Lab installs scoped. Lock and re-read application type before publishing a real eligibility change. - Keep failed inventory reads on their own retry deadline while healthy due batches continue. - Apply fixture migrations before using external disposable databases in all six producer suites. ## Verification Published head: `8c1d293d1ae2b7db753ab20f5a2a2edec1fb0a62`. Fresh CI and reviews are required for this exact head. - **623 distinct targeted cases pass.** The final four complete health, fair-batching, GitHub health recovery, and startup suites pass 80 cases after the retry fix. Five complete producer suites pass 543 cases: tool access 469, access 18, email 45, GitHub events four, and Smoke Lab seven. The earlier 26 health cases are part of this total, not additional cases. - The six producer suites first passed against separate, initially empty PostgreSQL 17 databases at `acbf262843`. They apply their own migrations. The subsequent repair changes only the health worker and its regression; its complete final consumer suites pass on that integrated parent. Server typecheck and build pass with the final repair. - **Installer amplification:** five real installs with 205 unrelated connections reproduced 1,035 example rows and 1,045 Smoke Lab rows. Repeated metadata installs now load only changed fixture connections. The same cases restore an excluded application through each installer and require all 205 unrelated IDs to re-enter the schedule. - **Concurrent eligibility:** a real PostgreSQL lock-wait case requires the installer to notify after a concurrent type change. A mutation that retains the lock but uses the old lookup fails the settled-hint assertion; original source was restored byte for byte. - **Retry isolation:** the failing baseline read one unavailable company three times while another company's 60 due checks drained. Final tests require one initial read, at least 30 seconds between failed reads, prompt healthy progress, eventual recovery, and SQL quiet after settlement. - Real-service producer coverage includes AgentMail activation/pause, GitHub installation changes, and member access removal. Lifecycle coverage includes outer commit/rollback, startup reconstruction, drain/resume, failure, and shutdown. - Failure history and exact evidence scope: [health acceptance](https://github.com/paperclipai/paperclip/blob/8c1d293d1ae2b7db753ab20f5a2a2edec1fb0a62/doc/idle-tool-connection-health.md#acceptance-follow-up-after-the-plugin-parent-merged-october-27), [producer audit](https://github.com/paperclipai/paperclip/blob/8c1d293d1ae2b7db753ab20f5a2a2edec1fb0a62/doc/plans/2026-10-24-tool-connection-health-sleep-audit.md). Prior remote failures and earlier 706-case evidence remain historical. Startup tests emitted listener-count warnings; all assertions passed. No full local repository-suite, deployed sleep, fleet savings, or live provider qualification claim is made. ## Risks - Supported live writers must use the transaction signals. Independent SQL writers and other replicas need their own wake integration. - Configured eligible connections still have a real proactive health obligation. This does not prove Cloud alarm coverage or host/database suspension. - Existing manual probes and catalog operations can overlap. This migration does not introduce a credential-version CAS or change provider failure policy. - Voice remains eligible under the previous health policy. Its generic probe can still report an unsupported transport; changing that behavior is separate work. - No database migration, new table, workflow, or lockfile change. This PR includes its merged parent and current inspected master; later stack PRs and all current-head acceptance gates remain separate. ## Model Used OpenAI Codex, GPT-6 family, with repository tools, code execution, and reasoning. The exact serving version and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
4e27a82a42 |
fix(server): wake approval queue seeding from committed changes (#17510)
## Thinking Path > - Paperclip manages work and human decisions for AI-agent companies. > - Automatic decision queues store membership for pending attention sources. > - The old attention maintenance scan created that membership as a side effect. > - Moving approval retention out of the scan also removed approval queue discovery. > - Queue API consumers do not build the attention feed first, so they can miss pending PR approvals. > - This pull request gives approval queue discovery its own worker and commit signals. > - The worker becomes quiet after it saves every missing membership. ## Linked Issues or Issue Description Refs #17497. Stacked on #17508, following #17505. This is a correctness dependency for the approval retention migration in #17497. Do not treat that migration as complete without this queue-population path. A real PostgreSQL regression found no PR queue when a pending approval had a linked PR work product and the maintenance feed excluded approvals. Related: #13364 proposes pruning stale seeded memberships. This PR preserves existing membership retention and does not implement pruning. #15149 documents the decision-queue APIs. ## What Changed - Add an approval queue worker that selects missing PR-queue memberships from existing durable records. Bound each pass to 100 rows. Continue past failed rows and revisit them on retry. - Reuse the queue service's seed writer through a normalized source contract. Keep queue rules, uniqueness, and audit writes in that service. Keep feed readers compatible. - Wake from committed approval and link changes, issue/company visibility, PR work-product changes and imports, and queue-rule changes. Reuse the existing transaction boundaries. - Recheck approval status under a row lock. Serialize queue-rule changes with the seed writer. Preserve required recovery during drain and shutdown. - Register startup reconstruction under the existing maintenance switch. Leave other attention-source seeding with its current owner until its own migration. - Document the complete producer audit and stack dependency. Add a validated external PostgreSQL option to the existing queue route suite so it runs on hosts without embedded database support. ## Verification Current tested revision: `7948136ef8`. This PR targets master after #17508 merged as `8406047b4e`. Normal ancestry reconciliation preserved the complete tested integration tree, before the review repairs below. Only the original 25 seeding paths differ from master, and database schema/migration metadata is identical. - **56 cases pass in the three complete affected suites:** 28 approval seeding, 6 queue routes and 22 task-watchdog cases. Server typecheck and production build pass. - Three real PostgreSQL retry regressions failed then passed: new matching arrivals cannot postpone a recovered failure indefinitely; successful partial progress resets old failure penalties; an empty cursor rollover does not reset outage backoff. Exact one- and two-second retry assertions, eventual complete recovery and existing capped-backoff/company isolation tests pass. - Three genuinely empty external databases failed before setup repair because tables were missing. Each suite now applies migrations itself. All 55 cases passed on a second set of fresh databases; adding the final rollover guard then passed all 56 cases using those now-migrated fixtures. No production migration or table is added. - The initial expanded fresh run passed 54 cases but failed a cleanup expectation that omitted a required cursor rollover pass. The fixture now permits that explicit continuation while preserving exact deadline and complete-membership assertions. All failure records remain in `doc/idle-approval-queue-seeding.md`. - The prior integrated revision `4e725e2055` passed **522 cases in fifteen complete producer/consumer suites**, plus types/build. Its unchanged consumer code and historical deadlock, cross-company amplification, backoff, gateway integration and setup evidence remain documented; those counts are not relabeled as current acceptance. Commands: server Vitest for approval queue seeding, queue routes and task watchdogs; server typecheck/build. Fresh exact-head Greptile/Ponytail reviews and all CI checks remain required. No full local repository-suite, deployed suspension or cost claim is made. ## Risks - Signals cover the supported in-process service writers. Independent SQL writers and multiple server replicas need a separate notification contract. - The source predicate must stay aligned with unscoped attention maintenance. The first linked issue, company scope, pending status, and execution visibility are tested. A pending approval itself keeps its linked external conversation active. - Existing enabled seed rules can re-add manually removed matching memberships. This preserves the old scan's behavior. Completed or archived membership is not pruned. - Queue seeding and retention are separate owners. Other attention sources still need their own event-driven discovery before the aggregate scan can be retired. - This PR does not extend Cloud wake coverage, relax sleep admission, or claim measured host/database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8406047b4e |
fix(server): schedule join request retention from durable deadlines (#17508)
## Thinking Path > - Paperclip manages agent companies and their access requests. > - Pending join requests appear in the attention feed and can age into automatic archival. > - The maintenance tick reads those requests even when no archive is due. > - Existing request rows already store the activity timestamp needed for retention. > - Committed changes can wake a worker that registers the next actual deadline. > - This PR gives join-request retention its own bounded worker and removes its recurring feed scan. > - Idle and resolved history stop producing retention traffic without changing access approval. ## Linked Issues or Issue Description **What happened?** Every retention maintenance pass reads pending join requests through the attention feed. Empty, resolved, kept, and future-due states still participate in recurring discovery. **Expected behavior** Committed request and retention changes wake the owner. It waits until the next actual archive deadline and stops querying when no eligible request remains. Existing approval and visibility behavior is preserved. **Steps to reproduce** Start an idle company with no pending join requests, or with requests younger than the 90-day archive threshold. Observe the maintenance feed repeatedly select join_requests. No state change is needed to cause those reads. This PR targets master after #17505 merged. Approval retention is #17497. Other attention source kinds remain separate migration duties. No duplicate open join-retention PR was found. ## What Changed - Add a join-request retention owner that preserves pending status, active-company visibility, matching invite ownership, and keep/archive controls. Invite expiry or revocation retains the existing feed policy. - Reuse one `createDecisionRetentionSweep` for approval and join paging, atomic retention and finite scan cycles that revisit failures despite continuing arrivals. Eligibility and source-row locks stay with each owner. - Register real saved deadlines, rounded upward in SQL to millisecond precision, with the existing coordinator. Preserve 100-row pages and thirty-second notification coalescing without idle polling. - Materialize unseen join state inside the existing authorized keep/archive/revive transaction, preserving archive choices and source locking. - Notify before request inserts and updates in the existing acceptance and Slack transactions. Use a small SQL-only helper for replay and approval/rejection updates. Company status, budget pause/resume, deletion, and retention controls notify this owner too. - Preserve the approved-only API-key consumption path without a new wrapper; it cannot produce pending retention work. - Exclude join requests from maintenance feed collection while retaining normal UI reads. - Add an idempotent concurrent partial pending index and prune old schema snapshots through the existing generator. - Document producer coverage, boundaries, failure evidence, and remaining work. ## Verification Test revision: `df1815fbba`. Normal integration of landed workspace precision `d23dd19750` has an identical complete source tree. This remains only the join-retention duty and its shared approval-retention mechanics, targeting master. - **55 worker cases pass**, including two new PostgreSQL regressions that failed before repair: continuing overdue arrivals no longer postpone a recovered failure indefinitely, while an entire failed page still allows later healthy rows to progress. - The nine complete suites passed 341 cases but failed an attention cleanup hook after its 16,400-run case. After vacuum/analyze of only that disposable fixture, the full 31-case attention suite passes unchanged. Together with the original eight passing suites (311 cases), all 342 distinct cases pass across complete-suite executions. Recovery is established; the original cleanup slowdown is not claimed explained. Server typecheck and production build pass. - Earlier CI at `df1d4d0` failed the unchanged decision-expiry restart deadline assertion (a thirty-second retry appeared before the saved expiry). All twenty unchanged expiry cases pass locally, but the CI timing cause remains unproven. No retry, timeout relaxation or assertion change is used to dismiss it; fresh final-head CI remains required. - **Five regressions failed before repair:** three unseen join controls returned 404, and both owners advertised a microsecond deadline one millisecond early. All pass. Two old real-producer expectations failed after the correct rounding change; they now derive exact expected milliseconds independently from database integer microseconds instead of truncating a JavaScript Date. - The initial timing-only concurrency test still passed a missing-lock mutation. It now observes the actual PostgreSQL wait before releasing the writer. Production passes; removing the lock fails with zero waiters instead of one. Source is restored before commit. The archive-first committed-order case confirms that later activity stays archived until explicit revival; no broader race guarantee is claimed. - Generated concurrent/idempotent `0393_open_grandmaster.sql` adds only the pending join index. All 391 parent journal entries and historical SQL files remain unchanged. Five generated snapshots remain (0389–0393). Migration safety/numbering, schema drift and actual apply/replay on six isolated PostgreSQL fixtures pass. - Existing lifecycle cases retain real HTTP invite acceptance/approval/rejection, Slack identity linking, rollback/outer commit, source visibility, failure rotation, drain/resume and shutdown coverage. Earlier stack setup and CI failures remain in the owning audit and stack ledger; they are historical evidence, not this head's acceptance. Commands: server Vitest for `join-request-retention-work`, `approval-retention-work`, `approvals-service`, `attention-service`, `decision-retention-service`, `budgets-service`, `tool-gateway-service`, `server-startup-feedback-export`, `scoped-deadline-work`; server typecheck/build; DB migration checks, snapshot drift and actual apply/replay. Fresh exact-head CI, security and Greptile/Ponytail 5/5 remain required. No full local repository-suite, deployed suspension or cost claim is made. ## Risks - A future supported producer must signal its transaction. Independent SQL writers and another active server process are outside the process-local notification contract. - A writer that wins the source lock first is rechecked before archival. If archival commits first, later activity preserves the archived state until explicit revival. Existing invite replay expires before the retention cutoff and Slack replay deduplicates. Future producers needing renewed visibility must request revival. Cross-table visibility retains the existing contract. - The worker registers only under the existing maintenance enablement switch. The internal deadline does not add Cloud alarm coverage or relax conservative sleep blockers. - New eligibility can wait within the thirty-second aggregate notification window; known deadlines preempt it. - Archival preserves pending approval status. Other attention sources and aggregate maintenance remain separate work. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
553a004926 |
fix(server): schedule approval retention from durable deadlines (#17497)
## Thinking Path > - Paperclip manages agent work and should become idle when it owes no background work. > - The attention feed archives old pending approvals under the existing 90-day policy. > - The server rebuilds the feed on each maintenance tick, even when approval state has not changed. > - Some approvals have never been read, so retention rows alone cannot define the work. > - Approval records and committed eligibility changes define the real deadline and wake triggers. > - This PR gives approval retention its own deadline worker and reuses the existing retention and archive-delivery contracts. > - The approval lane stops idle scans while other attention sources remain separate migration work. ## Linked Issues or Issue Description Stacked on #17490. This PR owns approval retention only. Archive notification delivery is already separated in #17475. Other attention-source retention and remaining recurring server duties remain in the stack inventory. The related-PR search found no overlapping approval-retention worker migration. **What existing behavior does this improve?** Automatic archival of old pending approvals from attention. **Current behavior** The shared maintenance tick builds the attention feed for each active company. That query discovers pending approvals and materializes retention records before checking their age. An unchanged or empty approval source is read repeatedly. **Proposed behavior** Reconstruct eligible approvals at startup. Schedule the earliest approval activity timestamp plus 90 days. Wake after committed changes to approvals, links, issue visibility, company activity, or retention controls. Stop scheduling when no eligible approval remains. Archive records through the existing version checks, audit writes, and notification outbox. **Reason and benefit** Idle approval retention should generate no recurring SQL. Unseen approvals still need eventual archival. A stale candidate must not archive an approval that was resubmitted while the worker waited for its source lock. **Breaking changes** None intended. The archive age, eligibility, keep controls, audit, and notification policy remain. Normal attention reads still include approvals. No Cloud sleep blocker is relaxed. ## What Changed - Restrict tool-review signals to their existing terminal transition owners (approve/reject/cancel). Creation/link/revision/resubmit signal retention only. - Coalesce retention notifications under one thirty-second aggregate read budget. Known deadlines still preempt it; no notification or durable deadline means no scan. - Add a bounded approval-retention worker. It uses the shared deadline coordinator and reads pending approvals directly. - Share first-linked-issue ordering and execution visibility through `approval-visibility.ts`, used by both the feed and worker. Preserve actor-specific read filtering and locked archival rechecks. - Materialize unseen approvals on demand inside manual keep/archive/revive transactions. Use the saved activity timestamp and existing versioned state; retain company scoping and rollback on failure. - Add awaited pre-write signals to approval, budget, tool, link, visibility, company, and retention mutation owners. Preserve outer transaction ownership; keep provider calls outside SQL wrappers. - Narrow retention inputs to the source fields the existing service uses. Signal materialization only for inserted or changed state so unchanged reads converge. - Exclude approval collection from the maintenance feed before its query. Other source kinds remain in that sweep. - Add the pending-only approval deadline index with concurrent, idempotent creation. Keep the five newest migration snapshots. - Add real-Postgres lifecycle tests and startup wiring tests. Enable validated external test databases for existing attention and budget suites. - Document producer coverage, quiet conditions, recovery and remaining scope in `doc/idle-approval-retention.md` and the stack ledger. ## Verification Current test revision: `dd844f4e8cbbf1db681a774dc31cd552301bef48`, followed by docs-only `3efda33812`, which received both 5/5 reviews and passing CI. This candidate normally integrates landed Grok `2adcde162e`. All runtime and acceptance-test sources are byte-identical to that tested/reviewed tree; only four independent Runner E2E accounting files and the ledger differ. Migration 0392 and the parent journal prefix are unchanged. Local checks are reused for their unchanged source; fresh final-head CI and reviews are required against master. This remains the approval-retention duty. - **315 cases pass** across eight complete suites: approval retention/service, attention, decision retention, budgets, tool gateway, server startup and scoped deadlines. All 30 real-PostgreSQL lifecycle cases pass. Server production typecheck and build pass. - **Four regressions failed before repair**: unseen young keep/archive/revive returned 404; the manual keep rollback case could not reach its update. They now pass. A failed keep update leaves no retention row or audit; retry succeeds with one audit. - Both linked-issue orders are checked against the feed: hidden first/visible later is excluded; visible first/hidden later remains eligible. The feed and worker use the same owner query, with actor read filtering preserved. - Generated concurrent/idempotent migration `0392_small_frightful_four.sql` adds only the pending approval index. Every historical parent SQL migration and all 390 parent journal entries are unchanged. Five generated snapshots remain (0388–0392). Schema drift, migration safety/numbering and actual fresh apply/replay on five isolated PostgreSQL databases pass. - Existing precision, transaction, drain/resume, failure fairness and shutdown tests remain. Earlier transaction-stub, budget-import, diagnostic-client and CI failures are recorded in the owning audit and stack ledger; later passes do not erase them. - Three additional real regressions failed before the wake repair: ordinary mutations emitted tool-review signals in both resolution cases, and sequential tenant writes caused repeated global reads. Tests now prove terminal approve/reject/cancel signals and one aggregate reconciliation per thirty-second window with a different company's pending approval retained. The aggregate budget is stricter than a separate per-company budget and matches the old default cadence; known deadlines preempt it. - The initial coalescing fixture expected the concrete pending-work hold to clear before reconciliation; it now preserves that hold and measures SQL. The first broad repair passed 313 cases but failed the existing parked-review cancel case. The original cancel producer is restored; all 315 now pass without weakening assertions. At old head `5991f257bb`, security was action-required and Greptile returned service error `8fb1835c-e564-476b-b802-d5be164ea915`. Fresh repaired-head results are required, not assumed. Commands: server Vitest for `approval-retention-work`, `approvals-service`, `attention-service`, `decision-retention-service`, `budgets-service`, `tool-gateway-service`, `server-startup-feedback-export`, `scoped-deadline-work`; server typecheck/build; DB snapshot drift, migration checks and fresh apply/replay. Fresh exact-head Greptile/Ponytail and CI remain merge gates. No full local repository suite or deployed host suspension/cost claim is made. Earlier stack verification below is historical evidence at the explicitly named revisions, not this head's acceptance: Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/20557d2c3dd0949c734eaa8095fd78b9b07d7fd5/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Earlier published head: `20557d2c3dd0949c734eaa8095fd78b9b07d7fd5`. This remains a single-duty PR stacked on #17490. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/76a49810c7ff403ae7720bef0d453747bd844731/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Signals cover the application database owner and its child pools. Independent SQL writers and other active processes need an explicit wake integration. Startup reconstructs committed durable state. - The source row lock prevents stale approval-activity archival. It does not add cross-table serializability for concurrent link, visibility, or company-policy changes. - Newly changed eligibility can wait up to the thirty-second notification window, matching the former default maintenance cadence. Saved deadlines still preempt; no new idle polling is added. - Failed records retain recovery debt. Bounded cursor progress lets later rows proceed without dropping failures. - The shared retention transaction now groups materialization and activity-version updates. Existing attention and retention consumers are tested against PostgreSQL. - Kept, archived, resolved or inactive-company approvals can become quiet, but their supported mutations must retain their wake notifications. - The local scheduler cannot wake stopped compute. This PR neither adds approval deadlines to Cloud's external alarm contract nor claims measured host or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8dfed893d8 |
fix(server): wake decision target checks from committed changes (#17486)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Decisions can depend on strict references to other tasks. > - A cancelled, deleted, or completed target can make a decision obsolete. > - The server currently scans open decisions on every maintenance tick to find those changes. > - The owning issue writes already have transaction boundaries that can signal the change. > - This pull request reconciles affected decisions after commit and reconstructs open targets at startup. > - Valid open decisions and completed history stop causing periodic SQL. ## Linked Issues or Issue Description **What existing behavior does this improve?** Decision target invalidation. This is slice 12 of the recurring-server sleep stack, based on #17482. TTL expiry is #17479 and continuation delivery is #17478. Retention remains separate work. **Subsystem affected** Server decision policy, issue mutation and deletion, and startup wiring. **Current behavior** The aggregate sweep repeatedly reads open decisions and their strict targets. It shares a maintenance tick with unrelated jobs even when every target remains valid. **Proposed behavior** Committed decision creation and terminal issue updates request a scoped check. Deletion requests a bounded full scan because its target links can cascade away. Startup reconstructs open target state. Once reconciliation settles, no target scan remains scheduled. **Reason and benefit** Remove normal empty polling without losing target invalidation, transactional delivery, or restart recovery. Reuse existing decision rows and target links instead of adding a table. **Breaking changes** No API or schema change. The main tick stops calling the combined decision sweep after its four duties have separate owners. The public combined method retains its default behavior. **Additional context** The initial decision implementation stored secondary links as well as primary links. This migration does not assume a new legacy backfill. Independent SQL writers still require an explicit signal integration. ## What Changed - Replace the periodic target scan with committed-change activation and bounded startup reconstruction. Keep the existing decision policy and expiry/activity owner. - Require production notifications to identify an issue or decision. Before deletion, lock the issue and capture its linked open decisions in the same company; emit their IDs before links cascade. This avoids resetting full scans and fences concurrent proposal creation. - Reuse that owner helper for issue removal and routine failure cleanup. No linked decisions means no target signal. Remove company deletion's ineffective global signal; its pre-existing restrictive decision foreign keys remain unchanged. - Retain PostgreSQL timestamp text in paging cursors so rows within one millisecond cannot repeat indefinitely. - Remove the unused combined internal decision sweep after its final duty is split; exercise each owner directly in the existing behavioral tests. - Keep query policy and notification discovery in `decision-targets.ts`. The coordinator registration contains lifecycle wiring only. Add no queue table, per-route recovery wrapper or external-wake exemption. ## Verification Current candidate normally integrates landed effect recovery `e9e84bb8a1`. Its runtime and acceptance tests are byte-identical to tested target head `85ce6545bf`; the only new source files are four independently merged Runner E2E accounting files, preserved exactly. The 248-case/type/build evidence comes from test revision `54936a919f36c2ea421e7ce8e15912b1105b8dcc`, followed by documentation-only changes. Fresh current-head CI and reviews are required. This remains the target-invalidation duty only. - **248 cases pass** in all seven complete suites: 25 target, 80 routine, 54 decision, 26 effect-recovery, 20 expiry, 42 startup and one scoped-deadline case. Database suites use separate migrated PostgreSQL fixtures. Server production typecheck and build pass. - **Four regressions failed before repair:** paging across microsecond timestamps, deletion behind a full valid startup page, unnecessary global work from unrelated deletion, and unnecessary work after deleting an empty company. All four pass. - A real concurrent proposal is observed waiting on the issue-row lock between pre-delete discovery and deletion commit. Removing that lock made the test fail because the proposal committed instead of waiting. Restoring the implementation passes. The mutation was restored before commit. - Existing real producer tests cover commit/rollback, deletion with scoped hints in one outer transaction, secondary targets, re-opened tasks, valid/lenient history quietness, startup paging, failure retry, drain/resume and shutdown. The routine failure case now requires no signal when its cleanup has no linked decision. - The earlier dropped-hint and drain-fixture failures remain in [the behavior audit](doc/idle-decision-targets.md). Extra upstream embedded-only tests attempted on the parent could not initialize locally; CI must cover that environment. No new schema migration or table is added. The final review cleanup removes the combined internal `sweepExpired` method, cursor, flags and unused target-policy export. A repository/package/SDK audit found no remaining production caller or public exposure. Behavioral tests now call the dedicated workers; isolation tests prove TTL does not recover stale effects and effect recovery does not expire open decisions. HTTP and plugin SDK contracts are unchanged. The full 248-case selection passes after this cleanup and parent reconciliation; types/build pass with the updated Runner vendor. Current-head CI run 40602159553 (attempt 1) failed in the workspace CLI group: company import/export startup exceeded its 30-second health deadline, and the dense ZIP case exceeded its unchanged five-second timeout. The preceding UI group passed all 4,169 cases. Both CLI files and the transfer implementation are unchanged from the landed parent. The complete ZIP-transfer file passes all 17 cases locally on this exact head; the CLI startup E2E is skipped because the host's embedded PostgreSQL bootstrap fails. This does not prove Linux startup locally. The failure logs remain recorded; no timeout or assertion is changed. One bounded retry of the failed jobs is requested after the workflow completed. Its result and the current-head gates remain required before merge. Commands: server typecheck/build and Vitest for `decision-target-work`, `routines-service`, `decisions-service`, `decision-effect-recovery-work`, `decision-expiry-work`, `server-startup-feedback-export` and `scoped-deadline-work`. Local targeted acceptance does not establish deployed suspension or fleet savings. Fresh exact-head CI and Greptile/Ponytail remain merge gates. ## Risks - Deletion now holds the issue row lock while discovering dependent decisions and completing the existing removal transaction. This is needed to serialize new FK references. Provider calls are not placed inside that transaction. - Company deletion with decision-origin references still has the pre-existing restrictive FK limitation. Removing the futile notification does not implement a new company-data cleanup policy. - Supported writers share the application DB owner and maintain existing target links. Independent SQL writers, damaged links or multiple active processes need their own wake contract; startup still reconstructs all open targets. - Cloud sleep blockers and durable external alarms are unchanged. Worker quiet is not proof of deployed host/database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e9e84bb8a1 |
fix(server): recover decision effects from saved claim deadlines (#17482)
## Thinking Path > - Paperclip manages agent work and the decisions that authorize it. > - A committed decision can leave effects unfinished after an interrupted execution. > - Recovery must preserve that debt without scanning settled history forever. > - Claims already store the state and time needed to schedule recovery. > - A failed claim must get another attempt even while newer claims arrive. > - Malformed claims need a durable failure outcome instead of an endless retry. > - This PR uses committed signals, saved deadlines, and bounded recovery rotations. ## Linked Issues or Issue Description **What happened?** Periodic decision maintenance queries for unfinished effects even when none remain. A cursor that only wraps after a short page can postpone an old failure indefinitely under continuous new work. Missing saved deciders or chosen options can also leave non-executable claims scanning forever. **Expected behavior** Open decisions and completed history produce no recurring effect-recovery SQL. Committed claims wake recovery, and restart reconstructs their deadlines. Genuine failures retain retries. Known malformed outcomes settle with an audit and preserve completed effects. Recovery must not invent authority, repeat committed effects, or lose continuation delivery. **Steps to reproduce** Create an old failing claim with a one-row recovery page, then keep adding newer due claims. Before the rotation fix, the old failure is never revisited. Separately, recover a claim with a missing decider or chosen option: the old implementation skips or throws on every attempt without settling it. Creating an ordinary open decision also caused an unnecessary effect scan. Stacked on #17479. Other decision expiry, continuation, target and retention duties have separate PRs in the stack. ## What Changed - Coalesce sequential committed notifications for up to one second without delaying already saved deadlines. This bounds instance-wide recovery reads and adds no idle polling. - Register effect recovery with the shared scheduler and transaction-aware notifications. Derive deadlines from the existing claim timestamp and recovery grace. - Remove effect recovery from recurring aggregate maintenance when the worker owns it. Preserve the explicit combined recovery entry point. - Page within a frozen final timestamp/ID and retain PostgreSQL precision at both boundaries. Failed claims and healthy later claims both make progress. - Load the current claim and derive recovery authority inside the shared execution flight. Preserve foreground joining, current authorization and per-effect locks. - Recheck known malformed claims under a row lock. Save a failed outcome, or partial when an executed effect receipt exists, with `effectRecoveryError` and an atomic `decision.recovery_failed` audit. Publish activity after commit and use normal continuation delivery. - Avoid effect-recovery notifications for open-decision creation. Claim and settlement writes retain their existing notification boundary. - Add generated migration `0390_salty_killraven.sql`, the concurrent, idempotent partial index for running-claim lookup. No queue table is added. - Cover quietness, transaction and lifecycle boundaries, retry fairness, malformed outcomes, and concurrent repair. Document the contract in `doc/idle-decision-effect-recovery.md`. ## Verification Current candidate normally integrates landed expiry `3421366903`, including expiry-only creation signals and all 21 independent master changes. The test revision is `dba8a79d1e1a5b97b8b5f6dac0f7c7837536531a`; the publication adds only this evidence documentation. - **143 current cases pass** across all 26 recovery, 54 decision, 20 expiry, 42 startup and one scoped-deadline cases. Server types/build pass with the rebuilt Runner vendor (1,524 inputs and 1,563 outputs). These results follow the landed dependency reconciliation; prior 69- and 141-case results remain historical evidence for their stated revisions. - Generated migration `0390_salty_killraven.sql` adds the existing running-claim partial index through the concurrent executor. All 388 parent journal entries are unchanged; five generated snapshots remain. Schema drift, numbering/safety and actual migration apply/replay pass. - Existing regressions exercise failed-claim rotation under sustained arrivals, malformed decider/option outcomes, atomic audit failure, row-lock repair races, current authority, foreground effect joining, coalesced tenant notifications, transaction commit/rollback, drain/resume, paging with PostgreSQL timestamp precision, and shutdown/restart. Completed effects are not repeated. - The first integrated run passed 140 cases and failed an overlap test that assumed the effect deadline was the scheduler's earliest deadline. Continuation coalescing now creates an earlier registered wake. The test services that earlier wake before asserting the original effect grace deadline; it still requires one shared effect flight, two lock attempts including the test gate, one committed comment and one continuation. No production timing was relaxed. - Earlier red/green fairness, malformed-claim, notification amplification and fixture failures remain recorded in [the owning audit](doc/idle-decision-effect-recovery.md). Extra upstream embedded-only tests attempted during parent integration could not initialize locally; CI must exercise that environment. - Previous candidate `f54a707d74` received both 5/5 reviews but its CI server shard 6 was interrupted (run 40597921712). The step was cancelled, with no failing assertion or suite summary, and the verify gate also failed. No retry or passing claim was made for that run. The necessary parent repair is included here; fresh CI and both reviews are required. Commands: server typecheck/build and Vitest for `decision-effect-recovery-work`, `decisions-service`, `decision-expiry-work`, `server-startup-feedback-export`, and `scoped-deadline-work`; DB migration checks, schema snapshot drift and real migration apply/replay. No full local repository-suite or deployed sleep/cost claim is made. Fresh current-head CI and Greptile/Ponytail are required before merge. ## Risks Missing deciders and missing chosen options now produce durable failed or partial outcomes instead of remaining running forever. Their audit and receipts remain available; no effect executes under invented authority. Only these known malformed states are settled this way. Database, audit, and other execution failures retain retries. Continuations retain their existing at-least-once delivery contract. The supported writer model remains process-local notifications plus restart reconstruction. Independent SQL writers and multiple replicas gain no new wake protocol. Cloud admission and durable external alarms are unchanged. These tests establish worker behavior, not deployed host suspension or fleet savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3421366903 |
fix(server): schedule decision TTL expiry from saved deadlines (#17479)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Open decisions have a saved expiry time. > - The main maintenance tick scans those times even when none is due. > - Committed creation and resolution can maintain the actual next deadline. > - The existing scheduler can run expiry only when that deadline arrives. > - This pull request gives decision TTL expiry its own worker. > - Terminal history becomes quiet while expiry and continuation delivery remain recoverable. ## Linked Issues or Issue Description Targets master after #17478 merged. This is slice 10 of the recurring server work migration. It owns decision TTL expiry. Target invalidation, interrupted effects, and retention remain separate migrations. The full ledger is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. **What existing behavior does this improve?** Time-based decision expiry in the server. **Current behavior** The combined decision maintenance sweep queries expired open decisions on each tick. It mixes that query with target invalidation, effect recovery, and continuation delivery. **Proposed behavior** Creation, bundle creation, and existing decision updates notify the shared scheduler. A dedicated worker runs at the first saved open expiry. Resolving an open decision removes its deadline. The combined tick opts out of TTL selection while its other duties remain intact. **Reason and benefit** A future decision no longer requires repeated TTL queries before its deadline. Terminal history and an empty decision table leave no recurring TTL work. No queue table is added. ## What Changed - Add an expiry owner for bounded due selection, next-deadline lookup, and expiry settlement. Reuse it for expiry-on-use and the existing target expiration transition. - Use the central expiry-only creation notification before insert, including bundles and caller-owned transactions. Update and terminal transitions continue to notify expiry and continuation. - Register the enabled-heartbeat TTL worker before decision housekeeping. Preserve disabled-heartbeat behavior and the public combined sweep's default batch limit. - Honor `PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE` (default 100). Keep a fair expiry/ID cursor across failures. Stop between candidates on shutdown and reconstruct saved work after restart. - Recheck open status and due TTL during settlement. Merge metadata and persist activity atomically with expiry. Publish activity and deliver continuation after commit. Handle exact timestamp equality. - Continue successful overdue batches after one second through the existing deadline adapter; preserve thirty-second failure retry and all default callers. - Add generated migration `0389_brave_hulk.sql`, an idempotent concurrent partial index on open expiry and ID. Preserve five generated snapshots. - Add 20 real-PostgreSQL lifecycle cases and startup ownership assertions. Document contracts and limitations in `doc/idle-decision-expiry.md`. ## Verification Current candidate: `aa03d2c4c25af4e6c2acf49d3160bbaf6ada030b`. This PR targets master after #17478 merged. Creation now signals only its real expiry obligation; terminal updates preserve continuation delivery. - **117 current cases pass** across all 20 expiry, 54 decision/continuation, 42 startup and one scoped-deadline cases. Server typecheck/build pass. Separate migrated PostgreSQL fixtures exercise real transaction and notification behavior. - Single and bundle creation each failed a regression before the expiry-only signal repair: open rows unnecessarily notified continuation recovery. Both now arm their saved expiry without a continuation signal. Cancelling each row afterward still signals and delivers continuation normally. The distinction is in the existing central notification owner; there are no per-route flags or counters. - **Three new real PostgreSQL regressions failed before repair:** a configured batch of two was ignored, a successful 101-row backlog waited for the thirty-second error retry, and an activity insert failure left the row expired without its audit. All three pass. The audit failure rolls back expiry; retry records one activity, preserves unrelated metadata and wakes once. - The shared deadline adapter has optional `continuationMs`. Only this owner opts into one-second successful overdue continuation. Other callers keep the previous default, and errors still retry after thirty seconds. The archive consumer passes with the option absent. - Migration `0389_brave_hulk.sql` is regenerated from the parent's metadata. Parent SQL/journal entries are preserved, five generated snapshots remain, schema drift and migration numbering/safety pass. Fresh migration apply and replay pass. - Lifecycle coverage includes first producer activation, future-deadline quiet, terminal history, exact equality, commit/rollback, restart, drain/resume, failed expiry and wake delivery, fair paging through failed rows at PostgreSQL timestamp precision, shutdown and recovery. - Failure accounting: the first full repair selection passed 113 cases and failed a startup assertion that observed a pending follow-up notification before reconciliation. The assertion now waits for the exact saved future deadline, then still proves no early expiration. Earlier fault-fixture and timestamp-cursor failures remain recorded in [the behavior audit](doc/idle-decision-expiry.md). Extra upstream embedded-only tests attempted on the parent could not initialize locally; CI must cover that environment. Commands: server typecheck/build and Vitest for `decision-expiry-work`, `decisions-service`, `decision-retention-service`, `server-startup-feedback-export` and `scoped-deadline-work`; DB migration checks, snapshot drift and actual migration apply/replay. No full local repository-suite or deployed sleep/cost claim is made. Fresh current-head CI and Greptile/Ponytail remain required before merge. ## Risks - Expiry uses `expiresAt <= now`; the old combined loop skipped exact equality. The deadline boundary is now covered explicitly. - TTL expiry records the TTL reason. Target-driven expiry preserves its own reason. When both causes apply, the first terminal transition wins. - Supported producers share the application DB signal owner. Independent SQL writers or multiple active processes need a separate notification contract. New deadline-edit APIs must use the existing helper. - The index requires top-level concurrent migration execution. It is registered with the migration owner and replayed in verification. - Future in-process deadlines are not external Cloud alarms. Existing sleep guards remain conservative. This PR does not claim host/database suspension or measured savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a63d19816f |
fix(server): wake decision continuations from committed changes (#17478)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A completed decision can owe a wake to the agent that proposed it. > - The decision row already stores the pending continuation. > - The main maintenance tick scans this debt even when no continuation is ready. > - Committed changes can activate recovery and leave settled history quiet. > - This pull request gives continuation delivery its own worker and shares foreground delivery ownership. > - Pending work stays recoverable without an unconditional continuation scan. ## Linked Issues or Issue Description Targets master after #17475 merged. This is slice 09 of the recurring server work migration. Each PR owns one reason for background work. The remaining inventory is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. **What existing behavior does this improve?** Decision continuation delivery and recovery in the server. **Current behavior** Foreground paths send a wake and clear a pending flag. The combined decision sweep scans pending continuations on every maintenance tick. It can overlap foreground delivery. **Proposed behavior** Terminal decision writes notify a dedicated worker through the existing transaction boundary. Foreground and recovery calls share one in-flight delivery per decision and DB owner. Recovery re-reads readiness, pages past failures, and retries only unresolved debt or errors. **Reason and benefit** Open decisions, running effects, and delivered history no longer cause an independent continuation scan. No queue table is added. Decision expiry, target invalidation, interrupted effects, and retention remain separate duties. ## What Changed - Extract continuation readiness, delivery, and paged recovery into the decision owner. Preserve the wake payload and at-least-once crash behavior. - Signal the ten decision update sites before SQL in their existing transaction scope. Keep effects and wake calls outside those transactions. - Explicitly create the continuation obligation on all three archive-proposal failure settlements, including recovered running rows without inherited pending metadata. - Serialize concurrent decision effect execution through final settlement. Coalesce foreground and worker wakes. Re-read the durable flag before sending. Fence effect completion to running state so competing completion cannot restore acknowledged debt. - Expose an opaque DB work-scope key. Child pools and nested transactions share that identity; independent roots remain separate. Existing signal consumers keep their behavior. - Bound each recovery pass to 100 candidates, retain a fair cursor across failures, and stop between deliveries on shutdown. Forward the existing AbortSignal through the shared deadline adapter. - Register enabled-heartbeat continuation recovery before decision housekeeping. Remove only its independent scan from the main tick. Preserve the public combined sweep and disabled-heartbeat behavior. - Add generated migration `0388_ambitious_micromax.sql`, an idempotent concurrent partial index for pending continuations, and keep five generated snapshots. - Add real producer, transaction, failure, concurrency, paging, quiet, and shutdown tests. Document ownership and remaining limits in `doc/idle-decision-continuations.md`. ## Verification Current candidate: `18f682863cadb91281a86b4da2553a4579ee054a`, normally integrated with landed archive squash `5785090ff1`. All 31 independent incoming master files are preserved exactly, including controller-authority recovery. All 97 decision/startup/deadline cases plus server types/build pass after the required Runner vendor rebuild. The preceding full five-suite selection passed 293 cases; the continuation owner behavior is unchanged by master reconciliation. - **97 cases pass at this master integration** across decision, startup and scoped-deadline suites, plus server types/build. The preceding parent candidate passed the full 293-case decision/gateway/archive/startup/deadline selection; those results remain evidence for that revision. All selected cases executed with separate PostgreSQL fixtures where needed. - Tenant-driven global scan amplification reproduced six sweeps for six sequential real cancellations. Existing thirty-second notification coalescing now permits one recovery sweep per window while retaining pending work. The first green fixture wrongly expected the intent hold to clear before coalesced reconciliation; the hold remains and quiet is checked after settlement. Foreground delivery stays immediate. - The archive contention test uses a signaled dedicated single-connection read and two FIFO SQL round trips instead of a fixed delay. Removing effect coalescing makes it fail with two contender reads; restoring production code passes. No production test hook was added. - **Three new real PostgreSQL regressions failed before repair**: recovered archive proposals with invalid signatures, mismatched manifests or archive execution failures and no inherited pending flag. Each now persists failure, wakes once, acknowledges continuation delivery and remains quiet on replay. Running-state fencing is preserved. - The unmerged continuation index is regenerated as `0388_ambitious_micromax.sql` after parent migrations 386/387. All parent journal entries are unchanged, five generated snapshots remain, numbering/safety and snapshot/schema-drift checks pass. Fresh database migration apply and replay pass; the concurrent index is valid. - Existing lifecycle coverage includes foreground/worker coalescing, child pools, transaction commit/rollback, failed acknowledgement, restart, drain/resume, shutdown and fair 100-row recovery. Earlier bounded-pass and concurrent archive-result regressions and their repairs remain recorded in [the owning behavior audit](doc/idle-decision-continuations.md). - Failure accounting: the first combined command used an incorrect retention fixture variable and skipped 15 cases. The corrected separate external-PostgreSQL retention run passed all 15. The historical nine embedded-only work-signal cases were not executed by this selection. Extra upstream embedded-only suites attempted on the parent could not initialize locally; they remain CI verification, as recorded in the parent PR. Earlier reused-fixture, assertion and compile failures remain in the behavior audit and stack ledger; they are not erased by later passing checks. Commands: `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/server build`, server Vitest for `decisions-service`, `decision-retention-service`, `server-startup-feedback-export` and `scoped-deadline-work`; `pnpm --filter @paperclipai/db check:migrations`, migration apply/replay, and the generated snapshot-drift test. Local targeted evidence does not establish a full local repository run, deployed sleep or fleet savings. Fresh exact-head CI and both automated reviews remain merge gates. ## Risks - Notification-driven recovery remains global but is coalesced to thirty seconds. This bounds request-driven scans; it does not introduce company-scoped selection. Foreground delivery remains immediate. - Delivery remains at least once. A crash or acknowledgement failure after accepting a wake can send it again. Local coalescing is not a cross-process claim. - Supported writers share the application DB signal owner. Independent SQL writers or multiple active server processes need a separate wake contract. Wake-producing service methods do not gain a new guarantee when called inside an uncommitted caller transaction. - The concurrent index needs top-level migration execution; it is registered with the existing migration owner and can be replayed safely. - No Cloud sleep blocker or external alarm coverage changes. Worker quiet does not prove deployed app or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5785090ff1 |
fix(server): wake archive notification delivery from committed changes (#17475)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Auto-archive writes an existing notification outbox in its archive transaction. > - The maintenance sweep polls that outbox even when it is empty. > - The committing producer can wake delivery. Saved rows and claim timestamps can restore unfinished work. > - This PR separates delivery from retention scanning and uses the existing coordinator for retries and deadlines. > - Settled delivery becomes quiet while archive and notification behavior remain intact. ## Linked Issues or Issue Description **What happened?** The retention maintenance sweep calls archive notification delivery on every tick. An empty outbox and retained delivered history still cause database work. **What did you expect?** Committed auto-archive should wake delivery. Pending work and existing claims should retain retry or expiry deadlines. Settled history should cause no recurring delivery SQL. **Steps to reproduce** Start a configured instance with an empty or fully delivered archive outbox. Leave it idle. The old maintenance tick still checks the outbox. This is slice 08 of the sleep migration. It remains stacked on projection repair #17468. Retention eligibility and scanning, decision TTL expiry and decision continuation recovery have separate owners and PRs. ## What Changed - Notify archive delivery before the existing outbox insert in the auto-archive transaction. Preserve outer transaction commit, rollback and uncertainty handling. - Register a delivery worker with the existing coordinator. Remove notification delivery from retention sweeps. - Keep claim recovery, selection, company/agent grouping and timestamp-fenced settlement in the delivery service. Keep the existing service API and payload. - Count a notified group only when it settles rows. Use one timestamp for failure reset. - Add generated concurrent migration `0387_boring_professor_monster.sql`. Its partial index matches pending order: `last_attempt_at NULLS FIRST, created_at, id`. Add no table. - Test real startup registration and prove a retention tick does not send notifications. Preserve the disabled-scheduler no-notification contract. ## Verification Current candidate: `457c70fc6acbd681fcccd90d08b6391efb202bce`, normally merged with pinned master `8161f2a813` after projection #17468 landed. All 39 independent files from that master integration are preserved exactly. Archive owner behavior and migration 0387 are unchanged. - **58 archive/startup/scoped-deadline cases pass** after master reconciliation. Server typecheck and build pass. The preceding projection-parent integration passed all **239** gateway/projection/archive/startup/deadline cases. - All 385 parent journal entries remain an exact prefix, with own migration 0387 appended; the five newest generated snapshots remain. Generated index apply, replay, schema drift and real pending-order query measurements passed in the prior owner verification. - Failure accounting: the broader local command also attempted six upstream heartbeat-session-preparation and 55 native control-plane cases. The embedded PostgreSQL runtime was unavailable: the six heartbeat cases skipped, and native setup failed before its 55 cases could execute. This is an incomplete additional local check, not a passing full run. The owning 58 cases passed with the existing external fixture where applicable. CI must exercise its supported embedded environment. - The first independent-master comparison used the shared mutable remote ref, which another worktree advanced during verification. Pinning the actual merged master SHA confirms all 39 independent files match exactly. No incoming source change was dropped. The behavior audit in `doc/idle-decision-archive-delivery.md` records earlier fixture failures, the stale-claim regression and the index probe. Local tests do not prove deployed suspension or fleet savings. Fresh current-head CI and both reviews remain required. ## Risks - Delivery remains at least once. A wake accepted before an acknowledgement is lost can be retried. - Each pass keeps the existing 500-row limit. Failed or remaining pending work retains the existing thirty-second retry. Saved delivery claims retain their five-minute deadline. - Auto-archive is the existing outbox producer. Manual archive/revive and reviewed-manifest actions do not gain notifications. - Signals use the existing single-active-writer database owner. Independent SQL or another active server needs a wake contract. - Concurrent index creation uses the existing top-level executor and invalid-index retry path. Existing SQL and migration receipts are unchanged. - Cloud sleep guards and external-wake coverage do not change. Local worker quiet is not proof of database or host suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge The final projection efficiency parent is integrated normally. All 239 affected gateway/projection, archive, startup and scoped-deadline cases pass, with server typecheck/build. The archive owner behavior and index remain unchanged. Fresh reviews and CI are required on this parent integration. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8161f2a813 |
fix(server): wake tool-review card repair from committed changes (#17468)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tool requests and invocations store the execution outcome. Review cards show that outcome. > - The maintenance tick repeatedly joins these tables even when every card matches. > - Committed changes can wake repair. Existing rows can restore work after restart. > - A large company backlog also needs a bounded pass so other companies and shutdown can proceed. > - This PR uses the shared coordinator for scoped repair, continuation and failure deadlines. Settled history becomes quiet. ## Linked Issues or Issue Description **What happened?** The aggregate maintenance tick scans tool-review projections on every tick. Matching terminal history still causes database work. A company with many stale cards can also hold one worker in an unlimited drain. **What did you expect?** Relevant writes and startup should wake repair. Failed or unresolved cards should retain recovery. Matching history should cause no recurring SQL. Each normal worker pass should honor drain and shutdown. **Steps to reproduce** Start an instance with completed tool requests and matching cards. Leave it idle. The old maintenance tick keeps querying for mismatches. A large mismatch backlog also drains without a row limit or cancellation check. This is slice 07 of the sleep migration. It now targets master after #17458 landed. Related owners are pending expiry #17453 and approved recovery #17454. Their execution duties remain separate from card repair. The prior open-PR search found no duplicate projection migration. ## What Changed - Register company-scoped projection repair with the existing delivery coordinator. Remove its recurring aggregate-tick scan. - Signal through the existing review-write transaction owner and linked interaction mutations. Ordinary unlinked confirmations do not wake repair. - Keep interpretation, locks, current-state checks, activity and continuation callbacks in the gateway. A repair never repeats a provider tool call. - Limit each normal worker pass to 500 rows. Check abort and admission between rows. Save company progress through truncated and failed passes so blocked early pages cannot hide later cards. - Schedule truncated work after one second. Back off unresolved cards from thirty seconds to five minutes; fresh committed hints reset the delay. Clear obsolete cursors after empty reconstruction. - Skip repeated company discovery for scoped hints and the extra candidate query on an empty sweep with no saved cursor. Attempted or resumed passes still verify remaining mismatches because reflection returns no settlement result. - Add generated migration `0386_steady_epoch.sql` for the existing `(company_id, interaction_id)` lookup. Use the existing concurrent-index executor. Add no table. ## Verification Current candidate: `d4706fbf2a04a00ade30181eb1bd9322e8455818`. All 222 cases in the full gateway/projection/interruption, startup and scoped-deadline suites pass without skips after the query/retry efficiency repair; server types and build pass. A single committed company hint with matching retained history and a separate empty direct sweep each perform one candidate query. The unresolved-card regression failed before repair and now verifies backoff through the five-minute cap, prompt producer activation and quiet after settlement. An initial assertion incorrectly assumed an entire approval emitted one wake; its separate real writes require isolating the single-hint measurement. The earlier master integration `fe6e10629b` additionally established the following evidence before this efficiency follow-up: - **320 cases pass** across full gateway/projection/interruption, startup, scoped-deadline, narrowed identity-read and authenticated gateway HTTP suites. The HTTP selection includes all 81 cases. - **Six real PostgreSQL regressions failed before their fixes:** pass bounds, abort, drain, progress past unprojectable pages, progress past failed pages, and first repair after empty reconstruction. The complete owning selection passes all 220 cases. - Server production typechecking and build pass. Generated index migration, snapshot drift and numbering/safety checks pass. The five newest snapshots remain; master journal entries are unchanged. - A real missing-link query on 30,000 history rows changed from a sequential scan that filtered all 30,000 rows and touched 429 buffers to the new index with two buffers. Both treatments returned no rows. - Normal master integration preserves 35 independent performance files exactly. The gateway also retains master's narrowed identity read. Commands include `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/server build`, and server Vitest selections for `tool-gateway-service`, `server-startup-feedback-export`, `scoped-deadline-work`, `run-identity-projections`, and `connection-management.integration`, with separate PostgreSQL test fixtures. [The behavior and failure audit](doc/idle-tool-review-projection.md) records the producer contract, earlier fixture failures, index measurement, red/green regressions and integration evidence. Fresh current-head CI and Greptile/Ponytail remain required before merge. ## Risks - Signals use the existing single-active-writer database owner. Another replica or independent SQL writer needs a wake contract. - An already issued database query or reflection finishes before cancellation takes effect. Checks between rows bound further dispatch. - Legacy manual/global and disabled-scheduler one-shot repairs keep their complete-sweep behavior. The normal coordinator worker always supplies its row budget and admission controls. - Unprojectable cards and thrown failures retain recovery with a five-minute backoff cap. The company mismatch query still grows with terminal history. Empty history does not retain a recurring scan. - The index migration uses concurrent creation and the existing invalid-index recovery path. Prior SQL and journal entries remain unchanged. - Cloud sleep eligibility and external-wake coverage do not change. Local worker quiet is not proof of host or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f6bc653f7f |
fix(server): schedule interrupted tool claims by their deadlines (#17458)
## Thinking Path > - Paperclip governs tool execution and keeps durable approval records. > - An interrupted execution may have produced an external effect without a saved final request state. > - The maintenance tick currently scans executing requests even when none exist. > - Every executing claim already has a ten-minute stale deadline. > - This PR schedules that deadline from committed claim changes and clears it after settlement. > - Recovery preserves saved provider outcomes and quarantines unknown outcomes without another provider call. ## Linked Issues or Issue Description Refs #17454. This is the sixth recurring-server sleep slice, stacked on `codex/sleep-05-approved-tool-recovery`. **Problem** The aggregate maintenance tick repeatedly searches for stale executing tool requests. Its settlement can overwrite a saved successful invocation with an unknown-outcome failure after a crash between invocation and request writes. **Proposed Solution** Use the existing review notification boundary and shared scheduler. Read the earliest executing claim deadline through a partial index. At that deadline, lock invocation then request, recheck claim identity, and atomically settle from the saved outcome or quarantine the unknown outcome. **Alternatives Considered** A longer polling interval still queries empty work. A new queue would duplicate existing records. Replaying an interrupted call can duplicate an external effect. Treating every stale request as unknown discards already saved outcomes. ## What Changed - Coalesce sequential review notifications for up to 30 seconds while preserving already loaded recovery deadlines. Do not schedule scans when no notification or execution deadline remains. - Extract interrupted-execution deadline lookup and settlement into the owning tool-review service. - Register its worker at startup. Use committed claim/result changes to update the existing ten-minute deadline; retry failed settlement after 30 seconds. - Preserve terminal invocation results, reasons, and completion times when repairing request state. - Settle uncertain request/invocation state atomically with the existing unknown-outcome reason, without dispatching a provider call. - Notify all gateway claim and settlement paths through the existing review transaction helper. Keep framework wiring and the shared scheduler free of provider-specific behavior. - Add an idempotent concurrent partial index for executing deadlines. Preserve the combined manual sweep and scheduler-disabled startup pass. - Remove interrupted scanning from the aggregate tick. Interaction projection repair remains the next separate duty. ## Verification Current head: `648d8a9a64ded15deeeb331545b92a12d69e277d`. The final commit changes no files and requests CI after the master retarget. A preceding run appeared while that trigger was being published; it was observational delay, not an established CI outage. No test rerun or bypass was used to turn a failure green. Source remains identical to the 198-test passing integration, apart from the already recorded ledger update. Fresh checks and both automated reviews must qualify this current head before merge. Current master-targeted head: `d55e44b376555e4926bb2c2a0e4dcfb431324162`. Parent #17454 landed as `77295faebc8`; a normal merge reconciles that squash. Every conflicted master file was checked against the already-integrated parent, and all matched byte-for-byte. Each interruption-only hunk was retained; the complete integrated source tree matched the 198-test passing `848f1b87b3` exactly before the sole final documentation update. All master migration journal entries are unchanged, with only this PR's executing-deadline index appended. Server types pass. Fresh CI and current-head Greptile/Ponytail are required before the next authorized merge. Latest published integration: `848f1b87b3a6646ae4e56e8ab142f64889d7a69c`, normally merging approved-recovery parent `23868c1751`. All 198 gateway/interruption/startup/deadline cases and server production typechecking pass. Both parent approval fixes and all interruption tests are retained. The two merge conflicts preserved the interruption import alongside the parent's claim-bound hash helper and preserved the parameterized historical test helpers. This remains stacked on #17454; fresh CI and both automated review gates are required after integration. - Review documentation correction at `c985eb9d1b48a481909de6a1c2d89d7af11bfc03`: the focused real-PostgreSQL interruption-worker selection passes all 16 cases; 124 unrelated cases were filtered out. The docs now correctly name 16 cases and six saved terminal statuses. No production behavior changed in the correction. Fresh remote CI/review is pending after integrating the verified parent. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/f6a5949fb726f71c247950714adecb275655284f/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Current head: `f6a5949fb726f71c247950714adecb275655284f`. This remains a single-duty PR stacked on #17454. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/eaac8d343e5ef4d2c13cdf7cb7607cb5a498e9a2/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Signals cover the existing single-active-writer database owner. Independent writers and replicas need an explicit wake contract. - The ten-minute policy is unchanged. A provider call that outlives its claim remains an uncertain outcome; this worker never retries the external call. - Interaction projection follows authoritative settlement outside its transaction. Its separate existing repair duty still handles an interrupted projection. - Non-executing review writes can cause one empty reconciliation. Without an executing claim or failed attempt, no recurring deadline remains. - Cloud eligibility and external-alarm coverage remain conservative. Worker quiet does not prove actual host or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
77295faebc |
fix(server): recover approved tool actions from committed changes (#17454)
## Thinking Path > - Paperclip manages agent work and its governed tool calls. > - A human approval can persist before the approving process dispatches the tool. > - Recovery currently scans approved requests on every maintenance tick. > - Completed and deliberately inert approvals do not owe another execution. > - Test-tab approval also needs a shared execution claim before recovery can safely race the foreground path. > - This PR wakes recovery from committed changes and retries only unresolved approvals. > - Idle history becomes quiet while existing approval and execution checks remain with the gateway. ## Linked Issues or Issue Description Refs #17453. This is the fifth slice of the recurring-server sleep migration, stacked on `codex/sleep-04-tool-review-expiry`. **Problem** Approved tool requests are scanned periodically even when no executable work remains. The Test-tab foreground path lacks the request claim needed to admit concurrent recovery safely. **Proposed Solution** Use the existing transaction-aware review notification boundary and shared deadline worker. Keep signed-payload interpretation, current policy and target checks, and provider dispatch in the gateway. Share a claim between Test-tab foreground and recovery, then persist both authoritative outcome rows atomically. **Alternatives Considered** A longer interval still scans idle history. A new durable queue would duplicate existing request/invocation state. Dispatching Test-tab approvals without a claim risks duplicate external calls. ## What Changed - Recheck both principals for delegated Test-tab approvals. Save their rate receipts in the same claim transaction, share only the same bucket, and roll back new charges on denial. - Preserve delegated gateway authorization and both issuer/target rate receipts inside the existing claimed-invocation transaction; retain the durable actor during approved replay. - Expire stale managed connector payloads before consuming approved-tool admission. - Persist current rate-policy charges and their invocation receipts atomically before dispatch, preserving exact approval claim fencing and read-only preflight. - Recover approved requests at startup and after committed review changes; retry only unresolved approvals. - Leave legacy agent approvals without execute-on-approve intent inert. Repair completed Test-tab history without another provider call. - Fence Test-tab execution and outcome settlement with an atomic claim shared by both callers. Revalidate the approved target and current policy before dispatch. - Keep provider calls outside database transactions. Retain quarantine for interrupted execution and unknown outcomes. - Add a concurrent, idempotent partial index for approved-request pagination. Reuse the parent's company-scoped review signals and shared scoped deadline worker. - Remove approved scanning from the aggregate tick. Preserve the public combined sweep and scheduler-disabled startup behavior. - Document producer ownership, quiet states, lifecycle behavior, and remaining recovery duties. ## Verification Latest published head: `23868c1751c100a27f58087f7ce16b93c52d4eba`, targeting master. The review-found delegated gateway pool deadlock is repaired by passing the admission transaction through token and issuer-authority reads. Foreground and recovery each reproduced the deadlock with a one-connection pool; all three constrained-pool cases, including revoked authority, now pass. All 81 HTTP integration cases pass. The historical foreground rate-slot regression also reproduced before its repair; verified preview hashes are accepted only by read-only preflight, while consuming admission still requires the complete identity under the exact claim. Five full suites passed 329 cases; the gateway service then passed all 141 cases after two additional mixed historical hash controls. Production types/build passed before the final normal integration of master `35820cf398` (including independent Grok/Runner changes); integration validation is running. Initial fixture/assertion errors and the prior E2E restoration/cancellation deadlock remain recorded in [the owning audit](doc/idle-approved-tool-recovery.md#gateway-admission-uses-its-transaction-connection). Fresh-head CI, Greptile 5/5, and passing Ponytail are still required before merge. - Master landing reconciliation at `8dfe3958db6e4e8a0b7499f3f5be0a396ca77361` after #17453 merged: production and server test files are byte-identical to the accepted 320-case repair. The landed migration journal prefix is unchanged, and #17928's independent eval reporting changes are preserved. Fresh CI and reviews on this exact head are required before merge. - Historical-identity repair published at `096b7c7198c3ca864aca9377e261cbaa0a8aad6f` (production/test source identical to integrated acceptance `81231824d773a79fc50c835eccf7d9b3977e79fc`): all 320 accepted gateway, policy, startup, preparation-timing and authenticated HTTP management cases pass. Server types and the production server build pass. Four real historical sensitive-argument regressions failed before repair; invalid-signature, foreign-hash and stale-claim controls remain denied without dispatch or identity changes. Both agent and Test-tab calls deliver the original signed arguments once, and new rate policies still apply. - The first integrated combined run passed 319 cases and hit a Slack bot ownership collision left in a reused HTTP test database. The guard correctly returned 409. The full 78-case HTTP suite passed on a fresh dedicated fixture without product or assertion changes. Earlier fixture-construction errors are distinguished from the four real regressions in the audit document. Fresh remote CI and reviews remain required. - Parent reconciliation at `736e7f56097055e21ce13b4e45d9033391483af8`: 299 cases pass (221 full gateway/policy/startup cases and 78 assembled-server connection management cases), with no skipped cases. Server types pass. This includes real delegated Test-tab approval admission for newly added shared and caller rate rules. Fresh remote CI and Greptile review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/bcca5b006976a471f4afd1f70c4031aa40a2b3f7/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Current head: `bcca5b006976a471f4afd1f70c4031aa40a2b3f7`. This remains a single-duty PR stacked on #17453. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/a021467f04d22be42ee5b0c87b6583b2300a62a9/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Notifications cover the existing database-owner, single-active-writer contract. Independent SQL writers and other replicas need an explicit wake path. - Unknown provider outcomes cannot be replayed safely. Failed result persistence retains an executing claim and the existing stale-execution quarantine. - Interrupted execution and terminal interaction projection repair remain periodic until their separate slices. This PR changes only approved recovery. - A pending-only review write can trigger one empty approved-recovery pass. Settled idle work schedules no further scan. - The partial index uses the existing top-level concurrent migration executor. Snapshot pruning follows repository policy. - Cloud sleep eligibility and external alarm coverage remain unchanged. Worker quiet is not proof of deployed instance suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d108aeeaaf |
fix(server): schedule pending tool-review expiry from committed changes (#17453)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Governed tools can wait for a human review before execution. > - Pending reviews already save a deadline and must reject approval after expiry. > - The server currently scans for expired reviews on every maintenance tick. > - Empty scans keep the database active without advancing work. > - This pull request schedules pending review expiry from committed changes and saved deadlines. > - Configured idle state and resolved history no longer cause this expiry scan. ## Linked Issues or Issue Description **What happened?** The combined tool review sweep repeatedly queries pending expiry even when no pending request has a deadline. **What did you expect?** Signing or resolving a request should update the expiry schedule. A saved expiry should trigger settlement. Once no expiry remains, this duty should stop querying. **Steps to reproduce** Start a claimed instance with heartbeat scheduling enabled. Leave it idle with no pending tool reviews. Observe pending-review expiry discovery in the maintenance sweep. Stacked on #17450; earlier slices are #17446 and #17443. These all reuse the delivery and transaction foundations from #15625. This PR owns pending-review expiry only. Approved-action recovery, interrupted-execution quarantine, and terminal interaction projection repair remain separate migrations. No duplicate pending-review expiry migration was found in the open PR search; credential lifetime changes in #17221 and #17274 concern the parent credential slice. ## What Changed - Replace repeated company review-deadline aggregates with an indexed single-row lookup. Add and replay a concurrent covering-index migration and generate combined snapshots throughout the stack. - Extract pending-expiry selection and settlement into its owning service. Keep provider execution and interaction publication with the gateway. - Signal deadline assignment and pending resolution inside the owning SQL transaction. Preserve outer commit/rollback and central uncertainty handling. - Register company expiry deadlines with the shared scoped coordinator. Reconstruct on startup; normal producer hints reload only their company. Preserve writes during drain and concurrent reconciliation. - Settle request expiry and invocation failure atomically. Keep interaction projection recovery in the existing separate sweep. - Remove pending-expiry scanning from the main tick. Preserve the combined manual API and startup-only behavior when scheduling is disabled. - Add a partial pending-expiry index through an idempotent concurrent migration. No queue table is added. - Document supported producers, quiet states, remaining duties, and external wake limits. ## Verification - CI run `40574661575` at `f5c935d1b89877033eb82bf890f867baeb372ef3` lost the runners for general-server shard 9/12 and E2E shard 2/8 at 20:09:03 UTC. Both job logs explicitly show a runner shutdown signal followed by cancellation; no test assertion failure was reported by those jobs. A single infrastructure retry is justified once the run completes; it has not been issued yet. Both current-head reviews are 5/5, and CI must still be green before merge. - Current master landing at `f5c935d1b89877033eb82bf890f867baeb372ef3`: 128 complete gateway expiry, startup, scoped deadline and preparation-timing cases pass with no skips. Server types and the production server build pass. Landed tracing changes from #17918 are preserved. The master migration journal prefix remains semantically identical, with only this PR's two migrations added. Fresh remote CI/review is required before merge. - Parent reconciliation at `17539e2a6cf8ac4922b3cbe06af6df361bfe5665`: 116 tests pass across the full tool gateway service, startup and scoped deadline suites against a dedicated migrated PostgreSQL database. No cases were skipped. Server typecheck passes. Fresh remote CI and review are pending. Current head: `ec28076f825e76a267f05517ff0c44a3e2283ce7`. This remains a single-duty PR stacked on #17450. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/ec28076f825e76a267f05517ff0c44a3e2283ce7/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - All supported producers must share the database signal owner. Independent SQL writers and other replicas need a live notification contract; startup alone does not cover them. - A crash after authoritative expiry and before interaction projection still relies on the existing projection-repair duty. That duty, approved-action recovery, and uncertain execution recovery remain periodic in this slice. - Unknown external outcomes remain quarantined. Provider calls stay outside SQL transactions. Expiry is still checked before approving a request even if physical cleanup waits during drain. - The new index uses the existing top-level migration executor because PostgreSQL concurrent index creation cannot run in a file-wide transaction. Snapshot pruning follows the repository rule. - This establishes worker quiet and reactivation, not deployed host suspension. Existing Cloud idle-safety and external alarm coverage remain unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Latest-head CI is now green after one evidence-backed retry. Greptile is 5/5 and all review threads are resolved. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5398275d2a |
fix(server): schedule gateway credential expiry from committed changes (#17450)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The tool gateway stores credentials for sessions and named gateways. > - Those credentials already contain expiry dates. > - The server repeatedly scans their tables even when no expiry work exists. > - Creation signals and saved deadlines can drive cleanup without idle scans. > - This PR registers that work with the existing coordinator. > - Authentication continues to reject expired credentials independently of physical cleanup. ## Linked Issues or Issue Description Refs #17446, #17221, #17274. This is the third sleep-migration slice, stacked on #17446. **What problem does this solve?** Gateway credential cleanup queries PostgreSQL on every maintenance tick, including empty tables and retained non-expiring tokens. **What is the proposed solution?** Signal expiring token/session creation from its transaction. Restore the earliest saved expiry at startup. Retry only while cleanup or reconciliation remains outstanding. **Why is this useful?** Configured idle gateways no longer create recurring expiry traffic. Existing credential data remains the durable source of truth. Related work: #17221 and #17274 change run-token lifetime and cleanup eligibility. They remain separate. This PR preserves current master authorization and deletion rules. When those PRs are combined, update the deadline query and signals to match their new operator-token expiry predicate; run-token idle markers must not keep this worker scanning. Retain the separate run-liveness reaper and run both regression suites. Text mergeability is insufficient evidence for that integration. ## What Changed - Make the credential test clock follow real scheduled work without overshooting future-expiry assertions; production behavior is unchanged by this fixture follow-up. - Coalesce sequential credential notifications within the former cleanup cadence, preserve loaded expiry deadlines and immediate authentication rejection, and bound scheduled deletion to 500 rows per pass. - Added transaction-aware intent before the two existing credential inserts. Non-expiring named tokens do not signal expiry work. - Added earliest-expiry selection in the gateway service and a narrow registration wrapper for the shared deadline helper. - Removed gateway credential cleanup from the aggregate maintenance tick. Startup reconstructs saved deadlines. Heartbeat-disabled instances retain their startup-only cleanup behavior. - Preserved expiry/revocation checks, run authorization, audit behavior, and the existing 500-token cleanup batch limit. - Added real PostgreSQL tests and a lifecycle/producer document. Tool-action review recovery remains a separate slice. - Drain all expired session/token pages in explicit and scheduler-disabled startup cleanup, while retaining one-page scheduled passes and authentication enforcement. ## Verification - At `907098cd3656bac59c60220c73b085d0349d4b5f`, all 290 Runner transport tests pass locally (including all seven prepared-environment variants). CI run `40570815360` attempt 1 had one provider-drain settlement failure in the `close-attachment` case; the identical source passed the preceding CI lane. A single failed-job rerun is in progress. The failure remains recorded; no shutdown assertion or production behavior was changed to suppress it. - Master landing reconciliation at `907098cd3656bac59c60220c73b085d0349d4b5f` after #17446 merged: all production and test files are byte-identical to `24318d9b99ee7f044a864ff3981fc1929bf3e509`, where the 150-case acceptance and server types passed. Only commit ancestry and the plan's merged status changed. Fresh CI/review is pending on this head. - Parent reconciliation at `24318d9b99ee7f044a864ff3981fc1929bf3e509`: 150 affected tests pass across gateway credential expiry, the complete tool gateway suite, custom-image worker and startup. The accepted PostgreSQL runs skip no cases. Server types pass. Existing direct tests cover sequential-write coalescing and saved-deadline precedence; the older Ponytail finding needs a fresh review. Remote CI is pending. Current head: `10c85a5eaa288867aa2997a27aee5f675c2a713d`. This remains a single-duty PR stacked on #17446. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/10c85a5eaa288867aa2997a27aee5f675c2a713d/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/10c85a5eaa288867aa2997a27aee5f675c2a713d/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Shared-owner signals cover supported local producers. Independent SQL writers or another replica need a separate ingress contract; startup recovery is not live discovery. - This is scheduling only. The open run-token lifetime PRs need semantic integration as described above before combining their behavior. - Expiry cleanup can wait during drain; authentication still enforces expiry on use. Existing Cloud safety gates are preserved. Local deadlines do not prove external wake coverage or deployed suspension. - Revocation does not move expiry. Non-expiring tokens remain retained by design; this PR does not invent a retention deadline. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Latest-head CI is now green after one evidence-backed retry. Greptile is 5/5 and all review threads are resolved. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
623f1dc289 |
fix(server): schedule secret-proposal expiry from committed changes (#17446)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can propose secrets and bindings for approval. > - Pending proposals already contain their expiry and encrypted value. > - The server scans for expired proposals on every maintenance tick, even when none exist. > - Committed proposal changes and saved expiry times provide the required triggers. > - This PR uses those triggers and preserves scrubbing, approval checks, and recovery. > - Empty and completed proposal history no longer causes recurring expiry queries. ## Linked Issues or Issue Description Refs #17443. This is the second PR in the sleep migration stack and targets its branch. **What problem does this solve?** The periodic proposal scan contacts PostgreSQL when no proposal needs maintenance. That obstructs idle database behavior across tenant instances. **What is the proposed solution?** Use transaction-aware notifications for creation and resolution. Reconstruct the earliest pending expiry at startup. Run cleanup at that deadline, or retry while reconciliation remains unfinished. **Why is this useful?** It removes an idle scan while retaining security checks and physical ciphertext cleanup. No queue table or migration is required. ## What Changed - Added intent notifications before secret/binding creation, approval, terminal transitions, and linked interaction resolution. The existing outer transaction owns settlement. - Kept expiry selection, ciphertext scrubbing, dependent rejection, and authorization in the secret-proposal service. - Registered a startup and deadline worker in place of the aggregate maintenance sweep. Empty/terminal states disarm; overdue batches and failures retain retries. - Reused shared scoped deadline dispatch for secret expiry. The service owns company-filtered SQL and expiry policy. Custom-image expiry continues to use `registerDeadlineWork`. - Added real PostgreSQL producer, outer-transaction, concurrent-write, expiry, restart, interaction resolution, and drain/resume tests. Updated startup wiring fixtures and documented the supported writer/lifecycle contract. ## Verification - Landing reconciliation at `c10eb1bcf752b058713e05d3345e5cf38dd3fa72`: 160 tests pass across the complete secret-proposal, scoped deadline, company batch, coordinator, startup, custom-image service and custom-image worker suites; no cases skipped in the accepted runs. Server typecheck passes. This integrates master after #17443 landed; fresh remote CI and Ponytail review are required before merge. Current head: `5bbe17ad9ebc4573b08ed1a9323644ef9608ca09`. This remains a single-duty PR stacked on #17443. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/5bbe17ad9ebc4573b08ed1a9323644ef9608ca09/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/5bbe17ad9ebc4573b08ed1a9323644ef9608ca09/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Notifications cover supported writers sharing the database signal owner. Independent SQL writers and other process roots require a separate ingress contract; startup recovery alone is not live replica support. - The shared coordinator and scoped worker also serve later stack duties. Their defaults remain intact; existing voice, chat, native, and startup consumers are included in integrated verification. - Startup now checks overdue proposals immediately. During drain, physical cleanup waits; approval still rejects expired proposals synchronously. - Each due company pass remains bounded to 100 rows. Failed scopes back off from 30 seconds to five minutes; healthy overdue work keeps draining. Self-generated write hints coalesce and do not erase a failed sweep's backoff. - Heartbeat-disabled behavior and Cloud sleep-safety gates remain unchanged. This does not establish deployed suspension or external wake coverage. Tool review and gateway token cleanup remain separate slices. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |