2767 Commits

Include renames

Author	SHA1	Message	Date
Nicky Leach	fba0ab8162	fix(runner): preserve GitHub helpers and Codex probe policy (#18212)	2026-10-30 07:42:10 -07:00
3 people	e345b9217d	fix(runner): paginate native task searches without hidden truncation (#18282) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents search tasks to find work that already exists. > - The search schema permits 200 results, but the server returns at most 100. > - The response does not tell the agent that more results exist. > - This pull request adds pages and reports the page size and continuation cursor. > - Agents can read the full accessible inventory before they decide that a task does not exist. ## Linked Issues or Issue Description **What happened?** A native `search_tasks` request with `limit: 200` returns only 100 tasks. Historical scans recovered 167 and 173 tasks with status partitions. This proves missing inventory and extra work. It does not prove duplicate task creation. **Expected behavior** Honor the declared limit. Report more results and give the caller a continuation cursor. **Steps to reproduce** Create 205 matching tasks in a test company. Call `search_tasks` with `limit: 200`. The baseline returns 100. The fixed path returns 200, then five when the caller uses `nextCursor`. **Paperclip version or commit** Baseline: `81ebe479e8db8a3d59c86cdfb894bd917734492f`. Related changes: #16979 changes UI status filters before pagination. #13699 adds public task URLs. Neither repairs this native limit. An exact `search_tasks` plus pagination PR search returned no result. Pattern: [Task searches omit cases beyond the first hundred](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/c20462d6-05fd-4862-bfbd-7bec1f2bc159). ### Hypothesis and thesis for a fix The native authority uses `boundedLimit` with its default maximum of 100. The tool schema permits 200. The response contains only `tasks`. The raw request and result counts agree with this cause. Fix the native authority and its canonical tool contract. Keep the existing `tasks` field. Add `limit`, `returnedCount`, `hasMore` and `nextCursor`. Use ascending task IDs so task activity does not move results between pages. Apply current company, privacy and Muse read checks before each page. A cursor grants no access. Callers must keep the same filters and read every page. Concurrent task changes require a fresh scan. ## What Changed - Honor the native search limit of 1 to 200, with a default of 50. - Add ID cursor pagination and page metadata. - Describe complete inventory reads in the tool contract and access map. - Refresh the generated semantic contracts and seeded catalog identity. - Add regression coverage for complete inventories and access-filtered pages. ## Verification - Red: the database integration test requested 200 of 205 persisted tasks. Baseline returned 100. - Green: the same regression returns pages of 200 and five without duplicate IDs. - The native authority and Muse suites cover company exclusion, status filters, empty pages, invalid cursors and readable-only metadata. - Command: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts src/services/native-runtime/muse-native-visibility.test.ts`. - The test uses the real run-bound native authority and disposable PostgreSQL. It does not use an external model or change a deployed scanner. - Both native authority and Muse suites pass: 71 tests. Server direct `tsc --noEmit`, Runner `typecheck:typescript`, `node scripts/check-maps.mjs` and generated semantic contract checks pass. - Full typecheck and build cannot finish here because `cargo` is absent. The full local test command was stopped after about five minutes when the complete CI matrix was running. It has no complete verdict. Current-head CI passed, including typecheck, build, all general/serialized/Runner test suites and E2E. Greptile is 5/5 with no actionable findings at `392cb7424a1fa0ec73401d5e8a10a22b504fde32`. Optional Storybook and manual Docker workflows were skipped and are not executed coverage. The success-example correction also passed all 82 contract tests locally. ## Risks Task order changes to ascending ID. Pagination is not a snapshot. New or changed tasks can require a fresh scan. Each page retains the existing full company-list read and Muse authorization cost. This change adds no database query loop or new background work. Existing callers can still read `tasks`, but must follow `nextCursor` for a complete inventory. No scanner implementation was supplied. No migration or deployment is included. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. The exact provider deployment ID and context limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge The task requires the supplied branch to stay unchanged. All applicable CI and review checks pass at the current head. Unmerged and not deployed. --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 09:23:10 -05:00
DottaandPaperclip	6da112ab40	perf: avoid flushing separate Git cache seed archives (#18308) ... ## Thinking Path > - Paperclip manages work done by AI agents. > - Native sandbox runs prepare a workspace before the agent can respond. > - A shared seed cache avoids repeated Git copies and archive preparation. > - Cache publication also flushes a branch-neutral Git seed that is separate from recovery archives. > - This cache can be rebuilt. Each reader checks the archive bytes before reuse. > - This change omits that separate cache file flush and measures the remaining preparation gaps. > - The benefit is less work before agent execution and clearer timing evidence. ## Linked Issues or Issue Description Refs #18240. **What existing behavior does this improve?** Workspace seed cache publication and latency diagnostics for native sandbox preparation. **Current behavior** Publication flushes both cache archives to disk under the company lock. Slow storage can delay the publishing run and cache readers. A cache hit also combines private copying, hashing, and last-used bookkeeping in one span. Early controller preparation has gaps before environment selection. **Proposed behavior** Keep the existing flush for recovery-backed workspace and Git archives. Omit the flush only for a separate branch-neutral Git cache archive. Reject an alternate cache file that is the recovery file or its hardlink. Keep archive checksums, private copies, atomic publication, quotas, and company locking. Measure cache read steps and the existing early preparation operations. ## What Changed - Skip the flush only for the separate branch-neutral Git seed used by the company cache. - Preserve the existing flush for recovery-backed archives, including hardlink storage. Reject cache-only files that alias recovery bytes before or during publication. - Add a regression with a blocked cache-only Git flush. The deployed code waits; the new code publishes verified cache bytes without waiting. Test storage failure for both recovery-backed archives and same-path, hardlink, and replacement alias cases. - Add cache copy, hash, and bookkeeping spans beneath seed lookup. - Add timing around existing agent, conversation, runtime, issue, identity, trust, and task context preparation. Keep their execution order. - Update the trust-order guard to recognize the timed call while continuing to assert trust resolution precedes workspace selection and credential setup. - Document the cache contract and the new spans. Keep the old flush span in the allowlist for compatibility with older observations. ## Verification - The blocked-flush regression fails on the old code and passes with this change. - Focused tests (`vitest run` on the six affected cache, timing, and native-sync suites) cover cache corruption, private-copy integrity, quota and eviction races, branch binding, per-run recovery, native workspace synchronization, trace parentage, and tracing disabled or failing. - 115 focused tests and both typechecks pass after rebase on current master. The tests include recovery storage failures and an archive replaced with a recovery hardlink during verification. - All seven trust tests pass with the actual service and migrations, using a fresh database on an existing local PostgreSQL server. The local embedded server could not allocate host semaphores, so only its test startup helper was substituted; assertions and service code were unchanged. - A real SDK export with a retained public seed verifies that cache-read detail spans have the lookup as their parent. The endpoint-off control produces no spans. - Current revision `28dea63432d462cad160906f8dfc2d21c81d3446`: all 53 checks pass, two optional checks are skipped, Greptile is 5/5, and no review threads are open. - No new staging latency claim is made by this PR. Hosted measurement remains necessary after delivery. ## Risks A crash or power loss can discard the cache-only Git seed. A later read rejects missing or corrupt cache bytes and prepares the workspace again. Recovery-backed archives retain their previous flush behavior. The change adds no flush to warm per-run persistence. Hashes and private-copy checks remain required. The new spans use the existing opt-in exporter and closed names; they record no task content. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The session does not expose the exact model SKU or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 09:13:08 -05:00
Devin FoleyandClaude Opus 5.5	2a221e8e80	fix(sentry): keep scheduled native model capacity retries out of run-failure reports (#18295) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The run-failure reporter sends terminal run failures to Sentry when an operator sets `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` > - In `unexpected` mode, the reporter skips failures that a bounded automatic retry already owns. The Codex ACP model capacity failure (`codex_transient_upstream`) is one of these > - The native runner has the same capacity failure: Codex `serverOverloaded` ("Selected model is at capacity"). Native finalization schedules a bounded automatic retry for it (#15347) > - The reporter has no matching exclusion for the native path. Each scheduled capacity retry also sends a Sentry error, even when the retry recovers the task > - This pull request skips the report only when the native failure has typed capacity evidence, a committed retry decision, and the retry run that the decision created > - The benefit is that Sentry shows capacity failures only when automatic recovery stops (retries are exhausted, held, or not scheduled) ## Linked Issues or Issue Description Refs #15347 (native capacity retry). Refs #16407 (Codex ACP capacity exclusion). **What happened?** A native Codex run failed with "Selected model is at capacity. Please try a different model." (`native_provider_overloaded`). Native finalization scheduled the automatic capacity retry. The retry run succeeded and the task completed. The run-failure reporter still sent a Sentry error event for the first run. **Expected behavior** When the automatic capacity retry owns recovery, the run failure stays in the local run log and does not go to Sentry. This is the same as the Codex ACP capacity path. When retries are exhausted or held, or when no retry is scheduled, the failure goes to Sentry. **Steps to reproduce** 1. Set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` with a backend Sentry DSN. 2. Run a native Codex agent turn that ends with a Codex `serverOverloaded` terminal error. 3. See that native finalization creates a `scheduled_retry` run with `scheduledRetryReason=native_provider_overloaded`. 4. See that Sentry also receives a run-failure event for the failed run. ## What Changed - `server/src/services/run-failure-report.ts`: add `isScheduledNativeCapacityRetry`. In `unexpected` mode it skips the report only when all of these are true: - the run is native, `failed`, has `errorCode=native_provider_overloaded`, and has a native task (`nativeIssueId`); - `resultJson.nativeProviderFailure` is the closed `codex`/`server_overloaded` diagnostic; - `resultJson.finalizationPhase` is `committed` and `finalizationReasonCode` is `native_provider_overloaded`; - the report is not from setup, and the run has no signal; - a successor run for the same task exists with `retryOfRunId=run.id` and `scheduledRetryReason=native_provider_overloaded`. The same status transaction creates this run. - The successor lookup filters on `company_id`, `context_snapshot ->> 'issueId'` and `created_at >= run.createdAt`, so the existing `heartbeat_runs_company_ctx_issue_created_idx` bounds it to that task's runs since the failure. No migration is needed. - A failed read of the retry evidence logs a warning and reports the failure. - `doc/observability.md`: document the new exclusion in the run failure context section. - Unit tests: the exclusion, and twelve cases that must still report (exhausted, held, no retry run, another retry lane, successor for another task, run with no native task, finalization not committed, wrong or missing diagnostic, legacy run, signal, timeout). Also setup phase, `all` mode, and an unreadable retry record. - Integration test (real native finalizer): no Sentry event for retries 1 and 2, one event when retries are exhausted, and one event when reassignment prevents a retry. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/__tests__/run-failure-report.test.ts src/services/native-runtime/native-provider-overload.integration.test.ts src/services/native-runtime/native-provider-failure.test.ts src/services/__tests__/status-arbiter.test.ts`: 359 tests pass. - Red/green: before the fix, the new unit test captured 2 events, and the integration test captured an event on the first scheduled retry. After the fix, both pass. - `EXPLAIN` on the migrated schema: `Index Scan using heartbeat_runs_company_ctx_issue_created_idx` with company, context issue and created time in `Index Cond`. - CI provides the full workspace typecheck. ## Risks - Low risk. The change applies only in `unexpected` mode and only to native runs with typed capacity evidence and an existing retry run. `all` mode still reports every failure. - The check adds one index-bounded read of `heartbeat_runs` for runs that match the in-memory capacity checks. Other runs do not do this read. - If a scheduled retry later fails, that retry run reports its own outcome. When retries are exhausted, the final failure reports. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with tool use (shell, file edits, test execution) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-30 05:42:43 -07:00
DottaandPaperclip	965184d296	fix(runner): reuse accepted confirmations on native continuation (#18283) ... ## Thinking Path > - Paperclip coordinates agent work and human decisions. > - Native agents receive accepted confirmations when their task resumes. > - A resumed run can repeat the original confirmation with the same key and scope. > - The interaction service rejects that replay because the source run changed. > - This PR returns the original acceptance only to its bound native continuation. > - Changed requests and newer proposals keep their existing approval gates. ## Linked Issues or Issue Description **What happened?** A native agent received a human acceptance, then repeated the same confirmation key and request. Paperclip returned a conflict because `sourceRunId` differed. The agent then used a new key and created a second pending confirmation. The task waited for another answer. **Expected behavior** An exact replay from the native run that received the accepted response returns the original accepted record. It creates no request or wake. Changed scope, another creator, an unrelated continuation, and a newer proposal must still fail. **Steps to reproduce** 1. Create a confirmation from a native run. 2. Accept it as a human. 3. Resume the same task with that accepted response in its saved native input. 4. Repeat the original confirmation key and request. Before this fix, step 4 returns `409 Interaction idempotency key already exists for a different request`. **Paperclip version or commit** Reproduced against source `81ebe479`. The deployed failure was also confirmed against Core `6a674716`. **Deployment mode** The failure occurred on Cloud. The regression uses the real native tool authority and PostgreSQL 17 with synthetic data. The full fresh Cloud journey is not yet verified with this fix. No duplicate open PR or issue was found in the related confirmation-replay and interaction-idempotency searches. ## What Changed - Let the current run's parsed, identity-bound accepted response nominate an internal replay of a plain confirmation or checkbox. Exclude governed tool, secret, connection, and quality-review cards. - Reuse the original accepted record only when the key, creator, and full normalized request match. Preserve its source run. Reject a newer confirmation. - Keep ordinary source-run comparison and mutation receipts. Recheck accepted cross-run reuse on cached calls and lost-ack recovery. An absent original cannot create a replacement. - Add real database regressions and document the replay contract. ## Verification - Real PostgreSQL regression: the original replay failed with the observed conflict, then passed. - Cached replay regressions: two controls first returned stale acceptance. Both now reject it. - Final replay and recovery suite: 27 passed. This covers both plain and checkbox confirmations, governed-payload exclusion, identity, company, request changes, newer proposals, revised plans, absent receipts, absent originals, and tool-key conflicts. - Existing native-tool and onboarding admission suites passed with the first 24 replay controls: 109 tests at `3cfd58e7`. The scope follow-up reran all 27 replay controls. - Existing interaction service and policy suites: 122 passed. - Full `pnpm build` and `pnpm -r typecheck` passed at `3cfd58e7`. After the scope follow-up, the focused 27 tests and server TypeScript check passed. Required full Linux CI passed on the final head `eabdbd6b`. - The full local Vitest suite was not repeated because the macOS embedded PostgreSQL bootstrap failure is already documented. Relevant suites used the supported external test database paths. Required Linux CI remains pending. - Final-head CI attempt 1 failed only the unchanged Antigravity login success fixture. Its implementation and test match a previous passing head. The exact failure cause is not retained. The original [failed job](https://github.com/paperclipai/paperclip/actions/runs/41014308876/job/123927423551) is preserved. The one failed-jobs-only retry request passed on the unchanged final head. This confirms recovery, not the cause of the original failure, and is separate from the Cloud journey. - Full fresh Cloud onboarding and its dependent journey remain unverified at this revision. ## Risks An approval replay must not authorize new work. The service keeps exact request and creator checks under the task lock. Newer confirmations and ordinary downstream task and target checks remain effective. Normal API callers cannot supply the internal replay nomination. No database migration or public request-schema change is included. ## Model Used OpenAI `gpt-6.1-sol`, ultra reasoning, with repository and shell tools. The parent runtime metadata supplies this model identity. The context window is not exposed. Independent review used the same model family. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 01:53:45 -05:00
Nicky LeachandClaude Opus 5.5	fa760dbce4	fix(runner): reuse image packs only with matching contract digest (#18237) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native runs use a provider pack. The controller reuses a protected pack from the sandbox image when the pack is compatible. > - #17749 gates this reuse on a hand-maintained interface revision in `provider-pack-contract.json`. > - Later changes altered the runner/pack boundary (#17922, #18188), but nobody bumped the revision. A stale Claude or Codex image pack can still be reused across a sidecar contract change. This is the failure class of #17972. > - A manual revision has a "forgot to bump" failure mode. Source-byte hashes change on every refactor, so they are too brittle. > - This pull request declares the sidecar `session.open` contract in its JSON schema and computes a contract digest from the JSON contracts and Rust shapes. > - Image-pack reuse now requires an equal digest. > - The benefit is that an incidental change keeps image packs reusable, and a real contract change retires them automatically. ## Linked Issues or Issue Description Refs #17749, #17972, #17922, #18188. **What happened?** The controller reused an image provider pack when `compatibility.acpx` and the selected agent's profile digest matched. For Claude and Codex, that profile digest binds only the upstream executable. The ACPX interface revision stayed at 1 after later sidecar contract changes. A pack built before those changes passed the reuse check. **Expected behavior** The controller reuses an image pack only when the pack uses the same runner/pack contracts as the controller's bundled pack. Comments, formatting and refactors do not change that result. **Steps to reproduce** 1. Build an image pack from a commit before a sidecar input change, such as a new `session.open` field. 2. Run a controller from a later commit that sends the new field, with the same interface revision and Claude version. 3. Before this fix, `verifyRemoteProviderPack` accepts the old image pack. ## What Changed - Declare the sidecar `session.open` parameter set as a closed `sessionOpenParams` definition in `acpx-sidecar.schema.json`. - `generate-acpx-sidecar-contract.mjs` now generates `GENERATED_ACPX_SIDECAR_OPEN_PARAMS` from it. - `parseOpenParams` uses the generated list instead of an inline `Set`. The accepted set is unchanged. - This is the surface of #17972. The schema is now its source of truth. - Declare the nested `session.open` contracts that the sidecar parses strictly: - `protocol/provider-schemas/runtime-context.schema.json` closes every runtime context object. `parseNativeRuntimeContext` uses the field lists that `scripts/generate-runtime-context-contract.mjs` generates from it. - The sidecar schema declares the closed `providerPolicy` object, and `parseProviderPolicy` uses its generated field list. - The accepted fields do not change. - Add `packages/paperclip-runner/scripts/contracts/contract-fingerprints.mjs`. It fingerprints these contracts: - the PRP and sidecar JSON schemas, with annotation text removed - the generated catalogs in `generated/` - the Rust serde shapes, `json!` payload keys, match literals and env keys in `runner-core/src`, without test-only modules - the reserved env key lists in `configured_environment.rs` - Read only the trees that every pack build context contains. Fail when a registered source is missing, so two build contexts cannot silently produce different digests. - Use only Node built-ins. The registry adds no dependency. - Write `contractDigest` as a top-level payload field in `build-provider-pack.mjs`. The manifest digest algorithm and the existing field layout do not change. - Require an equal `contractDigest` for image-pack reuse in `remote-provider-pack-verification.ts`. A pack without a digest is not reused. The controller uploads its bundled pack, as it does today for any incompatible pack. - Re-attest the source-hashed profiles, because this PR changes their bound sidecar sources: - Grok becomes profile 15 and now also binds the generated allowlist, so a Grok field change in the schema still invalidates the Grok identity. - Copilot becomes profile 88, and also binds the generated runtime context fields. - Earlier profiles stay decodable. - Add the registry script and `generated/` to the Daytona image content inputs. - Document the digest and the schema-first rule in `docker/daytona-runner/README.md` and `doc/DEVELOPING.md`. ## Verification - `node --test scripts/contracts/contract-fingerprints.test.mjs` (in `packages/paperclip-runner`): 4 tests pass. - These changes keep the digest: - schema description text - schema reformatting - renaming `parseOpenParams` - Rust comments, `#[serde(skip)]` fields and `#[cfg(test)]` modules - edits to the `#[path]` test module `tests/provider_api_validation.rs` - These changes alter the digest: - a new `session.open` schema field - a changed constraint on the `remainingWork[].description` property - a new reserved env key - a new sidecar env key - a new checkpoint field - A build context without a registered source fails. - `node --test test/acpx-sidecar-contract.test.mjs`: 13 tests pass. A new case rejects an unknown `session.open` parameter and accepts each declared parameter. - `node --test scripts/grok-profile-identity.test.mjs`: 4 tests pass. A change to the Grok field in the generated allowlist invalidates the Grok identity. - `generate-acpx-profiles.mjs --check`, `generate-acpx-sidecar-contract.mjs --check` and `generate-protocol-manifest.mjs --check` pass. - Runner vitest suites pass: 470 passed, 1 skipped. They cover the sidecar, sidecar input and protocol, Copilot and Pi installation, profile compatibility, native execution, recovery identity, the backend factory and the environment. - Server tests: pack verification, executor and artifact preparation pass (1,004 tests). - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. - `tests/runner-e2e/daytona-image.test.ts`: 10 of 11 tests pass. The failing test also fails on `master`. It expects a literal `profile-v19-identity.json` line in `.dockerignore`, which uses a `profile-v*` glob. - Not run locally: - A complete `build-provider-pack.mjs` run. The builder rejects the local Homebrew Node because it is not portable after relocation. - Rust tests and `acpx-controller-recovery.test.ts`, because `cargo` is not installed. - The full `pnpm -r typecheck`, `pnpm test:run` and `pnpm build`. ## Risks - Only JSON contracts and Rust shapes are part of the digest. The remaining TypeScript-only definitions are the `paperclip_finish` and `paperclip_block` tool schemas in `completion-result.ts` and the TypeScript env key lists. They are not covered until they move into a schema. The nested `expectedIdentity` and `tools` values ignore unknown fields, so a change there cannot cause a startup rejection. - A change there can let a stale image pack be reused. The sidecar then rejects the input with its strict parser error. - Today, Claude and Codex image packs have no contract check at all. - The digest covers all registered contracts for the whole pack. A contract change for one provider kind also retires image packs for the other. Each replacement costs about 16 s per remote start until the image is refreshed. - Existing image packs have no digest. After deploy, the controller uploads its bundled pack until the image is rebuilt. - The Grok and Copilot re-attestations start fresh sessions for those agents once after deploy, as each profile bump does today. - The Daytona image content ID formula changes, so the next image build does not reuse a cached image. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code in the desktop app, with repository tools, shell execution and research subagents. The session does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 22:55:11 -07:00
DottaandPaperclip	7df1656dda	fix(server): continue retention pages after overdue grace deadlines (#18270) ... ## Thinking Path > - Paperclip manages AI agents and their shared work. > - Skill snapshots need safe cleanup after their retention grace period. > - The retention worker scans large directories in pages. > - Another company's grace deadline can expire while a page is still running. > - The shared deadline helper then uses the worker's 30-second failure retry for successful continuation. > - This pull request gives this worker a one-second continuation delay. > - Directory scans keep moving while failure retries retain their existing delay. ## Linked Issues or Issue Description Refs #18266. Its unrelated fixture checks exposed this scheduling bug. **What happened?** A successful directory page can wait 30 seconds before the next page when another company's grace deadline becomes overdue during the scan. **Expected behavior** Continue successful paging within one second. Process the overdue company. Stop scheduling when all work settles. Keep the existing 30-second failure retry. **Steps to reproduce** Hold the second successful page with a deferred promise. Advance the fake clock past another company's saved five-second deadline. Release the page. Before this fix, the next wake is still 25 seconds away. The regression also requires all pages, the overdue rescan, and an idle worker after settlement. **Paperclip version or commit** Reproduced on `b3c2172d417ea6b783e0c796aaed4a91e1f9d381`. **Deployment mode** Core retention worker. The deterministic test uses the real coordinator, deadline helper, and retention worker with controlled service results. It needs no external database or provider. ## What Changed - Set `continuationMs: 1000` on the skill retention worker. - Add a controlled regression for a grace deadline that expires during a successful page. - Keep the shared deadline helper and the worker's failure retry unchanged. ## Verification - Red: the new regression fails on unchanged production code with a 25,000 ms remaining delay. The existing control passes. - Green: retention worker, deadline worker, and scoped deadline worker suites pass: 7 tests. - `pnpm build` and `pnpm -r typecheck` pass. All required checks pass on the current head. Greptile reports 5/5 with no review threads. - The full local test suite is not complete. The bundled macOS PostgreSQL runtime lacks `libzstd.1.dylib`, as confirmed in the existing host diagnostic. The deterministic worker tests do not need PostgreSQL. Full repository tests must pass in CI. - No live Cloud, provider, model, or deployment test is selected. ## Risks Successful overdue continuation can run sooner. The existing one-second page interval still applies. The test requires the remaining pages, the overdue rescan, and no more work after settlement. Failure retry remains 30 seconds. This change adds no schema, API, or product command. ## Model Used OpenAI `gpt-6.1-sol`, ultra reasoning, through Codex. Parent runtime metadata confirms this model. The context window is not exposed. Terminal tools assisted source review and testing. A separate agent reviewed the source and regression. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 23:50:08 -05:00
DottaandPaperclip	ed8fae8dbb	fix(workspaces): preserve completed shallow sandbox exports (#18221) ... ## Thinking Path > - Paperclip coordinates agent work and preserves its results. > - Remote execution copies completed workspace changes back to the host. > - An agent can fetch a published branch with shallow Git history. > - A full Git bundle omits the shallow boundary metadata and can fail host connectivity checks. > - This pull request carries the boundaries through the full-bundle fallback and imports through normal Git fetch. > - Recovery must also recognize completed native work on a reusable lease without treating it as ordinary executor reuse. > - Completed work can be saved without another provider turn, while existing host history remains connected. ## Linked Issues or Issue Description **What happened?** Sandbox finalization can fail with a missing parent commit after an agent checks out a separately fetched shallow branch. Re-exporting a full bundle repeats the failure because Git bundles do not encode shallow boundaries. **Expected behavior** Import the completed branch and saved files. Preserve existing host ancestry and concurrent host edits. Keep normal Git connectivity and workspace merge checks. **Steps to reproduce** 1. Prepare a sandbox from one host commit. 2. Fetch a different published branch with depth one and check it out. 3. Save a result file and restore the sandbox workspace. **Paperclip version or commit** Reproduced against master before this repair. The regression fails with the missing-parent error before the production change. Related public context: #15268 covers rewritten history integration. This repair handles shallow bundle transport before integration. Searches found no equivalent open repair. ## What Changed - Export shallow boundaries beside the full bundle when the delta import lacks history. - Unpack into a temporary shared bare repository. Keep parent connections already available on the host and import with Git's shallow fetch protocol. - Reject malformed boundary values and retain existing connectivity, branch, and concurrent-work merge guards. Keep boundary probes within 15 seconds without consuming the separate 60-second Git import budget. - Let source-bound preservation recover reusable leases only for the finalizer-selected, accepted successful native result and matching completion contract. Retain provider-stop, physical ownership and host-writer checks; ordinary reusable sources remain blocked. - Classify invalid Git metadata separately from omittable unsafe archives. Keep native finalization failed and preserve service-owned file recovery; carry the sender's dedicated rejection across the command boundary without transient retries. - Share the failure allowlist across bounded run/activity projections and cancelled-preservation selection. Migration `0416` rebuilds the matching partial index; task chat retains the non-retryable recovery notice. - Add real Git, filesystem sandbox and database recovery regressions. Update execution semantics and affected verification maps. ## Verification - Red: real filesystem sandbox restore failed with the same missing-parent/revision-walk error before the export repair. - Green: 240 focused Git, workspace restore, sandbox transport and native-finalization tests pass. Coverage includes both transfer paths, malformed/oversized metadata, a genuinely shallow host with its original boundary intact, and concurrent host work with both merge parents verified. - Red/green: a 14-second simulated probe left the final real Git fetch only 45,893 ms before the budget split. It now retains over 55 seconds after the probes; a 16-second probe exhausts its own budget before host import. - Red/green: an oversized read progress report was retried three times before its distinct metadata classification. It now aborts the first read before payload consumption, leaves the host tip unchanged and does not retry. - Red/green: the real metadata error was swallowed by best-effort export and native finalization incorrectly returned success. Both regressions now pass; all 240 focused Git/restore/native-finalization tests pass. A PostgreSQL/filesystem fault-injection test also verifies retained files, failed run, service-owned recovery barrier, no preservation receipt and no new wake. - Red/green: duplicate metadata produced 47 Git process starts before deduplication; the bounded-work regression now passes. Oversized byte and boundary inputs fail before Git probes. - Recovery red: both reusable lease policies exhausted recovery despite a selected accepted successful result. The assembled PostgreSQL/filesystem path now passes for retained and both reusable policies, including idempotent re-admission after exhausted automatic recovery, saved files, no agent wake, and once-only existing-result finalization. - Source verification controls reject ordinary reusable leases, rejected/failed/unselected results, another physical allocation owner, and an active host writer without changing recorded state. - The 78-test PostgreSQL/filesystem recovery/source suite passed before the metadata classification correction. Its new persisted metadata-failure regression also passes. Full workspace typecheck and build pass at final head `69f10daab2`. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Full local `pnpm test:run` attempts did not finish. They are not counted as passes. Early focused runs exceeded 5-second and 30-second fixture deadlines; the final serial suite passed all 176 tests with a 30-second per-test budget in 56.8 seconds. - Red/green: both bounded PostgreSQL projections dropped the new code in oversized results. All 19 source/projection cases now pass, plus an actual index-predicate test that excludes a metadata repair hold and still selects ordinary cancelled work. - Red/green: an oversized sender `.git/shallow` lost its error type before host validation. The real transport test now verifies one typed rejection, no shallow-payload read, and an unchanged host tip. All 465 command/sandbox/task-chat tests pass, and token/migration gates pass. - Full typecheck/build pass after these final contract corrections at `69f10daab2`; all 20 PostgreSQL source/projection/index contracts pass. All current-head CI checks passed; Greptile is 5/5 with no unresolved review threads. Merged #18230 supplies the CI-cache and Linux shell-fixture baseline repairs; their duplicate changes have been removed from this PR. The final Runner lane passed after one justified rerun of an unchanged authentication fixture timeout (all 11 fixture tests also pass locally). Previous CI failures and incomplete local runs remain recorded; live deployment and retained-run recovery remain separate verification. ## Risks - The fallback creates a temporary shared bare clone. Metadata is limited to 64 KiB and 256 unique boundaries, duplicate/parent probes are cached, and Git import has a 60-second command budget plus a separate 15-second boundary-probe budget. It runs only after a missing-history import failure. - Incorrect boundaries could hide real host ancestry. The importer retains available parent connections, and regressions cover that case. - The full-bundle fallback can transfer a bundle twice. This preserves the existing fast delta path; removing that extra transfer is outside this correctness repair. - Preservation eligibility expands only for completed native exports on reusable leases. It creates no provider turn and leaves ordinary reusable leases ineligible. Database/filesystem regressions cover result selection and competing physical/host ownership. - The migration rebuilds one partial index; no tables, columns or user records change. There are no agent execution admission, model, trust, recovery permission, workflow or lockfile changes. ## Model Used OpenAI Codex, GPT-6, with reasoning, code execution, and browser tools. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 20:02:53 -05:00
DottaandPaperclip	ebed7f5237	test(evals): protect Antigravity setup and two-turn delivery (#18190) ... ## Thinking Path > - Paperclip manages agents and their work. > - A provider integration must work through ordinary agent setup and task controls. > - The recorded Antigravity walkthrough exposed setup, delivery, and continuation failures. > - A repeatable check must verify real work and retain failed attempts. > - This pull request converts all 47 recorded steps into one fixed Product E2E journey. > - Maintainers can rerun the case and inspect independent delivery, cleanup, and accounting evidence. ## Linked Issues or Issue Description Builds on merged #18187, #18188, and #18189, and consumes the shared repairs from merged #18230 and the equivalent manual-policy declaration from #18222. Refs #18214 for the CI task wrapper. Refs #12503 and #14938 for related legacy adapter work. **Agent or provider** Google Antigravity native Runner with a saved managed Google subscription and the exact Low model. **Why this adapter is useful** A user must be able to add an agent, connect its account, and receive real task results across a follow-up. Picker visibility alone is insufficient. **How the agent is invoked** A fixed Playwright driver uses ordinary product controls. The agent runs through native ACP. Independent checks use public APIs, actual downloads, owned process observations, and native lifetime fences. ## What Changed - Register one explicit-only case: `antigravity-agent-journey.runner-antigravity.local.saved-subscription-two-turn`. - Map all 47 source steps and declare saved-account prerequisites, finite limits, and same-host macOS ARM64 scope. - Create an agent through the ordinary picker, connect the saved account, select Low, run Test, and wait for readiness. Save an instruction marker, deliver a file, and update it through an ordinary follow-up. - Require each run's visible completed file activity, exact downloaded bytes, committed Done result, same tagged conversation, distinct physical owner birth, released lease, and free native fences. Verify reload persistence. - Stop failed native runs that still own a retry. Replay an existing Stop with its durable request identity and wait for the matching audited acknowledgement. A terminal reservation without a native intent remains unconfirmed. - Keep private evidence in the existing result, report, dashboard, and accounting pipeline. Retain the paused QA agent and financial history while preserving the shared account and budget policies. - Journal only the ordinary wizard's future ID, target, name, intended harness and expected creator before forwarding its POST unchanged. Bind the response and interrupted recovery to exact identity reads; a pending 404 stays unconfirmed. - Serialize private resource snapshots through temporary-file fsync, atomic rename and directory fsync. Abort forwarding on storage failure and preserve legacy receipts. If later journal or diagnostic writes fail, retain the primary error, always close the browser, and attempt the ordinary failed report. - Add production-component browser controls and harden task-menu readiness, alias selection, scroll restoration, failure packaging, and one hermetic auth-test budget. - Wait for the existing GitHub recovery fixture to register its next deadline before asserting the original range; preserve its nested-transaction and zero-request checks. In the decision-expiry fixture, verify the exact saved expiry separately from the earlier global wake of another queue. ## Verification - Fresh registered acceptance passed on eval source `9060f3cd01b7c3445ecdf703d72e9cf30fe42b59` and merged serving product `61bd4ac6919c2309f766a25e62273b074cee3261`. It completed ordinary setup and two turns with native `.5` and `gemini-3.8-flash-low`. All 16 matchers, 14 checkpoints, and cleanup passed in **165,613 ms**. Actual downloads were **61 bytes** and **119 bytes**. Both turns passed their own run-bound browser activity checks and reload persistence. - Independent read-back confirms Done, a paused retained agent, the same connected saved default account, and no active owned run. Both accepted results have the same full tagged conversation and distinct primary process births. All 21 observed processes retired; both leases released and all three native fences were free after each turn. The selected `.5` checkpoints also showed equal agent/backend session IDs; this is an observed case requirement, not a generic ACP invariant. - Receipts report 339,273 input tokens, 7,224 output tokens, and 202,846 cached input tokens. Subscription prices are unavailable. Billing coverage remains incomplete, not free. - The normal report CLI passed. All 310 files from nine original attempts and two separate recoveries retained their hashes. The twelve current screenshots are byte-identical, and 194 local report links are contained and valid. The eight earlier reports were copied without regeneration. - Delivery source `1bf4fb04df04710b82844b3c9355767f46b5f68f` branches from `d3f2a60e795fc60d64c8b19d1eed4d48657d5492` and merges cleanly with current master `1d571de5f9f886d66e520800baec8fc2f67d0948`. All 41 prior `c8b332` authored file blobs remain identical; the 42nd path is the decision-expiry fixture contract correction. Creation ownership, the strict durable writer, and the ordinary successful workflow remain unchanged. Of 43 measured file blobs, 30 remain identical. The Antigravity oracle, artifact, billing and report gates and its suite-specific validation block remain unchanged. The actual retained agent matches its original creator; its original request `setupId` was not captured and is not reconstructed. - Later master changes include model-visible workspace guidance and rejected-terminal-report handling. Their bounded source assessment is separate: the first actual finish failed, then a same-turn replacement was accepted and committed. Later read-only results match both original native-result hashes, and the complete event catalogs have no `session.failed` event. Master #18228 also expands detection of explicit JSON file requests. The measured create/update-and-attach `.txt` task already required publication under both parsers; the publication validation gate itself is unchanged. Master #18240 changes remote workspace cache staging and adds an OpenCode index refresh. The selected local lane returns before those paths. Master #18213 adds optional numeric turn timing and trace boundaries. Its selected Antigravity model, text, IDs, result and retirement semantics remain unchanged in the bounded source review. Local diagnostic rows and awaits can change event ordinals and latency; the oracle checks relative semantic order. Collector coverage and latency improvement remain outside this proof. This does not claim that the whole current runtime is byte-identical to serving `61bd`, or relabel the measured result as a new run. - The adopted upstream cache controls passed 3/3 with pinned real Turbo. Forced work disables local and remote reads and writes; ordinary work reuses local results. The original genuine-pnpm/Turbo controls remain historical to `9060`'s earlier write-only policy. The duplicate cache patch is superseded by #18230. Its qualification-only pnpm addition is removed because the adopted upstream controls do not require it. - The E2E typecheck, map checks and 277-control replay are carried from the unchanged eval source paths. That replay passed, including the 183 Antigravity controls, catalog/connection checks and the separate product-journey validator. The carried creation controls include 91 config/recovery cases. The final launcher suite passed 19/19, including five real campaign finalization fault cases; the report suite passed 7/7. These final checks did not rerun the earlier 277-control group. The canonical production-component browser spec passed in 27.4 s under its unchanged 60 s limit before the replay. Four paused-composer cases and the product base's build/typecheck plus 376 focused regressions were verified earlier. - The shipped wizard and actual API client cover persistence before forwarding, lost response JSON, foreign/malformed requests, wrong response identity/creator and cleanup-error preservation. Real filesystem controls failed 5/5 before the writer repair and passed 5/5 afterward, preserving the old journal on partial-write, temporary-file sync and rename faults. Three persistent-sync campaign controls failed before the finalization repair and passed afterward. All five final campaign fault cases preserve the primary failure, cleanup failure, usage and failed report while attempting browser closure. The strict journal still blocks the next mutation. These are provider-free ownership controls; the Google task proof remains the original `9060` → `61bd` acceptance. - The decision restart check passed in the local 20-case fixture run. A current-source real-PostgreSQL rollback control reproduced the exact 90,000 ms earlier global wake: the old equality rejected it, while the exact durable expiry and new future-wake bound passed. The original 60 s open-state, exact-timestamp, continuation and quiet checks remain. The broader local run was **FAILED: 19 passed / 1 failed**, with a 15 s timeout and a 30 s cleanup timeout in an unchanged timestamp-pagination case. That case passed in canonical `c8b332` CI; the local timeout cause remains inconclusive. An extra local server typecheck failed against stale cached Runner declarations; it did not check this excluded test file. Both limits remain recorded for fresh canonical CI. - A broader local GitHub recovery group passed 21/22. Its unchanged pagination case observed an unexpected config request; the log does not establish the cause. That result remains retained, and fresh current-head CI is the final broader gate. - Exact delivery-head CI completed with 55 successes, two skips, and one optional neutral review. Required `ci / verify` and `ci / e2e`, build, typecheck, and Runner checks passed. Fresh Greptile is 5/5 with zero unresolved threads. Optional Paperclip is neutral 4/5; its carried recommendations are dispositioned against the public Stop contract and selected live identity evidence. Seven earlier converted attempts retain their failed grades and partial evidence. Historical Paid8 and its later read-only per-run UI observation remain separate from the fresh acceptance above. Unknown prices and the older partial startup accounting remain explicit. Full lineage and raw evidence remain private in the normal report bundle. ## Risks - This case requires an authorized saved personal default Google subscription and finite operator candidate admission. It does not test fresh OAuth or establish global qualification. - The physical observer supports a same-host loopback macOS ARM64 target. Cloud, Linux, and other models remain outside this case. - A known-cost stop cannot cap missing prices or in-flight subscription charges. Existing operator policies remain unchanged. - Raw account/native identities stay private. Missing retirement proof keeps resources retained. The selected cleanup mode does not test agent purge. ## Model Used OpenAI Codex (GPT-6). The host did not expose the exact model revision or context limit. The model used tools for code edits, tests, browser automation, and review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 19:51:32 -05:00
Nicky LeachandPaperclip	1d571de5f9	Add shared native turn attribution boundaries (#18213) ... ## Thinking Path > - Paperclip manages work performed by AI agents. > - Native execution crosses the plugin worker, Runner, sidecar, and durable event boundaries. > - Existing outer spans leave repeated preparation and admission work unattributed. > - Optimizations need a common baseline with accurate ownership and event meanings. > - This change extends the existing opt-in tracing contract with bounded timing receipts. > - It preserves execution policy and keeps provider internals opaque where the API provides no timing proof. ## Linked Issues or Issue Description **What existing behavior does this improve?** Attribution of native task preparation, retained/cold Runner handoff, and completion-to-text work. **Current behavior** Worker spans use a flat provider contract. Workspace calls lack a consistent dispatch envelope. Submitted-to-started timing is named as a provider queue even though it includes other work. The transport does not expose a shared breakdown of its waits. **Proposed improvement** Add invocation-scoped nesting, bounded numeric remote receipts, current-run transport timing, durable receipt intervals, and distinct completion/text milestones. Reuse the existing exporter. No latency optimization is included. Related public work: #17842 consolidates existing startup tracing tests; it does not add this runtime coverage. ## What Changed - Extend the optional SDK tracer with nested async scopes and completed timing receipts. Keep the host responsible for parent identity, capability checks, attributes, clocks, depth, and count limits. - Add dispatch and bounded host-read timings for environment and workspace providers. - Resolve transport parents on each call so retained sessions do not retain an earlier run's trace. - Split cold/resume/attach/prepare/registration/command/started-event waits. Add opt-in numeric ACPX Runner/sidecar status and prompt-start receipts. - Distinguish first completion request, validated tool receipt, final result commit, first nonempty text, and event persistence. Rename the submitted-to-started interval to match its meaning. - Link completed earlier remote lifecycles to their host-owned observer rather than placing them under a later polling request. Preserve sampling, timestamp limits and budgets. - Document the versioned coverage map, causal clock limits, and baseline acceptance requirements. ## Verification - Current head: GitHub CI is fully green. Standard Greptile and Apex are both 5/5, with no unresolved review threads. Full-suite validation was performed in CI; follow-up local validation was limited to affected packages and tests. - Review follow-up: early rejection, missing provider capability and startup-parent regressions were reproduced and fixed. The affected worker/SDK/timing set passes 128 tests; 28 timing/context tests, SDK typecheck and direct server TypeScript checking pass. The server typecheck wrapper reached an unrelated Runner rebuild and stopped because Cargo was absent from PATH; full Runner/image validation remains in CI. - CI failure accounting: the earlier Turbo cache/force conflict and Linux OpenCode shell assertion were reproduced and diagnosed. A direct proc probe ruled out permissions; a portable shell comparison identified Bash skipping BASH_ENV with stripped SHLVL and socket stdin. Upstream #18230 now supplies the CI and ignored-stdin fixture fixes; this PR adopts them without duplicate CI changes. The strict Linux OOM assertion passed before this source-equivalent merge. - Retry follow-up: the three insertion/replay/failure timing cases and server typecheck pass. The existing database regression now asserts a single marker across identical result replay; the current-head CI integration gate passes. - Upstream CI alignment: all three real-Turbo 2.11.10 integration cases pass. The repository policy gate rejected committing a generated lockfile; the final PR contains no lockfile edit and uses the maintained inline stale-lockfile recovery. - Linked-lifecycle follow-up: 44 focused host/export tests pass (three optional SDK-dependent cases skip); SDK and server typechecks pass. The API contract test verifies an explicit root context, host-owned observer link and unsampled suppression. - SDK typecheck and focused worker/invocation tests: 30 passed, including clock changes, bounded legacy IPC, and manual-span creation-scope parentage. - Focused reviewed host/native/transaction-boundary checks: 44 passed. The 86-test worker-manager suite passes after updating its exact assertions for the host-owned timing budget found by CI. Earlier preparation/first-text checks also passed. - Affected server and Runner TypeScript checks pass after building local workspace declarations. - Merge follow-up: the profile/candidate startup subset passes 94 tests, the four Grok source-identity checks pass, and Runner TypeScript checking passes. A subsequent execution-factory merge passes 21 focused timing/trace tests and server TypeScript checking. - Pinned Rust build and the ACPX Rust subset pass. The hostile receipt projection test passes. Source profile compatibility checks: 127 passed, one existing skip. Grok identity checks: four passed. - Broad transport/ACPX suite: 410 passed after building all fake provider binaries. Earlier attempts failed on missing Runner/fake-provider outputs; building the required fixtures resolved those failures. - The first CI run correctly rejected stale source attestations. After integrating current master, Grok advances 13→14 and Copilot 84→85; master’s historical fixtures and Pi 35 stay byte-identical. Provider dependencies, models and permission defaults are unchanged. Identity generation/checks now pass. - The added real-Postgres result lock regression could not initialize its local embedded cluster. The insertion/replay/failure unit regressions pass. The current-head CI integration gate passes. - Comprehensive repository checks and image builds are delegated to CI, as requested for this instrumentation pass. No deployment, model run, collector-ingestion proof, or latency improvement is claimed. ## Risks The new SDK methods are optional. Legacy span calls and endpoint-off behavior remain supported. The host accepts only closed timing names and bounded attributes. Remote durations are anchored at local receipt; transport/scheduling uncertainty remains explicit. Trace trees have count/depth limits. The first rollout must inspect actual collector exports before treating the baseline as accepted. Fresh live qualification for the new source identities remains a rollout requirement. Revert these commits to remove the new measurements; no schema migration is required. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and local code/tool execution. The exact runtime model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the enhancement template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Standard Greptile and Apex are both 5/5, with no unresolved findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 17:45:56 -07:00
DottaandPaperclip	2aa3ea47e3	Reduce restored workspace overhead and trace cache publication (#18240) ... ## Thinking Path > - Paperclip runs agents in isolated workspaces. > - A native sandbox turn must restore its workspace before the provider starts. > - Restored Git metadata can cause expensive work in the provider's first snapshot. > - Cache replay also hashes the same archives twice, and workspace helpers open sessions that do not need output. > - This pull request removes that repeated work and refreshes the restored index before OpenCode starts. > - It also adds cache-publication timings so the next measurement can separate quota checks, lock waiting, and archive IO. ## Linked Issues or Issue Description **What existing behavior does this improve?** Native sandbox workspace preparation and copyback. **Current behavior** Cache replay validates shared archives, copies them, and validates them again. Exit-status-only helpers use persistent sessions. A restored checkout can make OpenCode's first snapshot recheck much of the repository. Cache publication has an unexplained interval after archive verification. **Proposed behavior** Validate the private archive copy that staging will consume. Use the existing one-shot command path for helpers that only need an exit status. Refresh the Git index after workspace transfer for a new native OpenCode sandbox. Keep the refresh inside the awaited inbound barrier and use the existing command timeout. Add nested publication spans. **Reason and benefit** Two matched local component pairs reduced preparation plus provider time from 8.563/8.553 seconds to 7.313/7.511 seconds, including the refresh cost. Two alternating remote copyback pairs reduced the median from 9.110 to 7.943 seconds. These are component results. They do not establish an end-to-end hosted improvement. Related: #18181 and the earlier closed #18195. This implementation uses the existing fully awaited transfer boundary. It does not use an optional preparation deadline. ## What Changed - Materialize cache hits into caller-owned staging and validate those exact bytes once. Remove incomplete copies on failure. - Select the existing one-shot path for exit-status-only run and removal helpers. Output-reading helpers keep their existing behavior. - Refresh restored Git stat metadata before OpenCode starts. Enable this only for newly acquired native OpenCode host-current workspaces. Skip recovery and retained sandboxes. Keep dirty files unstaged, accept Git's dirty-worktree status, and propagate real command errors. - Trace cache capacity checks, eviction, the body after lock acquisition, archive copy, sync, and hashing. Keep the operator endpoint gate and bounded phase names. ## Verification - On the rebased branch: 202 focused tests pass: 152 workspace seed/cache/runtime tests, 27 managed command tests, and 23 native workspace/tracing tests. Adapter TypeScript build and server typecheck pass. - Tests cover corrupted copies, cache changes after private materialization, dirty working trees, quoted paths, recovery, the awaited refresh barrier, real command failure, and company-lock contention. - A real-filesystem probe through the server phase mapper verified parentage and endpoint-off behavior. A controlled lock wait stayed outside the locked-publication span. No external export was used. - A full local Core/Daytona replay on the pre-rebase implementation saved its synthetic reply. It retained the pinned large public workspace, all 209 skill files, positive Git context, and 66 tool schemas. Both model calls stayed on sandbox loopback. Index refresh took 0.644 seconds and the first provider snapshot took 0.499 seconds. The run took 162.004 seconds, including large laptop uploads; it is not a hosted acceptance result or a matched end-to-end comparison. The failed installer attempt remains recorded separately. Normal cleanup removed all owned sandboxes. - The current head passed 53 CI checks, with two optional skips. Greptile is 5/5 on the exact head, with no open review threads. No deployment or hosted latency claim is included. ## Risks - Index refresh adds one awaited command. It uses the existing timeout. A real Git failure stops preparation. - Cache hits need temporary disk space for a private copy. Existing corruption fallback and per-run recovery archives remain. - One-shot command transport must preserve exit status and timeout behavior across providers. The agent's native session and output-reading helpers are unchanged. - Local component gains are small relative to the remaining hosted latency. Batch the fixes with the diagnostic spans; measure the actual deployed path before claiming success. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The session does not expose the exact model SKU or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 19:32:28 -05:00
DottaandPaperclip	6a67471644	Require publication for explicit JSON file outputs (#18228) ... Extend the native file evidence rule to explicit JSON creation and publication requests, with conservative input and destination handling and a real PostgreSQL completion regression. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-29 19:17:46 -05:00
DottaandPaperclip	1e984cc9ca	fix(runner): guide missing deliverables to the task workspace (#18226) ... Explain missing task-workspace deliverables and preserve safe remote file errors across output streams. Keep the existing file confinement and granted-location contracts. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-29 18:34:08 -05:00
ARTeMobAgentsandARTeMobAgents	4de82bc12e	test(server): give the description-edit admission poll the same retry budget as its siblings (#17487) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One subsystem is the chat channel control plane. It admits inbound GitHub mentions and then creates a chat conversation. > - Admission is durable. A transient database error defers the wakeup and retries it later. > - The retry delay has a floor of 1000 ms (`1000 * 2 ** attemptCount` in `server/src/services/chat-channels/inbound/wakeups.ts`). > - One integration test waits 1000 ms for the conversation. Any deferred retry therefore fails the test, although the product works. > - This pull request gives that one wait the same 10 s budget that the neighbouring tests already use. > - The benefit is a stable `chat (2/3)` CI shard. No production code changes. ## Linked Issues or Issue Description No public GitHub issue exists for this test defect. This is a bug report. **What happened?** The test "authorizes description edits as the editor, never inheriting the original author's access" failed in CI at `server/src/__tests__/chat-channels.integration.test.ts:5636`. The assertion was `expected +0 to be 1`. It is an `expect.poll` call without a timeout, so the budget is the 1000 ms default. The server log from the same job shows that the application did not drop the conversation. It shows a deferred admission instead: ``` WARN: accepted inbound chat wakeup will retry {"deliveryId":"..."} caused by: PostgresError: could not obtain lock on row in relation "chat_endpoints" ``` `acceptInboundWakeup` sets `retryAt = now + Math.min(30_000, 1000 * 2 ** Math.min(attemptCount, 5))` on such errors. The floor is 1000 ms. One deferred retry always exceeds the poll budget. **Expected behavior** The test waits long enough for one deferred retry. It then passes, because the conversation is created. **Steps to reproduce** 1. Check out `master`. 2. Run `cd server && ../node_modules/.bin/vitest run src/__tests__/chat-channels.integration.test.ts -t "authorizes description edits as the editor" --config vitest.config.ts`. 3. Add database contention on the `chat_endpoints` row during the second `send()`, so the first admission attempt fails with `could not obtain lock`. 4. The wakeup is deferred with a 1000 ms delay. The poll stops at 1000 ms and fails. **Paperclip version or commit** `master` at `991c47b`. The failure was observed on a pull request branch that does not change the chat code. **Deployment mode** Local trusted, embedded Postgres test runner. ## What Changed - Add `{ timeout: 10_000 }` to the `expect.poll` call at `server/src/__tests__/chat-channels.integration.test.ts:5636`. - This matches the three neighbouring tests that poll the same conversation count (lines 5577, 5658, 5721). ## Verification - Before the change, one deferred admission fails the assertion, because the default budget is 1000 ms and the retry floor is 1000 ms. - After the change, on `master` at `991c47b`: ``` cd server && ../node_modules/.bin/vitest run src/__tests__/chat-channels.integration.test.ts \ -t "authorizes description edits as the editor" --config vitest.config.ts → Test Files 1 passed (1); Tests 1 passed | 1421 skipped (1422) ``` - The complete file was not re-run locally. The targeted test covers the changed assertion. All other lines of the test stay the same. ## Risks - Low risk. The change touches test code only. - A longer budget can hide a slow admission, but only up to 10 s. The neighbouring tests already accept that budget for the same behaviour. - A real loss of the conversation still fails the test. ## Model Used - Provider: OpenCode. Model: `deepseek-v4.1-flash-free`. Context window: 200K. Modes: tool use, code execution, file access. No extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: ARTeMobAgents <11891309+ARTeMobAgents@users.noreply.github.com>	2026-10-29 18:33:15 -05:00
DottaandPaperclip	61bd4ac691	feat(agents): set up Antigravity with saved Google subscriptions (#18189) ... ## Thinking Path > - Paperclip manages agents and their work. > - Users choose providers and managed accounts while adding agents. > - Antigravity was missing from the ordinary additional-agent picker and its saved Google path did not complete native readiness. > - A user needs a successful Test, an enabled task action, and a stable account binding. > - This pull request connects the ordinary setup flow to the native candidate. > - It also preserves retired credential refreshes across database failures and server restarts. ## Linked Issues or Issue Description Built on merged #18188 and #18187. Refs: #12503, #14938 for related legacy CLI proposals. **Agent or provider** Google Antigravity ACP 1.4.3. **Why this adapter is useful** Users need to set up an Antigravity agent and run real work through their managed Google subscription. **How the agent is invoked** Native Paperclip Runner uses the pinned Antigravity ACP runtime. Exact operator admission remains required; the local candidate result does not enable general release. ## What Changed - Show Google Antigravity in the ordinary adapter grid and preserve the draft choice when setup resumes. - Reuse the same user’s saved personal default Google subscription and discover the exact low model through the native account. - Run a prompt-free native Test and wait for authoritative readiness before enabling task assignment. - Carry the managed account through native runs and recover retired refreshes through the existing encrypted, fsynced journal with exact generation checks. - Retain the latest rotated credential through verification, storage, and partial home-removal failures. Retry the exact account’s retained cleanup before another provider use. - Reconcile cleanup before taking the credential mutation lock, fence owners arriving during lock acquisition, and support nested lock holders without a join cycle. - Bound sign-in retirement outside database transactions and preserve claims for the existing cleanup owner; one pending sign-in does not block unrelated accounts. - Cover Gemini and Pi API-key recovery, keep API-key bulk manual-model editing usable, and keep adapter discovery available after candidate expiry. - Skip taskless native timer work before run or credential admission. Preserve current master monitor and deadline scheduling. ## Verification - The registered local saved-Google-subscription journey passed all 16 matchers in 196,520 ms using eval source `12aba` against actual serving source `4a37ee7c`. Both turns stayed in one conversation, produced exact downloaded files of 61 and 119 bytes after reload, and used distinct native primary processes. All observed owners retired, leases released, and credential fences settled. Independent readback confirmed Done work, no active owned runs, a connected saved account, and the retained agent paused for inspection. - The measured runtime is production-equivalent to pre-restack `65ef31dc` except its test-only transport fixture correction. Current-master head `ff7c9249` preserves all 99 product paths: 98 exact blobs and only eight inherited plugin-panel lines in `agents.ts`. Its native binary was rebuilt for inherited Codex-only diagnostic changes; the changed Rust behavior is outside Antigravity's ACPX execution path. Shared native/UI changes have focused regression coverage. The recorded live result is not represented as a measured replay of `ff7c9249`. - Full workspace build and typecheck passed on `ff7c9249`. All 376 focused tests in 14 files passed, covering Google credential/retirement controls and the inherited native-tool, redaction, route, and UI recovery changes. Token gates, maps, and diff checks passed. The required-close fixture was separately demonstrated red, then passed all 317 selected provider-wiring cases. - [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40959387972) is green: 55 successful checks, two skipped workflow checks, and an optional neutral review check. Required `ci / verify` and `ci / e2e` passed. Greptile and Paperclip Review both report 5/5 on exact `ff7c9249`, with zero unresolved review threads. - This qualifies the selected operator-authorized local saved-subscription candidate. The original exploratory journey remains historical evidence; the repeatable fixed-driver eval is in the next PR. ## Risks - Google refresh recovery must preserve the exact grant, account, and credential generation. An uncertain native owner or failed encrypted handoff retains the private home and blocks cleanup. - A successful local saved-subscription candidate does not qualify fresh OAuth, API keys, Cloud, Linux, other models, or general release. - Timer filtering changes only taskless native Runner admissions. Existing task-bound monitors and legacy adapters retain their owners. ## Model Used OpenAI Codex (GPT-6). The host did not expose the exact model revision or context limit. The model used tools for code edits, tests, and review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 16:51:50 -05:00
DottaandPaperclip	54cc522e86	feat(runner): add gated native Antigravity ACP execution (#18188) ... ## Thinking Path > - Paperclip manages agents and their work. > - Native Runner must bind each provider turn to the selected task, account, model, and instructions. > - Antigravity needs native ACP admission and physical process ownership across continuation. > - A saved conversation must not reuse an old process owner or silently switch its effective model. > - This pull request adds the native candidate behind exact operator admission. > - The default release gate remains closed while qualification is pending. ## Linked Issues or Issue Description Builds on merged #18187. Product account setup and the candidate producer are in dependent #18189. The repeatable browser journey is in #18190. Refs: #12503, #14938; those proposals use legacy CLI adapters. **Agent or provider** Google Antigravity ACP 1.4.3. **Why this adapter is useful** Users need to set up an Antigravity agent and run real work through their managed Google subscription. **How the agent is invoked** Native Paperclip Runner uses the pinned Antigravity ACP runtime. Exact operator admission is required while qualification remains pending. ## What Changed - Add native Antigravity ACP model and instruction admission, selected-account homes, and fenced refresh ownership. - Carry exact candidate authority from the control plane through Runner, Rust, the sidecar, and each current run. Clamp daemon lifetime to the absolute permission expiry, including replacement owners. - Preserve a rotated OAuth refresh token before later identity verification can fail, including failed or cancelled startup admission. Make private lease-close retries safe. - Bound native probe retirement and retry the exact original owner; do not delete credential homes before physical proof. - Preserve Gemini API-key recovery when shared Google subscription capability is added. - Resume the same tagged native conversation with fresh per-turn process ownership and physical native retirement. - Include native manifests in isolated Rust image stages, release cache keys, and dev rebuild detection. The workflow changes only extend existing cache input lists. - Keep current master provider behavior and append Grok 13, Pi 35, and Copilot 84 input identities. Preserve prior identities, including saved Copilot 82 and 83 runs. ## Verification - Runner TypeScript main checks, profile generators, Rust formatting, and offline workspace checks pass. The native backend integration target passes 28 tests. - The complete transport and cleanup group passes 387 tests without unhandled rejections. These checks use local scripted children and actual process and port ownership. - Saved-profile decoding, native-execution parsing, installation, and selected-credential controls pass 208 focused tests. All 109 pre-existing profile identity files remain byte-for-byte unchanged. Copilot, Grok, and shared TypeScript/Rust generation checks pass. - The final recovery fixture passes all seven cases. After the master rebase, the combined recovery and owned-Node-runtime group passes all 10 cases. Each expiry case sets one 45-second absolute grant before startup. It includes the deliberate six-second cold delay, four existing eight-second command budgets, and active-turn headroom. The test proves an active turn before expiry and then checks the daemon, owned process group, child processes, and all three native port fences. The 60-second test deadline does not renew provider authority. - The consolidated model-discovery and pending-release group passes 22 tests. Google credential compatibility and auth recovery pass 25 tests, including existing Gemini API-key recovery. The full product settings test group remains intact in #18189. - Earlier failures remain in their original CI runs and local records. They exposed the Copilot 82 decoder gap, an expired five-second fixture bootstrap ticket, and a fixture grant consumed before activation on loaded CI. Each repair has a separate result. Production ticket limits and admission behavior are unchanged by the final test commits. - An unrelated decision-expiry check failed in [the earlier CI run](https://github.com/paperclipai/paperclip/actions/runs/40941986698/job/123688668997). Its owning code is unchanged from the base. The unchanged suite passes 20 tests on isolated PostgreSQL. A controlled database rollback reproduces the earlier wake: clean startup schedules the 120-second decision expiry, while rollback reconciliation leaves a 30-second continuation-coalescing wake. The decision stays open, with no wake delivery or error. The fresh final-head server shard passes; the failed result is preserved. - Final native commit `ec6bc7e4d77f5ae0cc4946dbba6c662e95a87f5b` passed `ci / verify` and `ci / e2e`. Fresh Greptile and Paperclip reviews both score 5/5, with no open findings. Antigravity retains the recorded native `.5` closure and exact low-model profile. This PR does not claim global or cross-platform qualification. ## Risks - The new transport carries private managed credentials. Exact ownership, account generation, model, instructions, and native retirement are fail-closed. - Only local macOS ARM64 has a verified complete Antigravity runtime. Default Antigravity task execution remains disabled without current trusted operator admission. The Google server sign-in lane and production candidate producer ship in #18189; this native slice does not expose the subscription catalog choice. - Shared launch inputs change provider identities. New identities are append-only; existing recorded identities remain available. ## Model Used OpenAI Codex (GPT-6). The host did not expose the exact model revision or context limit. The model used tools for code edits, tests, and review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 16:36:12 -05:00
Dotta	6df9fc2221	test(server): consolidate duplicated startup-tracing unit suites (#17842) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/48/changes/paperclip/48 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server can export manual OpenTelemetry startup spans through `server/src/instrumentation.ts` (`resolveProtocol` plus the `getStartupTracer` / `getStartupTraceContext` accessors). > - Two suites covered that one module in every server Vitest run: `server/src/instrumentation.test.ts` (unit cases added with the manual-span seam in #10565) and `server/src/__tests__/instrumentation.test.ts` (the bootstrap machinery from #3735/#12249). > - `resolveProtocol` and the startup tracer accessors were registered twice per run with overlapping but unequal tables, so the same contracts were executed by two files while each held a few unique rows the other lacked. > - This pull request moves the unique assertions into the `__tests__` owner and retires the colocated file, so one suite exercises the full contract once. > - The benefit is one fewer suite registration per server run and one less duplicated unit surface to maintain, with no lost assertion. ## Linked Issues or Issue Description **What happened?** The server Vitest project ran two test files against `server/src/instrumentation.ts`. Both registered `resolveProtocol` and startup-tracer accessor cases. The colocated file held unique coverage (protocol input trimming, warn-once, the `getStartupTraceContext` accessor, the `setAttribute`/`setStatus` span surface) while the `__tests__` file held the bootstrap, peer-version, shutdown, and span-recording coverage. The split duplicated executions and made the contract's owner ambiguous. **Expected behavior** One suite (`server/src/__tests__/instrumentation.test.ts`) owns every `server/src/instrumentation.ts` unit contract. Every assertion from the retired file still runs, inside the owner. No production code changes. ## What Changed - Retired `server/src/instrumentation.test.ts` after migrating its unique assertions into `server/src/__tests__/instrumentation.test.ts`. - Added the `OTEL_EXPORTER_OTLP_PROTOCOL=" HTTP/PROTOBUF "` row, proving surrounding space is trimmed (previously only the letter-case row existed). - Strengthened the unknown-protocol case to assert the warning fires exactly once. - Extended the no-op tracer case to exercise `setAttribute` and `setStatus` before `end`. - Added the `getStartupTraceContext` accessor case (tracer surface plus the `contextWithSpan` helper), previously covered only by the retired file. Six production callers use this seam (heartbeat runtime execution, native run trace, duplex observability recorder, and others). ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/instrumentation.test.ts` — 21 passed, 2 skipped (SDK-gated span-recording cases skip without `@opentelemetry/*`, as before). The two new/extended cases pass inside the owner. - Deliberate failure controls, both restored byte-exact afterward and re-run green: - removing `.trim()` from `resolveProtocol` made exactly the migrated `" HTTP/PROTOBUF "` row fail; - making the no-op `contextWithSpan` throw made exactly the migrated accessor case fail. - The retired file's six remaining `resolveProtocol` rows and one `getStartupTracer` case were exact-value duplicates of owner rows (same inputs, same expected protocol/package pairs, same no-op tracer shape). - Full repository typecheck/test/build were not run locally (no Rust toolchain for the Runner build in this environment; server-wide `tsc --noEmit` exceeds the session window). CI is the authoritative gate for those; this diff touches only test code under `server/src`. ## Risks Low. Test-only change; no production behavior, routing, or CI configuration touched. Known stale mentions of the retired path are non-functional: `.devin/wiki.json` lists it as a non-exhaustive wiki anchor (its own instructions say to relocate removed anchors at generation time) and `scripts/general-server-shard-durations.json` keeps a historical duration key that no longer matches any selected file. ## Model Used - Provider: Z.ai via OpenRouter (agent: Paperclip Test Gardener) - Model ID: `openrouter/z-ai/glm-5.3` - Reasoning: standard chain-of-thought mode, tool use (shell, file edit, GitHub CLI) - Context window: 200k tokens (provider default for this model) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (none required: test-only change) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge	2026-10-29 14:08:23 -07:00
Ethereal49	649354d8c4	test: wait for Slack workers before checking settled queues (#16698) ... ## Thinking Path > - Paperclip is an open source app that manages AI agents for work. > - Slack delivery uses scheduled workers to process queued actions. > - The queue test checks that a settled queue does not poll the database. > - The next scheduled wake can be empty while a worker still runs. > - This pull request waits for the tracked workers to stop before that check. > - The test still rejects all queries during the 1200 ms quiet period. ## Linked Issues or Issue Description **What happened?** The test `wakes Slack session sync from publication commit and leaves settled action queues quiet` can report three unexpected `select` calls. The worker can clear its scheduled wake before its final deadline read and empty queue scan end. The test starts its query spy in that interval. The failure was observed in [this CI job](https://github.com/paperclipai/paperclip/actions/runs/38854774831/job/116823717452) on [the SSH summary-slot pull request](https://github.com/paperclipai/paperclip/pull/16535). This test race also reproduces on `master` without the SSH change. This pull request contains only the independent test fix. **Expected behavior** Start the quiet-period assertion after the scheduled work and the active workers are both idle. Continue to fail if any query occurs during the next 1200 ms. **Steps to reproduce** 1. On the base commit below, run the target test. The failure depends on timing. 2. To expose the interval, add this temporary control after the coordinator is created in that test, before work is registered: ```ts const readDeadline = f.service.nextChatActionAt.bind(f.service); vi.spyOn(f.service, "nextChatActionAt").mockImplementation(async (kind) => { if (kind === "slack_session_sync") await new Promise(resolve => setTimeout(resolve, 150)); return readDeadline(kind); }); ``` 3. Run `pnpm exec vitest run server/src/__tests__/chat-channels.integration.test.ts -t 'wakes Slack session sync from publication commit and leaves settled action queues quiet'`. The old wait fails with three `select` calls. 4. Apply this change and run the same control. The test passes with zero queries during the quiet period. 5. Remove the temporary control. It is not part of this pull request. **Paperclip version or commit** Base: `4415665d67b16ffefbe4519c4bdd521f37bb87b9` on `master`. **Deployment mode** Local test run from source. The fixture uses embedded PostgreSQL and the test Slack runtime. No live Slack workspace was used. ## What Changed - Require both `coordinator.nextWakeAt() === null` and `idleWorkSnapshot().active === 0` in the existing wait. - Keep the query spy, 1200 ms observation period, and zero-query assertion unchanged. - Reuse the existing import and wait pattern. No production code, dependencies, or configuration changed. ## Verification - Controlled reproduction on the base: one target test failed with three unexpected queries. With the same 150 ms control and this fix: one target test passed. The control was then removed. - `pnpm exec vitest run server/src/__tests__/chat-channels.integration.test.ts -t 'returns two answered Slack turns to Idle in the same thread|wakes Slack session sync from publication commit and leaves settled action queues quiet|sleeps behind an unresolved Slack tool send and wakes when its receipt settles'`: three tests passed; 1344 tests were skipped by the name filter. This run used the final source without the temporary control. - `pnpm exec vitest run server/src/__tests__/delivery-work-coordinator.test.ts server/src/__tests__/work-scheduler.test.ts`: 20 tests passed in two files. - `git diff --check`: passed. Environment: macOS ARM64, Node 26.12.0, pnpm 9.15.4, Vitest 5.0.3. Existing dependencies were reused in an isolated worktree. The shared/plugin SDK build passed. The CI chat shard was attempted with `pnpm exec node scripts/run-vitest-stable.mjs --mode general --group general-chat --shard-index 2 --shard-count 3`. The runner verified its collected filters, then reported `No test found ... in lines ...`. That invocation skipped all 1347 tests. Its sibling invocation was stopped after this runner failure. The shard did not pass. The first name-filtered retry stopped in fixture setup with `ENOSPC`; task-owned scratch directories were then removed before the final retry. Full workspace typecheck, test, and build were not run. The isolated environment does not have the full CLI/workspace dependencies or Cargo. Linux with Node 24, new pull request CI, and Greptile were not run. ## Risks Low risk. This changes a test wait only. It does not change delivery behavior or database data. The existing global idle counter is valid for this sequential fixture. If the suite later runs fixtures concurrently, the wait may also wait for unrelated tracked work. ## Model Used OpenAI Codex assisted with diagnosis, implementation, and validation. The session configuration was `gpt-6.1-sol` with `xhigh` reasoning. Read-only investigation and review used `gpt-6-luna` with `max` reasoning. Browser preparation used `gpt-6.1-sol` with `medium` reasoning. Tools included shell execution, Git, GitHub, Vitest, and browser automation. The context window size was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-10-29 15:58:27 -05:00
Nicky LeachandPaperclip	35ab90ad0f	Map runtime activity to authorized task plugin panels (#18131) ... ## Thinking Path > - Paperclip manages agents and tasks. > - Plugins contribute views to the task sidebar. > - A ready view can belong to an earlier task. > - Runner records activity under its authenticated run. > - This PR maps that activity to installed task-panel declarations. > - The sidebar checks readiness after current-run use and restores use after reconnect. ## Linked Issues or Issue Description Builds on merged contract #18129 and Runner publisher #18050. This PR targets master and is ready for review. Earlier combined verification is retained in [the verification archive](https://github.com/paperclipai/paperclip/pull/18050#issuecomment-6489369676). **Subsystem affected** Run-event reads, plugin task contributions and the task sidebar. **Problem or motivation** Readiness polling can show a view left ready by another task. Reload and reconnect must recover current-run use without losing concurrent live activity. **Proposed solution** Map recorded activity to installed task-panel declarations. Merge authorized history and live activity, then perform bounded readiness checks for the running agent. Preserve the existing read-only view. ## What Changed - Add optional `{ namespace, activityKey }` activation mappings to task-panel declarations. Legacy declarations remain valid but contribute no task panel. - Add `view=plugin-panels` to the existing run-events route. Preserve run-read and telemetry-read authorization, secret redaction and private-run denial. Map activity to installed contributions and return at most 128 panel identities. - Scope activation caches to current panel versions and activity declarations. Preserve unchanged panels during unrelated plugin updates and history reads. - Merge history and live hints without losing concurrent activations. Filter by company, task, run and agent. Restore history on mount and reconnect. - Replace periodic availability checks with up to three readiness attempts after activity. Fresh activity restarts settled transient failures. Pending checks, successful readiness and HTTP 4xx failures are left alone. - Reuse live run updates with the panel cache key. Recover only panel history after a pending subscription read. Mark expected not-ready query results so the global error reporter skips them; real failures remain reportable. - Preserve current-run dismissal and the read-only plugin view. Update manifest comments and producer/consumer documentation. ## Verification - Current head: `5e04d757c`, based on master `09c05c550`. Range comparison confirmed all six feature commits were unchanged by the rebase. The final follow-up keys readiness by the plugin installation ID and availability path. - `cd ui && node ../node_modules/vitest/vitest.mjs run src/hooks/useTaskPluginPanels.test.tsx` — all 45 cases passed. `cd ui && node ../node_modules/typescript/bin/tsc -b` passed. - The shared query-client and live-update recovery selection passed 96 cases on `49c544cd2`. Token gates passed before the final hook-only changes. - Earlier focused verification passed 57 manifest cases, 69 database/history cases, event-registry cases and route authorization cases, including private-run denial. - Regression coverage includes subscription/history races, task changes during recovery, unfocused run completion, bounded readiness retries, denied reads, expected not-ready responses, changed activity mappings, unchanged panel preservation, old cache scopes, and same-version route or installation changes. - [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40942001707) passed: 54 successful checks, two skips, no failures or pending checks. - [Standard and final review](https://github.com/paperclipai/paperclip/pull/18131#issuecomment-6489842168) both scored 5/5 on `5e04d757c`. The final review used the Apex tier, added no comments, and left no unresolved findings. The PR is conflict-free and ready for review. - Local checks stay targeted. GitHub CI provides the broad build, test, typecheck, browser and Runner coverage. No deployment or live-provider acceptance is claimed. <details> <summary>Retained failures and recovery evidence</summary> - `ac9329548` twice failed a browser reload assertion while conversation history was concealed. Master fixed its readiness wait in #17923. The approve journey then passed locally on `fea63a418` with installed Chrome (1 passed, 39 seconds). All eight browser shards passed in later CI. The browser test is outside this diff. - A canary checkout failed with Git HTTP 408; its unchanged-head retry passed. - `de07d09fa` failed the mobile Cases card click after a ten-second stability wait. The same read-only journey passed locally on that exact head (1 passed, 26 seconds). Its unchanged-head CI retry passed 22 cases, with one skip. The original artifact is retained; the cause of that click timeout is not established. - [CI on `49c544cd2`](https://github.com/paperclipai/paperclip/actions/runs/40903679075) passed 54 checks with two skips. CI also passed on `de07d09fa` after the recorded retry. - The `1e59ab441` merge checkout passed all browser and server suites. Runner shard 2 passed 2,252 cases and timed out waiting for a controller-recovery command. The test had landed on master in #18161 after the former branch base. Rebasing aligned the local checkout with that code; Runner shard 2 then passed on `fcf7380d3`. The original timeout cause remains unproven; no unrelated timeout was changed. - [CI on `fcf7380d3`](https://github.com/paperclipai/paperclip/actions/runs/40940235193) recorded runner shutdown signals in typecheck and a workspace shard. A chat recovery test observed a null wake deadline, and one Runner transport fixture failed during close. Those separate assertion failures remain unexplained. All of those jobs passed on the final readiness-key head. Those later passes establish recovery; they do not prove the cause of the earlier assertion failures. - Local browser setup first stopped at pnpm version-download verification, then a sandbox IPC restriction, then a missing bundled browser. Installed pnpm 9.15.4 and Chrome completed the journeys. No dependency files were changed. - Earlier optional-property TypeScript errors were corrected. Embedded PostgreSQL needed its install hook run from its package directory. Older CI also recorded fixture-cleanup lock timeouts, attention-fixture timeouts, and canceled server shards after runner shutdown. The parent fixed its FIFO snapshot-copy failure before merge. The linked verification archive retains earlier combined evidence. </details> ## Risks Activity is a hint and grants no access. Missing plugins, denied reads and unavailable views contribute no panel. Older runs without activity records show no task plugin tab. A used view stays available for the running task; reconnect reconciles readiness. Matching producer and consumer builds are required. No migration, dependency, image publication or deployment is included. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools and code execution. The exact serving snapshot is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 13:56:48 -07:00
DottaandPaperclip	9e09d8f51f	fix(runner): clarify native approval argument recovery (#18205) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - First-task onboarding requires a saved human proposal decision before child work can start. > - Native proposal tools reject interaction metadata placed outside their advertised schema. > - The existing rejection text focuses on question forms and does not explain the misplaced approval metadata. > - This pull request explains the field placement and keeps approval recovery in its original confirmation or checkbox kind. > - It also retains safe schema guidance in the run log so operators can identify the rejected field. ## Linked Issues or Issue Description Refs: #18025 for native first-task proposal recipes and #16889 for safe native schema guidance. Related open PR #18105 changes separate server API-client guidance. **What happened?** A native first-task confirmation copied `version` and `supersedeOnUserComment` to the tool root. Both belong inside `payload`. The runner correctly rejected the call before execution, but its feedback described question forms. Later calls kept the misplaced field and changed the approval into a question. No proposal card was created. The run log retained only a generic schema error. **Expected behavior** Keep strict validation. Explain the nested field locations. Keep a confirmation or checkbox approval in that kind while correcting its arguments. Save the normal pending card and wait for an actual human decision. Retain bounded schema guidance without storing rejected input. **Steps to reproduce** 1. Send an otherwise valid native confirmation with root-level `version: 1` and `supersedeOnUserComment: false`. 2. Observe the pre-dispatch `invalid_tool_call` rejection. 3. Move both fields into `payload` and keep `interactionKind: "confirmation"`. 4. Verify that the corrected call creates one pending confirmation, exact replay creates no duplicate, and child creation still needs saved human acceptance. **Paperclip version or commit** The observed argument shape and initial regressions came from public source base `c32e1ba6`. The repair is based on maintained master `4a6f6a65`. Public tests use synthetic prompts and identifiers. ## What Changed - Give rejected confirmations and checkboxes approval-specific guidance. It names `payload.version` and `payload.supersedeOnUserComment` without suggesting a question-form recipe. Other human-input errors retain complete question-form guidance and stable question/option ID instructions. - State the allowed native root fields in the first-task skill and put the metadata inside both executable proposal recipes. - Retain the existing safe guidance in `semantic_tool_denied` diagnostics. Keep the original rejection identity and `paperclipExecuted: false`. - Preserve only the exact bounded scalar diagnostic payload after verified event admission. The rest of the envelope and all API request/result bodies keep their existing redaction. - Add a real native Codex transport regression and strengthen advertised-schema, card replay, human-admission, persistence, and privacy controls. Update the execution contract and onboarding maps. ## Verification - Red: the native transport reproduced the malformed confirmation and showed that the old hint omitted the metadata correction. A second red confirmed that approval guidance still included the question-form recipe. The corrected transport case now rejects before execution, retains a safe hint, and dispatches one corrected confirmation that returns a pending card. - Red: both shipped recipe checks lacked explicit nested metadata. All 49 first-task admission tests now pass, including advertised-schema validation, idempotent card replay, actual human acceptance, agent-only rejection, and question-answer controls. - Updated native checks pass on the current repair: 95 native Codex transport tests with 2 existing ignored cases, all 32 provider-bridge tests, and the schema-feedback unit control. - API/case persistence regressions were red before the event-boundary correction. All 71 affected port/redaction tests pass in a clean run at frozen head `3be3a70f`, including the corrected envelope privacy regression. The focused 16-test persistence/privacy group and server typecheck also passed earlier. - Full local `pnpm build` and `pnpm -r typecheck` pass at exact head `3be3a70f` in a clean detached checkout with Node 24.22.0 and pnpm 9.15.4. Exact-head CI passes: [full PR CI](https://github.com/paperclipai/paperclip/actions/runs/40950360142) and [isolated native Runner readiness](https://github.com/paperclipai/paperclip/actions/runs/40950359835). - The full local unit attempt recorded a 15-second timeout in the unrelated Microsoft Teams 100,000-character existing-comment publication case. That failure remains recorded. The exact isolated rerun passed (1 test; 1,814 unselected). The aggregate timeout cause is unproven. That long local run straddled subsequent source edits and also observed the two port regressions during their red phase; it was stopped and is not final-head evidence. The clean affected-suite run above and current-head CI provide the final verification. No timeout or assertion was weakened. - Release-registry CI initially rejected the native test's compile-time schema include outside the release crate's hashed input closure. The guard was reproduced locally. The test now reads the current canonical schema at execution. All 10 crate-cache controls and the real native approval regression pass afterward; the cache guard and workflow keys remain unchanged. - Current-head Greptile review is 5/5 with no unresolved review threads. All required checks pass (56 success, 4 skipped). The advisory Paperclip Review remains neutral/incomplete after one normal re-review request; no completed assessment or score is claimed. - `cargo fmt --check`, map structural checks, and `git diff --check` pass. - Source and scripted-provider checks do not prove that a model will follow the guidance. The original live onboarding attempt remains failed; acceptance at a deployed revision is separate work. ## Risks - This changes repair guidance and shipped instructions. It does not make malformed arguments valid or supply a human decision. - Diagnostic retention adds no rejected arguments, values, or unknown input keys. The API receipt allowlist is unchanged. The validated-event exception restores only the safe scalar payload; optional debug and other envelope metadata still pass through redaction. - The existing 512-byte guidance bound, strict tool schema, company/run binding, idempotency checks, and proposal admission remain effective. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex, with tool use and code execution. The context-window size is not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 15:48:37 -05:00
LKL-ZREO	6c1ee40af0	fix(runner): keep assigned-tool pagination stable across revocations (#14258) ... ## Thinking Path > - Paperclip manages AI agents and the tools they can use. > - Native runs can receive assigned app tools through governed tool access. > - Large tool catalogs use a search tool with offset paging. > - The server refreshes grants before each page so revocations take effect at once. > - The old paging code applied the saved offset after this live list changed. > - A revoked earlier tool shifted the list and caused a later allowed tool to be skipped. > - This pull request keeps the offset on the fixed session catalog and filters grants during the scan. > - The benefit is complete tool discovery without weakening live revocation. ## Linked Issues or Issue Description **What happened?** `paperclip_search_assigned_tools` used an offset against the current authorized tool list. If an earlier tool was revoked between pages, the list became shorter. The next page then skipped a tool that was still authorized. **Expected behavior** Paging must return every matching tool that remains authorized. A live revocation must hide the revoked tool without changing the position of later tools. **Steps to reproduce** 1. Create an assigned tool catalog with `alpha`, `bravo`, `charlie`, and `delta`. 2. Request two tools from offset `0`. The result contains `alpha` and `bravo` with next offset `2`. 3. Revoke `alpha` before the next request. 4. Request two tools from offset `2`. 5. The old code returns only `delta` and skips the still-authorized `charlie` tool. **Paperclip version or commit** `0f14d261233c545aa6a8a38ec253c498a5130fff` **Agent adapter(s) involved** Not adapter-specific. This bug is in the native assigned-tool discovery path. Related work: Refs #14218. ## What Changed - Keep search offsets on the fixed tool catalog for the session. - Apply fresh grant checks while scanning that fixed sequence. - Add a regression test for a revocation between two pages. ## Verification - `node node_modules/vitest/vitest.mjs run server/src/services/native-runtime/assigned-mcp-tools.test.ts` - `node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit` - `git diff --check` ## Risks - Low risk. The API and numeric offset format do not change. - Revoked tools remain hidden immediately. - No database, migration, or public documentation change is required. ## Model Used - OpenAI GPT-5.6, Codex agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-10-29 15:36:46 -05:00
Devin FoleyandPaperclip	88a35cf4e7	fix(recovery): retain workspaces when restart recovery blocks (#17839) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native restart recovery decides whether an interrupted provider session can continue. > - An unusable checkpoint must stop provider replay. > - The failed run can still own a remote sandbox and files that need saving. > - The blocked restart path recorded failure but omitted the existing workspace preservation obligation. > - Saved user comments then waited for cleanup that had no file-recovery owner. > - This change records that obligation in the same transaction as the failure. ## Linked Issues or Issue Description **What happened?** A native run failed after its provider reported a terminal connection error. Restart recovery rejected its failed checkpoint and marked the run failed. Its prepared workspace and active sandbox lease had no preservation record. A later user continuation waited for the old environment to stop and could not start. **Expected behavior** When restart recovery blocks a run, preserve its exactly bound workspace through the existing stop-and-save service. Keep provider replay blocked. Stop proof and saved files must still precede any successor. **Steps to reproduce** 1. Prepare a native run with a remote workspace descriptor and its matching lease. 2. Persist a permanently failed provider checkpoint. 3. Run native restart recovery. 4. Observe a failed run with an active lease and no required workspace-recovery record. Refs #17801 and #17439. Those PRs add lease-settlement notifications and remote controller adoption. This change records the missing preservation obligation in the current restart failure producer. ## What Changed - Call the existing native workspace preservation service before the blocked restart transaction commits. - Keep the existing company, coordinator ownership, source lease and descriptor checks. - Add real PostgreSQL cases for exact binding, an unrelated source, a live owner, and rollback when preservation cannot commit. - Verify that preservation creates no replacement run or wake and retains the provider attempt count. - Document the transaction and recovery contract. ## Verification - Updated the existing branch with master at `09c05c55050fede448196637cbaa543a154da76e`. Both sides of the overlapping test additions remain. The production fix is still two added lines. - Current source verification: macOS arm64, Node 25.6.1, and disposable Homebrew PostgreSQL 16.14. No deployment or live task replay was performed for this branch update. - **113 tests pass** across `native-failed-workspace-recovery.test.ts` and `native-restart-recovery.test.ts`. Run them with `PAPERCLIP_TEST_DATABASE_URL` set to a disposable database ending in `_test`. Coverage includes exact source binding, live-owner protection, transaction rollback, no replacement run or wake, cancellation preservation, and orphan preservation. - Full local `pnpm -r typecheck` and `pnpm build` pass. Map checks, module-boundary checks, and the PR diff whitespace check pass. - Local verification limits remain explicit. The embedded PostgreSQL probe fails during `initdb`. The full `pnpm test:run` invocation hit that startup failure and was stopped. A broader restart integration invocation against disposable PostgreSQL did not finish while checkpoint-backlog work remained pending and was stopped. Neither invocation is a passing acceptance result. Hosted CI must qualify the updated head. - Prior-head evidence retained: the preservation regression first failed because its required fields were absent. The original verification passed 105 locally available cases and skipped nine real-runner process cases because the local binaries were unavailable. Previous-head hosted CI passed full typecheck/build, test shards, Runner checks, and browser shards. Those results do not qualify the updated head. - Prior live-repair evidence retained: an audited call to the existing preservation service registered the missing obligation for an exactly bound source. Normal workers confirmed termination, released the source lease, and preserved the prepared files before an assigned continuation started. This was an earlier repair, not acceptance of the current branch update. - Updated-head hosted CI passes on `a22feeb54daa2e9325a55d0ccd86515e64cbd47d`, including typecheck/build, all test shards, Runner checks, and browser shards. Apex is 5/5 on the same commit with no actionable findings or unresolved review threads. GitHub reports a clean merge state. The patch is not deployed. The original provider connection failure remains unexplained by this change. ## Risks - Preservation adds database work only when a restart classification becomes terminal. It adds no recurring scan or timer. - A failed preservation write now rolls back failure settlement. The existing owner can retry instead of leaving cleanup without an owner. - A live coordinator or mismatched source remains protected. This change neither destroys the sandbox nor starts provider work in the recovery transaction. - Existing cleanup must still prove termination and preserve files. A terminal run alone does not authorize continuation. ## Model Used OpenAI `gpt-6.1-sol` through Codex in T3 Code, with xhigh reasoning effort, repository inspection, tool use, and code execution. The runtime does not report the context-window size. Earlier implementation used the OpenAI Codex model recorded in the PR history. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 13:06:28 -07:00
Devin FoleyandPaperclip	5d79d88d68	fix(server): let settled native finalizations permit idle sleep (#18184) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs retain finalization records for recovery and audit history. > - Idle sleep checks durable work before it stops an instance. > - The current check treats every retained finalization as pending work. > - A committed record can remain after its run and status effects have finished. > - This pull request recognizes complete records without deleting them. > - Instances can sleep while unfinished execution and recovery work remain protected. ## Linked Issues or Issue Description **What happened?** A retained `native_run_finalizations` row prevents idle sleep even after the run has finished and its status decision has been applied. The current gate checks whether any row exists. **Expected behavior** Complete finalization history should remain available without counting as pending work. Active leases, retries, accounting, recovery, and incomplete status effects must still prevent sleep. **Steps to reproduce** 1. Finish a native run and retain its committed finalization. 2. Confirm that its accepted result, assessment, applied decision, and delivered effects agree. 3. Request an idle report under a valid quiet admission hold. 4. Observe that the retained row reports background work before this change. **Paperclip version or commit** Reproduced on master at `cd0c86e5e1d427c8fa8aa8c3a9499391ee9b425b` with a local PostgreSQL fixture. **Deployment mode** Server with PostgreSQL. No provider calls are needed to reproduce the gate failure. Related PR and issue searches found no duplicate for this check. ## What Changed - Replace the blanket finalization-row check with a completion proof bound to the same company, issue, run, accepted result, assessment, and applied decision. - Require the owning run's successful workspace-finalization receipt; retain unresolved leases, retries, accounting, recovery, and status effects. - Share final-reply candidate discovery with the presentation recovery worker, so a completed run cannot hide a required reply. The pure query stays with native presentation; idle inspection uses it in the existing read-only transaction and five-second statement limit. - Register open warm sessions and in-flight close/checkpoint promises with local idle inspection. Historical controller/provider process metadata remains preserved. - Add PostgreSQL and local-warm regressions and document the retained-history rule. ## Verification - PostgreSQL sleep/finalization and chat-monitor continuation suites: 300 tests passed, including 56 finalization cases. - Local-warm regression: passed; an open session and its pending close/checkpoint both block sleep until retirement completes. - Server typecheck and server build: passed after review fixes. - Full workspace typecheck and build: passed before the review additions; hosted checks cover the updated revision. - An earlier local full invocation encountered two chat recovery failures that passed in isolation, and two existing Git workspace fixture failures caused by Xcode system Git initializing `main`. Those two pass with `init.defaultBranch=master` scoped to the test subprocess. That superseded full invocation was stopped for review revisions; no full local pass is claimed. No machine configuration or unrelated source was changed. - Fresh Apex review: 5/5 on `6846203fe2383bae1ede888fbaaedf83e9946e73`, with all findings addressed. - Hosted CI: fully green on `6846203fe2383bae1ede888fbaaedf83e9946e73`, including the final verify gate. Twelve jobs were interrupted by a simultaneous runner shutdown; rerunning the failed jobs completed successfully. - No hosted deployment of this change is claimed. ## Risks This changes an automatic-stop gate. An incomplete proof could stop an instance with pending work. The SQL fails closed and retains the existing admission, local work, plugin, schedule, backup, and provider-allocation gates. PostgreSQL cases cover unresolved leases, retry/recovery intent, accounting, projections, rejected results, unapplied decisions, and pending effects. No migration or history deletion is required. Reverting the predicate restores the conservative gate. ## Model Used OpenAI GPT-6 through Codex, with repository tools and local PostgreSQL verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 11:11:22 -07:00
Nicky Leach	e2cfdc7e0c	feat(plugins): expose GitHub credential broker to task providers (#18178)	2026-10-29 10:52:44 -07:00
DottaandPaperclip	9f3fc2094f	Fix shared workspace seed capacity and trace deferred copyback (#18181) ... Apply the existing company quota to company-shared workspace seed buckets. Preserve individual seed limits and recheck size after verification. Trace each deferred native workspace finalization check and its resumed copyback under the same hashed run ID. Keep existing admission, ownership and retry behavior. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-29 12:44:34 -05:00
DottaandPaperclip	f1911a4c53	perf(skills): move repository packages out of Postgres (#18082) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/205/changes/paperclip/205 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents receive selected company skills before each turn. > - Imported skills can need files from the whole source repository. > - The current path stores those files in Postgres and reads large JSON payloads during preparation. > - This pull request stores verified packages in the configured file or object store and keeps preparation queries small. > - It also converts old rows automatically and restores missing files from the exact saved commit. > - The result reduces database traffic while preserving skill versions and repository dependencies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Company skill import, version storage, and preparation before agent turns. **Current behavior** Repository snapshot bodies live in JSONB. Skill preparation can read those bodies again even when the files already exist on disk. Historical packages and native assets also need a bounded cleanup path. **Proposed behavior** Store package archives and manifests through the existing storage service. Keep version identities, receipts, hashes, and recovery metadata in Postgres. Convert legacy rows after the stored package passes read-back verification. A missing package can use legacy bytes or its exact Git commit under current repository authorization. No separate data-migration command is required. **Reason and benefit** The warm path sends no repository bodies through SQL and reads no object bytes. Cleanup runs through the existing deadline coordinator. Current versions, active runs, explicit pins, and authored content retain their protection. **Breaking changes** The normal application schema upgrade is required. Single-version and skill-file responses keep their existing shape. Version lists, current-version summaries, and test-run version summaries now contain file names and kinds without file bodies. Diff and restore fetch selected versions through the existing single-version endpoint. Older server builds must not write after conversion unless they support the new package format. The storage override and rollback limits are documented in `doc/skill-package-storage.md`. **Additional context** Related work: #15731 preserves repository dependencies; #17923 changes version-cache scheduling; #18080 changes delivery to remote Codex homes. This change owns package persistence, acquisition, and expiry. It does not replace those delivery or scheduler changes. ## What Changed - Add company-scoped archive receipts, exact-commit recovery recipes, migration deadlines, and a durable object-deletion queue. - Add bounded archive upload, read-back verification, extraction, concurrent request coalescing, and verified local caches. - Convert old snapshots and versions automatically. Keep legacy bytes when verification fails. - Recover missing files through current source authorization and the original commit. Validate repository identity, file hashes, modes, and the aggregate hash. - Expire unused recoverable Git packages and native assets after 30 days. Preserve active pins and the durable copy of authored or unproven content. - Keep warm preparation queries metadata-only. Bound file digest caches and avoid repeated body hashing when file identities are unchanged. - Extend the existing storage contract with cancellation. Register package work with the existing coordinator. Optional conversion retries retain the original bytes and may wait for ingress without blocking sleep. - Add regression tests, operating guidance, a deep trace, benchmark receipts, and an optional maintenance command. ## Verification ### Final master integration and review repair — October 29, 2026 Current head: `a6065424249e763dfeac37161f1b52a9f25bd9b7`. Commit `f8bbdd3dbd` resolves the conflicts with master `76a796e964`; the final two commits repair the Apex deletion findings below. - Keep both package-maintenance and version-retention notifications in their write transactions. - Keep the shared run-pin predicate and master's controller-lease deadline metadata. Package expiry reads the updated protected-version result. - Preserve every master migration entry. Generate migration **0415 (`0415_zippy_zemo.sql`)** after master's 0414. Its skill SQL is byte-for-byte unchanged from the previously tested idempotent migration. Retain only the five newest schema snapshots. Migration numbering and safety checks pass. - Repair failed file/folder deletion: preserve a complete temporary subtree until transaction commit, restore on upload or SQL failure, and restore callback failures before releasing the mutation lock. Both PostgreSQL regressions reproduced missing files before the fix and now pass. They cover binary data, executable permissions, empty folders, unlisted files, unchanged database/version rows, successful retry after upload-claim expiry, and historical reads. - Reconcile lost commit replies under the same advisory and row locks. Check the saved version and source identity before restoring. Keep committed deletions intact and preserve backups when the database outcome is unknown or a newer edit exists. Four real-PostgreSQL outcome tests pass; the prior code failed three of them. - Full workspace typecheck and production build pass on the final commit. All **204** focused PostgreSQL package/cache/retention/authoring and filesystem-boundary tests pass, with none skipped. - Apex review **18 passes 5/5**, with zero new comments on the exact final head. All **34** review conversations are resolved. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/40914116508) completes successfully on attempt 2. The head has **55 successful checks**, two neutral reports, two skipped checks, and a successful Snyk status. No checks are pending or failing. All eight browser shards pass. - Runner Vitest 2/2 timed out once in the unchanged ACPX controller-replacement fixture. The same job passed on the preceding commit. One unchanged CI retry passes all five ACPX recovery cases, **2,253** Runner package cases and **126** native server integration cases; their standard one and five skips remain. The initial timeout's root cause is not established. Local diagnosis also hit the fixture's eight-second session.open timeout with an authenticated Runner; no local Runner pass is claimed. - Both full local `pnpm test:run` attempts were stopped when Apex identified deletion defects and source repair became necessary; neither is claimed as a passing full local run. The final **204** focused tests, full workspace typecheck, and production build pass. Full-suite acceptance uses the final-head CI jobs. The superseded merge-head CI included runners terminated by shutdown signals; each source repair started a new full run. - Auto-merge is enabled for squash merge. The final head is conflict-free and all checks pass. GitHub's active master ruleset still requires a code-owner approval for the dependency files. The existing review requests are with `devinfoley`, `nickyleach`, and `forgottendev`; no policy bypass was used. - The earlier hosted stress and full native receipts below remain tied to `a2e597d800`; they are not relabeled as a deployment of this final integration. The earlier local PostgreSQL startup failure remains recorded below; PostgreSQL ran successfully for this integration verification. ### Full native skill execution — October 29, 2026 The full native-agent acceptance gap is closed on candidate `a2e597d800eb989d565780e7aa947d5368676444`. Two qualifying native turns used OpenAI API / `gpt-5.4-mini` in real Daytona sandboxes, on the same disposable hosted TEST tenant as the stress test. A follow-up on the same task consumed a newly saved skill version. The task text asked for the assigned skill workflow. It did not contain the proof nonce or expected receipt. The assigned authored skill contained a Python verifier and a random package-local nonce. The agent read the actual installed skill files and executed that verifier against the imported, automatically converted 1,360-file repository package. It checked the repository file count and reference/binary bytes, and directly executed the repository's executable probe. Independent verification matched the emitted JSON to the nonce and expected byte hashes. The agent persisted the same receipt as its task response; native finalization committed, workspace finalization succeeded, and task state became Done. | Qualifying turn | End-to-end run time | Recorded skills.prepare | Repository files | Binary hex | Shell receipt SHA-256 | |---|---:|---:|---:|---|---| | Initial task | 47.881 s | 1822 ms | 1360 | `00ff07800a` | `a1deb30dc095f2306c7aa9cc5ec396da12c68961cf5b809f4d7c20f7fc68493b` | | Corrected follow-up, new nonce/version | 37.974 s | 377 ms | 1360 | `00ff07800a` | `dbcdc33fa1376b4edc136feae1cdf770560a73292467e93619efdda20e4282a7` | Both successful shell receipts reported `probe: STRESS18082` and `assetSha256: e8ea7a2f15f7bb3b332a46adbef848465884c8996595aa234748befd544d1f3a`. Their proof nonces, selected version IDs and receipt hashes differed, and both ran on the same task. These are observed timings, not latency SLOs; they include real provider work. The service stress-test timings remain a separate measurement boundary. Earlier failures remain in the evidence: - The original five-minute Daytona allocation timeout never reached model execution. After normal owned-stack restore, its durable cleanup record was `expired` / `success`. The cause of that original allocation timeout remains unknown; a later success establishes recovery only. - Initial continuation setup used unsupported generic agent-update fields and returned HTTP 409 before dispatch. Setup was corrected to use the supported skill-assignment and versioned instruction commands. - The intervening completed native turn read repository bytes and ran the probe, but omitted the verifier JSON under the older test agent's overly-specific instruction file. Its strict receipt validator failed; it is not counted as a qualifying turn. Correcting the test agent's instruction file and sending an explicit task-scoped wake produced the qualifying follow-up above. - The pinned application's immutable rate card omits `gpt-5.4-mini`, so completed usage triggered the normal block-on-unpriced budget policy. Supported append-only accounting corrections estimated the three completed runs using the [official OpenAI model rate card](https://developers.openai.com/api/docs/models/gpt-5.4-mini): $0.75 input, $0.075 cached input, $4.50 output per million tokens, assuming standard processing. Estimates totaled 12.482655 cents. Original usage and missing-price records are preserved; the $1 hard stop and unpriced-usage guard stayed active. - Model shell attempts included a missing `xxd` command and incorrect guessed file paths. Those failed calls and subsequent successful actual verifier calls remain visible in complete event streams. No zero-tool-error claim is made. No product source changed during this continuation. The previously measured contention, cold-start and memory limits remain. The later merge resolution is covered by the separate integration checks above; these hosted receipts qualify their recorded candidate. ### Hosted stress test — October 29, 2026 **Hosted acceptance:** full native-agent execution and a changed-skill follow-up pass on the recorded hosted candidate. The final integration section above records the subsequent conflict resolution and its separate checks. Tested baseline `929e840595c8933ca485ac7a2e99413e1e11ca64` and this exact candidate sequentially on one disposable TEST tenant, with external Neon PostgreSQL and actual S3. The fixture contained **1,360 files / 22.4 MB / ten skills sharing one repository**. Measurements cover deployed skill-preparation and native-context services in an adjacent diagnostic process, not a complete agent turn. The container had **24 vCPU / 24 GB**. | Warm workload | Median before → after | p95 before → after | DB returned value bytes/preparation before → after | |---|---:|---:|---:| | One skill, serial | 1,060 → 125 ms | 1,303 → 155 ms | 22,610,389 → 1,771 | | One skill, four concurrent | 4,415 → 297 ms | 5,143 → 328 ms | 22,610,389 → 1,771 | | Ten skills, serial | 5,723 → 535 ms | 6,057 → 600 ms | 224,739,030 → 10,671 | | Ten skills, four concurrent | 24,410 → 1,986 ms | 24,833 → 2,221 ms | 224,739,030 → 10,671 | - Candidate warm preparation performed **zero S3 GETs**. Ordinary-pool queries fell from 16 to 9 for one skill and 61 to 9 for ten, excluding dedicated lock connections. DB numbers measure returned values before Drizzle decoding, not wire bytes. Baseline/candidate sample counts were 30/100, 32/100, 5/50 and 8/48 respectively; the small baseline ten-skill samples are comparative evidence, not SLO estimates. - **Application-UID soak:** 800 ten-skill preparations over 10.3 minutes, four workers paced by 2.5 seconds; **zero failures**, median **559 ms**, p95 **775 ms**, p99 **910 ms**, zero warm S3 reads. All 726 accompanying health/company-list requests returned HTTP 200. Their end-to-end p95 was 569/502 ms, including network latency. The probe overlaps most of the soak. Soak DB-response capture was limited to its first 60 seconds. - **Important remaining costs:** an unpaced 16-way ten-skill burst had p95 **8.32 seconds** because company-level preparation serializes. Cold preparation took approximately **1.1–2.5 seconds** for one skill and **7.4–10.0 seconds** for ten. Ten-skill diagnostic RSS remained about **1.17 GiB** (baseline 1.18 GiB), and the soak peaked at **1.35 GiB**. A concurrent whole-container sample was 3.45 GB. This does not establish suitability for a small-memory host or eliminate per-entrypoint native asset duplication. - **Upgrade without a manual data migration passed:** legacy snapshot JSON shrank from 22,465,691 bytes to `[]`, with a stored package and zero pending conversions. - **Recovery passed:** sixteen concurrent cold requests coalesced into two S3 GETs; full bytes/binary/executable modes matched; same-size corruption repaired; an injected storage failure preserved warm access and failed cold access safely. A disabled public-Git source denied recovery, and after the actual five-minute backoff its exact SHA reconstructed all 229 files. Cross-company descriptor access was denied. Private-Git credential revocation was not tested. - **Retention passed:** reclaimed 27,754,496 allocated cache bytes, deleted an unreferenced package row and both objects, preserved aged authored history, and expired/reacquired reconstructible Git content. A version pinned by the failed native run was correctly retained. Only owned fixture timestamps were aged. - **Browser UI passed:** authored skill creation, saved edit surviving reload, version history, and restoration of earlier content. - **Initial native attempt (superseded by the qualifying turns above):** one real run failed at Daytona allocation after five minutes (`setup_failed`), before any model execution. Provider cleanup also reported failure. That attempt does not count as a passing native turn. Its receipt and all evidence remain preserved; its cleanup later recorded success after normal restoration. - Initial A/B commands ran as SSH root. Diagnostic-owned lock/cache permissions caused one setup HTTP 500 and the first UID-1000 soak to fail; exact fixture paths were repaired. The final soak, retention and recovery ran as application UID 1000. A fresh UID-1000 one-skill batch also passed (30 samples, median 116 ms, p95 321 ms, 1,771 DB value bytes/preparation). Earlier harness-export, SSH, npm provenance, backoff and UI-wait failures remain recorded alongside their successful retries. - Private evidence includes raw timings, assertions, HTTP probes, screenshots and failure/cleanup receipts. Initial failed-campaign cleanup suspended and retained the owned TEST tenant and released the shared QA account lease at **15:22:41 UTC**. The tenant was subsequently resumed under its existing ownership for the native execution qualification above. Final cleanup completed at **15:53:09 UTC**: the account lease is released and the tenant is suspended/retained under the original October 30, 12:24:52 UTC retention deadline. Its separate stack lease remains held; no active or pending-cleanup environment lease remained. The original retention window is deliberately preserved even after successful continuation. - Earlier hosted-tested head: `a2e597d800eb989d565780e7aa947d5368676444`, integrated with master `929e840595`. The earlier merge conflicts were resolved at this head. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40884087629) passed; all checks are complete with 55 successes, two neutral reports, two skips, and no failures. [Apex review 15](https://github.com/paperclipai/paperclip/pull/18082#issuecomment-6480257258) passed 5/5 with zero findings on this exact head. All review conversations are resolved. - Ten real-filesystem boundary cases and server typecheck pass locally for the security fix. Coverage includes cache roots, descendants, individual files, symlink aliases, recursive ancestors, a future concurrent cache fill, and allowed sibling directories. [Current-head private-project integration tests](https://github.com/paperclipai/paperclip/actions/runs/40884087629/job/123483306405) passed all 29 cases, including authorized and unauthorized imports of a cached private package. Full typecheck, build, server suites, all eight browser shards, and Runner checks passed in current-head CI. Local database tests could not run because PostgreSQL could not initialize; this limitation is recorded below. - The earlier generated migration preserved the newest five snapshots. The final integration renumbers its unchanged SQL to 0415. Migration safety/numbering checks pass. Its SQL is byte-for-byte identical to the previously verified idempotent skill migration; repeated application on this latest schema was blocked by the same local database startup failure. - The broader verification before the final migration-only rebase passed 1,025 package, lifecycle, remote-launch, native-session, and asset-retention tests, 52 focused UI tests, full workspace typecheck/build, token gates, and map checks. The linked complete CI run passed before the latest focused fixes. - A disposable MinIO instance passed multipart upload, verified read-back, and cold restoration of an 18 MB archive. - Twenty warm samples with a synthetic 1,349-file, 22.8 MB repository retained five metadata queries and zero object reads. Median preparation was 256 ms for one selected skill and 691 ms for ten, with 5 ms emulated network delay each way. These are local measurements, not production latency guarantees. - No production instance was changed. Earlier verification failures and their disposition are retained below. <details> <summary>Earlier failures, review findings, fixes, and rerun evidence</summary> - The first CI pass found a stale inventory-span expectation and a missing repository-read span. Both are fixed; all three observability tests pass. One browser test copied page text into its YAML parser after reload; its first unchanged-head retry passed, but the race recurred after rebasing. The test now waits for the renamed URL and completed reload before copying the editor, retaining its original assertions. The earlier full local suite did not complete cleanly: it reported setup timeouts and assertions, while isolated relevant reruns passed. The complete current-head CI run closes that broad verification gap. No production instance was changed. - The first Apex review scored 2/5 and identified six issues. The fixes protect authored history even when it shares an in-flight Git upload, recheck upload expiry during orphan deletion, bound failed-upload cleanup, retain per-version kind/encoding metadata, add sanitized failure diagnostics, and correct the migration guide. The final focused run passes all 140 tests; the preceding eight-file run passed all 149 tests, including local storage and S3 diagnostics. Server typecheck and migration safety/numbering checks pass after the fixes. The second Apex review scored 1/5 and found unbounded maintenance storage calls and mismatched package/Git batch limits. Those are fixed with a 30-second cancellable cleanup deadline, shared 100 MiB/10,000-file scanner and storage limits, and bounded Git read batches. The proposed manual lockfile edit was rejected with evidence of the enforced CI-owned lockfile policy. All 19 storage/recovery unit tests and server typechecking pass. The new database-backed cleanup/conversion tests were locally skipped because the host exhausted its PostgreSQL semaphore capacity (confirmed directly by initdb). That gap is closed by current-head CI: [package-store tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109300889) pass all 18 cases and [package-worker tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109301157) pass all 7, with no skips in either suite. The corrected YAML browser shard passes. A separate planning-fixture test failed its cleanup DELETE with an activity-log foreign key violation after its behavior assertions passed; its one unchanged-head retry passed. The third Apex review scored 1/5 and identified a one-slot pool deadlock, projected path limits, cleanup holding the preparation lock during archive reads, and a publication/expiry race. These are fixed with transaction-local reads, prefix-aware bounds, isolated verification, and shared package row locks with a publication grace. The separate Paperclip review also found stale historical inspection, eager history downloads, and conversion retries blocking sleep. Those are fixed. Local PostgreSQL capacity recovered: the new database regressions ran, including the one-slot fork, paused storage during a warm turn, and a locked publication during expiry. The expanded run passed 144/145 tests; its remaining assertion still expected file bodies from the version list and was updated to fetch the selected version. The corrected company-skill suite passes all 81 tests, and the final package/projection run passes all 27. All 52 focused UI tests pass after updating their query-provider fixtures; the first UI run exposed a fixture remount, and the follow-up exposed an editing syntax error, both corrected. Full workspace typecheck and build, token gates, and map checks pass. The sandbox initially blocked the build tool’s local IPC socket; the authorized build rerun passed. Master then claimed migration 0401. The branch is rebased on f8226b0515, the unchanged SQL is regenerated as 0402, and repeated application passed twice on the combined schema. Post-rebase workspace typecheck passes. All 1,025 package, lifecycle, remote-launch, native-session, and asset-retention tests pass on the combined head. All CI checks passed for head 9d80a3b52d. The fourth Apex review scored 3/5 and found a save/edit lock-order cycle, a retry retaining an uncommitted source ID in its recovery recipe, and a summary read racing conversion. Commit b7d9237c6a fixes all three. The new concurrency and retry tests pass within all 61 package/source tests, and server typecheck passes. Master then claimed migration 0402. The branch is now rebased on 65fa1cd617 and the unchanged SQL is regenerated as 0403. All 68 package, source, and worker regressions, server typecheck, migration safety/numbering, and repeated migration application pass on current head 62553671a0. Subsequent verification and review outcomes are recorded below. - The fifth Apex review scored 4/5 and found cached version reads unnecessarily depending on object storage and conversion reads missing the coordinator cancellation signal. Commit 1eb90d80be fixes both: cached reads verify bytes without the preparation lock; manifest and archive reads carry cancellation through conversion. The first focused run passed 70/71 tests, including all three new cases, but the existing diagnostic test observed no warning. Its fixture now sets an explicitly expired migration deadline; the final worker suite passes all nine cases. Server typecheck passes. - The sixth Apex review scored 4/5 with one finding: concurrent cached inventories were unbounded. Commit 4b7589e14c gives cached inventories and previews a two-reader/32-waiter limit independent of cold storage. The first expanded pressure test observed a third reader before the preview path had the same bound; both paths now share the local limit. All 73 focused cases pass, followed by all 29 package cases after the final overload adjustment; server typecheck passes. The separate Paperclip review found no must-fix or should-fix issues. Its optional suggestion to persist durability verification was considered: cleanup intentionally verifies current external bytes before discarding the last local authored copy, and the missing-object regression requires that check. Shared-package eviction happens on cold fills with at most two attempts and a ten-second budget. - The seventh Apex review reached 5/5 with one remaining P2: timeout-bearing manifest reads bypassed request coalescing. The security review also found repeated archive acquisition after a cold administrative read. Commit 79a80a39d2 shares manifest loads with per-caller cancellation and cancels the underlying work when the last caller leaves; coalesces identical cold version/file requests; and publishes verified restored packages into the shared cache. Publication uses a short nonblocking lock, while eviction verification runs outside it. The first expanded run passed 73/74 tests and exposed a missing originating DB for cache publication inside read-only inspection; transaction readers now pass that owner explicitly, keeping metadata on the transaction. The corrected run passed 75/75. A final eviction test initially expected the shared package to remain after only its separate runtime projection was reused; it now explicitly accesses the shared package during verification. All 32 final package tests pass, including independent caller cancellation, one acquisition across concurrent/repeated reads, and warm preparation during stalled eviction verification. Server typecheck passes. - The eighth Apex review scored 4/5 with one finding: decoded cold inventories outlived the storage semaphore while waiting for cleanup. Commit 9d7921d1ec adds a separate two-response/32-waiter lifetime bound held through cleanup, preserving the independent warm-read queue and forwarding cancellation through eviction verification. The new regression pauses cleanup after two responses decode, verifies a third archive stays queued, and completes warm preparation before releasing cleanup. All 77 package/source/worker cases and server typecheck pass. On the preceding head, serialized server shard 6/9 was interrupted with “The operation was canceled” after 57 seconds of tests; the log contains no failing assertion. Fresh latest-head CI reruns that shard along with the other gates. - The ninth Apex review scored 4/5 with one P1: cold responses could own both permits while waiting on a one-slot ordinary database pool held by a queued transaction reader. Commit a94f0e409f reads caller-scoped metadata before admission and isolates all acquisition/cleanup SQL on a private connection. The new real-PostgreSQL regression reproduces the deadlock against the previous head and passes with the fix; all 78 focused cases and server typecheck pass. The previous CI run completed with 54 successful checks, including both formerly interrupted/stuck serialized shards. - The tenth Apex review scored 4/5 with one P1: preloading legacy bodies before admission could retain many full repositories in queued requests. Commit 276e353a37 removes that preload and also bounds legacy manifest derivation/fallback, using private connections inside the I/O slot. All 80 focused tests pass; the final synchronized package suite passes all 36 tests and server typecheck passes. - Apex review 11 returned 5/5 with zero comments on 276e353a37. While it ran, master 8bc051245f added migration 0403, creating a metadata conflict. Merge commit e1783764d9 integrates master and generates 0404_clumsy_leo.sql; its skill SQL is byte-for-byte identical to the previously verified migration. The combined schema passes migration safety/numbering, repeated SQL application twice, all 80 focused PostgreSQL tests, full workspace typecheck, and full build. The preceding a94f0e409f CI run was superseded and cancelled while its final runner job was active; no old-head success is claimed for that run. - Final acceptance: Apex review 12 returned 5/5 with zero comments on e1783764d9. All 28 review threads are resolved. The final head has 55 successful checks, two neutral reports, two skipped checks, and no pending or failing checks; the PR workflow completed successfully. Superseded run 40789121902 was cancelled to free its workflow slot for the final head; normal cancellation did not finish promptly, so force cancellation completed it. - October 29 conflict resolution (867c00673b): master 929e840595 added installed-selection reuse and bundled relocation, routine cleanup notifications, a hardened YAML copy helper, and migrations through 0413. The merge retains those behaviors together with package read scoping and the public-selection fast path; restricted selections still reauthorize and the full row read rechecks scope. Generated migration 0414 retains the existing SQL unchanged. Workspace typecheck/build and migration checks pass. Local database verification is unavailable: initdb reports semget / “could not create semaphores: No space left on device”; all 168 cases skipped, and repeated migration application failed at startup. Docker returned HTTP 500 during container creation, so no Docker test pass is claimed. Current-head CI must close this gap. - Apex review 14 scored 3/5 with one P1: a local import could copy cached private package files into a company-wide skill. Commit a2e597d800 rejects cache trees and ancestors through a canonical filesystem boundary, including aliases and ancestors checked before a concurrent fill. Ten local filesystem tests and server typecheck pass. The existing private-project integration case now rejects package-cache imports for both authorized and unauthorized users. The preceding merge commit’s complete CI run passed, and logs confirm all 168 locally skipped cases executed successfully: package store 36, sources 35, package worker 9, runtime selection 4, observability 3, authoring 81. - Apex review 15 returned 5/5 with zero new findings on a2e597d800. The separate Paperclip review found no must-fix or should-fix issues. Its optional suggestions were considered: unexpected filesystem cleanup failures remain visible, the durable marker cannot replace checking that bytes still exist before deleting a last local authored copy, and administrative history batching remains outside the pre-turn optimization. The cache-path literal centralization suggestion is also optional; the current producer and guard use the same directory, covered by the new integration regression. All review conversations are resolved. - Earlier October 29 CI/Apex acceptance: a2e597d800 was mergeable at that check, Apex review 15 passed 5/5 with zero findings, and all review conversations were resolved. The hosted verification above records the later acceptance limits and renewed conflicts. Workflow 40884087629 completed successfully, with 55 successful checks, two neutral reports, two skips, and no pending or failed checks. The project-skill-sources job log confirms all 29 cases ran and passed, closing the latest integration gap. The preceding merge workflow also confirmed all 168 focused database cases ran and passed. No local PostgreSQL or Docker pass is claimed. </details> ## Risks - First use after eviction can require storage or Git access. Revoked grants, removed sources, and unreachable commits can prevent reconstruction. Recovery never substitutes a newer commit. - Historical packages without a proven recipe keep durable bytes. Cleanup favors retention when a run pin cannot be interpreted safely. - New uploads and conversion use bounded work and persisted retries. Large initial conversions still add temporary storage and database load. - Optional cleanup can wait until the next server wake. Supported notifications assume one owning server process. - Per-entrypoint native tree duplication and cold preparation cost remain. The final ten-skill cold sample took 47.6 seconds on the shared host. - Generic authored edits can still hold an existing edit transaction while a package uploads. - Explicit file/folder deletion uses temporary disk space for rollback. An uncertain commit with a changed skill or unavailable database retains that backup and logs its location for recovery. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and tests. The exact served model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 12:19:37 -05:00
Nicky LeachandPaperclip	247de89eae	fix(runner): preserve warm tasks during recovery (#18169) ... ## Thinking Path > - Paperclip manages agent work and the lifetime of its execution resources. > - A warm native task can keep one Runner alive between successful turns. > - Foreground completion selects whether that resource stays running or stops. > - Background recovery must use the same selected policy. > - Recovery currently reads that policy only from workspace copy-back records. > - A task without such a record can be stopped immediately after foreground retention. > - This pull request saves the resource policy with the native runtime and permits the retained task to refresh its control connection. ## Linked Issues or Issue Description **What happened?** Background native finalization can stop a successful warm Runner when its run has no Core workspace-sync record. Foreground completion retains that same task. The stopped Runner cannot receive the later drain and suspend commands. Cold resume then correctly rejects its unproven checkpoint. Hosted diagnostics confirmed the recovery reconciler attempted environment release immediately after foreground retention. The provider connection returned HTTP 404 within the same second. A database-backed regression reproduced the stop without a workspace-sync record. A separate authorization gap also prevents the current retained task from refreshing its connection endpoint after the run succeeds. **Expected behavior** Foreground completion and recovery must use the selected resource policy. A successful warm task stays running until its idle shutdown can suspend it. Failed or cancelled work still stops. The current retained task can refresh its existing control route, but cannot prepare or submit new execution. **Steps to reproduce** 1. Select a warm native runtime on a reusable remote execution environment. 2. Complete a turn without a Core workspace-sync record. 3. Let native finalization recovery settle the still-active run lease before foreground retention finishes. 4. Before this change, recovery defaults to `stop_and_retain` and invokes the provider stop path. 5. Separately, request the retained task's connection endpoint after successful completion. Before this change, Core rejects that request. **Paperclip version/commit** The hosted failure used Core 31b927e301. Regressions also reproduced on master at 6d59f5ef7. **Deployment mode** A remote native execution environment with reusable warm tasks. The resource policy is generic and applies across harnesses and providers. Related recovery coverage: Refs #18161. This change preserves the suspended-checkpoint admission guard. Existing tasks without the newly saved policy retain the previous conservative fallback; deployments must drain or recover them explicitly. ## What Changed - Save the selected native resource disposition in the existing runtime-selection transaction. Add no startup database round trip or network request. - Read this small policy in background settlement and recovery when no workspace-sync record exists. Preserve valid workspace-sync policies and export-retry behavior. - Keep failed and cancelled runs out of warm retention. - Permit connection refresh for a successful, dispatched warm task on its current retained reusable lease. Keep retirement, termination, expiry, saved-run identity, and plugin capability checks. - Add database-backed retention and selection regressions, failure/cancellation coverage, and bounded projection checks. Document the lifecycle and access contracts. ## Verification - Before the source edits, the no-workspace-sync regression invoked the driver with `stop_and_retain` instead of retaining the lease. Both runtime-selection cases failed because the resource policy was absent. - After the source edits, those regressions and both run-state/runtime-selection files passed. The three-file run had 77 passes and two new fixture expectation failures. Those fixtures were corrected. Final-head GitHub CI passed, including the server suites, Runner checks, typecheck, build, and E2E gates. The local rerun was deferred at the user's request. - Commands: `node_modules/.bin/vitest run server/src/__tests__/native-environment-settlement-work.test.ts server/src/services/heartbeat/runtime-selection.test.ts server/src/services/heartbeat/run-state.test.ts`. - The connection authorization regression failed before its edit. All 34 tests in `server/src/__tests__/environment-task-runtime.test.ts` then passed. - `node scripts/check-maps.mjs` and `git diff --check` passed. - No comprehensive local suite was run. Further local tests and VM reproductions were stopped at the user's request to conserve battery and mobile data. GitHub CI verifies the final head. - Hosted verification of this patch requires deployment after merge. Passing pre-patch retries did not establish repair of the original failure. - Standard Greptile and the subsequent requested Apex review both returned **5/5** on `360bcd7e789d65950c27484af237faec02553666`, with no inline findings. ## Risks - This adds one server-owned field to the existing run profile JSON; no migration is required. Invalid or absent policy keeps the previous conservative fallback. - Valid workspace-sync disposition remains authoritative. Pending export and failed finalization still use stop-and-retain safety behavior. - Retained connection access requires a successful run, matching saved run identity, dispatched warm task, unexpired reusable lease, and no retirement or termination. Superseded leases and new execution remain denied. - The cold-resume checkpoint proof is unchanged. This patch does not make the already-stopped task's unsuspended checkpoint resumable. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The host does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run targeted local checks; final verification passed in GitHub CI (local rerun deferred at the user's request) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 09:19:49 -07:00
DottaandPaperclip	6f278dd6d7	fix(runtime): settle recovery lease and connection lifecycles (#18167) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - A remote runner can survive a controller restart. > - Recovery renews its execution lease during slow preparation. > - The native executor then shares that renewal timer. > - Preparation kept its reference until all finalization finished. A tick after lease release treated completion as ownership loss and detached the retained runner. > - A Linux continuity test also found that buffered wire events could arrive before their connection was registered. Cleanup then kept a completed promise under an undefined key and never finished. > - This pull request transfers renewal ownership after execution acquires its reference. It also registers each connection before wire callbacks can run. > - Recovered turns can complete, retain the warm runner, and settle connection cleanup. ## Linked Issues or Issue Description **What happened?** A recovered remote run could complete, release its coordinator lease, then detach during instruction collection. The next turn could find the old runner still using the ingress port. Recovery then stopped because it could not prove process ownership. **Expected behavior** Preparation keeps the lease alive until execution owns renewal. Execution stops renewal before releasing the lease. Actual ownership loss still detaches the runner. Buffered wire callbacks must use the registered connection, and cleanup must finish. **Steps to reproduce** 1. Recover an in-flight remote runner after a controller restart. 2. Let the recovered turn complete. 3. Hold instruction collection open through the next lease renewal tick. 4. Send a follow-up that needs the retained runner. Related: #17439 introduced remote controller recovery. #18011 preserves checkpoint authority through warm preparation. This change fixes the lifetime of preparation's renewal reference. ## What Changed - Pass an internal lease-acquired callback from recovery through heartbeat dispatch. - Release preparation's renewal reference after the executor registers its own. Keep final cleanup for exits before dispatch. - Add a PostgreSQL regression for completion while finalization is still pending. Cover callback forwarding, acquire-before-release ordering, and failure before dispatch. - Register the authority connection before subscribing to wire events. Handle an immediate close before accepting buffered data. - Add regressions for a message or close delivered during subscription. - Extend Linux continuity coverage with preparation renewal and delayed instruction collection after recovery. - Update the continuity guide and session verification recipes. ## Verification - The new recovery regression failed before the fix because renewal detached the runner after lease release. The 76 recovery, renewal, and dispatch tests then passed. The final ordering and early-exit checks passed again after replacing fixed sleeps with controlled interval ticks. - Both buffered-wire regressions failed before the connection fix. All 170 control-plane tests passed afterward. Existing tests cover stale authority, ownership loss, pending operations, and closed ingress. - Linux ARM64 acceptance passed at `a7daefacceb8bd994d0b79d284cdf54f3e3829d1`, with Node 24.19.0, PostgreSQL, the real Rust runner, TLS ingress, and a deterministic Codex protocol fixture. The test interrupted the first turn, recovered it, held finalization open for a renewal interval, and completed two follow-ups. It retained one provider process, three turns in one history, and the same runner identity. Final suspension and connection cleanup completed. The ordinary warm-reuse case also passed three turns with one provider process. - Earlier Linux attempts completed the turns but hung during cleanup. A debugger found a fulfilled promise keyed by an undefined connection. The regression reproduces that defect. One later setup attempt timed out while staging an unstripped 373 MB debug binary. The passing run used the same binary with debug symbols removed. - Reproduce Linux acceptance with the `same-server` case in `doc/testing-native-remote-continuity.md`. - `pnpm -r typecheck`, `pnpm build`, map validation, and `git diff --check` passed at the final revision. The full local `pnpm test:run` command was attempted, but the serial run had not finished after 13 minutes. It was stopped after the equivalent CI lanes passed. It is not counted as a full local pass. - Greptile and the Paperclip review both scored the final revision 5/5 with no findings. All Runner verification lanes passed. The first CI attempt hit timing failures in voice recovery, agent YAML configuration, and the task-title outage browser test. The full YAML and voice-recovery files passed locally (88 and 19 tests). These paths were unchanged and passed on the prior revision. All three passed on the single retry. Final current-head status: 55 successful checks, one neutral check, and two skipped Storybook checks. No pending or failed checks remain. - Live Daytona/model acceptance and recovery of the hosted task remain unverified. This PR does not deploy the change. ## Risks - Releasing preparation before execution starts renewal could leave a gap in authority. The callback runs after the executor registers its reference. - Preparation errors must not leak renewal. The recovery promise retains idempotent final cleanup. - Wire callbacks can run during subscription. Message and close regressions check that registration is complete before either callback runs. - No schema, API, model, permission, lease duration, or retry policy changes. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser diagnostics. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 11:18:53 -05:00
3 people	07c6df260b	fix(server): schedule archived runner evidence from saved deadlines (#17925) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/85/changes/paperclip/85 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Retired native sessions leave continuity and quarantine evidence on disk. > - The old hourly loop mixes evidence retention with backup expiry, which has different database eligibility rules. > - Evidence already has saved creation timestamps and local publication signals. > - This pull request gives evidence its own registered worker and actual next deadline. > - The worker stops filesystem and SQL activity when its evidence obligations settle. ## Linked Issues or Issue Description Stack slice 65. Parent: #17923 (`codex/sleep-64-skill-version-retention`). Related foundation: #15625. **What happened?** At parent `7b94aa185c29673372646bbf2e07a2fbd16bc394`, an hourly backup/evidence tick still inspects native state roots. Evidence's private due time is checked by that recurring tick instead of the shared scheduler. Empty evidence and future retention windows have no reason to be rediscovered every hour. **Expected behavior** Reconstruct evidence once. Wake from the existing publication signal. Register the earliest saved retention deadline. Keep busy, failed, or incomplete evidence recoverable. Preserve backup eligibility as a separate duty. **Steps to reproduce** Start retention with an empty evidence root or archives inside the seven-day window. The old timer still visits the combined owner. The new evidence worker disarms when empty, or registers the saved expiry with no evidence filesystem or SQL calls between relevant changes and that deadline. ## What Changed - Add an evidence worker through the delivery coordinator and shared scheduler. Track startup reconstruction without delaying HTTP readiness. Unsubscribe local publication before delivery shutdown. - Select `backups` for the existing backup timer and `evidence` for the new worker. Existing callers that omit the selection retain combined behavior. Backup expiration remains the next separate migration. - Use hourly retries for concrete busy or failed evidence, with earlier saved expiries and one-second continuations independent. Restore candidate/removal/byte summaries. Schedule publication at the first removable instant; preserve legacy manual-tick semantics. - Expose the existing earliest evidence deadline. Preserve one-second bounded continuation, concrete failure/busy retries, cleanup-source archive exclusions, symlink protection, late activity checks, atomic rename, and interrupted removal recovery. - Preserve non-ENOENT inspection errors as recovery obligations. Retry a failed session root in a new evidence pass instead of continuing past it and forgetting it. - Register the first removable instant after the inclusive retention boundary. Keep the existing manual dry-run publication semantics. - Add real filesystem/isolated PostgreSQL lifecycle tests and disabled/held-startup wiring cases. Add no table or migration. The archive and quarantine writers already call `noteNativeRunnerEvidence`; this PR connects that existing signal to the scheduler. Shared code owns scheduling, while evidence rules stay with the retention service. See [`doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md`](doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md). ## Verification - Adopted actual prerequisite squash `b06ea111a254c62e0a55199b010d82e0adb33bf0` from #17923 through normal merges. Qualified runtime/package/UI/map trees remain unchanged. The final nine-path diff includes one browser-spec readiness repair. - Runtime qualification remains **652 passing TypeScript cases across 22 complete files**, plus **29 passing Rust cases** in native-backend, runtime-notification and configured-environment targets. One actual closed Pi snapshot launch is intentionally skipped because its external distribution root is not provided. Accepted source is `b80eee32f344ae0218d8b791c14f2ea174da09cc`; all runtime and earlier sleep-test bytes are preserved. - The complete outage-recovery browser file passes **all five journeys** on a fresh isolated database after four additive rendered-history readiness assertions. Every original assertion, outage control and timeout remains byte-identical when those new lines are removed. The intentionally unresolved first-session scenario keeps its pre-auth interaction and delayed-owner coverage. The prior CI/local timeout evidence and one rejected broader precondition remain in the audit. - Adapter/Runner/database builds, server/Runner typechecking, Rust formatting/workspace checks, server build, module boundaries, migration order and generated TypeScript/Rust profiles pass. PostgreSQL 17 fixtures apply all 413 journal entries through index 414; source hashes and temporary fixture imports restore exactly. - **Inherited validation exception:** Copilot profile 81's separate source-identity check fails on the merged Rust configured-environment hash. Its inputs and attestations equal upstream and stay unchanged here. Grok's identity check passes. No release attestation is regenerated or new live-provider qualification claimed. Required exact-head CI/security and Greptile 5/5 still must pass. - [The evidence audit](doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md) retains historical failures, counts and limitations. No full local repository-suite, live provider, external-alarm, measured suspension or fleet-savings claim. Fresh final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - Live publication is the existing server-owned executor in one process. Independent filesystem writers need an explicit signal contract. Startup reconstructs restart state. - Busy sessions and concrete inspection/removal failures retain retries. Unknown cleanup-source archives remain protected. - The existing scan enumerates directory names before applying its entry/removal budget. This PR preserves that behavior and does not claim a bounded total directory-listing allocation. - An in-process deadline cannot wake stopped compute. Cloud sleep-safety and external-alarm coverage stay unchanged. Physical evidence reclamation can wait for a supported host wake. - Backup expiration still has an hourly database eligibility scan. This slice removes only the archived-evidence duty from that loop and does not claim full instance suspension. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs and complete PostgreSQL/filesystem/startup suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 11:12:16 -05:00
DottaandPaperclip	c32e1ba6da	fix(onboarding): keep approved child task handoffs runnable (#18160) ... ## Thinking Path > - Paperclip manages work through tasks, agent runs, dependencies, and human decisions. > - The first-task workflow asks for approval before it creates an execution task. > - File-producing work must run on a child task before onboarding can finish. > - A child blocked by its onboarding source cannot start while the source waits for that child. > - This change rejects that dependency during task creation and later edits, and gives the agent a correction. > - The shipped recipe shows the dependency direction and requires the actual approved inputs in the child instructions. ## Linked Issues or Issue Description **What happened?** After human approval, an onboarding run could create a child with the onboarding source in `blockedByTaskIds`. Creation saved a blocked child with no wake. The parent could then wait for that child. A handoff could also refer to input from the parent conversation without including the input in the child description. **Expected behavior** The approved child can execute before its waiting parent. Invalid first-task dependency input fails before task creation and leaves the same command key available for a corrected call. The child instructions include the required approved input and constraints. **Steps to reproduce** 1. Accept a proposal on a server-created first task. 2. Call native `create_task` with `status: "todo"` and the onboarding source in `blockedByTaskIds`. 3. Observe a blocked child with no scheduled wake. Block the parent on that child to expose the stalled handoff. **Paperclip version or commit** The regression reproduces on `f8484a9f8d1f68e9534fee4f29e00e716dc680f4`. The repair is based on current master. **Deployment mode** Core native and REST task creation. The same server boundary serves self-hosted and hosted instances. Related: #18080 fixed remote delivery of the selected first-task skill. This change handles the later child handoff boundary. ## What Changed - Reject a task blocked by its authenticated first-task source during creation and later dependency edits. Later agent edits also check the locked target's server-owned onboarding parent, protecting a child running under its own execution. Compare UUID identity in PostgreSQL so alternate supported UUID spellings cannot bypass the check. The error explains the correction. - Add a native child/dependency recipe with self-contained approved input guidance. Clarify the general `create_task` input descriptions and update their generated contracts. - Cover native and REST rejection, corrected retry, preserved valid dependencies and replay, and an executable recipe with exact input bytes read from the child's own context. Thread the authenticated run through native dependency edits. Keep blocker corrections independent of a new proposal decision. - Update execution semantics and the existing onboarding map entries. ## Verification - Red on `f8484a9f`: the new native regression received a created blocked child instead of the expected rejection. - Review regressions were red on the original PR head: uppercase, unhyphenated and braced source UUIDs created blocked children; a later child PATCH returned 200 and saved the reversed blocker. - Review follow-up reproduced both native and REST child-own-run bypasses on `75fa75a6`: the native dependency call applied the reverse edge; the child's own PATCH returned 200. The regression uses the actual authenticated child execution and an already checked-out task. - All 95 tests across first-task admission, assets, and the full native tool-authority suite pass. The shipped recipe is validated against the native schemas and exercised through real task/dependency services with disposable PostgreSQL. Rejected PATCH tests preserve existing authorization and attempt-audit behavior and require no requested task-field, relation or wake change. Compatibility controls retain ordinary manual parent blockers, explicit board edits, other blockers and empty-list corrections. - Runner TypeScript build, generated contracts, capability inventory, protocol manifest, and map checks passed. - Server typecheck and all 95 focused tests pass with the child-run follow-up. All latest checks pass on `f24509ae`: 55 successful checks, one neutral review check, four skipped checks, and a successful legacy security status. Both automated reviews are 5/5 on this exact commit. - Prior-head CI failures are preserved: the Runner close fixture had unconfirmed provider drain while suspension succeeded; its exact fake-provider subprocess row passed locally with unchanged sources. The browser outage test lost its title textarea before any outage write. A bounded local control with a fully loaded composer passed every original outage assertion. The prior failure is an editor-readiness race; its exact asynchronous ordering remains unproved. No UI, browser-test, or timeout change is included. Neither failure is counted as a pass or silently discarded. - The first `f24509ae` CI attempt had two interrupted jobs: Typecheck and General server 9/12. Their logs record runner shutdown signals. The server job also records lost runner communication. The same-head failed-job rerun passed both jobs and the required aggregate. It reused the other successful results. No source, test, or timeout change was made for the rerun. The failed attempt remains preserved. Full repository checks pass in CI; the local full repository typecheck, test, and build commands were not repeated. - No external model or provider calls are used by these checks. Real-provider acceptance at this revision remains separate. ## Risks - The new rejection uses the authenticated run and the server-owned onboarding origin. Ordinary tasks, other blockers, empty-list corrections, and already committed creation replay retain their existing behavior. Blocker edits do not reapply the creation approval gate. - Nonempty blocker edits perform two indexed identity reads; the task lookup covers at most the run source and locked structural parent. Other patches and empty-list corrections add no source lookup. The update guard adds no parent row lock behind a child. Existing REST authorization checkout and attempt auditing remain in place before the domain mutation guard. - The explicit input recipe guides the model. Schema and persistence tests do not prove that a future model follows every instruction. - The tool guidance changes the generated semantic catalog identity. The maintained generated files are synchronized in this change. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with `ultra` reasoning effort, repository tools, code generation, and test execution. The context window is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and stated the available runtime detail - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 10:57:31 -05:00
Nicky LeachandPaperclip	6d59f5ef7e	fix: allow workspace changes after settled native bootstrap failures (#18166) ... ## Thinking Path > - Paperclip manages work assigned to AI agents. > - Workspace admission must wait for file persistence and execution cleanup. > - A native startup error can record a failed workspace finalization before provider work starts. > - The controller and environment can settle while that diagnostic remains. > - The workspace guard then blocks every later binding change. > - This pull request recognizes settled bootstrap failures with no output obligation. > - Real output, process ownership, and cleanup obligations continue to block changes. ## Linked Issues or Issue Description Related: #14234 handles recovery-hold regeneration for legacy failures. This change addresses the separate native workspace-finalization guard. **What happened?** A workspace change stayed blocked after two native startup failures. Both runs had explicit bootstrap receipts with `providerWorkStarted: false`. Their controllers were terminal and unowned. Their leases had completed cleanup. No provider events or workspace copyback descriptors existed. Each run still had a failed `workspace_finalize` diagnostic. Later successful turns did not remove that historical record, so selecting a task directory failed during setup on every retry. **Expected behavior** A confirmed startup failure with no provider work must not create a permanent file-persistence barrier. A real workspace export or cleanup obligation must remain fenced. **Steps to reproduce** 1. Fail native session startup before provider work, leaving a failed finalization diagnostic. 2. Settle the failed controller and its environment cleanup. 3. Schedule a different task workspace. 4. Start the next admission. Before this fix, it fails with “Previous task run still owns workspace recovery.” **Paperclip version or commit** Reproduced against the workspace guard on master at `27be85904`. **Deployment mode** Hosted application with external PostgreSQL and a plugin-managed agent computer. ## What Changed - Exempt a failed finalization diagnostic only for finished native bootstrap failures with an explicit no-work receipt and an unowned terminal coordinator without a result. - Keep provider events, process identities, restore failures, sync descriptors, copyback owners, unfinished operations, and unsettled environment cleanup as blockers. - Reuse the existing batched provider-evidence reader. - Test both pending workspace selection and legacy binding updates. Keep the original failed operation unchanged. - Document the admission invariant. ## Verification - The new reproduction failed on both workspace-change paths before the fix. - `pnpm exec vitest run server/src/__tests__/task-workspace-binding.test.ts`: 92 tests passed after the fix. - A first isolated test attempt needed the existing plugin SDK build prerequisites. Tests ran after `pnpm --filter @paperclipai/plugin-sdk ensure-build-deps`. - `pnpm -r typecheck` and `pnpm build` pass with pnpm 9.15.4 and Rust 1.97.1. The initial typecheck attempt lacked Cargo on PATH; it was corrected to use the installed toolchain. The unsharded local `pnpm test:run` did not finish during the validation window and was stopped after all PR CI test shards passed. It did not provide a completed local full-suite verdict. - Remote recovery used an audited, scoped transaction to append no-op reconciliation records for the two verified bootstrap failures. Original failures and workspace data were retained. The normal UI Retry then applied the pending selection and reached native provider execution. - Greptile reviewed commit `807d22be9e530e272d53def365a0cdd37bd142ea`: 5/5, with no review threads. All PR CI checks passed. GitHub reports no merge conflicts. ## Risks The main risk is misclassifying an unfinished workspace as a startup-only diagnostic. The exemption requires positive no-work evidence and rejects conflicting provider or output state. Existing ownership and cleanup gates run first. Provider evidence adds one batched database read only when eligible failed bootstrap diagnostics exist. There is no new background work, migration, dependency, or provider replay. Reverting this change restores the former guard. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, and browser tools. This session does not expose a more specific model build ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 08:34:52 -07:00
3 people	b06ea111a2	fix(server): schedule skill cache retention from changes and saved deadlines (#17923) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/83/changes/paperclip/83 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server caches saved skill versions on disk so runs can restore their exact instructions. > - The old retention owner scans company rows every hour, even after all caches are protected or removed. > - Cache publication, version changes, and pin release already have local or transaction-owned boundaries. > - Grace windows and terminal controller leases already have saved timestamps. > - This pull request uses those changes and deadlines to drive the existing bounded prune service. > - Idle retention stops SQL and directory reads while pending cleanup remains recoverable. ## Linked Issues or Issue Description Stack slice 64. Parent: #17876 (`codex/sleep-63-agent-timer-deadlines`). Related: #15625 establishes transaction-owned work signals. #17907 independently keeps an active cache-lock transaction alive; this PR preserves that lock and does not duplicate that repair. **What happened?** At parent `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`, skill-version retention has an hourly company scan, a startup timeout, and separate continuation timers. Empty or permanently protected caches still cause recurring SQL. **Expected behavior** Reconstruct once, inspect changed companies, and schedule saved access/staging grace or terminal controller deadlines. Stop recurring work when no timed or uncertain obligation remains. Keep current/newest/agent/run protection, restorable-byte validation, cache locking, final pin revalidation, atomic rename, and trash recovery. **Steps to reproduce** Start an enabled server with no cached versions, or only current/pinned versions. After the initial scan, advance an hour. The old interval selects companies again. The new worker has no deadline and performs no retention SQL or directory scan in that state. ## What Changed - Signal the missing-local-source reconciliation deletion in its transaction. Limit fresh scans of a successfully observed company to one per thirty seconds. Keep one-second page continuations and actual saved deadlines independent. Empty or uncertain passes do not delay first publication. - Restart a company scan if drain interrupts inspection or post-rename trash cleanup. Restore bounded candidate/byte logging with dry-run mode. Three controlled cases failed before repair and the complete 34-case retention suite passes afterward. - Register retention through the delivery coordinator and shared scheduler. Remove the hourly, startup, and continuation timers from `index.ts`. Track startup work without delaying readiness. Stop delivery work before closing retained iterators. - Keep one active company cursor and bounded 50-entry passes. Preserve observed deadlines across pages and bound company discovery to 51 rows per page. Retain concrete company failures and incomplete pins for retry. - Report saved grace and terminal controller deadlines from the existing cache owner. PostgreSQL remains the controller-lease authority. Current/newest/agent pins do not arm unrelated lease checks. - Wake on cache publication, rejected materialization, and access markers. Signal version creation/deletion and agent configuration/hire changes in their owning transactions. Extend existing instruction, finalization/environment, and retained-session fanouts with company hints. - Route reused provider-task prior-owner retirement through `writeEnvironmentCleanup`, so that supported pin-release write also settles a notification. - Add an optional scope callback to `registerDeadlineWork`. Existing consumers omit it and keep their behavior. The feature owns persistence predicates; startup only registers it. Add no table or migration. Full producer audit, shared-contract explanation, lifecycle evidence, and limitations: [`doc/plans/2026-10-27-skill-version-retention-sleep-audit.md`](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md). ## Verification - Reconciled all existing sleep changes with actual master `7e27bc5007c8f9cb8cdce06c24f9813e27f7120f` through normal merges. All 24 retention-owned and 95 unrelated upstream paths are preserved; package, UI, E2E and map trees equal master. - Combined-source qualification: **1,009 TypeScript cases across 42 complete files**, plus **28 Rust cases** in two targets. One intentional Linux-inotify-only adapter case is skipped on this macOS host. Coverage includes actual retention/cache/producer/runtime behavior, startup, shared deadline consumers, changed OpenCode/observability and native execution boundaries. - After the subsequent task-index/task-read upstream merges, **85 cases across six complete files** passed on final source `d9277b1858a2230bdb726974c90a0c9bc5c5ebe6`: real retention/cache/work, task reads/query behavior and concurrent migration repair. Counts overlap earlier coverage. Fresh PostgreSQL 17 schemas apply all 413 journal entries through index 414. - Database/adapter/Runner builds, production server and Runner typechecking, module boundaries, target migration order and generated TypeScript/Rust release-profile consistency pass. Selected source hashes and temporary fixture imports restore exactly. - Reproduced and repaired Greptile's own-company due-cleanup P2 with identical old-worker-failing/new-worker-passing assertions. The 30-second ordinary scan floor remains; already-due cleanup receives its owed scan and disarms afterward. Earlier 799-case and fresh-blank-plugin acceptance remain in the audit. - **Inherited validation exception:** the separate Copilot source-identity check fails after upstream OpenCode reserved-environment changes; all 72 identity inputs equal current master. No provider release attestation is regenerated or new live qualification claimed here. Required exact-head CI/security and Greptile 5/5 still must pass. - Historical results, repair details and limitations remain in [the retention audit](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md). No full local repository-suite, live provider, production-load, external-alarm, measured suspend/wake or fleet-savings claim. - CI on the prior published head found four connection-review assertions racing the concealed conversation after reload. Its fresh Greptile/Ponytail reviews were both 5/5, but the failing browser gate was not waived or retried unchanged. The fixture now foregrounds the task tab and waits for visible history before retaining all existing saved-result assertions. The full file passes four journeys locally; the controlled-restart case is intentionally skipped by the default configuration. Original local journeys also passed unchanged, so no deterministic local red reproduction is claimed. See the audit for CI failure artifacts and scope. Runtime and prior sleep-test sources remain unchanged. ## Risks - Supported live writers are the existing service/API owners in one process. Independent SQL/FS writers need an ingress contract. Startup reconstruction covers restarts. - Notifications cause a bounded inspection; they do not authorize deletion. Durable pin predicates and the cache lock remain authoritative. - A database clock behind application time can require another one-second check of the concrete expired controller lease. Unknown pins, lock contention, and failed inspection retain retries until resolved. - The in-process scheduler cannot wake stopped compute. This PR preserves Cloud sleep-safety gates and external-alarm coverage. It does not prove deployed suspension or fleet savings. Cache reclamation can wait until a supported wake when compute is stopped. - Native-harness backup retention and the other remaining duties are separate stack slices. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs, a controlled red/green regression and complete PostgreSQL/filesystem integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 10:14:12 -05:00
DottaandPaperclip	27fbdb2a74	perf(workspaces): reuse verified seeds across task branches (#18159) ... ## Thinking Path > - Paperclip manages AI agents and their workspaces. > - Remote runs stage a recoverable copy of the selected host workspace. > - New tasks use separate workspace IDs and Git branches. > - The verified seed cache currently treats those identities as different content. > - This causes repeated clones and seed verification when task files and Git commits are identical. > - This pull request shares immutable seed content within a company and applies each task's branch to a private archive. > - It also removes credential broker calls for two local Git snapshot commands. ## Linked Issues or Issue Description **What happened?** New remote tasks rebuild the same Git seed under different workspace IDs and branches. In a measured staging run, preparation took 29.7 seconds. Git clone took 6.1 seconds and optional seed publication took 7.8 seconds. Publication overlapped the 4.4-second transfer, so those values cannot be added. Local `diff-files` and `write-tree` commands also request GitHub credentials. **What did you expect?** Reuse a verified seed when filesystem contents, Git HEAD, origin, and nested repository layout match. Preserve each task's branch, recovery archive, ignored files, and copyback behavior. Keep local snapshot plumbing independent of GitHub credential capture. **Steps to reproduce** 1. Configure a remote native environment and a Git project workspace. 2. Create two tasks from the same source commit with different task branches and workspace IDs. 3. Inspect the workspace preparation spans. Both tasks currently clone and publish the same seed content. **Paperclip version and deployment mode** Reproduced on Core `25f1070fbdabd96b26cdf199e624a77905f3ca37`, with a managed remote sandbox and native Runner. The local comparison used a public repository at a fixed commit with roughly 13,000 entries. Refs #17886, #16398, #16399, #17357. The Git launcher extension builds on Jamie Buchanan's work in #17886. The other PRs establish the seed verification, recovery, and eviction contracts preserved here. A search found no open PR for sharing these verified seeds across task branches. ## What Changed - Use a company-scoped cache bucket for native host-current preparation. Keep the existing bucket and company storage limits and eviction rules. - Remove branch names from the shared generation identity. Keep file contents, Git HEAD, origin, and nested layout in that identity. Keep the default workspace-scoped identity unchanged for other callers. - Store a detached Git archive in the shared cache. Copy it and append the selected branch metadata before upload and private per-run persistence. Existing durable-seed replay can read the private archive without the new cache option. - Forward the scope through the execution target and both command-runtime paths. - Treat `diff-files` and `write-tree` as local Git commands. Network commands, unknown commands, and managed commit identity keep their existing credential path. - Test new workspace IDs, named and detached branches, nested repositories, dirty and ignored files, symlinks, recovery, corrupt caches, refused publication, and copyback. ## Verification - Current head `6087bac274d50804d1b332646e43f3b8d41d798f`: 53 checks passed, two optional checks skipped, Greptile 5/5, no unresolved review threads. Linux CI passed. - Adapter-utils TypeScript 7.0.2 typecheck passed. - Server package typecheck passed, including its Runner and plugin build prerequisites. - 140 seed-cache, seed-reuse, and sandbox-runtime tests passed after rebasing. - 44 Git launcher and command-runtime tests passed after rebasing. - 30 native workspace preparation, warm history, and retention-configuration tests passed. All 56 retention tests also passed (5.72 seconds) after repairing missing library aliases inside the installed macOS Postgres package. That local dependency repair changed no repository source. The earlier startup timeouts occurred before test assertions. - The new seed reuse regression failed before the change because the second branch cloned again. It passes through the execution target and both command-runtime paths after the change. - Clean local preparation comparison: two alternating measured pairs, with an existing verified company seed and new task branches. Baseline: 19.63 and 9.85 seconds. Shared cache: 5.18 and 4.80 seconds. Median: 14.74 to 4.99 seconds. Each baseline cloned and published; each shared-cache run skipped both. Branch and file checks passed. The complete comparison loop took 58 seconds. - The comparison used real Git, archives, snapshot workers, and local file transport. It used no model calls. It does not establish a complete staging response time. The first generation still pays the cold preparation cost. ## Risks - A shared cache hit requires an identical generation that is still present. The company bucket retains the existing 16-generation/1 GiB bucket limit; busy companies may miss more often. - The branch binding appends entries to a private tar archive. Tests cover the host platform; Linux CI must also pass before merge. - A cache miss, corruption, or capacity refusal must retain the private recovery path. Existing tests exercise those cases. - Git plumbing can still encounter repository-specific filters or missing objects. This extends the existing local-command policy by two commands; it does not grant network credentials to them. - The latest measured real staging reply took 77.8 seconds. This change targets a bounded part of that path and does not claim the under-15-second response goal is met. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and repository tools. The session does not expose the exact model revision or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 09:19:23 -05:00
DottaandPaperclip	7e27bc5007	fix(tasks): keep anchored opens compact and stabilize saved cards (#18092) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task pages must make the current instruction and recent conversation readable quickly. > - Compact startup avoids reading all earlier messages and run activity. > - Every URL fragment previously selected the full-history path, even for documents. > - Saved document cards also arrived after the conversation was visible. PR statistics could add another row later. > - This pull request keeps anchors compact and reveals the saved conversation structure together. > - The benefit is stable reading position without loading unrelated history. ## Linked Issues or Issue Description Related: #17593 introduced compact startup. #14667 covers earlier reveal and scroll work. This fix retains the bounded history contract. **What happened?** Opening a task document link loaded all task history. Ordinary cold opens could reveal messages before saved document cards and plans arrived. Later PR metadata added a statistics row and changed card height. **Expected behavior** Every URL fragment keeps compact startup. Document links open their pane. Older-comment links fetch only the target comment. Saved cards keep their geometry after first reveal. **Steps to reproduce** 1. Create a task with many comments, a saved plan, documents and saved PR work products. 2. Delay the enriched task response and open the task. Observe cards inserted after messages appear. 3. Open a document fragment. Observe full comment, activity and run reads. 4. Delay PR metadata refresh. Observe each card grow when diff counts arrive. ## What Changed - Remove fragment and tab checks from automatic history expansion. Resolve validated comment UUIDs through the authorized single-comment endpoint, encode their path components, retain fetched targets across document navigation, and preserve cancellation/deletion cache updates. - Include the saved plan, document metadata and local work-product projection in the core task read. Skip recovery, workspace and related-task enrichment. Do not refresh providers. Summary queries no longer select other document bodies. - Seed saved work-product and plan queries from core, including already pending or errored reads, without replacing known lists or an authoritative empty result. Coordinate the bounded recent reads, saved cards, latest transcript and settled initial pane width before first reveal. Keep the thread and composer mounted during refresh and explicit expansion. - Open continuation summaries in the document pane. Reserve PR statistics geometry while provider metadata loads. - Add a reusable synthetic local task, nine browser checks, frame measurements and video. Extend the PostgreSQL core-route fixture with a saved PR row and assert its identity, URL, state and stored metadata survive the response. Update design guidance and capability maps. ## Verification - Red: visible document/plan cards changed from 0 to 1 to 4; thread height grew from 690 to 1,432 pixels. Green: all seven saved cards appear before reveal, with constant thread height and zero per-card movement in the final reviewed browser run. The final 1,492-pixel layout includes the reserved PR statistics lines. - Red: delayed PR diff counts changed all three cards from 62 to 82 pixels. Green: their heights stay at 82 pixels before and after metadata refresh. - Nine browser journeys pass: delayed enrichment, document/comment/unknown/continuation anchors, document clicks and explicit history expansion with a retained draft, missing-comment retry, delayed PR statistics, and retained older-comment position after document navigation. - 583 focused UI tests passed on `c91fa107ba`. After the plan-cache follow-up, all 212 affected page/cache/plan-hook tests and UI typecheck pass, including 14 plan-hook cases. The original 16 focused PostgreSQL tests passed for the server changes. - Merge-review red: holding the duplicate stored work-products read blocked the first reveal past 10 seconds despite core already containing the saved cards. Both empty and populated core projections now reveal while that request remains pending. Malformed-comment tests prove no target request is made; API tests prove path encoding and abort-signal forwarding. - The populated core-route test now inserts a real saved PR work product alongside the plan and documents, then checks the complete saved projection through HTTP. All ten task cold-open PostgreSQL tests pass. - Plan-cache review red: an existing pending or errored plan query prevented the saved core plan from reaching the consumer. Four real-hook cases failed before the data-presence fix and pass afterward, for both present and absent plans. A separately fetched newer plan or known absent plan remains authoritative. - Further layout red: the initial properties pane reduced visible thread width from 775 to 742 pixels after reveal. The reveal now waits for two stable width frames; the browser check includes width, height, card counts and per-card geometry. - `pnpm -r typecheck` and `pnpm build` passed for the original change; UI typecheck and build pass after review fixes. Token, module-boundary and map checks pass. `pnpm test:run` was stopped after about 15 minutes in the serialized server lane. No full local test-suite pass is claimed. On the earlier head, hosted build/typecheck and all Vitest lanes passed; one E2E shard failed because the agent-configuration clipboard helper copied page text into its YAML parser. On `c91fa107ba`, Greptile was 5/5 and every E2E shard passed, including the earlier clipboard case. One unchanged Discord chat-binding test in server shard 8 returned false after 185 ms; its 16-test integration file passes locally without changes. The CI cause remains unestablished. On `be436ce492`, both server shard 8 and E2E shard 7 passed without changing those tests or their product code. On final head `e357b3bced`, all CI jobs pass, including full general and serialized tests, all eight E2E shards, typecheck, build and the aggregate verification gate. Exact-head Greptile is 5/5 with no findings. The overall status rollup contains 55 successes, two intentionally skipped Storybook checks and a neutral Paperclip Review. That advisory review ended without a complete assessment; all previously posted findings are resolved and it added no current findings. - Test startup exceeded an initial 5-second deadline once while development assets and other checks were active. The final browser checks allow 10 seconds. These runs establish geometry and bounded requests; they do not establish a reliable latency improvement. An initial 503 error fixture exercised global reconnect/backoff. The target-comment failure test uses a terminal 404. - Browser data is synthetic. Provider metadata is simulated for the delayed-response check. No staging deployment or live provider execution is claimed. ## Risks - Core task reads now include local saved-card queries and the saved plan body. Other document bodies and all historical run activity remain outside that path. - First reveal waits for the bounded current structure. The existing 15-second incomplete-history fallback and inline retry remain available. - Continuation-summary links open in the document pane in the current task interface. Classic tasks keep their existing surface. - No migration or mutation permission change. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code editing, shell tools and browser testing. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 09:13:01 -05:00
DottaandPaperclip	2a6d4d1694	perf(db): index canonical task activity lookups (#18043) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/177/changes/paperclip/177 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip helps people manage AI agents and their tasks. > - Task lists rank tasks by their latest canonical activity. > - The existing lookup reads every event for each task before pagination. > - Query-only alternatives failed a narrow visibility control or the performance goal. > - This PR adds an index that lets the unchanged lookup stop at the newest qualifying event. > - The result is less history work with the same permissions, ordering, and response. ## Linked Issues or Issue Description Refs: #13803. That broader proposal stores a new activity field with triggers and a backfill. This index was authored independently and uses the existing query. **Subsystem affected** Task-list activity ranking and the database schema. **Current behavior** The list computes `MAX(activity_log.created_at)` for every eligible task. The existing entity index reads 50 events per task across 1,199 candidates in the representative fixture. **Proposed behavior** Add `(company_id, entity_type, entity_id, created_at)`. Keep the production task service and permission helpers unchanged. PostgreSQL reads backward until an event passes the existing exclusions. **Reason and benefit** Fresh paired measurements on the current base improve the full-description scoped SQL from **44.60 to 20.33 ms (2.19×)**. Activity buffer accesses fall from **63,547 to 4,796**, and history rows per probe fall from 50 to one. The complete service median falls from 50.63 to 31.31 ms. Complete positional SQL results and service responses match. ## What Changed - Add generated migration `0414_closed_edwin_jarvis.sql` and the matching Drizzle index. - Use `CREATE INDEX CONCURRENTLY IF NOT EXISTS` through the existing top-level runner and invalid-index retry cleanup. The client change is one whitelist entry. - Preserve all upstream SQL and journal entries; retain snapshots 0410–0414. - Cover full and index projections, canonical ordering, invalid-index recovery, concurrent migrators, and incomplete fixture rejection. - Repair benchmark seeding, apply requested work memory to both connections, and record both settings. Hosted defaults now include six Slack conversations, 642 GitHub conversations, and 53 removed conversations. ## Verification - **All final-head CI gates passed**, including full workspace typechecks, tests, build, Runner checks, and all eight E2E shards. Greptile is **5/5** on `67ca7ae1cf15fc054ef515ffd8205b1fe30a0d55`; no unresolved review threads remain. - Local database/server typechecks and migration numbering/safety checks pass. Three server regression tests and three migration regression suites pass on real PostgreSQL. - The original pre-index work guard fails at eight rows per probe; the indexed guard now checks both the full-description query and index projection. The service oracle covers cleanup/inbox exclusions, raw text IDs, company scope, comments, ties, sort directions, visibility before pagination, empty lists, and one child per parent. - The documented pre-index seed command reproduced the review failure before the fix and now succeeds. A paired run with non-default memory confirms both independent connections use `16MB`. - Fresh proof pins pre-index base `929e840595c8933ca485ac7a2e99413e1e11ca64`, with identical cloned data, unchanged generated SQL and parameters, two warmups, and five alternating samples per arm. All **18 comparison cells** preserve every positional row and complete service response. The 15-case default-memory matrix passes its physical-work, speed, and control bounds. - The one-visible-task control is 6.64 → 7.25 ms SQL; the 20-task control is 3.71 → 3.72 ms. Both pass the declared 25% or 2 ms budget. Full-query memory controls use matching `4MB` or `16MB` settings. - Committed 1,000-row write batches measured 40.09 → 43.17 ms median (+7.7%). Ranges overlap (34.67–71.85 and 34.27–48.05 ms); this is an observed local cost, not a precise throughput guarantee. - The final-numbered migration builds a valid index through the normal runner on the fresh 300,000-row fixture in 0.55 s. Earlier large-fixture proof of the same index SQL used 5,000 tasks and 1.25M activity rows: 2.58 s first apply, concurrent write/read probes 5.16/2.43 ms, interrupted invalid-index cleanup, successful retry, and one receipt. Current migration recovery tests verify the renumbered path. - [The proof document](https://github.com/paperclipai/paperclip/blob/67ca7ae1cf15fc054ef515ffd8205b1fe30a0d55/docs/performance/task-list-activity-index.md) retains the historical baseline, rejected rewrites, and the earlier matrix invalidated by severe host contention. Those historical timings are not substituted for the fresh measurements above. ## Risks The migration is an exception after measured query-only alternatives failed. The index adds storage and work to every activity write: about 30 MiB for 300,000 rows and 123 MiB for 1.25M rows. Concurrent creation avoids the writer blocking observed with an ordinary index build, but long transactions can delay it. Local timing does not establish Neon build duration. The query still ranks all eligible tasks, and excluded newer events still need to be skipped. These are local SQL/service measurements; deployed HTTP latency remains unverified. No hosted DDL or deployment was performed. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, and tool use. The environment does not expose the exact model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue using the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 09:06:41 -05:00
3 people	ba07f85cad	perf: schedule agent timers from durable deadlines (#17876) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/58/changes/paperclip/58 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent timers must start due work and preserve queue admission rules. > - The server woke their scan on every heartbeat interval, even with no eligible timer. > - Idle scans keep the database and instance active. > - This pull request derives the next deadline from saved policy and work records. > - Supported producer transactions notify the shared coordinator when eligibility changes. > - An idle timer worker stops scanning and resumes when real work or a deadline requires it. ## Linked Issues or Issue Description Refs #17863. This is slice 63 of the recurring-work migration and follows the merged issue-monitor deadline PR. Related: #16632 changes timer issue scope; #16881 changes live-run eligibility. This PR preserves the current admission policy. Those policy changes are separate work. A future time-sensitive predicate must expose its next transition deadline. **What existing behavior does this improve?** Agent timer heartbeat scheduling in the server. **Current behavior** A recurring heartbeat wakes the timer scan even when companies are paused, timers are disabled, or no actionable task exists. **Proposed behavior** Register the earliest eligible interval or positive daily-cap reset deadline. Re-read on supported producer commits and startup. Leave no recurring database scan when no timer obligation remains. ## What Changed - Keep policy and deadline calculation in the heartbeat queue. Scheduling retains company, reporting-chain, worktree, and CAS gates. - Use the existing deadline coordinator. Remove only the timer wake from the server heartbeat loop. - Notify before existing transaction writes that change eligibility or deadlines. Cover configuration, lifecycle, company budgets, task ownership, recovery, conversations, accounting corrections, and completion. - Use the lifecycle status-change hook for saved hire rejection. Cover checkout's second adoption transaction after an initial failed CAS. - Preserve pending work through idle and maintenance drains. Check cancellation between due agents. Reconstruct on restart. - Document the supported writer model and producer inventory. Add no schema, table, or provider branch. ## Verification - **1,710 passing cases across 49 complete files**, no skips, on accepted source `32db86c1974dcbf794f01a8c5e42024e2ad071fc`. Coverage includes real timer policy/CAS/producers, monitor and shared deadline consumers, native reconstruction, Slack lifecycle, skills/settings/configuration, queue isolation and agent lifecycle. - Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. Twenty-nine fresh PostgreSQL 17 schemas plus per-file fixtures applied all 412 journal entries through index 413; all selected source hashes and temporary fixture imports were restored exactly. - Four new deadline composition cases verify future/overdue timing, minimum scan spacing and disarming. A controlled mutation dropping only the inherited continuation option fails; the composed source passes with identical assertions and timers. - Normal parent reconciliation preserves task-drain expiry and both deadline contracts. The final monitor-parent publication changes only its audit, preserving every qualified runtime and selected test source. - Historical failures and review corrections remain in [the timer audit](doc/idle-agent-timers.md). No full local repository-suite, live provider, measured suspend/wake, external-alarm or fleet-savings claim. Final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - Process-local notifications cover supported service/API writers in one server process. Direct SQL and independent replicas need an explicit ingress contract. Startup reconstruction is not live cross-process notification. - Notifications can batch for up to the configured heartbeat interval. Known deadlines retain their scheduler path; failed reconciliation retains bounded retry. - Enabled timers without an actionable-work requirement remain real recurring obligations. Zero caps require policy changes; positive reached caps use UTC midnight. - The timer fixture substitutes adapter enqueue. Existing admission tests also pass, but this is not deployed provider or sleep qualification. - Cloud does not gain legacy agent timer alarms in this PR. Existing sleep-safety blockers remain. No database/host suspension or fleet savings claim is made. - Parent-stack CI and review must pass before merge. This PR does not authorize merging its dependencies. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs, a controlled red/green regression and complete PostgreSQL integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 08:58:27 -05:00
DottaandPaperclip	b81b870732	Add QA smoke journeys and OpenCode run traces (#18098) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E journeys verify saved application behavior across deployments. > - Task creation and questions need reusable smoke selections and exact run attribution. > - Operators also need to inspect the OpenCode execution behind a measured question. > - Company observability does not currently support native OpenCode. > - This change adds the journeys and a bounded projection of normalized OpenCode run events. > - The projection follows the configured capture policy and retains the current run identity across warm reuse. ## Linked Issues or Issue Description Related foundation: #17262. Related alternative: #17989 exports metadata from OpenCode's native OTLP stream. This change uses normalized run events to support the existing content switches and per-run linkage. It does not forward the native stream. The two PRs overlap in launch registration and must be reconciled before both land. **What existing behavior does this improve?** Reusable Product E2E smoke selections and company observability for native OpenCode. **Current behavior** The journey catalog lacks reusable task creation and attached-agent question checks. Native OpenCode is marked unsupported for company observability, so a successful destination connection test does not prove that agent traces will appear. **Proposed behavior** Add explicit task-create and ask-question selections. Verify saved state and exact company, issue, agent, model, environment and successful run attribution. Export normalized OpenCode turns, tools and usage to the configured OTLP destination. Honor each capture switch and redact known secrets. **Reason and benefit** CI can run a small measured smoke selection, retain its screenshots and checks, and link results to traces belonging to the same run. **Breaking changes** Existing selectors remain unchanged. OpenCode export is opt-in through existing company settings. Legacy OpenCode stays unsupported. ## What Changed - Add reusable task creation and question journeys, configured-agent capability, owned-fixture checkpoints and strict evidence validation. - Add an OpenCode-owned projection of normalized run events. No provider SDK or raw payload schema enters heartbeat coordination. - Extend the private launch union and reserved environment allowlist for native OpenCode. With export disabled, no receiver request runs. - Keep destination headers in the parent wrapper and out of the actual provider child environment and configuration. - Bound each turn to 64 tool and 64 usage entries, 8 KiB per text field and a 1 MiB request. Use a three-second request deadline, four concurrent exports, and no retries or idle worker. - Pass the rotated runner authority to each proxy turn. A retained process no longer attributes later turns to its original launch run. - Update settings copy, observability documentation and capability maps. ## Verification - `pnpm -r typecheck` and `pnpm build` passed after the implementation changes. - Adapter launch/redaction: 9 tests passed. Projection and actual proxy: 12 tests passed before the review fix; the expanded projection suite passed all 12 cases after the fix. The proxy test exercises two runs in one process against a real local HTTP receiver and a fake provider. - Full OpenCode driver/projection suite: 72 passed, one existing response_wake test timed out at five seconds. A focused check then passed both existing omitted-continuation cases and the new trace export case. The timeout cause remains unconfirmed. - Runner package TypeScript and Rust `cargo check` passed. Map and UI token checks passed. - Prior journey validation: 273 tests across 12 files and the journey TypeScript check passed. - Broad local `pnpm test:run` encountered native-runtime integration failures with repeated embedded PostgreSQL initialization failures and was stopped. All 16 failing cases passed in an isolated run outside the filesystem sandbox (961 unrelated cases unselected). The broad local run is not a clean acceptance result. A later focused database-backed chat/heartbeat run passed all six selected observability cases, including native OpenCode launch and successful allocation (1,809 unrelated cases were unselected). - All CI gates passed for `21085f880d8babc9efeb22d0623db99613d33093`, including packaged-runner, clean-install, browser and build checks: https://github.com/paperclipai/paperclip/actions/runs/40884658995 . Greptile reviewed the same commit at 5/5 with no unresolved findings. - Verified OTLP JSON acceptance and indexing with an explicitly synthetic transport marker. No live OpenCode model trace or deployed smoke acceptance is claimed yet. ## Risks Export is best effort. Abrupt termination, recovered in-flight turns without a new submitted event, limits or receiver failure can lose traces. Export failure does not change task success. The projection represents observed normalized events, not raw native model spans or hidden reasoning. Known-secret redaction is not an OS isolation boundary. UI changes can break journey locators. Deployment adapters retain credential, ownership and cleanup responsibility. Both the controller and native runner bundle must be updated before live qualification. ## Model Used OpenAI Codex, GPT-6, with reasoning, code execution and browser tools. The exact serving model ID and context size are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted suites; broad local limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 08:57:12 -05:00
DottaandPaperclip	7d8d1fbc1d	fix(adapter-utils): provide a default author for local commits (#18094) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/209/changes/paperclip/209 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use managed launchers for Git and GitHub commands. > - These launchers exclude host credentials and host Git identity. > - A checkout without an author cannot save a local commit when managed GitHub identity is absent. > - A local author default belongs in the launcher that owns this isolation. > - This pull request supplies a private Paperclip default and preserves explicit and managed identity selection. > - Agents can save local work without manual author setup or extra GitHub access. ## Linked Issues or Issue Description **What happened?** An ordinary `git commit` fails with `Author identity unknown` in a fresh checkout without author settings or a usable managed GitHub identity. An agent must set its own author to continue. **Expected behavior** The managed launcher supplies a default local author. Repository and command settings override that default. A current managed identity retains its existing precedence. **Steps to reproduce** Run the managed launcher in a fresh Git repository without author settings or broker credentials. Add a file, then run `git commit -m work`. **Paperclip version or commit** Reproduced at `063cfd43813cabf18417605fb9eefb9aacf95c8d` on Linux with Node 25.9.0 and pnpm 9.15.4. **Deployment mode** Managed sandbox execution. The shared launcher also serves local and SSH execution that selects managed access. Related: #16662 changes precedence for a connected GitHub account. #8945 assigns identity to managed worktrees. This repair supplies the missing anonymous launcher default and keeps the current managed precedence. It does not require either change. ### Hypothesis and thesis for a fix **Hypothesis:** The launcher removes inherited author variables and host Git config. Without a broker identity or explicit repository settings, it has no author. Four real-Git cases reproduce the reported error on the base revision. **Thesis for a fix:** Put `Paperclip <noreply@paperclip.ing>` in a private Git config for each operation. Git treats it as global config, so repository and command settings can override it. The existing broker environment still takes precedence. Do not store a person's identity in shared checkout config, where it could remain after steering or revocation. The default grants no GitHub access. ## What Changed - Write the default author to the existing private operation directory with mode `0600`. Remove it when the operation exits. - Point managed Git at this file while keeping host global/system config, credentials and SSH identity excluded. - Add real-Git regressions for missing, absent, unavailable and rejected access. Preserve explicit overrides, identity changes and revocation controls. - Verify two shell commits through normal launcher staging and a commit after anonymous heartbeat cleanup. Check saved files, commit metadata and temporary config cleanup. - Update identity documentation and the two run map recipes. ## Verification - Red: the four new local-commit cases fail on the base launcher with `Author identity unknown`. - Green: all 80 focused tests pass in one run across `github-launcher.test.ts`, `github-launcher-environment.test.ts`, `heartbeat-github-launchers.test.ts` and `native-github-access.test.ts` (23.31 seconds). - A controlled filesystem failure first made the candidate discard valid managed credentials. After the correction, the same real launcher process retains them. Default-author file failure does not change operation-directory readiness. - Review found a second failure: a partial write can leave invalid Git config. The real-process test failed before the repair with `fatal: bad config line` from `git status`. The launcher now selects the default file only after the write succeeds. Both write-failure cases retain managed credentials, allow status, require identity when no default is available, and save a commit with explicit identity. The test reads the stored commit metadata with real Git. - Real shells, generated launchers and Git create and inspect saved commits. Sandbox transport and broker replies use controlled fixtures. No live cloud rollout or GitHub write acceptance is claimed. - The isolated test process excludes inherited task launcher paths, Git config and broker authentication variables. An initial run exposed those test-environment inputs; the isolated replay passed. The heartbeat assertion now checks the private default file and its cleanup instead of expecting `/dev/null`. - Adapter-utils typecheck and build pass. Map validation and `git diff --check` pass. - Conflict repair on 2026-10-29: merged master `fa5e70faf47ef2ca20bf2daf95b7503c94469b89` into the PR branch. Resolved 19 conflict paths by retaining master for unrelated files. All seven Git identity files remain byte-for-byte equal to the reviewed fix. The final diff is seven files, 145 additions and 15 deletions. - At head `c501e3f28369d404c91082b89188dc150c9362a9`, all 80 focused tests pass again (37.53 seconds). Adapter-utils typecheck and build pass. Map validation and migration checks pass. The PR diff passes `git diff --check`. - Full workspace typecheck and build were attempted again. Both stop because this sandbox has no Rust `cargo`. The duplicate full local Vitest run was stopped after about five minutes without a final result while full CI was running. The earlier full local attempt was stopped without a final result. No local full-suite pass is claimed. - GitHub now reports the PR as mergeable. All current-head checks have completed. [Full CI run 40878973679](https://github.com/paperclipai/paperclip/actions/runs/40878973679) passes, including typecheck, build, Rust, test shards and all eight E2E shards. The exact-head check inventory has 54 successes, one neutral Paperclip Review and two Storybook skips. Snyk passes. Greptile and Paperclip each reviewed `c501e3f28369d404c91082b89188dc150c9362a9` at 5/5 with no actionable findings. All review threads are resolved. GitHub reports the PR as clean and mergeable. - The prior CI run had one failed E2E test: `a lost creation response requires reload and cannot blindly replay commands`. Its YAML parser received page text that starts with `Skip to Main Content`. The new run passes E2E shard 7/8 and the full E2E gate. The prior failure remains part of this record. A passing run does not establish the earlier failure's cause. This change does not alter that test or the agent configuration UI. ## Risks - Commits without explicit or managed identity now use the generic Paperclip author. Operators can still set repository or command identity. - `git config --global` writes to temporary operation config and does not persist. The documentation states this behavior. - If the private config directory cannot be prepared, commits still need explicit identity. Existing redacted diagnostics and credential isolation remain. - One small synchronous file write is added to a directory the launcher already creates. No new subprocess, network request, persistent identity or authority cache is added. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, shell execution and tool use. The runtime does not expose the exact model ID or context window size to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 08:56:12 -05:00
3 people	0f6f0a3fee	fix(server): schedule issue monitors without heartbeat polling (#17863) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/56/changes/paperclip/56 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Issue monitors save a future check on an existing issue. > - The heartbeat timer still scans monitors on every tick, even when none are due. > - The saved deadline and claim already provide durable scheduling state. > - This pull request gives monitors their own deadline worker and connects state changes to it. > - Idle monitor work stops querying. Existing admission and recovery rules remain in force. ## Linked Issues or Issue Description Refs #17840, #16325. Related: #17171 addresses rearming a monitor after its admitted run fails. This PR changes when monitor inspection runs and follows the merged routine prerequisite. **What existing behavior does this improve?** Issue-monitor scheduling and idle database use. **Current behavior** Heartbeat ticks call the combined agent and monitor sweep. The existing next-deadline query only retains future monitors. Polling covers overdue leftovers, stale claims, and changes that make a saved monitor eligible. **Proposed behavior** Use the shared deadline worker. Retain overdue work with bounded retries. Wake after committed changes to monitor state, issue eligibility, company state, or native ownership. Keep legacy agent ticks separate for the next slice. ## What Changed - Add monitor-specific sweep and deadline methods to the existing scheduling owner. Keep the manual combined method. - Include overdue work and the strict five-minute claim fence. Exclude active native owners until their state changes. - Register monitors with the shared deadline worker. Heartbeat ticks wake only the legacy agent timer worker. - Connect direct issue-state writers and existing execution-control notifications to monitor work. Checkout uses a short transaction around its existing CAS. Provider work stays outside it. - Add real database lifecycle tests and retain routine, startup, admission, execution-control, and native-projection coverage. - Record the obligation, producer inventory, recovery behavior, and limits in `doc/plans/2026-10-27-issue-monitor-sleep-audit.md`. ## Verification - **1,233 passing cases in 35 complete files**, no skips, on accepted source `c24aadfc91e9394d73c799f9e05fc67202160557`. Coverage includes real monitor scheduling/admission, native settlement, committed producers, retained routines and all shared deadline consumer integration suites. - Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. Twenty-four fresh PostgreSQL 17 schemas plus per-file fixtures applied all 412 journal entries through index 413; every selected source hash and temporary fixture import was restored exactly. - Normal parent reconciliation preserves master's admitted-wake result and scoped issue deletion while keeping committed monitor notifications. All other paths match the automatic merge; packages remain byte-exact to the qualified parent. The final routine-parent publication changes only its audit. - Historical failures and corrections remain in [the monitor audit](doc/plans/2026-10-27-issue-monitor-sleep-audit.md). Old delegated-Slack rate and Runner drain CI failures are retained; the current inherited clock fixture and package source are byte-identical to merged #17835. Fresh final-head CI remains required. - No full local repository-suite, live provider, deployed suspend/wake, external-alarm, production-load or fleet-savings claim. Final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - A missing producer notification can strand a monitor after polling is removed. The audit covers ordinary writes, direct status/assignment writers, recovery, and native ownership changes. - The writer model is process-local. Independent SQL writers and other replicas need a supported notification or reconciliation contract. - Overdue work uses the configured heartbeat interval as a retry floor for the monitor worker. New notifications during that backoff wait for its next pass. Normal state-change bursts coalesce for one second. - Notification fanout can inspect monitors during other active work. An unchanged idle subsystem has no recurring monitor scan. - The legacy agent timer still polls. Cloud sleep proofs and external alarm coverage are unchanged. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, ordinary Git merges, composition proofs, PostgreSQL fixtures and complete local integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 08:40:56 -05:00
3 people	929e840595	fix(server): schedule routines without heartbeat polling (#17840) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/47/changes/paperclip/47 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Routines create work at saved cron deadlines. > - The heartbeat still wakes the routine worker when nothing is due. > - Its old next-deadline query also drops overdue work left by capped catch-up. > - Removing the heartbeat alone can strand that work and miss company reactivation. > - This pull request uses saved deadlines and committed producer changes. > - Idle routine scheduling stops querying until a change, retry, or real deadline. ## Linked Issues or Issue Description Refs #16514. Merged prerequisite: #17836. Related Cloud wake contract: #16215. **What happened?** Every heartbeat wakes the routine scheduler. The next-deadline query excludes overdue triggers. A 27-tick hourly backlog loses its deadline after the first 25-run pass. Company and budget lifecycle changes also lack routine schedule signals. **Expected behavior** Use the actual next saved deadline. Retain overdue work. Refresh after supported producer transactions commit. Keep idle schedules quiet between obligations. **Steps to reproduce** 1. Create an hourly routine with capped catch-up and 27 overdue ticks. 2. Run the registered worker once. The base processes 25 ticks and reports no next wake. 3. Create a future schedule and pause or reactivate its company. The base does not refresh its registered deadline until another wake. ## What Changed - Remove only the routine worker wake from the heartbeat tick. - Put routine eligibility and the next-deadline read in `routineService`. Include overdue triggers. Keep trigger/routine company scope consistent and require an active company. - Reuse `registerDeadlineWork` for retries, request generations, and passive drain behavior. Narrow its coordinator type to the `register` method it uses. Existing consumers keep the same runtime behavior. - Signal routine work from manual schedule invocation, company pause/resume/archive/delete, and budget-induced company pause/resume. Existing routine and trigger producers retain their transaction signals. - Stop between trigger batches during drain/shutdown. Finish a claimed batch so its advanced timestamp does not discard admitted work. - Add real PostgreSQL lifecycle tests and extend startup/query fixtures. Add a validated external database option to the company suite, including migrations and client closure. - Record producer coverage and limits in `doc/plans/2026-10-27-routine-schedule-sleep.md`. ## Verification - **417 passing unique cases across 14 complete files**, no skips, on accepted source `4659ef76d398193b15e7812d4a97f3da3b743226`. Coverage includes the real deadline worker, database eligibility queries, routine services/HTTP/routes/plugins, budget/company producers, git/device fixtures, startup and shared coordinator. - The first combined run passed 416 cases and had one UUID HTTP authorization timeout at its unchanged 20-second limit. One fresh complete 13-case E2E diagnostic passed (the affected case took 425 ms). No assertions, timeouts or production source were changed; the initial timeout's cause is not established. - Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. Fresh PostgreSQL 17 fixtures applied all 412 journal entries through index 413; all selected test hashes and temporary fixture imports were restored exactly. - Normal parent merges preserve the tested production and test source. Historical acceptance and review corrections remain in [the routine audit](doc/plans/2026-10-27-routine-schedule-sleep.md). - No full local repository-suite, deployed suspend/wake, external alarm, live provider or fleet-cost claim. Final-head CI, security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - Company pause now prevents due routine claims, matching the prior deadline query. Rows remain saved for reactivation. Existing routine reactivation, project-pause skip records, cron, timezone, catch-up, and concurrency policies remain with the routine owner. - Signals cover supported service writers in one process. Independent SQL writers and other replicas need their own integration. - This proves worker quiet, not host/database suspension or fleet savings. Cloud idle proof and external wake coverage are unchanged. - Issue-monitor and legacy-agent-timer polling remain separate duties. This slice does not finish the full migration stack or its CI/review work. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, ordinary Git merges, source-composition proofs, PostgreSQL fixtures and local lifecycle tests. No subagents were used in the current reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 08:08:04 -05:00
3 people	fa5e70faf4	fix(server): schedule automatic backups through shared lifecycle (#17836) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/39/changes/paperclip/39 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip manages agent work and its durable state. > - Automatic database backups protect that state on a configured schedule. > - Their independent interval is outside the shared deadline and shutdown lifecycle. > - An enabled backup schedule owes real work even when no queue changed. > - The existing coordinator can own this deadline without another table or notification protocol. > - This change preserves backup policy and moves scheduling into that shared lifecycle. ## Linked Issues or Issue Description Refs: #17835. Related work: #11756 (abortable backups), #3549 (stale-backup catch-up), #2120 (interval unref), #16364 (bounded shutdown steps). This slice changes scheduling; it does not implement those separate features. **What existing behavior does this improve?** Automatic database backup scheduling and shutdown ownership. **Current behavior** A standalone interval invokes backups, skips drain and warm standby, and is not registered with the shared work scheduler. Its future deadline is invisible there. The interval is not cleared or joined by the delivery coordinator during shutdown. **Proposed behavior** Register the configured backup deadline with the shared coordinator. Call the existing backup owner only when due. Keep the startup-relative phase, skip missed slots, retain the next interval after failure or a busy-owner skip, and join active scheduled work on shutdown. **Reason and benefit** One scheduling lifecycle owns overlap, admission, deadlines and shutdown. Waiting does not query the database. This is a true periodic obligation, so enabled backups still recur; it is not an empty-queue polling removal or a claim of measured savings. This existing slice now follows the merged import-spool parent #17835 and targets master. Merge authorization and exact-head CI/review gates are tracked separately. ## What Changed - Add a small backup scheduling owner using the existing coordinator and scheduler. - Replace the automatic-backup interval in startup. Register only when backups are enabled, independently of heartbeat scheduling. - Preserve the existing backup executor, manual serialization, retention reads, pre-sleep verification, restart marker and host-sleep gates. - Test deadlines, drain/standby, failures, busy-owner skips, slow attempts, restart and shutdown. Update startup-marker tests to exercise the registered deadline. - Record the audit and update inventory item 15. No schema or backup format changes. ## Verification - Current source acceptance: **130 passing cases in four complete files**, no skips: backup worker, startup checkpoints, scoped deadline work and delivery coordinator. Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. - Runtime and all four selected test sources are byte-identical to accepted source `11775eee7ed795b989cbb31052e69643c468c3cf` after normal parent reconciliation. The parent carries the tool-access test-order repair; it changes no backup behavior. - Historical broader checks and the real JavaScript backup/restore probe, including their intermediate failures and limits, remain in [the backup audit](doc/plans/2026-10-26-database-backup-sleep-schedule.md). They are not claimed as repeated current-head acceptance. - No full local repository-suite, deployed suspend/wake, pg_dump, live provider or cost-savings claim. Final-head CI, security, Greptile 5/5 and any actionable Ponytail findings remain separate merge gates. ## Risks - Enabled backups still require periodic database and disk work. This change centralizes the deadline; it does not reduce backup frequency or establish deployed cost savings. - The backup engine has no AbortSignal contract. Coordinator stop joins an active scheduled backup, so a stalled backup can delay graceful shutdown. Bounded backup cancellation remains separate work. - Restart still starts a new ordinary interval; it does not replay historical slots. The existing pre-sleep marker still requests an immediate verified restart backup and persists on failure. - A due backup waits during drain or standby and retries on the next coordinator attempt after admission reopens, within one configured interval. - The internal deadline is not a persisted Cloud alarm. Existing scheduled-backup and checkpoint sleep gates remain unchanged. The real probe qualifies the JavaScript engine only, not pg_dump or deployed shutdown latency. ## Model Used OpenAI Codex (GPT-6-based coding agent). Repository inspection, ordinary Git merges, source-composition proofs and local lifecycle tests were used for this reconciliation. No subagents were used in the current merge campaign. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 07:55:53 -05:00
3 people	1bceae0c91	Make import spool cleanup wake on changes and saved deadlines (#17835) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/38/changes/paperclip/38 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip manages agent work and the files that support it. > - Resumable company imports keep uploaded parts until apply or expiry. > - An hourly cleanup loop reads the database for every retained upload directory. > - Failed and applying imports can retain files without owing periodic cleanup. > - Existing transfer rows and file timestamps already identify real deadlines. > - This change schedules those deadlines and wakes cleanup after relevant changes. ## Linked Issues or Issue Description Refs: #17801. **What problem does this solve?** The hourly import sweep keeps querying retained uploads when no cleanup is due. It also treats a failed database read as a missing transfer row. That can cause deletion of a live upload whose directory is old. **What is the proposed solution?** Use the existing delivery coordinator with per-transfer deadlines. Keep commit-aware notifications in the transfer ledger service. Keep filesystem change notifications in the spool owner. Recover pending cleanup from existing files and rows at startup. **What alternatives have you considered?** A slower periodic sweep would still query idle history. A new job table would duplicate the existing ledger and directory timestamps. Retrying only after successful uploads would miss partial file writes that leave cleanup work. This is slice 59. Parent #17801 has merged. This publication targets master and preserves that landed prerequisite. Fresh current-head CI and review remain required. ## What Changed - Repair an inherited CI concurrency test. Observe exact PostgreSQL blockers instead of assuming lock order from two 50 ms sleeps. Keep production authorization and all fail-closed assertions unchanged. - Refresh verified-upload activity before writing bytes. Preserve the guarded cleanup claim without recording a part receipt before the file exists. - Isolate unreadable child directories so healthy cleanup continues. Use unique, self-cleaning route fixtures for external database reruns. - Replace the hourly timer with the existing scoped deadline worker. No new table or independent scheduler. - Schedule open uploads at saved activity time plus 24 hours, preserving the strict cutoff and guarded cancellation before deletion. - Keep failed and applying imports quiet while retaining their files. Clean completed/cancelled leftovers immediately. - Register run-scoped write intent before every supported ledger mutation, including nested transactions. The helper stays inside the existing ledger owner. - Wake from filesystem changes, including failed writes and removals. Reconstruct orphan deadlines from directory timestamps after restart. - Propagate database and permission errors. Only a missing directory proves absence. Read only the four ledger fields needed for cleanup. - Preserve startup-only stranded-apply recovery. The cleanup worker cannot fail live applies. Remove its filesystem subscription and join active cleanup on shutdown. - Record producer coverage and limits in `doc/plans/2026-10-26-import-spool-cleanup-sleep.md`. ## Verification Current repair source: `5d974e5eb28fc3e28b5e75e36fb18c61582348d5`. The publication adds a test-fixture repair and audit text to the previously tested import-spool source; runtime code is unchanged. **765 distinct cases pass across nine complete files:** the original 293 consumer cases remain hash-identical, and the full tool-access file passes all 472 in 20.86 seconds. Controlled evidence: a 500 ms cleanup-query delay fails all three old timing cases; observed lock barriers pass all three with the same delay and unchanged policy assertions. The artificial delay is not committed. Current-head CI's original audience-race failure is preserved. The first local full-file run also retains one separate 15-second stdio HTTP timeout; one fresh diagnostic full run passes all 472, without establishing that timeout's cause or changing its limit. Fresh CI and reviews are required on this new publication head. Accepted production source: `87549234b125464dc7fa216227fc9bfa1d93f3d7`. Publication head: `016b3af51a734ad0ddf574de701679a7bdc33d06`. Adopting the actual parent squash preserves the complete accepted child tree. The subsequent changes are the CI test fixture and its audit text. - **293 cases pass across eight complete server files, with zero skips:** spool worker 18; HTTP import transfers 25; transfer ledger 9; cloud floor 8; portability 114; startup 92; coordinator 26; scoped deadlines 1. Runtime: 12.28 seconds. - Real filesystem and PostgreSQL cases cover zero idle SQL/directory scans, actual deadlines, first/concurrent writes, nested commit/rollback, restart, failure, drain/resume, root and symlink boundaries, and joined shutdown. Three fresh PostgreSQL 17 fixtures and isolated PAPERCLIP_HOME keep the run separate. All eight source hashes remain exact; no assertion, timeout, or source adaptation. - **Static gates pass:** database build; server typecheck/build; module boundaries; migration order; profile identity. All 412 migration journal entries are present through index/tag 0413_tired_kronos. - **Composition:** 1,237 parent-only and eight owned-only paths remain exact; 11,521 identical paths and two automatic shared production results are preserved. The only manual conflict is the status ledger. Packages and other native worker services equal the parent. Its 4,192 selected cases are prior source-scoped evidence, not added to this count. - **Historical failures remain recorded:** earlier trusted-runner and Greptile service errors are not current passes. The older Ponytail Ship verdict names another head. Fresh CI, security, and exact-head Greptile 5/5 must pass before merge. - [Master integration and spool acceptance audit](https://github.com/paperclipai/paperclip/blob/016b3af51a734ad0ddf574de701679a7bdc33d06/doc/plans/2026-10-26-import-spool-cleanup-sleep.md#current-master-integration-and-spool-acceptance-2026-10-29). No full local repository, live provider, or deployed sleep claim is made. <details><summary>Earlier revision verification and failure history</summary> Receiver failure fairness: a controlled 1,100 ms batch reproduced a 30-second delay for healthy due receivers. The worker now supplies its existing per-item deadline through `nextRunAtOnError`. All 20 receiver lifecycle tests pass on the owner and combined tip `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production typechecking passes on the owner. The failed receiver retains its own backoff, and settled receivers return to zero recurring SQL. The five-second assertion budget is unchanged. See [the owning audit](https://github.com/paperclipai/paperclip/blob/7435a19f6d18f14466175d72b3b7d893ed39acd0/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch). Current-head CI and review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/1509a56a97f2ffb2ab59a9a27f6d7be16ab46569/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Inherited voice-cleanup verification: the controlled slow-provider regression and all 13 lifecycle tests passed on its owner and combined tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only the fixture clock advancement and its audit note. Current head: `7435a19f6d18f14466175d72b3b7d893ed39acd0`. This remains a single-duty PR stacked on #17801. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/47ba0a93c4b01484ad58d16bbeb5a372c76b5a0d/doc/plans/2026-10-24-recurring-server-sleep-stack.md), [2026-10-26-import-spool-cleanup-sleep.md](https://github.com/paperclipai/paperclip/blob/47ba0a93c4b01484ad58d16bbeb5a372c76b5a0d/doc/plans/2026-10-26-import-spool-cleanup-sleep.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. </details> ## Risks - Expiry and active apply must keep their guarded ledger claims. A stale read must never authorize file deletion after a concurrent resume. - Database and permission failures remain unresolved work. They cannot be interpreted as missing rows or empty directories. - Only supported ledger and spool owner writers emit these local notifications. Independent processes or SQL writers need ingress; startup recovery alone cannot wake stopped compute. - Cloud sleep blockers remain. No deployed suspension, external alarm coverage, live provider qualification, or cost savings is claimed. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, real PostgreSQL and filesystem tests, and GitHub tools. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 07:42:14 -05:00
3 people	c8b7afee06	fix(server): schedule native environment lease settlement (#17801) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/24/changes/paperclip/24 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip manages AI agents and the resources used by their work. > - Native run completion has several independent settlement duties. > - The recurring run reaper repeated environment-release discovery for completed native history. > - Existing lease rows already record releases that still need work. > - This pull request wakes the existing release owners from committed changes and schedules only unfinished release debt. > - Completed history becomes quiet while stop evidence, workspace preservation, and retention policy keep their existing owners. ## Linked Issues or Issue Description Refs: #17793, #17717. **What problem does this solve?** Periodic native finalization replay calls environment release even when no lease needs work. This adds recurring database traffic to an idle instance. **What is the proposed solution?** Register one scoped worker on the existing delivery coordinator. Discover work from native finalizations and existing environment leases. Wake it from native, lease, and settings transactions. Keep the release owner's saved pending-cleanup deadlines. Keep retries only while a concrete lease still needs settlement. **What alternatives have you considered?** A slower unconditional scan still queries idle history. A new job table duplicates existing lease truth. Replacing the compound callback would also change runtime-service and agent-status duties; those remain separate slices. This is slice 58. Parent #17793 has merged. This publication targets master and preserves that landed prerequisite. Fresh current-head checks and review remain required. ## What Changed - Keep ordinary company-only finalization hints out of environment settlement. Prove zero candidate SQL after quiet and preserve real lease delivery. - Reuse the existing durable destroy claim for run-owned cleanup, excluding concurrent attempts and fencing late receipts while preserving cleanup recovery. - Scope lease cleanup notifications to their company/run so settlement does not rediscover the entire backlog after each release. - Read only `nativeWorkspaceSync` and `workspaceExportRetry` JSON fields when discovering environment cleanup. Preserve saved run data and existing release decisions. - Reuse the environment runtime's release predicate for discovery. Add no table or independent timer. - Wake from existing finalization, projection, lifecycle, recovery, lease, and settings transaction boundaries. Simple lease acquisition now uses a transaction too. - Preserve recorded retention policy, pending-cleanup deadlines, original lease/provider ownership, and current release guards. - Re-read durable lease state after a release. A logged or swallowed per-lease failure does not discharge the obligation. - Delegate only periodic environment release. Preserve default live/manual behavior, runtime-service release, and agent-status callbacks. - Add an owner/producer/lifecycle audit and real database tests. ## Verification Accepted production source: `9596fa022b57feea026c4728c80bdc36037b804f`. Publication head: `fb252c8b0e52ce355e7d98db2e392e4831f99c61`. The actual parent squash is adopted by a normal merge with the complete accepted child tree unchanged. Only audit text follows. - **4,172 distinct cases in 64 complete server files pass**, plus **20 selected chat cases** (4,192 selected total). The main batch reports 3,440 passes and 49 skips. Both complete Stop/read files then pass 51 + 6 cases on separate fresh databases through their supported override. Their eight pure cases are counted once; skipped cases are never counted as passes. Separate process recovery passes 410; review response passes 273. The other 1,794 chat cases were not run locally. - **Quiet producer proof:** the controlled unscoped baseline fails with three candidate SELECTs after three routine company-only hints. The scoped case performs zero, releases a real eligible committed-run lease, and returns to quiet. The same test hash spans red and green. - **Static gates pass:** database build; server typecheck/build; module boundaries; migration order; current Copilot profile identity. Packages equal the accepted parent byte for byte. - **Fresh fixtures:** 39 prepared PostgreSQL 17 databases plus fresh helper-owned databases. The schema has all 412 journal entries through index/tag 0413_tired_kronos. All 65 original test hashes restore exactly; the indirect Stop fixture is also hashed. No assertion or timeout changes. - **Preserved records:** original 49 skips and their completed coverage remain documented. Parent handoff timeout and prior chat-idle failure records remain in their owning audits. This does not claim full repository, full chat, live provider, or deployed sleep acceptance. - [Current integration and native lease audit](https://github.com/paperclipai/paperclip/blob/fb252c8b0e52ce355e7d98db2e392e4831f99c61/doc/plans/2026-10-26-native-environment-settlement-sleep.md#current-master-integration-and-native-lease-quiet-proof-2026-10-29). Fresh CI, security, and exact-head Greptile 5/5 must pass before merge. <details><summary>Earlier revision verification and failure history</summary> Receiver failure fairness: a controlled 1,100 ms batch reproduced a 30-second delay for healthy due receivers. The worker now supplies its existing per-item deadline through `nextRunAtOnError`. All 20 receiver lifecycle tests pass on the owner and combined tip `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production typechecking passes on the owner. The failed receiver retains its own backoff, and settled receivers return to zero recurring SQL. The five-second assertion budget is unchanged. See [the owning audit](https://github.com/paperclipai/paperclip/blob/6f3a0b8de149fb5620f15909473eb7c4455b9824/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch). Current-head CI and review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/ae9bff1c06e34702f64a399e58b45a657c133bd1/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Inherited voice-cleanup verification: the controlled slow-provider regression and all 13 lifecycle tests passed on its owner and combined tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only the fixture clock advancement and its audit note. Current head: `6f3a0b8de149fb5620f15909473eb7c4455b9824`. This remains a single-duty PR stacked on #17793. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/7e619453567e078a7d745f28bfe8e5ede571f282/doc/plans/2026-10-24-recurring-server-sleep-stack.md), [2026-10-26-native-environment-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/7e619453567e078a7d745f28bfe8e5ede571f282/doc/plans/2026-10-26-native-environment-settlement-sleep.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. </details> ## Risks - Existing lease owners retain destruction claims, provider identity, stop evidence, retention policy, and pending-cleanup deadlines. Incorrect changes here can release a live resource or lose a cleanup obligation. - Logged or swallowed release failures remain concrete debt. Real retries must retain their saved floor across unrelated hints. - In-process signals do not cover independent SQL or other replicas without ingress. - This delegates only periodic environment settlement. Runtime-service and agent-status duties remain separate. Existing Cloud blockers remain; no deployed sleep, database suspension, external alarm coverage, or cost savings is claimed. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, real PostgreSQL tests, and GitHub tools. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 06:37:48 -05:00
3 people	829f9e7eaf	fix(server): settle Muse terminal assignments from committed changes (#17793) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/12/changes/paperclip/12 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its external-agent module owns Muse assignments, operation receipts, and worker holds. > - Running assignments and retained terminal history must not cause an empty maintenance scan every five seconds. > - Terminal run changes and operation results already have durable database transactions. > - This pull request wakes the existing maintenance worker from those commits and reconstructs work at startup. > - The worker retries only concrete settlement debt. It retains qualification deadlines and authority checks. ## Linked Issues or Issue Description Refs: #17772. This is slice 57. Parent #17772 has merged. Related: #17000, #17525, #17752, #17758. The publication targets master and preserves the merged parent. **What existing behavior does this improve?** Muse terminal assignment settlement and the existing Muse maintenance worker. **Current behavior** A running assignment can keep maintenance scanning every five seconds. Terminal reconciliation rewrites unchanged history. Native finalization also calls that reconciliation during periodic replay. Discovery excludes a revoked original binding even when its assignment still needs settlement. **Proposed behavior** Committed run changes and operation receipts wake the existing worker with the original company, run or binding identity. Startup reconstructs actionable work from existing rows. Maintenance processes bounded batches and disarms after settlement. Default live and manual callbacks remain. The periodic callback delegates this one duty. **Reason and benefit** Configured but idle instances stop querying for this obligation. Late receipts, restarts, drain and historical bindings still have recovery paths. No table or independent timer is added. **Breaking changes** No schema, endpoint or provider protocol changes. Explicit manual reconciliation retains its unbounded default. This does not alter Cloud sleep admission or claim that all instance work can sleep. ## What Changed - Add an owner predicate shared by discovery and locked settlement. Fenced private-worker uncertainty remains held without an automatic retry loop. - Preserve original binding generation and company/agent/run identity, including revoked bindings. Fence only the terminal assignment during reconciliation. - Extend existing maintenance with scoped hints, bounded batches, concrete-debt retry, startup reconstruction and quiet drain behavior. - Register notifications inside existing terminal, projection, cancellation, operation-result and send-failure transactions. Keep qualification and OAuth signals explicitly global when coalesced with scoped changes. - Serialize only four domain IDs. Real native-finalizer tests caught and prevent a circular-object serialization failure at this boundary. - Use existing restore/worktree suppression and settings reactivation. Delegate both periodic finalizer branches while retaining resource callbacks and default live/manual settlement. - Add the producer audit and regression evidence in `doc/plans/2026-10-26-muse-terminal-settlement-sleep.md`. ## Verification Accepted production source: `584015018ed4a9d57d2b0f192b75847503b54e53`. Publication head: `305014b030d6c5eb891c70bc57036f4192d4fdad`. Normal adoption of the actual parent squash preserves the complete accepted child tree. Only audit text, a migration-count clarification, and the proved Photon test extension follow. - **3,715 distinct cases in 52 complete server files pass**, plus **20 selected chat cases** (3,735 selected total). The main 49-file batch passes 3,008; process recovery passes 410; review response passes 273; the full Photon file passes 24. The other 1,794 chat cases were not run locally. - **Quiet producer proof:** moving only the Muse fanout outside its parent guard produces six candidate SELECTs and fails the regression. Keeping the guard yields zero and still settles a real terminal run. The same test hash spans the controlled red/green pair. - **Photon review evidence:** repeated active callbacks produce zero settlement hints/candidate SQL. Three concurrent attention recoveries produce exactly one of each. The existing locked attention guard was correct; production needed no fix. The parent security check accepted the resolved finding before merge. This PR keeps the additional settlement queue in the existing regression. - **Static checks pass:** database build; server typecheck/build; module boundaries; migration order; current Copilot identity. Packages equal the accepted parent byte for byte. No new live provider qualification is claimed. - **Fixtures and failures:** 34 fresh PostgreSQL 17 fixtures have 412 applied migration receipts through tag 0413_tired_kronos, plus fresh per-call helper fixtures. All 52 original test hashes restore exactly. The additional Photon test hash matches its passing proof. Earlier remote chat passive-idle failure remains recorded; both passive/missing cases pass in the new selection. Parent handoff timeout/fresh-file-pass accounting remains in its audit. No assertion or timeout was weakened. - [Integration and quiet producer audit](https://github.com/paperclipai/paperclip/blob/305014b030d6c5eb891c70bc57036f4192d4fdad/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md#current-master-integration-and-quiet-producers-2026-10-29). Fresh CI, security, and exact-head Greptile 5/5 must pass before merge. <details><summary>Earlier revision verification and failure history</summary> Receiver failure fairness: a controlled 1,100 ms batch reproduced a 30-second delay for healthy due receivers. The worker now supplies its existing per-item deadline through `nextRunAtOnError`. All 20 receiver lifecycle tests pass on the owner and combined tip `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production typechecking passes on the owner. The failed receiver retains its own backoff, and settled receivers return to zero recurring SQL. The five-second assertion budget is unchanged. See [the owning audit](https://github.com/paperclipai/paperclip/blob/f7477933469fdc960f555425dce66eff25a94e54/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch). Current-head CI and review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/bbd31dea5ef3a4875950a87b4636af8b6bde858e/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Inherited voice-cleanup verification: the controlled slow-provider regression and all 13 lifecycle tests passed on its owner and combined tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only the fixture clock advancement and its audit note. Current head: `f7477933469fdc960f555425dce66eff25a94e54`. This remains a single-duty PR stacked on #17772. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-24-recurring-server-sleep-stack.md), [2026-10-26-muse-terminal-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. </details> ## Risks - In-process signals do not capture independent SQL or other replicas. Supported external events still need ingress. - Pending assignments and operation receipts retain their original binding identity. A new agent configuration cannot take their ownership. - Genuine qualification, OAuth, and recovery deadlines remain obligations. They must not be discarded merely to make the worker quiet. - The parent coarse-hint guard prevents recurring Muse discovery; real company/binding authority changes retain their signals. - No full local repository suite, full chat suite, live Muse or Copilot qualification, deployed sleep, database suspension, external alarm coverage, or cost savings is claimed. Existing Cloud blockers remain. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, real PostgreSQL tests, and GitHub tools. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 06:18:40 -05:00
3 people	4d0847c4ca	fix(server): settle external conversations from committed work (#17772) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/6/changes/paperclip/6 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip manages AI agents and the tasks that they execute. > - External conversations must become idle after an authorized response is published and all related work is complete. > - Native replay and stranded-task recovery still discover this settlement work through recurring scans. > - A new deadline worker must preserve response ordering, monitor handoffs, and unfinished work. > - This pull request gives settlement one shared readiness predicate and activates its worker from committed changes. > - Existing rows provide recovery truth. No new table or per-feature timer is added. > - The benefit is a settlement worker that becomes quiet and resumes when relevant work changes. ## Linked Issues or Issue Description Refs #17758. This is slice 56. Slice 55 has merged. Related: #17606 (chat milestones), #17663 (bounded run reads), #17665 (monitor proof), and #13809 (answered Slack conversation lifecycle). The provider applicability remains Slack and Speko-owned conversations. This does not extend Telegram settlement from #16559. **What happened?** Conversation settlement depended on periodic native replay and generic task recovery. An exhausted monitor also cleared its schedule before recovery admission. During that gap, an answered conversation could look finished. Failed or deferred admission lost the monitor schedule. **Expected behavior** Relevant committed changes activate settlement. The worker uses the owning service's evidence and becomes quiet when no settlement is owed. Monitor recovery stays pending until admission, and restart does not admit an already accepted recovery twice. **Steps to reproduce** 1. Create an active external conversation with a successful run and an authorized published response. 2. Keep execution, approval, a goal, or a monitor pending. Let the settlement worker become quiet. 3. Resolve that prerequisite through its owning service. Verify that the conversation settles without another periodic sweep. 4. Delay or fail exhausted-monitor recovery admission. Verify that the monitor remains pending and the conversation cannot settle during the handoff. **Paperclip version or commit** Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`. Parent #17758 merged as `8ac5a128e66fb672e88106412f76cf278599bfc4`. The publication adopts that actual squash through a normal merge with an unchanged complete child tree. ## What Changed - Preserve quiet after coarse company-only maintenance hints. The shared inbound-notice and response-availability helpers keep settlement enabled by default. Only the milestone dispatcher suppresses their duplicate settlement fanout. Its scoped issue-review signal and the explicit company-authority signal still wake real work. Four database cases prove both quiet and real company-wide producer delivery. - Preserve source-generation checks for chat handoff, task-before-comment locking, issue-scoped decision invalidation, and current conversation event columns from master. - Track initial external conversation settlement through shutdown without blocking HTTP readiness on endpoint or task locks. - Reuse the real queue's coalesced recovery receipt after restart, without admitting the same monitor recovery again. - Share the existing conversation readiness predicate between bounded discovery and locked settlement. Preserve current provider policy, run/publication proof, company boundaries, and endpoint-to-issue lock order. - Register a scoped deadline worker through the delivery coordinator. Preserve startup, restore/worktree suppression, drain/resume, failure recovery, and joined shutdown. - Extend existing review, milestone, wake-notice, and finalization signals. Add notifications at independent issue, comment, goal, session, and wake-receipt unblocks. - Keep monitor owner recovery pending until guarded admission. Reuse an accepted receipt for the same monitor schedule after restart, even when its limit reason changes; retain compatibility with legacy receipt keys. Commit board escalation and its monitor clear together. - Delegate periodic native settlement while preserving resource callbacks. Keep direct/manual defaults and preserve generic recovery ordering with a read-only readiness guard. - Add the producer and lifecycle audit in `doc/plans/2026-10-26-external-conversation-settlement-sleep.md` and update the stack ledger. ## Verification Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`. Publication head: `b256bb52647affd9f5279ea8c40f48bbe40f031f`. The complete accepted child tree stays unchanged when adopting merged parent #17758. This publication adds only the audit document. - **3,129 distinct cases in 37 complete server files pass**, plus **18 selected chat-response cases** (3,147 selected cases). The other 1,796 chat cases were not run locally. Full settlement, queue, scheduling, run-state, goals, interactions, task Retry, decision targets, status reassessment, workspace, process recovery, and review consumers are covered. - **Preserved failure:** the first 35-file command passed 2,445 cases and timed out one restored-work handoff at its existing 15-second limit. That aggregate command failed. The unchanged full 89-case actions file then passed on a fresh fixture; the affected case took 22 ms. Root cause was not established. No timeout or assertion changed. The 34 other files passed originally. Distinct counts exclude duplicate actions passes. Fresh CI remains a separate gate. - **Indirect wake regression:** the original unconditional baseline failed two quiet checks. Guards alone left the chat case failing with three candidate queries. The final controlled indirect baseline fails one of four cases; the repair passes all four with the same test hash. Positive controls preserve real company-only producers. - **Source and fixture controls:** all 38 selected test files restore to their exact committed hashes after fixture imports. Fresh PostgreSQL 17 fixtures use 413 migrations. All five manual merge resolutions preserve current authority, locking, and event-column contracts. - **Static checks pass:** database build; server typecheck and build; module boundaries; migration order; Copilot profile identity. Runner packages equal the accepted parent byte for byte. No new Runner/package qualification is claimed. - [Current integration and wake audit](https://github.com/paperclipai/paperclip/blob/b256bb52647affd9f5279ea8c40f48bbe40f031f/doc/plans/2026-10-26-external-conversation-settlement-sleep.md#current-master-integration-and-indirect-wake-audit-2026-10-29). Latest-head CI and exact-head Greptile 5/5 must pass before merge. <details><summary>Earlier revision verification and preserved failure history</summary> Receiver failure fairness: a controlled 1,100 ms batch reproduced a 30-second delay for healthy due receivers. The worker now supplies its existing per-item deadline through `nextRunAtOnError`. All 20 receiver lifecycle tests pass on the owner and combined tip `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production typechecking passes on the owner. The failed receiver retains its own backoff, and settled receivers return to zero recurring SQL. The five-second assertion budget is unchanged. See [the owning audit](https://github.com/paperclipai/paperclip/blob/83204a67880d980cfbea3771cee19a3cda0022b7/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch). Current-head CI and review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/ef9ae5767430d0093efc736ae0b303ba5afaae65/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Inherited voice-cleanup verification: the controlled slow-provider regression and all 13 lifecycle tests passed on its owner and combined tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only the fixture clock advancement and its audit note. Current head: `83204a67880d980cfbea3771cee19a3cda0022b7`. This remains a single-duty PR stacked on #17758. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-24-recurring-server-sleep-stack.md), [2026-10-26-external-conversation-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-26-external-conversation-settlement-sleep.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. </details> ## Risks - Signals cover supported in-process writers. They do not supply change capture for independent SQL or other replicas. - Real company-wide changes can inspect active candidates. Coarse maintenance hints do not activate settlement. Configured idle and retained history must remain quiet. - Monitor receipt binding and admission order must preserve company, agent, issue, and exact schedule identity. Current parent handoff locks and source-generation checks remain intact. - The local restored-work timeout is retained above. A fresh full-file pass does not establish its cause; latest-head CI must also pass. - Generic recovery and Muse/resource duties remain separate slices. This does not make the whole server idle. Existing Cloud sleep blockers remain. No deployed sleep, database suspension, external alarm coverage, fleet cost, or live provider qualification claim is made. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, PostgreSQL tests, and GitHub tools. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 06:02:43 -05:00
3 people	8ac5a128e6	fix(server): resolve native finalization notices from committed wakes (#17758) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native finalization can commit before its recovery notice is resolved. > - The periodic orphan pass revisits committed runs to resolve those notices. > - Retained completed history should not require repeated discovery. > - This PR moves that one duty to transaction wakes and concrete failure retries. > - Existing notice rows reconstruct work after restart. The action service still owns resolution. > - Other completion and resource duties keep their current recovery paths. ## Linked Issues or Issue Description Refs #17752 (stack parent: committed run projection), #17748 (accepted-result commit), and #17734 (workspace finalization). These are distinct recovery obligations. This slice incorporates parent #17752 at `a49a2dac13b0e1c19da5d3d6b435c781703b1c0b`, now merged as `50fe32684d2d4e1db129e3c1bb455571344a0c09`. Related #17724 retires obsolete exact-session notices for terminal runs with no result and preserves their replay hold. This slice resolves notices after a result committed; it does not change that no-replay policy. **Current behavior** Committed native finalization recovery repeatedly inspects active watchdog notices. A notice can also be updated in place to describe a different run, so resolving a previously selected ID alone is unsafe. **Proposed behavior** Wake a scoped worker from relevant committed writes. Reconstruct active finalization notices at startup. Resolve only the current notice after the native owner rechecks its binding under lifecycle locks. Stop scheduled work when no matching obligation remains. **Reason and benefit** Notice recovery becomes quiet after settlement and reacts to real changes without a polling delay. Reused notices remain attached to their current incident. This does not claim that every remaining server duty or the host can sleep yet. ## What Changed - Track the initial recovery-notice sweep through shutdown without blocking HTTP readiness on task locks. - Require issue/run scope for ordinary finalization fanout. Routine company-only hints leave dormant notices quiet; dedicated owner and explicit settings signals retain their scopes. - Extract the native finalization notice predicate and resolution owner. Keep the existing recovery-action service responsible for writing the resolution. - Bind candidates by company, issue and run. Require a committed result and an active or escalated finalization notice. Exclude held ownership, pending status effects and unfinished run projection. - Recheck coordinator, issue, run and matching actions under locks. A newer incident cannot be resolved from a stale candidate snapshot. - Use the shared scoped worker, transaction signals, retry backoff and lifecycle handling. Add no table, per-feature timer or provider branch. - Signal action insertion, update, supersession and resolution; finalization and effect changes; run projection and ownership release; issue changes; settings activation; and the recovery route that reopens an action. - Preserve no-op resolution quiet. Serialize only the three supported notification IDs, even when a caller supplies a larger internal object. - Delegate only notice resolution from the periodic path. Keep manual/default finalization and other completion/resource work. - Add real PostgreSQL producer, concurrency and lifecycle coverage and a detailed audit. ## Verification - Current local source `730e00a7b007a36d188ca0a328ef050d3a25d1ee` incorporates the parent with a normal merge. All 1,209 parent-only, eight owned-only and 12 automatic shared paths are preserved. The two manual paths retain both chat import sets and scope the new notice fanout. The entire package tree matches parent profile 80 byte-for-byte. - Controlled real-database regression: original notice-only unscoped fanout produces three candidate SELECTs for routine company hints. The same zero-query assertion passes after restoring scoped fanout. The real notice writer then creates and resolves an action for a committed run and returns to quiet. Both phases restore production bytes exactly; the 22 cases outside this focused pattern are not counted as passing there. - The complete main command passes **1,873 cases in 21 files**, without skips, in 309.69 seconds. It includes all 23 notice-worker cases, full presentation/ownership read coverage, 88 startup cases, and the projection/result/physical-work consumers. Process recovery passes 410 in 163.54 seconds and external-chat response passes 273 in 53.48 seconds. Total complete-file acceptance: **2,556 cases in 23 complete files**, with no skips. The additional focused chat selection passes 18 cases in 13.44 seconds; 1,796 unselected chat cases do not count. Total selected passes: **2,574**. Normal merge `a0f2b3f4348b078219a53d8cc135c84202ba505e` adopts actual parent ancestry with this complete tested child tree unchanged. - The chat selection covers the real reply/notice row-lock regression, Slack and Telegram response recovery, current-authority denial cases, exact canonical response binding and internal GitHub attachment navigation. Other cases in the giant chat file remain unselected; this is not full chat-suite acceptance. - Production server typecheck, build, module boundaries, migration order and source identity checks pass. Twenty dedicated PostgreSQL 17 databases use the unchanged migration 413 schema. Embedded-only invocations use the existing fresh-per-call recovery helper. All temporary imports restore exactly and all 24 selected test hashes match after completion. No assertion or timeout is relaxed. - Parent package acceptance remains applicable: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One existing opt-in closed Pi snapshot case is skipped because its distribution root is unset. This does not qualify a live provider. - Failure history and scope are retained in `doc/plans/2026-10-26-native-recovery-notice-sleep-audit.md`. Current-head acceptance: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `2a8777bbed8b1e10810e3d6d99aae3f25b4854ef`, with no actionable findings. Prior scores do not substitute. No Ponytail verdict is present under the authorized missing-review policy. No full local repository suite, deployed sleep, expanded Cloud wake coverage or measured savings is claimed. ## Risks - Resolution takes coordinator, issue, run and action locks in the native recovery order. Contention must preserve the current incident; the real concurrent-upsert and competing-resolver cases exercise this boundary. - Process-local signals cover instrumented application writers. They do not add support for independent SQL writers or replicas without supported ingress. Startup reconstructs the existing durable rows. - Completed finalization is not proof that external conversation settlement, Muse reconciliation or physical cleanup finished. Those obligations retain their owners and recovery paths. - Discovery uses narrow keyset pages. No fleet benchmark, host suspension or cost reduction is claimed. Cloud wake coverage remains unchanged. - Parent #17752 has merged. This PR targets master at `50fe32684d2d4e1db129e3c1bb455571344a0c09`. Fresh current-head CI and review must pass before the authorized protected merge. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 05:35:01 -05:00
3 people	50fe32684d	fix(server): wake committed native run projection from durable changes (#17752) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run has separate result acceptance, task decision, run status, and resource settlement steps. > - A task decision can commit before its run status is saved. > - Recovery currently revisits committed decisions through the periodic orphan pass. > - This PR gives the existing run projection repair its own transaction-driven worker. > - Existing records reconstruct unfinished work after restart. Completed history becomes quiet. > - Other completion and resource obligations keep their existing recovery paths. ## Linked Issues or Issue Description Refs #17748 (stack parent: accepted-result commit), #17734 (workspace finalization), and #17717 (retained session cleanup). Related #17614 preserves terminal notifications. This slice is prepared on current parent #17748 at `b2efdb2dbbbef5b2227ffa2bd2a9a6cfe3867fe2`; the parent has merged as `28e38da7e6bdd6af4092dda56bc2be05bed47af5`. **Current behavior** The native recovery pass revisits committed decisions to repair the heartbeat-run status and metadata. A crash between decision and run writes creates real work, but periodic scanning also revisits retained history. **Proposed behavior** Wake projection repair after committed producer changes. Reconstruct mismatches on startup and settings activation. Retry concrete failures through the shared scheduler. Keep the run projection before resource settlement. **Reason and benefit** The repair worker performs no recurring SQL after its work settles. A real decision, effect settlement, or ownership release reactivates it. This is one worker migration; it does not establish host or database suspension. ## What Changed - Extract the existing projection predicate and write into their native-runtime owner. The finalizer keeps initial metadata enrichment and task decision authority. - Discover accepted, committed, mismatched run projections with company, issue and run bindings. Exclude pending status effects and held runner ownership. Read narrow fields instead of whole result payloads. - Use existing transaction hints, scoped delivery scheduling, failure backoff and lifecycle handling. Add no tables or per-feature timers. - Require issue/run scope for ordinary finalization fanout. Routine company-only hints leave dormant projection quiet; explicit settings and dedicated producer signals keep their existing contract. - Connect finalization, pending effects, lifecycle status writes, issue changes and worktree execution activation. Preserve outer transaction commit and rollback behavior. - Lock the current run before deciding which terminal event and completion time to save. Concurrent live and recovery projection must not announce the same transition twice. - Delegate only already-committed projection from the periodic pass. Its candidate scan uses the same pending predicate to defer remaining consequences and cleanup until projection finishes. - Add real PostgreSQL producer, race and lifecycle tests plus an owner audit. ## Verification - Current local source `a83b9449233a275176e637c765059f414305d7c1` incorporates the parent with a normal merge. All 1,208 parent-only, seven owned-only and ten automatic shared paths are preserved. The only manual conflict keeps new projection fanout inside the ordinary issue/run guard. The complete package tree matches parent profile 80 byte-for-byte. - Controlled real-database regression: the original projection-only unscoped fanout produces three candidate SELECTs for routine company hints. The same zero-query assertion passes with scoped fanout. A real committed result then repairs the run and emits one terminal event before returning to quiet. Production bytes restore exactly in both phases. Twenty cases outside that focused pattern are not counted as passing there. - The complete main command passes **1,830 cases in 19 files**, without skips, in 301.14 seconds. It includes all 21 projection cases, 85 startup cases, required quality review and heartbeat run lifecycle, plus physical-work and result-commit consumers. Separate process recovery passes 410 in 163.58 seconds and external-chat response passes 273 in 53.26 seconds. Total: **2,513 distinct passes in 21 complete files**, with no skips. Normal merge `d4b46fd29cb23126d556cc3aeb883e6b41f1a8a5` adopts the actual parent squash with this entire tested child tree unchanged. - Production server typecheck, build, module boundaries, migration order and source identity checks pass. Seventeen dedicated PostgreSQL 17 databases use the unchanged migration 413 schema. Embedded-only calls use the existing fresh-per-call recovery helper. All temporary imports restore byte-exact and all 21 selected test hashes match after completion. No assertion or timeout is relaxed. - Parent package acceptance remains applicable: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One pre-existing opt-in closed Pi snapshot case is skipped because its distribution root is not configured. No live provider qualification is claimed. - The owner audit at `doc/plans/2026-10-26-native-run-projection-sleep-audit.md` retains prior failures and their disposition. Current-head acceptance: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `a49a2dac13b0e1c19da5d3d6b435c781703b1c0b`, with no actionable findings. Old scores do not substitute. No Ponytail verdict is present under the authorized missing-review policy. - These checks do not claim a full local repository suite, deployed suspension, expanded Cloud wake coverage or measured cost savings. ## Risks - Process-local signals cover instrumented application writers. Independent SQL writers and replicas without supported ingress are not newly covered. Durable startup reconstruction remains enabled. - Projection must precede remaining committed resource settlement. The real reconciler test proves the periodic path defers cleanup while projection is pending and resumes it after repair. - Status decisions, external conversation settlement, watchdog resolution, Muse reconciliation and resource release keep their own authority and recovery. A committed result alone is not proof of completed cleanup. - Candidate discovery uses narrow bounded pages. There is no large-fleet performance or deployed sleep claim. Existing Cloud wake coverage and sleep blockers remain unchanged. - Parent #17748 has merged. This PR targets master at `28e38da7e6bdd6af4092dda56bc2be05bed47af5`. Current-head CI and review must pass before the authorized protected merge. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 05:10:16 -05:00
3 people	28e38da7e6	fix(server): wake native result commits from durable work (#17748) ... ## Thinking Path > - Paperclip manages AI agents and preserves their results through failures. > - Native completion must save workspace output before it commits the accepted result. > - The orphan recovery pass repeatedly revisits this work even when the subsystem is idle. > - Accepted results, workspace receipts, retry dates and leases already provide durable truth. > - This PR wakes a scoped worker from those writes and schedules only real unfinished work. > - It preserves existing finalization authority and keeps later projection and resource recovery separate. ## Linked Issues or Issue Description Refs #17734 and #17717. This is slice 53 of the existing sleep stack, targeting master after parent #17734 merged at `a019d00a393b57f531afe8dd8ab0b8a1820d1c4c`. A current search found no duplicate accepted-result commit PR. **What problem does this solve?** Native accepted-result recovery currently depends on periodic orphan scanning. Quiet instances keep doing work to discover that no result needs a commit. **What improvement do you propose?** Use the shared delivery coordinator with transaction-bound producer signals. Reconstruct unfinished results at startup. Preserve saved retry and lease deadlines. Remove only uncommitted-result execution from the periodic pass. **Why is this useful?** The accepted-result worker becomes quiet with no work or retained completed history. Real results and completed workspace operations reactivate it without a polling delay. This does not claim that the whole server or database can sleep yet. ## What Changed - Track the initial result-commit sweep through shutdown without blocking HTTP readiness on recovery locks. - Keep ordinary company-only finalization hints from reconstructing dormant result work. Existing company settings and dedicated owner signals remain supported. - Add a native result-commit worker. Bind candidates by company, issue, run, accepted result and a successful workspace receipt. Exclude committed, terminal, held and pending-effect work. - Signal both accepted-result writers and workspace receipt writes before their transaction changes. Extend the existing finalization fanout for retry, lease, effect and completion transitions. - Add an optional atomic due-date check to the existing finalizer claim. A lease collision or later retry does not consume a failed attempt. - Signal native retry cancellation so it removes the saved wake after commit. - Register through the existing heartbeat suppression and shared delivery lifecycle. Preserve outer commit/rollback, drain/resume, settings activation and shutdown behavior. - Delegate only uncommitted-result execution from the periodic orphan pass. Keep committed projection and resource settlement recovery enabled. - Add real producer/lifecycle tests and a detailed audit. No new tables, provider branches or per-feature timers. ## Verification - Current local source `2af88fa55f961a74340880240019f6cd31e21bf1` incorporates the parent with a normal merge. All 1,203 parent-only, nine owned-only and 12 automatic shared paths are preserved. The only manual conflict keeps the new result fanout inside the existing ordinary issue/run scope guard. The complete package tree matches parent profile 80 byte-for-byte. - Controlled real-database regression: the original result-only unscoped fanout produces three candidate SELECTs for routine company hints. The same zero-query assertion passes after restoring the scoped fanout. A real workspace receipt and accepted result then commit exactly one decision and return to quiet. Red and green restore production bytes exactly. Twenty cases outside the focused pattern are not counted as passing there. - The complete main selection passes **1,746 cases in 16 files**, without skips, in 277.97 seconds. This includes all 21 result-worker lifecycle cases and 82 startup cases, including held result recovery, HTTP readiness and shutdown joining. Separate process recovery passes 410 cases in 162.42 seconds and external-chat response passes 273 in 53.41 seconds. Total: **2,429 distinct passes in 18 complete files**, with no skips. Normal merge `998239b019defe52e7b5d64f884de4e871ccd736` adopts the actual parent squash and preserves this complete tested child tree. - Production server typecheck, build, module boundaries, migration order and source identity checks pass. Fifteen dedicated PostgreSQL 17 databases use the unchanged migration 413 schema. Embedded-only calls use the existing fresh-per-call recovery helper. All temporary imports restore byte-exact, and every selected test hash matches after completion. No assertion or timeout is relaxed. - Runner validation belongs to the unchanged parent package tree: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One existing opt-in closed Pi snapshot case is skipped because its distribution root is not configured. This does not claim live provider qualification. - Earlier Greptile 4/5 on `656f6722be1aedf3d52b7f853956e15838204278` correctly identified startup blocking on result recovery. The prior branch `108cf0dc8296bf451b721ddac55ecbcd035d00f0` already tracks its ready promise through the existing lifecycle owner. Current full startup acceptance verifies the fix in the merged context. The later Greptile service error is not a pass; current-head acceptance now passes: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `b2efdb2dbbbef5b2227ffa2bd2a9a6cfe3867fe2`, with no actionable findings. No Ponytail verdict is present under the authorized missing-review policy. - The owner audit at `doc/plans/2026-10-26-native-result-commit-sleep-audit.md` retains earlier failures and their disposition. These checks do not claim a full local repository suite, deployed app/database suspension, expanded Cloud wake coverage or measured cost savings. ## Risks - Process-local signals cover instrumented application writes in the owning server process. They do not cover independent SQL writers or another replica without supported ingress. Durable startup reconstruction remains enabled. - The initial native decision and its run projection are separate transactions. The periodic reconciler still repairs already-committed results and releases resources. This PR must not be used to infer that committed means fully settled. - Candidate discovery adds bounded, narrow reads at startup and after signals. No large-fleet query benchmark is claimed. - Existing Cloud alarm coverage, host sleep safety and resource blockers are unchanged. No deployed sleep or savings claim is made. - Parent #17734 has merged. This PR targets master at `a019d00a393b57f531afe8dd8ab0b8a1820d1c4c`. Fresh current-head CI and review remain required before merge. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 04:53:11 -05:00
3 people	a019d00a39	fix(server): wake native workspace finalization from committed work (#17734) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs can accept a result before workspace copy-back finishes. > - The periodic orphan pass repeatedly tries this recovery, including cases that require operator action. > - Copy-back needs exact ownership evidence. A generic timer cannot grant that authority. > - This pull request wakes workspace recovery from committed changes and local owner completion. > - Saved retries use the existing scheduler. Empty work, completed barriers, and operator holds become quiet. ## Linked Issues or Issue Description Refs #17717. This is slice 52 of the recurring-server sleep stack, targeting master after parent #17717 merged at `4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Related work: #17709 (status reassessment), #17687 (unsafe export recovery), and #16478 (workspace repair holds). This PR covers native workspace copy-back and validation. Result commit/projection and resource settlement remain separate recovery duties. It does not claim full-instance sleep. ## What Changed - Register a company/run/task-scoped workspace worker. Reconstruct accepted, unfinished work from existing records. Use saved retry and lease deadlines. - Notify from both accepted-result writers (control-plane and PRP coordinator), workspace receipts, coordinator changes, operator confirmation, retained workspace repair, and relevant task updates. Notifications follow the outer transaction. - Distinguish active local ownership, busy external ownership, pending Cloud evidence, operator-required evidence, and changed bindings. Only concrete external ownership/evidence waits receive a scoped retry. - Preserve advisory locks, signed stop proof, exact binding checks, physical callback joins, and existing failure policy. Join physical operations during shutdown. - Respect the existing heartbeat execution-suppression policy before discovery and execution; wake suppressed work on settings activation. - Skip physical workspace recovery in the enabled periodic orphan pass. Keep direct/startup reconciliation and the remaining result/cleanup recovery paths. - Keep routine company-only finalization hints quiet. Ordinary work requires issue/run scope; explicit settings and owner-release events keep their company-wide contract. - Preserve the parent's entire Runner/package tree, profile 80 and all historical identities. Add the dated obligation, producer, lifecycle and limitation audit. ## Verification - Current local source: `4059b24e3ea5c623be54ade18ec9b6ab78eb0289`, prepared by a normal merge of the current parent. The sole manual conflict keeps ordinary workspace fanout inside the existing issue/run guard. All 1,198 parent-only, 15 owned-only and 10 automatic shared paths are preserved. The complete package tree matches the parent byte-for-byte. - Controlled regression: the original workspace-only unscoped fanout performs three candidate SELECTs for routine company hints. The same zero-query assertion passes with scoped fanout. A real accepted result then produces one successful workspace settlement and returns to quiet. The failed baseline remains in the audit. - The first complete local group passes **1,693 cases in 14 files**, including all 19 workspace-worker cases. It covers both accepted-result producers, physical ownership, export retry/failure, finalization recovery, arbiter corpus, issue recovery actions, control-plane port, workspace recovery, heartbeat recovery, startup, retained cleanup and native execution. - Separate process-recovery and external-chat response files qualify the changed recovery/settlement paths without sharing their general test database with other files. Both pass in full: 410 process-recovery cases in 162.79 seconds and 273 external-chat cases in 53.79 seconds. The total is **2,376 distinct passes in 16 complete files**, with no skips. Normal merge `5c46c52873158529736b8349e4d44658865108ee` adopts the parent squash with this entire tested child tree unchanged. - Production server typechecking, build, module boundaries, migration order and source-profile checks pass. Thirteen dedicated PostgreSQL 17 fixtures use the unchanged migration 413 schema. Embedded-only paths use the existing recovery helper with fresh per-call fixtures. All temporary imports are restored byte-exact. No test assertion or timeout is relaxed. - Runner validation belongs to the unchanged parent package tree: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One existing opt-in closed Pi snapshot case is skipped because its distribution root is not configured. These are source-scoped checks, with no live provider qualification claim. - The owner audit at `doc/plans/2026-10-26-native-workspace-finalization-sleep-audit.md` preserves earlier failures and their disposition. Current-head acceptance: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `7df5c9c44ea9e171be3c61b4430d3727b10f3337` with no actionable findings. Earlier scores do not substitute for this result. No Ponytail verdict is present under the authorized missing-review policy. - No claim of a full local repository suite, deployed app/database sleep, expanded Cloud wake coverage or measured cost savings is made. ## Risks - Supported wake producers are in-process application services. This adds no cross-replica or independent SQL-writer transport. - A real unresolved export still needs recovery. Busy owners and pending Cloud stop evidence retain a 30-second scoped check. No timeout authorizes copy-back. - Result projection and resource release still use their existing recovery. Some existing cleanup failures remain best effort; this PR does not infer cleanup success from a workspace barrier. - Cloud alarm coverage and host-sleep safety remain unchanged. No deployed suspension or cost savings have been measured. - Parent #17717 is merged. Current base is master at `4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Fresh current-head CI and review must pass before this slice merges. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 04:40:01 -05:00
3 people	4c56702e06	fix(server): schedule retained native-session cleanup (#17717) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs can retain a local session after an interrupted durable suspend. > - The orphan reaper repeatedly scans completed history for that cleanup duty. > - Idle scans keep the database active. Removing them requires complete wake and ownership paths. > - This pull request moves retained-session cleanup to committed signals, local release events and saved deadlines. > - Existing physical proof checks still decide whether cleanup is safe. The shared scheduler does not grant execution authority. > - The benefit is quiet retained history with recoverable cleanup and owned shutdown. ## Linked Issues or Issue Description Refs #17709 and #18135. This is slice 51 of the existing recurring-server sleep stack. Both referenced dependencies have merged. The target is `master`. **What happened?** The periodic orphan pass scans retained native sessions even after their cleanup is complete or no automatic action is safe. Some eligible sessions also depend on a local session, environment lease or original maintenance callback releasing ownership. An earlier decision notification can be consumed before the run's later terminal projection commits. **Expected behavior** Reconstruct existing work at startup. Wake from committed writes and actual owner release. Register the saved claim expiry or a probe for a specific live process. Stop recurring discovery when no obligation remains. Preserve original callback ownership and all existing cleanup proof checks. **Steps to reproduce** 1. Retain completed native runs with a durable-suspend close failure. 2. Let the periodic orphan pass run with no new work. 3. Observe repeated cleanup discovery. Separately, delay terminal projection after its decision notification is consumed and observe the missing cleanup wake. **Paperclip version or commit** Parent #17709 at reviewed head `3b75f4bc1da1370304f069fc287af3d693575716`, landed as `df6156c2e37f542a69f393dde31e357b82b7cac1`. Current integrated master: `31b927e3013a8a4b3266854b740cf40a24753788`. ## What Changed - Move scoped, paged discovery to the retained cleanup owner. Preserve explicit manual recovery. - Register the duty with the shared scheduler and remove it from the periodic orphan pass. Keep concrete lease expiry and live-process probe obligations. - Use typed outcomes for settled/refused work and process, session, environment, maintenance or lease waits. Keep physical proof and quarantine decisions in the executor. - Wake from committed finalization and terminal projection writes, durable process/lease release, and local session or callback settlement. Routine company-only finalization hints leave dormant cleanup quiet. - Join duplicate cleanup and serialize inspections on one database lane. Keep original Runner callbacks owned until settlement and shutdown. - Preserve all master profile fixtures and the branch's historical identities. Save the colliding sleep profile 79 separately. Generate combined Copilot profile 80 from its source and retain master's other provider profiles. - Add the owner audit and regression coverage. No new table, migration or independent maintenance timer is added. ## Verification - Server acceptance: **2,342 distinct passing cases in 18 complete files**. The 14-file unaffected group contributes 1,532 passes. Full startup contributes 76, process recovery 410, environment service 51, and external-chat response 273. The 29-case cleanup worker suite and 977-case executor suite run in full. No case in these files is skipped. - Controlled scope regression: removing only the ordinary finalization issue/run guard wakes dormant retained cleanup; the same assertion passes after restoring it. A real local owner release still resumes the durable candidate. - Preserve the first broad failure: 1,607 passed and one startup shutdown case failed because the coordinator mock lacked `stop()`. Add only that mock method. All 76 unchanged startup cases then pass. Do not treat the first broad command as green. - Dedicated PostgreSQL 17 databases use the existing migration 413 schema. Embedded-only fixtures use the existing recovery helper with fresh per-call databases. All temporary imports are restored byte-exact. No assertion or timeout is relaxed. - Current-master integration preserves both complete source histories. The server, UI and database trees are identical to the server-tested source. All master historical profile fixtures remain byte-exact. The prepared sleep profile 79 remains independently inspectable. - Current runtime validation at `d08f764e3efe75e96929173afe7ee8739121ad9f` passes all 12 commands (22 installed ACPX/Grok contract cases and 26 native-provider Rust cases): `pnpm install --frozen-lockfile --ignore-scripts`; Copilot/Grok identity and generated profile checks; Runner and server typechecking; server build; Copilot/Pi installation, profile compatibility and transport settlement files (130 passed; one opt-in closed Pi snapshot case skipped because its distribution root is not configured); installed ACPX model/reconnect package contract and Grok identity tests; Rust native-provider backend tests; module boundaries and migration order. - Earlier 94-case installation/settlement acceptance applies to the prepared profile 79 source. It is retained as history and does not substitute for the current profile 80 validation above. - Failure history and duty-specific evidence remain in `doc/plans/2026-10-26-retained-native-session-cleanup-sleep-audit.md`. Exact-current-head acceptance: all 60 checks are clean, including protected CI and security; Greptile gives 5/5 for `281d579de0d08e907194b8b790a0597b07d16b7b` with no actionable findings. No Ponytail review is present under the authorized missing-review policy. - These checks do not claim a full local repository suite, live Copilot or Daytona qualification, deployed host/database suspension, or measured cost savings. ## Risks - Missing a supported producer could strand cleanup. Tests exercise real finalizer, heartbeat, environment and process writers. Process-local signals cover in-process writers; no multi-replica or external SQL notification transport is added. - A timeout can leave an original callback active. Cleanup and shutdown must keep waiting for it. The existing global maintenance drain is conservative across roots. - A retained live process still has a scoped 30-second absence probe. A saved lease keeps its actual deadline. Operational errors retain retry work. Immutable proof refusals and operator-required histories do not gain automatic retry authority. - Cleanup keeps all existing artifact, identity, claim, no-launch and quarantine checks. Terminal status alone does not authorize physical cleanup. - This PR does not expand Cloud alarm coverage or relax host sleep eligibility. It proves worker behavior, not deployed app/database suspension. - The Copilot source identity has changed; these checks do not claim new live provider qualification. - The parent has merged. This PR is merged only after its current-head protected CI and review gates pass under the existing campaign authorization. ## Model Used OpenAI Codex, GPT-6 family, with code execution and repository tools. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 04:24:57 -05:00