1916 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	124b2cd0ad	fix(server): certify background deadlines for idle tenants (#19001) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Background services own future checks, retries, and recovery work. > - An idle tenant can stop only when a durable wake covers its remaining duties. > - The current alarm catalog covers several domains but leaves these owners blocked. > - A saved future timestamp alone cannot prove that reconstruction or admitted work has settled. > - This pull request binds durable domain snapshots to the actual registered worker revisions. > - The tenant can hand off supported future duties and retain all checks for uncertain or active work. ## Linked Issues or Issue Description No matching public issue was found. Related work: [domain deadline handoff #18656](https://github.com/paperclipai/paperclip/pull/18656), [owned blocker diagnostics #18632](https://github.com/paperclipai/paperclip/pull/18632), and [completed history proof #18646](https://github.com/paperclipai/paperclip/pull/18646). The separate [native idle-session change #18988](https://github.com/paperclipai/paperclip/pull/18988) covers runtime retirement. [Far-future retry re-probes #13534](https://github.com/paperclipai/paperclip/pull/13534) changes recovery responsiveness after an upstream success; this change preserves existing retry decisions. **What happened?** Future status-card, external-object, skill-sync, watchdog, retry, and quota duties lacked complete owner-bound coverage in the idle deadline proof. A completed quota task could also retain its monitor policy and timestamp and remain blocked after the scheduler had disarmed it. **Expected behavior** A settled registered owner can hand off its exact supported future deadline. Active work, dirty reconstruction, transaction uncertainty, changed revisions, claims, unknown state, and unsupported duties keep the tenant awake. Completed history retains its data and stops recurring work when the owner has no remaining duty. **Steps to reproduce** 1. Produce a supported future duty through its normal service. 2. Let the registered worker finish reconstruction and admitted work. 3. Request an owned idle report with the negotiated domain deadline catalog. 4. Check the saved earliest deadline and generation before durable alarm registration. 5. Stop and reconstruct the worker. Check that ordinary admission and claims resume useful work once due. 6. Complete a quota task. Keep its saved policy and timestamp. Check that linked unfinished work still blocks sleep and that settled history does not. **Paperclip version or commit** The owned patch was qualified at `699c8569fa34338dd75cdc5731381ec1e2efdb45`. Publication head `491fac368ee72eaac8ae105d53e6bf6b305d8a6a` normally merges actual master `039e0dad2c51c80251e1302a19c5566e039c357a`. All 43 owned files are byte-identical. All 29 new upstream paths are preserved. The later normal current-master integration is `8c912e1ba4fc66397a7341ade59373af8075068f`, described separately below. **Deployment mode** Source checkout with real PostgreSQL 17 and a controlled local Node process. No deployed alarm, tenant suspension, database suspension, or fleet result is claimed. ## What Changed - Add registered-worker reconciliation revisions and invalidate them on stop. - Keep status-card, external-object, and skill deadline interpretation with their owners. - Reuse the watchdog classifier against the supplied database snapshot. - Validate the exact saved retry run and paired queued wake. - Validate legacy quota source bindings and the current assignee or reviewer policy. - Preserve common issue scope, dependency, sibling, and state checks for covered waits. - Compose normalized owner snapshots within the existing bounded deadline proof and recheck revisions after inspection. - Bind actual service and worker instances from server startup. Missing bindings preserve existing blockers. - Retain completed quota policy and timestamps while independently rejecting linked unfinished runs or wakes. - Block unconsumed budget evaluation inputs through the owning queue predicate. Accounting policy and settlement stay unchanged. - Bind completed quota history through fixed-size UUID-array parameters, preserve exact company/task/source associations, and guard external references before the run-ID join. - Batch watchdog subtrees and shared classifier facts without changing the 64-query or two-second proof budget; preserve exact company pairing for first-run evidence. - Extend production-path regression controls and document supported coverage and limits. ## Verification Current integration head: `688abc6cac6016b9fb50e5832f81e2975ed5d49f`. Fresh exact-head checks completed with **58 successes, three intentional skips and one neutral review**, including aggregate `ci / verify`. The complete current Greptile review is **5/5**. The complete current Ponytail verdict is **Ship (4/5)**, with only optional cleanup of an unused MCP proof input; both prior correctness findings are resolved. All review threads and nested pages were inspected. The normal master merges preserve the registered retry deadline owner and current startup completion log after the same readiness await. The earlier focused evidence below remains tied to its tested source revisions. Current-master integration `8c912e1ba4fc66397a7341ade59373af8075068f`: - Normally merge actual master `46fb933b0155d183ce64a8bdc4950d1b58c926f6`. Every nonoverlapping tree blob retains its owning parent across 14,044 paths. All 42 owner-only files and 15 critical binding/proof/readiness files remain byte-identical. Three automatic overlaps preserve the existing watchdog binding, conservative unavailable-reader fallback and optional actual owner arguments. The two scoped-deadline conflicts retain the entire upstream resource-reservation/mid-pass admission/abort algorithm inside the existing reconciliation revision/running/failure fence, with both parent test sets. - Actual TypeScript 7.0.2 server noEmit passed in 9.434 seconds, normal source declaration layers in 12.102 seconds, and four selected test/helper/config layers extending the actual CI configuration in 3.167 seconds. No compiler option weakening, heap override, donor declarations or SDK replacement stub. - Focused PG0 controls passed: 102 scheduler/proof/owner controls in 16.451 seconds, including all 21 scoped-deadline cases; 121 startup controls; and all 107 unchanged instance-settings route cases in 6.456 seconds. The latter first hit a local prewire-guard listener-registration race. Only the ignored guard was corrected to register the same actual OS-bound server port immediately after native listen returns. Closed, unknown, foreign, offhost and PG destinations remain blocked. Original failure evidence is retained; product code and assertions were unchanged. - The exact current UI policy command now reports zero findings (0.362 seconds); its seven scanner controls pass. The earlier dependent PR used an old scanner/source baseline under a newly enforcing trusted workflow and reported 397 findings. Carrying the genuine current master resolves that mismatch without editing unrelated UI sites or relaxing the gate. - Every meaningful worker checked its executing PID/thread/preload hash and matching receipt. PG0 controls opened no PostgreSQL sockets; all owned processes exited. Original dependency-cache/DNS, ignored-harness syntax/type and guard failures remain separate from product results. - This is fresh semantic and focused runtime qualification, not a local whole-repository or current-head PostgreSQL acceptance rerun. The upstream workspace-busy parked-retry/terminal-holder notification path remains unchanged by this merge and needs its separate current-graph producer check. Historical six-case/scale results below stay tied to their measured sources. A matching controlled default-cadence follow-up is in #19191; no fresh current-head two-cycle or deployed suspension claim follows from this merge. - All 59 exact-head check contexts now pass (55 success, one neutral, three intentional skips), plus the legacy security status. The complete edited Greptile review is 5/5 for this exact commit, with all root/nested pages inspected and no unresolved threads. Ponytail stopped without completing its new turn; it supplied no new verdict or finding. Prior actual findings remain addressed. Earlier scores/checks below belong to their explicit earlier commits. On frozen clean merged source `699c8569fa`: - Real PostgreSQL acceptance: **6/6 passed**,83.483 seconds wall. Actual retry producer creates the exact saved run/wake; stopped and reconstructed owner resumes through current admission, one durable start, real local process output and normal task completion. Includes cancellation, drain arrival, retained quota history with queued-run/wake negatives, watchdog review settlement and budget-input empty/pending/consumed control using the actual owner. - Affected coordinator, deadline, scoped-deadline, recovery and composed-proof controls: **78/78 in five files passed**,32.114 seconds wall, worker 1/no file parallelism. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: **passed**,17.467 seconds. Native TypeScript 7.0.2 and Node 24.22.2. - PostgreSQL tests use one dedicated test database, matching fixture/admin URLs, pool limit four, one worker, and no embedded fallback. Every run recorded unchanged source hashes and a clean checkout. - The eight-second real-clock retry is a domain execution check. It is below the Cloud minimum wake lead and is not a positive accepted hosted sleep cycle. - The completed quota-history product failure and original fixture/protocol failures were preserved; the owning-reader fix passed a real PostgreSQL red/green regression before the fresh merged six-case pass. - After runtime completion, the dedicated database had zero other clients and no owned runtime processes. All three qualification process PIDs exited. The coordinator independently audits and drops the database. - Comment-only cleanup has equal strict TypeScript parser ASTs across 15 files. The final lexical audit finds no added comments relative to actual master. `git diff --check` passed. - Earlier retry fixtures omitted responsible-user membership, advanced JavaScript time past the real PostgreSQL lease clock, or produced stdout without normal task completion. Corrected fixtures use real membership, an eight-second real deadline, and ordinary task comment/completion. The last case exposed legitimate disposition repair rather than duplicate retry promotion. - The quota and watchdog controls first ran before admitted work or the existing 30-second reconstruction duty settled. The stronger tests require an interim blocked report and then exact settled-owner proof. Production deadlines were not changed. The separate completed-quota-history defect was fixed in its reader and verified with linked unfinished-run and wake controls. - The status/object and skill owners reviewed the actual final startup, route and post-backup bindings. Their domain source evidence carries where token-equivalent. The master coordinator coalescing change was reviewed separately. Historical domain PostgreSQL results are not presented as fresh full-suite results on this head. The later master merge adds retry predecessor cleanup checks under the existing issue claim lock. The source audit confirms unchanged saved deadlines and run/wake identities. The three actual retry lifecycle/cancellation/drain cases passed on the batched candidate with this parent (31.817 seconds). Those local dependency controls preceded the current exact-head remote run. Native backup preservation and TestOps changes from master are retained. No pass is inferred for changed dependency behavior solely from unchanged owned files. Review repair published as `643f1e7400`: - The original reader returned `unknown` after seven actual watchdog reviews settled across seven companies. The unchanged composed-report regression passed after batching; the final reader passed in 61.841 seconds, including active queued-work and stopped-owner rejection. - A separate two-company red/green control rejects terminal-run context from another company as first-run grace evidence. Shared facts are serialized once, with per-root IDs and review summaries. - The affected recovery, proof and diagnostic unit controls passed 47/47 (14.575 seconds). Startup owner-contract fixtures passed 121/121 (10.589 seconds). - Diagnostic PostgreSQL controls passed7/7 (12.406 seconds), including the normal budget-input producer and its exact blocker reason. Initial empty-state setup uses the same cleanup as per-case teardown when a dedicated test database is reused. - Actual repository `pnpm exec tsc --noEmit -p server/tsconfig.json` passed 12.710 seconds with no heap override. Final source has no added code comments; whitespace checks pass. - The original query-budget failure, two-company discriminator, initial cross-suite fixture failure and an interrupted incorrect local-driver selector are retained separately. The interrupted run is not a pass. Independent cleanup found zero owned clients or runtime processes. Quota-scale repair published as `6696a37f8a`: - The exact 10,000-row completed-history case reproduced `MAX_PARAMETERS_EXCEEDED`:90,002 bound parameters exceeded the driver's 65,534 limit, and the composed report returned `unknown` despite settled owners. Original failure and actual driver diagnostics are retained. - Fixed pending-work queries use three typed UUID-array parameters plus the result limit, keeping exact company/issue/source tuple bindings. The primary run-ID column stays uncast; a guarded CASE validates the raw external reference before conversion. The 10,000-row cap and 64-query/two-second proof budget are unchanged. - The same 10,000-row composed-report case passed 18.860 seconds, with unfinished run/wake rejection, malformed-reference safety and conservative 10,001-row overflow. Its fixture derives retained history from the actual quota producer and normal task completion, expands valid persisted issue/source pairs, then starts the actual registered owners. - The existing actual quota producer case passed 14.187 seconds, including exact future deadline, retained completion history, unfinished work, real foreign-company source references and noncanonical/malformed references. - Affected owner/proof controls passed 22/22 (26.444 seconds). Production server noEmit passed 16.039 seconds. - CI on 643 identified TS7034/TS7005 in the settled-watchdog fixture's initially untyped collection. It is now explicitly typed. A separate affected-entry semantic check extending the repository's actual `server/src/__tests__/tsconfig.json` passed 15.093 seconds: six affected tests, two owner files, ambient declarations and current imported source, with unchanged strict/module/lib/path settings. The production server config excludes that test directory and is not presented as test-layer proof. No custom suppression or replacement SDK stub was added. - All runs were sequential with pool max 4, worker 1, source/config/command hashes and cleared inherited provider/LIVE/other-database variables. Cleanup found zero owned database clients or local runtime processes. At prior head `6696a37f8a5718cd0f79054969a0084462d41d47`, all required CI gates pass. The exact-head inventory has 55 successes, one neutral review and three intentional skips. Greptile is 5/5. The complete latest Ponytail verdict confirms the quota finding is fixed; its only remaining note is optional unused MCP proof state, not a behavior defect. All three review threads are resolved. The earlier 643 result remains historical. The repository-wide typecheck, tests, build and browser gates passed in the prior `6696a37f8a5718cd0f79054969a0084462d41d47` CI run. The new `8c912e1ba4fc66397a7341ade59373af8075068f` full CI also passes. They were not run as a full local workspace command in the focused PostgreSQL lease. Local evidence above uses bounded external databases and does not claim embedded-cluster, live-provider or deployed-alarm qualification. ## Risks - An unsupported or uncertain owner state must remain blocked. An optimistic exemption could lose work while compute is stopped. - Changes during reconstruction or inspection must invalidate the proof. A memory-only timer is not durable alarm coverage. - Current authorization, budgets, cancellation, accounting, and resource ownership still apply on dispatch. The proof does not grant execution authority. - The supported writer model uses in-process transaction signals and startup/resume reconstruction. Independent SQL writers and other replicas need an explicit wake contract. - Native, review, coalesced, and taskless retry variants remain conservative until their owner cases qualify. Reviewer quota and hosted alarm qualification remain open. - The shared catalog remains opt-in. Legacy callers and missing owner bindings retain their old blockers. ## Model Used OpenAI `gpt-6.1-sol` was the configured model for the delegated service implementations and reviews. OpenAI Codex, based on GPT-6, performed managing review and integration. Both used repository tools and code execution. The exact served backend snapshots and context-window limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass on the final merged head - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — exact current-head run passes - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — complete current-head review and all paginated threads inspected - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 13:48:55 -06:00
Nicky LeachandPaperclip	1391f64551	Add gated release/resume methods to plugin Runner owners (#19584) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core owns Runner tasks through a lease-bound plugin owner. > - The SDK already defines release and resume in #19468. > - The owner still expects an accepted result for both operations, so it cannot use those SDK results. > - This pull request adds both methods behind the existing capability, warm-task and account checks. > - It uses the SDK parser to verify the result and task ID, and preserves the owner's cancellation checks. > - This prepares the owner boundary for later lifecycle integration. It does not start automatic release or resume. ## Linked Issues or Issue Description **Problem or motivation** The plugin Runner owner cannot forward the existing SDK release/resume pair. Its local result mapping rejects their valid results. Later lifecycle code needs both operations on the same checked task owner. **Proposed solution** Pass the existing description capability to the owner. Add explicit release and resume methods and use the SDK's operation-result parser. Require warm-task support and a selected account. Reject calls and late receipts after owner cancellation or detachment. **Alternatives considered** Adding automatic release here would also require connection quieting, settled-event drain, pending-ticket retry ownership and safe disposal. Keep those changes together in the next integration. Do not add a new flag, schema, timer or lifecycle mode here. Related: #19468. I checked ROADMAP.md and found no duplicate planned item. An open-PR title search for release/resume found no overlapping pull request. ## What Changed - Add the capability-gated owner facade for Codex and ACPX. Resume returns only `resumed` or `unavailable`. - Use the SDK parser for exact task-ID and operation-result validation. - Pass `releaseResumeVersion` from the checked description and test that it does not cause an automatic operation. - Cover account-preserving owner rebind, missing prerequisites, malformed results, ambiguous errors, cancellation and late replies. - Document the caller's remaining settlement, connection, ticket and attachment duties. ## Verification Passed locally on Node 26.12.0 with the pinned workspace dependencies: - In `server/`: `pnpm exec vitest run src/services/native-runtime/plugin-runner-task.test.ts` — 74 tests passed. - In `server/`: `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts -t 'uses a task-only'` — 3 cases passed; 1,160 unrelated cases were not selected. - Built the real workspace SDK dependencies and Runner TypeScript. A focused strict TypeScript program for the owner and its tests passed. - `git diff --check` and the added-line secret/PII scan passed. The first root test command selected a nonexistent Vitest project name and ran no tests. Running the test from the server package fixed the command. The first focused TypeScript setup could not find Node types; pointing it to the package's installed type definitions fixed that setup. The full regression suite and release checks run in GitHub. ## Risks The facade has no automatic caller. Existing warm timeout, bootstrap, outbound connection and pool behavior stay in place. No plugin capability is advertised or task control channel enabled by this pull request. Future integration must serialize release with new work and disposal. It must quiet the owned connection, keep the exact pending ticket after an ambiguous reply, authenticate the source again and finish ordinary run attachment before new work. This change does not qualify provider parking, cold recovery, retention policy or deployment. ## Model Used OpenAI GPT-6.1 Sol, exact model ID `gpt-6.1-sol`, High reasoning. Used code execution and targeted tests. The context-window size was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 11:42:32 -08:00
Nicky LeachandClaude Opus 5.5	7dfdc97b9f	fix(deps): let Dependabot update the sandbox providers (#19489) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox providers (`packages/plugins/sandbox-providers/*`) are standalone published packages. `pnpm-workspace.yaml` keeps them out of the root workspace on purpose. > - `.github/dependabot.yml` watches only `/`, so Dependabot never saw the providers. > - Because of this, the Kubernetes provider stayed on zod 3 (GHSA-mrhx-c7xc-m3w5) after the rest of the repo moved to zod `^4.6.6`. #18839 fixed it by hand. > - We do not want to keep these versions current by hand in many places. > - This pull request lets Dependabot update the providers, and keeps each provider standalone. > - The benefit is that a provider dependency that falls behind gets a pull request that can pass CI. ## Linked Issues or Issue Description Refs #18839 (adds `scripts/typecheck-sandbox-providers` and fixes Kubernetes zod by hand). Refs #18998 (a Dependabot pull request for the Pi distribution that fails CI by design). **What existing behavior does this improve?** Dependabot version updates. They skip every package that the root pnpm workspace excludes. **Subsystem affected** CI and dependency automation: `.github/dependabot.yml`, `scripts/typecheck-sandbox-providers`. **Current behavior** Dependabot does not update the sandbox providers. A provider can stay on an old or unsafe version, and nobody gets a pull request. **Proposed behavior** Dependabot updates the providers each week. Each pull request changes the provider manifests and the lockfiles that CI installs frozen, so the pull request can pass CI. **Reason and benefit** We stop having to find and fix old provider dependencies by hand. ## What Changed - `.github/dependabot.yml`: add an npm entry for `/scripts/typecheck-sandbox-providers`. Dependabot treats this workspace as one unit. One pull request per dependency changes every provider that uses the dependency and the shared `pnpm-lock.yaml` together. So the entry needs no `groups`. One entry per provider would open a separate pull request for each provider, and each one would fail the frozen install in CI. - `.github/dependabot.yml`: add an npm entry for `/packages/plugins/sandbox-providers/daytona`. Daytona is the only provider with its own lockfile, and the release build and the Daytona end-to-end job install it frozen. Each lockfile has exactly one Dependabot entry, so every pull request changes a manifest and its lockfile together. - Take Daytona out of the shared typecheck workspace. This pull request does not commit `scripts/typecheck-sandbox-providers/pnpm-lock.yaml`: since #19522, only the lockfile refresh workflow writes it. After merge, the bot removes Daytona's packages from it (no other version changes; checked with pnpm 9.15.4). Until then CI resolves it in place. `scripts/typecheck-test-programs.mjs` now installs Daytona from its own lockfile (`LOCKED_PROVIDERS`, with `--ignore-workspace`). A stale lockfile is resolved in place for the run, for both installs. - `.github/dependabot.yml`: add a comment that tells why the Pi and Kimi runtime distributions have no entry (see Risks). - `scripts/typecheck-sandbox-providers/pnpm-workspace.yaml`: list each provider by its full path. Dependabot does not expand a workspace glob that starts with `../`, so with the glob it saw no providers. - `scripts/typecheck-test-programs.test.mjs`: add tests. They check that the workspace list has no glob and matches `SANDBOX_PROVIDERS` without `LOCKED_PROVIDERS`, that each locked provider has its lockfile, that a locked provider installs frozen with `--ignore-workspace`, and that a stale locked-provider lockfile is resolved in place. - Add `.npmrc` (`auto-install-peers=true`) in `scripts/typecheck-sandbox-providers` and in the Daytona provider. Without it, Dependabot reads the root `.npmrc` (`false`). That rewrites the lockfile settings, and the frozen install then fails. npm and pnpm never publish `.npmrc`. - `scripts/typecheck-sandbox-providers/package.json`: add `"packageManager": "pnpm@9.15.4"`, the version CI uses. Without it, Dependabot guesses pnpm 10. - `doc/DEVELOPING.md`: say that contributors do not refresh or commit Daytona's lockfile either, and that a new provider goes in both lists. No provider moves into the root workspace. There is no pnpm catalog. The root `pnpm-lock.yaml` does not change. ## Verification I ran the real Dependabot updater on this branch with the Dependabot CLI (`dependabot update ... --local`, updater image `dependabot-updater-npm@sha256:c5a9451c…`): - Before the change, the workspace entry loaded only `cloudflare/bridge-template` (the one path without a glob). It skipped all the other providers. - After the change, `--dep typescript` gave one pull request. It changed the 9 non-Daytona provider `package.json` files and the shared `pnpm-lock.yaml`, and did not touch Daytona. - `--dep wrangler` changed only the wrangler closure in the shared lockfile. Before the `.npmrc` fix, the same run changed `autoInstallPeers` to `false` and 757 lockfile lines. - The Daytona entry (`--dep @daytonaio/sdk`) changed Daytona's `package.json` and its own lockfile in one pull request, and kept `autoInstallPeers: true`. It keeps the exact pin exact: `0.203.0` becomes `0.227.3`. - Both frozen installs (shared workspace, and Daytona with `--ignore-workspace`) pass with pnpm 9.15.4. - Dependabot installed `pnpm@9.15.4` from the new `packageManager` field. Other checks: - `.github/dependabot.yml` passes the SchemaStore `dependabot-2.0` schema (Ajv). A wrong key fails the same check. - With pnpm 9.15.4, `install --resolution-only --ignore-scripts --no-frozen-lockfile` in `scripts/typecheck-sandbox-providers` (the lockfile bot's command) gives exactly the lockfile that passed the frozen installs earlier in this pull request. - The lockfile rule from #19522 allows this pull request's diff: it changes no lock content. - `node --test scripts/typecheck-test-programs.test.mjs`: the new test and the provider tests pass. Two other tests need a root `node_modules` (`tsc`), which my worktree did not have. CI runs them. These checks cannot run before merge. GitHub reads `dependabot.yml` only from the default branch. After merge, open Insights → Dependency graph → Dependabot and make sure that the two new entries show no errors and run on Monday. ## Risks - A dependency in both a non-Daytona provider and Daytona (for example `typescript`) gets two pull requests: one from each entry. Each one passes alone, and they can merge in any order. - The typecheck job now typechecks Daytona's tests against Daytona's own lockfile instead of the shared one. Its versions are a little older (for example vitest 4.1.11), so a type difference can show up there. - Majors of shared dev tools (for example TypeScript 7, Vitest 5) come as one pull request across all providers. It can need code changes before it passes. That is the same as the root entry. - Pi and Kimi get no version updates. Every direct dependency there is a pinned provider release, and the pins feed the provider's ACPX `commandDigest`. The Runner checks compare the Pi lock with `PI_DISTRIBUTION_PINS`, and the Kimi build compares the lock SHA-256 with a pin. So any Dependabot change there fails CI (see #18998), and a real upgrade starts saved sessions fresh. GitHub security alerts still cover both lockfiles. - The Pi zod `3.25.76` pin is still affected by GHSA-mrhx-c7xc-m3w5 (open alert). It is deliberate: `pi-acp@0.0.33` has `zod ^3.25.0` as a direct dependency, not a peer. A fix needs a `pi-acp` release that allows zod 4. This pull request does not change it. - `packages/plugins/examples/plugin-orchestration-smoke-example` is also outside the root workspace. It uses `workspace:*` dependencies, so it cannot install alone, and this pull request does not cover it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker runs of the Dependabot CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 11:08:39 -08:00
DottaandPaperclip	f513f0efd9	fix(recovery): admit certified process-loss preparation retries (#19552) ... ## Thinking Path > - Paperclip keeps task work durable across server restarts. > - Restart recovery records a certificate when a Runner preparation ended before provider work started. > - Automatic recovery already verifies that certificate together with saved files and stopped leases. > - Explicit Retry called a preparation helper that only reached this proof for status `interrupted`. > - A certified `failed/process_lost` source therefore stayed blocked after its files were recovered. > - This pull request makes the helper reuse the same existing proof. > - A queued user Retry can then admit one new turn while all current safety and policy gates remain effective. ## Linked Issues or Issue Description **What happened?** A task had a certified process-lost preparation, verified saved-workspace receipts, a destroyed source sandbox, and one or two later attempts refused before dispatch. Its accepted Retry request stayed deferred with `execution_recovery` and created no successor. The explicit admission helper excluded this source before it could use the preparation proof added in #19464. This also prevented the refused-attempt continuation path from #19466 from advancing. **Expected behavior** Explicit Retry must use the same positive pre-provider proof as restart recovery. It must preserve source history, refuse contradictory execution evidence, honor current policy gates, and admit at most one successor for the existing request. **Steps to reproduce** Persist a legacy preparing run as failed with `process_lost` and the host bootstrap certificate that provider work did not start. Record its canonical committed snapshot, recovery/finalization operations, clean workspace authority, and exact destroyed-lease receipt. Add one or two unstarted cancelled descendants with the dispatcher's execution-wait receipt. Accept one Retry request and resume it concurrently twice. Before this fix, no successor exists and the same request remains deferred. **Paperclip version or commit** Reproduced on master `a6c22806308e8a3c7b21145f80fb471756bb443d`. Related repairs: #19464 and #19466. ## What Changed - Reuse `isInterruptedNativePreparation` before the existing failed-preparation checks. The existing no-coordinator and historical-ancestor restrictions remain. - Add real-database regressions for one and two refused descendants, duplicate clicks, concurrent resume, continuation dispatch, preserved history, and unchanged recovery attempt counts. - Add late-gate checks for provider work, provider events, a newer native checkpoint, missing bootstrap proof, missing stop proof, a changed snapshot, formal approval, user Stop, agent/company pause, and a spent budget. - Clarify that explicit admission uses the existing certified preparation proof. ## Verification - Red: both real-database cases passed the saved-file, interrupted-preparation, and refused-dispatch proof checks, but the original successor-count assertion found zero queued runs instead of one. The accepted requests retained `execution_recovery`. - Green: the same cases admit one successor after two concurrent resume calls. Continuation dispatch resolves to the latest refused attempt. Source and cancelled-attempt history stay intact. - Complete relevant local suites: **456/456 passed** before rebase. Command from `server`: `../node_modules/.bin/vitest run --config vitest.config.ts --fsModuleCache src/services/task-retry.test.ts src/services/explicit-native-continuation.test.ts`. - Final head `9788382c019df57899af06e81211e7c5165e7f0d`: the thirteen new cases passed **13/13** again after rebase. Required CI and review remain pending. - Environment: macOS ARM64, Node 26, Vitest 5.0.3, disposable real PostgreSQL. Tests exercise production request, admission, and continuation services. No provider executes and no hosted task is changed. - Hosted recovery still requires rollout and verification through the original queued requests. Final reviewed head: `9788382c019df57899af06e81211e7c5165e7f0d`. [Required CI](https://github.com/paperclipai/paperclip/actions/runs/42273663624) passed (56 successful checks, four expected skips, none pending or failed). Greptile and Paperclip Review both rated this head 5/5 with no findings; all review threads are resolved. ## Risks The change accepts only the existing verified preparation contract. An error label alone is insufficient. Provider-started or contradictory state remains blocked. Saved-file, lease, approval, Stop, pause, and budget checks remain effective. No schema, retry limit, provider setting, account selection, or production scheduler changes. ## Model Used OpenAI GPT-6 family via Codex, with coding and tool use. The exact model ID and context window are unavailable in this session's metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 13:03:21 -06:00
DottaandClaude Opus 5.5	c6eecac1e0	fix(heartbeat): keep a shutdown-suspended native recovery for the next boot (#19530) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner Full-Stack E2E workflow tests the Runner in Daytona sandboxes from a signed, content-addressed image > - Every change to an audited Runner input needs a new image before a Daytona cell can test it > - The image job took 10 to 16.5 minutes today, and a restart chaos campaign waits on it after each runner-side fix > - Measurement shows two serial waits: a full Rust dependency compile (about 5 of 6.6 minutes) and a 3.5 to 4 minute download of cached base layers that starts only after the builder stages finish > - There is also no way to build only the image: every dispatch reaches a paid gate that the operator must reject > - This pull request removes both serial waits and adds an `image_only` dispatch input > - The benefit is a shorter fix-to-Daytona-test loop, with every image security check unchanged ## Linked Issues or Issue Description **What happened?** The `Publish verified Daytona image` job is on the critical path of the control-plane restart chaos campaign. Builds today took 16.5 min (run 42240434011), 15 min (42253920727), 10.5 min (42259118276) and 16 min (42265637650). Run 42264611573 failed: its RunsOn runner lost communication with GitHub 7.6 minutes into the build step, during the Rust compile. GitHub has no log for that job. **Expected behavior** A Runner source change should not recompile the locked third-party Rust graph. Cached image layers should download while the builder stages compile. An operator should be able to publish only the image, without a paid gate. **Steps to reproduce** Read the BuildKit timeline in the logs of job 127970125495 (run 42240434011). See the stage table below. ## What Changed - `docker/daytona-runner/Dockerfile`: compile the locked Rust dependency graph in its own layer against placeholder targets. The layer depends only on `Cargo.toml`, `Cargo.lock` and the crate manifest. The step deletes the placeholder outputs and fingerprints, so the next step always compiles the real Runner sources. The final stage still receives only the stripped executable. - `docker/daytona-runner/Dockerfile`: move the content-ID / source-revision check above the two `COPY --from` steps. It does not depend on either builder stage, so BuildKit now downloads the cached base and CLI layers in parallel with the compile. Both copies use `COPY --link`. - `.github/workflows/runner-full-stack-e2e.yml`: add the boolean `image_only` dispatch input. `authorize`, `target_lock`, `catalog` and `daytona_image` run unchanged. Every other campaign job gets `&& inputs.image_only != true`. `authorize` rejects `image_only` together with TestOps inputs. - `tests/runner-e2e/workflow-image-only.test.ts` (new): pin the job classification, the top-level guard on each campaign job, the guard on every protected environment, and the TestOps rejection. - `tests/runner-e2e/README.md`: document the layer layout and the `image_only` command. ## Verification Measured on the same RunsOn `m7i-flex.xlarge` (4 vCPU, 16 GiB, native x64) runner class. "Before" is master job 127970125495 (run 42240434011). "After (cold)" is this branch, job 128077687214 (run 42271236140), with no dependency layer in the registry cache yet. Its paid gate was rejected. | Stage | Before | After (cold dependency cache) | |---|---|---| | Whole image job | 16.5 min | **11.6 min** | | Rust dependency graph | 311 s (inside one 399 s step) | 395 s (own layer; reused from cache after a master build) | | `paperclip-runner-core` compile | ~87 s | 65 s | | Cached base and CLI layer download | 236 s, **after** cargo finished | 257 s, **in parallel** with cargo | | Copy runnerd + provider pack | 20 s + 50 s | 9 s (`--link`) | | Provider pack stage | 143 s (parallel) | 152 s (parallel) | | Export + push | 82 s + 10 s | 81 s + 11 s | | Dependency layer size | n/a | `target` 852 MB + registry 333 MB (builder stage only) | With a warm dependency cache, a Runner source change compiles only the 65 s crate step. The critical path then becomes the ~4 min cached-layer download or the provider pack. The expected job time is about 7 to 8 min. I measure this after merge with an `image_only` dispatch (master first, to write the cache, then a Rust-only branch change). Disk: the instance starts with 38.76 GiB free. The dependency layer adds about 1.2 GB uncompressed to builder storage. The final image does not change. - `vitest run tests/runner-e2e/workflow-image-only.test.ts tests/runner-e2e/workflow-security.test.ts tests/runner-e2e/daytona-image.test.ts`: the new test and all image tests pass. Three tests in `workflow-security.test.ts` (`gates every provider-secret job with stable actor IDs`, `keeps provider credentials inside explicitly gated paid workflows`, `isolates private archive trust from workflows that execute target-branch code`) fail on `origin/master` without this change, because of `chat-catalog-testops-case.yml`. This pull request does not change them. - `node --test tests/runner-e2e/product-regression-testops.test.mjs`: 15/15 pass. - No local Docker build. The Dockerfile is measured with a branch-targeted CI dispatch whose paid gate was rejected. ## Risks - The dependency layer keeps the Rust `target` and registry trees in a builder-stage layer, and default-branch builds export it to the registry cache. This adds builder disk and cache upload when `Cargo.lock` changes. The build prints the size (`du -sk target /usr/local/cargo/registry`). - The dependency cache helps only after a default-branch build writes it. Branch builds read it but never write it. This is the existing cache policy. - `image_only` changes the workflow, so it works only after merge to master. - No check is removed. Lockfile verification, cosign sign and verify, the anonymous pull check, the provider-pack manifest match, the default-branch-only cache write and the actor allowlist are unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, GitHub CLI). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 13:01:16 -06:00
DottaandClaude Opus 5.5	7afb57c022	test(runner-e2e): repair restart-cell harness drift (#19538) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner Product E2E suite proves that agent work survives a control-plane restart > - A restart chaos campaign found three restart cells that fail in the test harness, not in the product > - A harness failure hides the real restart result, so the campaign cannot classify these cells > - This pull request repairs the harness drift in those cells and names the one failure the harness cannot prevent > - The benefit is that these cells now test restart behavior, or fail with a clear cause ## Linked Issues or Issue Description Refs #14811 (connector setup became one screen). Refs #19475 (completion-update restart fixture). **What happened?** 1. `lifecycle-baseline.runner-codex.local.tool-review-restart` clicked a "Save and continue" button. #14811 removed that button. The cell failed during setup. After that fix, the cell also failed after the restart: the expanded review card does not survive the post-restart reload, so "Approve & run" was not visible. 2. `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` failed with `page.goto: net::ERR_ABORTED` after the controller restart. The old document's Vite client reloads when the replacement server starts, and this aborts the test navigation. The code tolerated only `TimeoutError`. 3. `completion-updates.runner-acpx-claude.local.handoff-completion-restart` waited for the source agent to run an instructed wait command. Claude replied without the command (2/2 attempts), so the restart never occurred. The cell then failed with a generic timeout. **Expected behavior** Each cell reaches its restart boundary and tests restart behavior. If the boundary cannot be reached, the cell names the cause. **Steps to reproduce** Run `pnpm test:e2e:runner -- --max-parallel 1 --max-automatic-retries 0 --id <cell>` for each cell above on `master`. ## What Changed - `tests/runner-e2e/connection-reviews.ts`: accept the one-screen "Check link" step and the older "Save and continue" step. This uses the same regex approach as `connection-flow.ts`. - `tests/runner-e2e/runner.spec.ts` (tool review): click "Review request" after the restart reload, on the document that acts on the card. - `tests/runner-e2e/chat-restart.ts`: add `gotoAfterServerRestart`. It retries only `net::ERR_ABORTED`, within the caller's deadline and at most 5 attempts. It still accepts `TimeoutError`. It rethrows all other errors. The structured-question restart uses it. The existing sentinel and durable-state assertions still prove the new document. - `tests/runner-e2e/completion-updates.ts` and `completion-update-flow.ts`: add `completionRestartGateSkipped` and `completionRestartGateTimeoutDetail`. The gate poll stops at once with `restart boundary not reached (agent skipped gate)` when no completion turn is active and the turn that carries this task's Done facts (`chatCompletionUpdates`, as in `completionDelivery`) succeeded or published its reply before the gate reports `waiting`. A failed or cancelled turn is not blamed on the agent, because delivery can retry it; the timeout detail names it instead. The post-restart chat navigation now uses `gotoAfterServerRestart`. - A fixture-level gate is not possible without a product hook. The native run finalizer publishes the completion reply inside its own transaction, through the issue service closure. The isolated-server entrypoint can patch only class prototypes and HTTP responses, and the reply uses no tool call or HTTP request. - `tests/runner-e2e/README.md`: document the skipped-gate failure. - Docs: use `pnpm --filter @paperclipai/paperclip-runner run build:runner-binaries` in `tests/runner-e2e/README.md`, `tests/restart-chaos/README.md`, `tests/restart-chaos/main.ts` and `doc/DEVELOPING.md`. - No product code changes. ## Verification - Unit: `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-restart.test.ts tests/runner-e2e/completion-updates.test.ts tests/runner-e2e/connection-reviews.test.ts` (36 passed). - Typecheck: `node scripts/typecheck-test-projects.mjs tests/runner-e2e` passed. - Live cells, local, `--max-parallel 1 --max-automatic-retries 0`, macOS, load average 35–65: - `lifecycle-baseline.runner-codex.local.tool-review-restart`: PASS (first run found the review-card reload drift, fixed in the second commit). - `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`: PASS. A first attempt failed before the restart with `PRP command session.open timed out` at load ~36 (environment). - `completion-updates.runner-acpx-claude.local.handoff-completion-restart`: FAIL as designed, twice (before and after the review fix), with `restart boundary not reached (agent skipped gate): completion turn … ended succeeded without the reference gate reporting waiting`. Claude skipped the wait both times, so this cell still does not exercise a restart. ## Risks - Low risk. Test harness and docs only. - The navigation retry could hide a real abort. It is limited to `net::ERR_ABORTED`, 5 attempts and the deadline, and the test still asserts that a new document loaded. - The ACPX Claude completion restart cell still does not exercise a restart when the agent skips the gate. It now reports this cause by name. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, browser test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 12:59:54 -06:00
Devin FoleyandPaperclip	fd4b197f83	fix(ui): hide runner selection during agent setup (#19517) ... ## Thinking Path > - Paperclip lets people manage AI agents for work. > - Agent setup connects a harness, model, account, and computer. > - The system already selects a qualified runner for a new agent. > - Setup still asks users to select that internal runner. > - The runner dropdown can be the only field in Advanced. > - This PR removes the dropdown and hides an empty Advanced section. > - Users keep their saved execution settings and see fewer setup controls. ## Linked Issues or Issue Description **What existing behavior does this improve?** Agent creation, onboarding, built-in agent setup, company import, and agent configuration. **Subsystem affected** The UI, setup error messages, tests, and Storybook fixtures. **Current behavior** Users can select a runner from a dropdown. Some setup screens place only that dropdown in Advanced. Errors can direct users to that control. **Proposed behavior** Remove runner dropdowns from setup and configuration. Use the existing automatic selection for new agents. Retain saved runner settings. Show Advanced only when other fields remain. **Reason and benefit** Users can configure agents without an extra internal execution choice. Setup also avoids an empty Advanced section. **Breaking changes** Users can no longer select a runner through these UI dropdowns. The API and CLI retain their runner options. Existing agents keep their saved execution settings. **Additional context** Related: [Default new agents to the qualified runner for their harness](https://github.com/paperclipai/paperclip/pull/15422). This PR changes the existing setup UI. It adds no harness or runtime default. The master merge retains the custom gateway placement from [#19510](https://github.com/paperclipai/paperclip/pull/19510). ## What Changed - Remove the shared runner dropdown and its setup/configuration callers. - Hide Advanced when it has no remaining fields. Keep command and argument settings where supported. Preserve custom gateway setup under Advanced on the Claude and OpenAI API key paths. - Keep setup failures visible and keep the retry action outside the removed control. - Update errors and documentation that referred to the removed dropdown. - Update component tests and Storybook fixtures for the new UI states. - Make Legacy first-task evals resume an API-configured setup record. Record that path separately from fresh automatic onboarding. ## Verification - Latest master integration (`187c0f6d6e5`): 276 onboarding/provider-connection tests passed across six files. UI typecheck and build passed. Token gates, query-error rendering checks, and the Storybook catalog check passed. The conflict resolution retains master's bounded connection tests and cancellation while removing the obsolete runner instruction. The first test command used the wrong working directory and failed before collecting tests; the corrected invocation from `ui/` passed. - Earlier branch verification passed: full workspace build and typecheck, 1,037 focused UI/service tests, 49 heartbeat provider contract tests, eval typecheck, and 30 map/catalog tests. - The full local `pnpm test:run` run passed all six chat integration shards (1,931 tests). It was stopped for the master merge before the complete suite finished. The [complete CI run](https://github.com/paperclipai/paperclip/actions/runs/42264662642) passed on the previous head `72e1699`. - Current head `a097c8369d079c839e78661a2206cfa62c5210e0`: [full CI](https://github.com/paperclipai/paperclip/actions/runs/42273168268) passed. PR checks total 55 passed and 3 skipped, with none failed or pending. Greptile reviewed this head at 5/5 with no actionable findings or unresolved review threads. - Live browser/provider evals have not run. The Legacy eval change has no live qualification claim. ## Risks - Removing the dropdown changes the setup UI. Existing API and CLI runner choices remain available. - Conditional Advanced sections must retain valid command and argument fields. - Legacy first-task evals now prove setup resumption. They do not prove the fresh-agent automatic default. - No schema change or data migration is included. ## Model Used OpenAI `gpt-6.1-sol`, through Codex in T3 Code. Reasoning effort: `xhigh`. The model used file editing, shell commands, and test tools. The harness does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-06 10:58:42 -08:00
Devin FoleyandPaperclip	f0006c7783	fix(workspaces): contain inventory deadline aborts (#19543) ... A capture deadline can expire while file identity validation awaits the filesystem. Recheck cancellation before creating the inventory read stream so the expired capture rejects without terminating the server. All 224 affected capture and archive tests pass, including real Node subprocess cancellation regressions. Final-head CI passes 54 checks, and Greptile scores 5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-06 10:45:47 -08:00
DottaandClaude Opus 5.5	07ddf9a5c2	fix(native): keep accepted results when runner close cannot prove suspension (#19525) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs persist an accepted result first. The Runner then closes and suspends its session. > - Under heavy host load, the Runner close can fail to prove a durable suspension after the result is accepted. > - The executor then kept its finalization lease. The heartbeat finalizer got `native_finalization_lease_busy`, and the run ended `failed/adapter_failed`. > - The provider work was complete. Only Runner bookkeeping failed, so the run must not fail. > - This pull request hands the exact lease to finalization and completes the run from its accepted result. > - The benefit is that an accepted result stays accepted, independent of host load. ## Linked Issues or Issue Description **What happened?** A native ACPX Claude run asked a structured question. The control plane accepted its result (`Waiting for…`) and recorded `run.terminal` with `runTerminalState=succeeded`. The Runner close then took about 8.6 s. Its `runner.suspend` command ended `failed`, but the durable Runner state was `suspended` with a matching identity. The close threw `NativeSessionCloseUnrecoverableError`. The executor emitted `native.post_result_failure` (`category: runner_settlement_incomplete`, `resultAlreadyAccepted: true`) and rethrew `native_session_lease_lost`. Its finalization lease stayed held (its TTL is 20 minutes). `finalizeNativeRun` got `native_finalization_lease_busy`, and the run ended `failed/adapter_failed`. No restart was in progress. Host load average was about 40. **Expected behavior** After the control plane accepts a result, finalization bookkeeping must not fail the run. The run settles from its accepted result. Repair of the Runner checkpoint stays with the existing next-run admission rules. **Steps to reproduce** Run `pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts -t "accepted local result"` in `server/`. The test uses the real executor, the real control-plane result commit, and embedded PostgreSQL. The provider boundary accepts a result and then throws `NativeSessionCloseUnrecoverableError`. On the base revision, the finalization row keeps the executor lease (`phase: workspace_finalizing`) and the finalizer reports `lease_busy`. **Paperclip version or commit** Base `ff1d1dea5d`. The incident was on `819420a60a`. ## What Changed - `native-session-executor.ts`: after the rejected failure write proves an accepted result under the exact, unexpired lease, the executor releases that lease to finalization. It uses the same exact-authority predicates as the normal release. Before, the lease stayed held until its TTL expired. - `runtime-execution.ts`: for `runner_settlement_incomplete` on a local workspace (no workspace sync and no remote workspace authority), Paperclip records `workspace_finalize` as succeeded, finalizes the accepted result, and returns it to the ordinary completion path. - New `native-accepted-result-settlement.ts`: rebuilds the adapter result from the durable accepted result row, its terminal, the run's native session, and the accepted usage checkpoint. The result carries the closed `nativePostResultFailure` diagnostic. It contains no provider text. - Other post-result categories (for example `io_not_found`) keep their workspace-failure handling. They now also reach finalization at once, instead of after the lease TTL. - `doc/execution-semantics.md`: describes the settlement. ## Verification - Red, then green: `settles an accepted local result when runner close cannot prove suspension` and `keeps workspace failure handling for other accepted-result failures on a local workspace` in `server/src/services/heartbeat/runtime-execution.test.ts`. Both failed on the base revision with the executor lease still held. Both pass with this change. - Safety case `settles an accepted local result ... (workspace-check-failed)`: when the `workspace_finalize` check fails, the run keeps the existing failure path. Case `terminal-unreadable`: when the accepted result cannot be rebuilt, only a failed `workspace_finalize` record is written. - `pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts`: all pass (some cases time out at 30 s only under host load above 40; they pass on a narrow rerun). This includes the 14 existing post-result diagnostic and authority cases (expired, replaced owner, generation, attempt, boot, pid, process start, stopped, lock busy, and others). - `pnpm --filter @paperclipai/server typecheck`: pass. - Live cell `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` (real ACPX Claude, local runner): 3 runs at host load average about 40. Runs 1 and 2 passed. Run 3 hit the fault naturally: the first run emitted `native.post_result_failure` (`runner_settlement_incomplete`) and now ended `run succeeded` (before this change it ended `failed/adapter_failed`). The cell then failed later, in the resume run after the answer: `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=target_not_sandbox_runner`. That is a separate follow-up (see Risks). ## Risks - A changed lease handoff. The executor releases the lease only after the rejected failure write proves an accepted result under the exact owner, attempt, controller generation, controller identity, and an unexpired lease. A stale or replaced owner does not release anything. A failed release is bounded and best effort. The old behavior then applies. - The run now completes as `succeeded` while the local Runner checkpoint is not proven suspended. The next run on the same session then meets the existing fence: the continuity-break retirement (`retireFinishedPerTurnRunnerdOwner`) accepts only sandbox Runners, so a local Runner is refused with `target_not_sandbox_runner` and its state root is quarantined. This PR does not change that fence. Extending the retirement to local Runners needs its own proof that the local Runner process stopped, and is a follow-up. - Settlement happens only when the accepted result and its terminal are rebuilt first, and the `workspace_finalize` record then succeeds. A failed branch or plugin workspace check keeps the failure path. - Synced and remote workspaces are not changed. They keep their copy-back retry path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #19126 added the closed post-result diagnostic that this change uses. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 12:36:34 -06:00
Nicky LeachandClaude Opus 5.5	3b3037aab5	fix(ci): run the sandbox provider tests in required CI (#19533) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox provider plugins (Daytona, E2B, Cloudflare, Kubernetes, Modal, and others) run agents in remote sandboxes > - The providers are outside the root pnpm workspace, so `pnpm test:run` and the CI test lanes never run their Vitest suites > - paperclipai/paperclip#18839 typechecks those tests, but a test that fails at runtime still fails nothing > - The providers have about 1,040 tests, and they finish in about 15 seconds on Linux > - This pull request adds a required `Sandbox provider tests` job that runs them when a change can affect them > - The benefit is that a broken provider, plugin SDK, or shared contract fails CI on the pull request instead of in a release or a paid end-to-end run ## Linked Issues or Issue Description Refs #18839 **What happened?** No CI job runs the sandbox provider tests. `runner-full-stack-e2e.yml` builds the Daytona plugin for a paid nightly run, but it does not run any provider's `pnpm test`. **Expected behavior** A pull request that can break a provider runs that provider's tests, and a failure blocks `ci / verify`. **Steps to reproduce** Add `expect(1).toBe(2)` to a test in `packages/plugins/sandbox-providers/e2b/src`. Open a pull request. CI stays green. ## What Changed - Added `scripts/test-sandbox-providers.mjs` and its tests. The script installs the providers through `typecheckTestPrograms` with no programs, because that function already owns the provider install and the plugin SDK link. Then it runs each provider's `pnpm test`, four at a time, and prints each provider's output as one block. It fails if a suite fails or if no provider has a test script. - The script skips `cloudflare/bridge-template`, because Cloudflare's Vitest config already runs those tests. - Added the `sandbox_provider_tests` job to `.github/workflows/pr-trusted.yml`, and made `ci / verify` require its result. - Added the `//#ci:sandbox-providers` task to `turbo.json` and its lane to `.github/ci-task-matrix.json`. The CI planner runs the job only when a change touches `packages/plugins/sandbox-providers/**`, `packages/plugins/sdk/**`, or `packages/shared/**`. `packages/shared` is the plugin SDK's only workspace dependency. Global CI inputs, such as `scripts/**`, `.github/**`, `pnpm-lock.yaml`, or any `package.json`, still select every job. - The job turns off core dumps. A Daytona capacity test kills a process at a file-size limit inside the exported directory. If core dumps are on, the core file joins the export, and the test sees 12 files instead of 11. - Updated the job lists in `scripts/__tests__/e2e-shard.test.mjs` and the job count in `.github/scripts/tests/ci-profile.test.mjs`. Added planner tests to `.github/scripts/tests/ci-plan.test.mjs`. One of these tests fails if the plugin SDK gets a new workspace dependency that is not in the job's inputs. - Added the new test file to `test:release-registry`, and documented the script in `doc/DEVELOPING.md`. ## Verification - `node scripts/test-sandbox-providers.mjs` on macOS: 8 of 9 suites pass in 31.6 s, including the install. Kubernetes fails 12 `file-sync` tests, because macOS has no `/proc/self/fd`. Some Daytona tests skip, because they need GNU `tar` and coreutils. - The same script in a Linux container (`node:24`, a non-root user, core dumps off): all 9 suites pass, with 1,012 tests passed and 26 skipped. The script took 15 s after the install. - Kubernetes: 205 passed. - Daytona: 590 passed and 16 skipped. - With core dumps on, Daytona's `preserve literal capacity repair` test fails. The export includes a `core` file. With `--ulimit core=0`, it passes. - To look for timing failures under load, I ran the whole script 12 more times in the container, limited to 2 CPUs (8 runs) and 1 CPU (4 runs). All 12 runs passed. They took 45 to 76 s on 2 CPUs and 73 to 102 s on 1 CPU. - `ubuntu-latest` has `zstd`, and my container did not, so some Daytona tests skipped in those runs. With `zstd` installed, Daytona passed with 598 tests and 8 skipped, and only the live tests skipped. One earlier `zstd` run failed while the stress runs used the same CPUs, and I did not capture its output. The next run passed. - `node --test` on every test that reads `pr-trusted.yml`, the CI planner, or the task matrix: 816 passed. These include `.github/scripts/tests/*.test.mjs`, `scripts/__tests__/e2e-shard.test.mjs`, `scripts/ci/test-selection-workflow.test.mjs`, and the new tests. - `pnpm run test:release-registry` on macOS: 251 of 255 pass, including the new tests. The 4 failures are `wait_for_npm_package_versions` tests in `scripts/release-lib.test.mjs`. They fail the same way on master on the same machine, and CI runs this suite on Linux. - `actionlint` 1.7.7 reports no errors on the changed workflow. - I called `planTasks` from `.github/scripts/ci-plan.mjs` with one changed file at a time. `server/src/index.ts` and `ui/src/App.tsx` leave `sandbox_provider_tests` off. `packages/plugins/sandbox-providers/e2b/src/plugin.ts` and `packages/shared/src/index.ts` turn it on. This pull request cannot run its own new job. `pr.yml` calls `pr-trusted.yml@master`, so the job first runs on pull requests after this one merges. After merge, check the first pull request that changes a provider or `packages/shared`. ## Risks - **First real run is after merge.** I measured the time on Linux in Docker, not on a CI runner. CI runners are about 2 times slower than my machine, so I expect about 2 to 3 minutes for the job, including the dependency restore. The job runs in parallel with the other jobs, and the slowest required job takes about 13 minutes. So the job should not make CI slower. - **Open provider pull requests.** After this merges, open pull requests that change a provider run its tests. If those tests fail, `ci / verify` blocks them until they are fixed. - **`packages/shared` changes often.** Any change there runs this job. A Linux-only provider failure would block those pull requests. All suites pass on Linux today. - **One unexplained Daytona failure.** One Daytona run with `zstd` failed under CPU contention, and I did not capture its output. Some Daytona tests have short deadlines. If the job is flaky after merge, quarantine the failing test or revert the job, and do not leave `ci / verify` red. - **Path rules can be too narrow.** If a provider starts to import another workspace package, a change to that package does not run this job. The plugin SDK dependency test covers the SDK. It does not cover a provider that imports a workspace package directly. Today no provider does. - **Overlap with paperclipai/paperclip#19489 and paperclipai/paperclip#19522.** Both pull requests change the provider install in `scripts/typecheck-test-programs.mjs`. This pull request does not edit that file. It calls the same install function, so it uses their install changes when they merge. All three pull requests edit the same section of `doc/DEVELOPING.md`, so a small text conflict is possible. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits, GitHub CLI, Docker). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 10:34:03 -08:00
Nicky LeachandClaude Opus 5.5	e3041f5481	fix(ci): let the lockfile bot own the sandbox provider lockfiles (#19522) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The repository has a supply-chain rule: contributors do not commit lockfiles. A bot and verified Dependabot commits write them. > - The rule checks only the root `pnpm-lock.yaml`. > - The sandbox providers are outside the root workspace, so they have two more pnpm lockfiles. Pull requests edited them by hand, for example #12791, #18839 and #19489. > - This pull request makes the same rule and the same bot cover those two lockfiles. > - The benefit is one lockfile rule for every managed pnpm lockfile, and no lockfile edits in contributor pull requests. ## Linked Issues or Issue Description Refs #19489 **What existing behavior does this improve?** The lockfile policy and the lockfile refresh bot. **Subsystem affected** CI: `.github/scripts/lockfile-trust.mjs`, `.github/workflows/refresh-lockfile.yml`, and the sandbox provider installs. **Current behavior** `lockfile-trust.mjs` blocks contributor edits to `pnpm-lock.yaml` only. A contributor can edit `scripts/typecheck-sandbox-providers/pnpm-lock.yaml` or `packages/plugins/sandbox-providers/daytona/pnpm-lock.yaml`, and CI passes. The refresh bot does not update these files. `typecheck:test-programs` fails on a stale provider lockfile and tells the contributor to commit a new one. **Proposed behavior** The policy blocks contributor edits to all three lockfiles. The refresh bot regenerates all three after merge. CI resolves a stale provider lockfile in place for the run, as it does for the root lockfile. **Reason and benefit** Lockfile content comes only from the bot or from verified Dependabot commits. Contributors do not need to run pnpm in special folders. **Breaking changes** None. A pull request that edits a provider lockfile now fails the policy check. Remove the lockfile change from the pull request. The bot commits it after merge. ## What Changed - `lockfile-trust.mjs` now exports `LOCKFILES`: the root lockfile, the shared sandbox provider lockfile, and the Daytona lockfile. The policy blocks contributor edits to each of them. A Dependabot pull request must match a verified Dependabot commit for each lockfile that it changes. The `merged` command and the reporting-lock proof still use only the root lockfile. - `check-pr-lockfile.mjs` (advisory check) uses the same list. - `refresh-lockfile.yml`: - A job-level `LOCKFILES` env lists the same three files. A test checks that it equals the policy list. - A new step regenerates the two provider lockfiles with `--resolution-only --ignore-scripts --no-frozen-lockfile`. - The unexpected-changes check, the change check, and `git add` cover all three lockfiles. - The release-age gate checks the added versions in every lockfile. - A held refresh saves all three lockfiles at their repository paths, with a `source-locks.sha256` file. The next run reuses them only if every master lockfile is unchanged. An artifact in the old format is not reused. - The PR title is now `chore(lockfile): refresh pnpm lockfiles`. - `check-lockfile-release-age.mjs` accepts more than one `--base`/`--head` pair. - `scripts/typecheck-test-programs.mjs`: if the frozen install of the provider workspace fails, the script resolves the lockfile in place and installs frozen again. - `scripts/build-standalone-public-packages.mjs`: the same fallback applies to a package with a committed lockfile (Daytona). `release.sh` already restores tracked files when it exits. - `doc/DEVELOPING.md` and `doc/RELEASE-AUTOMATION-SETUP.md` describe the managed lockfiles. They also say that the Pi and Kimi distribution lockfiles stay hand-managed, because they hold attested provider pins. ## Verification - `node --test .github/scripts/tests/*.test.mjs scripts/__tests__/build-standalone-concurrency.test.mjs`: 731 tests pass. New tests cover: - the policy for provider lockfiles, including a real git history with a Dependabot commit and a later hand edit - the workflow and policy lists match - the unexpected-changes check, run with bash against a fixture - a held refresh round trip, and its invalidation when a provider lockfile changes on master - the release-age gate across lockfile pairs - the inline fallback in both install scripts - `node --test scripts/typecheck-test-programs.test.mjs`: the install tests pass. Two tests fail locally only because this worktree has no root `node_modules` (no `tsc`). They also fail without this change. - I ran the new bot commands on master with pnpm 9.15.4. No package version changed. pnpm 9.15.4 removes `libc:` fields and adds `deprecated:` notes, so the first refresh is a metadata-only change. The release-age gate passes it because it adds no versions. - I ran the release-age and held-lock steps locally with bash against this commit. The gate reported `eligible=true`, and `sha256sum --check` accepted the saved hashes. - The workflow YAML parses, and `bash -n` accepts every `run` block. - Not testable before merge: the policy job loads `lockfile-trust.mjs` from the base branch, so the new rule applies only to pull requests opened after this merges. The refresh workflow runs only on master. ## Risks - The release build now resolves a stale Daytona lockfile in place, from the registry. The root release install already resolves without a frozen lockfile, so this is the same level of trust. - The paid runner e2e job still installs Daytona with `--frozen-lockfile`. After a Daytona manifest change merges, its Daytona cases fail until the bot's refresh merges. This matches the rule that paid jobs refuse a new root lockfile until its refresh merges. - Dependabot pull requests for the provider entries in #19489 change a provider lockfile. They pass only if each lockfile matches a verified Dependabot commit, the same as for the root lockfile. - Low risk otherwise: the bot keeps its existing token, environment, and merge rules. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, and git). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 10:07:57 -08:00
DottaandPaperclip	5d25117ffa	fix(heartbeat): recover deferred dependency completion wakes (#19457) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tasks wait for other tasks through saved dependency edges and wake requests. > - A dependency can finish while its assignee answers a question about the blocked task. > - Paperclip saves the completion wake behind that active turn. > - Recovery did not find that saved wake unless it also contained a comment or an interaction response. > - This pull request restores that wake through the existing release and dispatch checks. > - The task can then continue after the active turn and its cleanup finish. ## Linked Issues or Issue Description **What happened?** A task stayed blocked after its dependency finished. The completion wake was saved while an audit turn was active. The wake stayed deferred after that turn ended. It had no comment or interaction response, so both recovery queries and the input guard excluded it. **Expected behavior** Recover the saved dependency wake when the prior turn and its cleanup finish. Start one dependency continuation under the current execution rules. **Steps to reproduce** 1. Run a question response on a dependency-blocked task. 2. Finish the dependency while the response is active. 3. Save the completion wake behind that response. 4. Finish the response and its environment cleanup, or restart the workers. 5. Observe that the saved wake has no run. **Paperclip version or commit** Reproduced against `70d23612dd`. Rechecked against `2f35eb1080` with the final regression fixture. **Deployment mode** Source checkout on Linux x64, Node 25.9.0, pnpm 9.15.4, and isolated embedded PostgreSQL. Related work: #16673 added recovery for saved messages. #17288 and #16966 address assignment wakes after cleanup. This change covers dependency completion receipts with no message. ## What Changed - Include saved `issue_blockers_resolved` receipts in worker reconstruction and stranded queue selection. - Let these receipts pass the input guard without a comment or interaction response. - Use the existing release transaction and dispatch checks. Add no timer, migration, or direct task status change. - Add process-backed regression cases for restart, cleanup release, cancellation, agent pause, pending cleanup, and unresolved blockers. - Update the dependency wake recovery document. - Share the dependency-wake payload predicate and SQL fragment across reconstruction and stranded receipt admission, so these eligibility checks have one owner. ## Verification - The final regression fails on the base code: the receipt stays `deferred_issue_execution` and has no successor. The same test passes with the fix. - Six focused cases pass. They use real PostgreSQL and a real process adapter. They check saved output, one dependency continuation, clean task locks, repeat recovery, and execution gates. They do not call a model provider. - Three nearby suites pass: 125 tests across queue wiring, dependency scheduling, and dependency recovery. - Final-head server layered typechecking passes with `PAPERCLIP_TYPECHECK_LOW_MEMORY=1`. Whitespace and module boundary checks pass. Two parallel checker attempts reached the sandbox memory limit; the completed low-memory check covers all source and test layers. - The full deferred-receipt suite passes: 45 tests. Combined with the nearby suites, 170 local tests pass. These results cover head `8f9d3aa6ac19a6d2532bd869eb8adda7cf122a5a`. Required CI remains in progress. - Local workspace typechecking and build stop at the Runner Rust step because Cargo is absent. The full local test suite was not run. Required CI must supply broad coverage. - The hosted incident has not been recovered or deployed from this branch. - Review follow-up at `182385e9bc433adac06440997de7e4cfbf889a72`: normal merge of master `0d03610d2f73b7cd799547e43929bb2303319024`; no conflict or history rewrite. All 54 receipt/helper cases pass on macOS arm64, Node 26.4.0, real embedded PostgreSQL and process adapters. The six dependency continuation and safety cases also pass separately. Module boundaries and whitespace checks pass. No assertion, timeout, execution gate or retry policy changed for this extraction. - Current shared, SDK and adapter-utils source builds pass. The first broad local server typecheck read stale borrowed adapter declarations; rebuilding them exposed nominal type duplication between the borrowed workspace and this checkout. It is not counted as a server typecheck pass. All 30 SDK and 169 adapter runtime JavaScript outputs match the previously tested artifacts. Final revision `182385e9bc433adac06440997de7e4cfbf889a72` now passes all 55 applicable CI checks (three skipped, one neutral review), including clean-workspace typecheck, build and broad tests. [CI result](https://github.com/paperclipai/paperclip/actions/runs/42262173960). Greptile and Paperclip review both give 5/5 with no open findings. ## Risks - Recovery now considers a saved dependency receipt with no message. Current ownership, source finalization, cleanup, pause, cancellation, and dependency checks still control execution. - The scan remains bounded and uses existing transaction notifications and restart reconstruction. The integration checks verify that it becomes quiet after settlement. ## Model Used - OpenAI Codex, based on GPT-6. The exact provider model ID and context size are not exposed in this session. Used reasoning, tool calls, TypeScript edits, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 11:49:38 -06:00
DottaandPaperclip	e3fb4bc73a	fix(native): recover saved listener startup failures (#19479) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Native recovery must preserve saved work and respect current task authority. > - A closed local API listener can stop startup before a provider starts. > - #19462 gives new failures a bounded bootstrap retry. > - Older saved failures still have a generic terminal code and a recovery hold. > - This pull request verifies the old producer outcome and uses the existing safe replacement path. > - Eligible tasks continue once. Unknown provider outcomes and user holds remain blocked. ## Linked Issues or Issue Description Refs: #19462 **What happened?** A restart can leave an old listener startup failure in `terminal_failure` with `native_session_interrupted`. Startup recovery does not claim it after #19462. A completed file recovery can also retain its no-replay hold. **Expected behavior** Continue automatically when durable records prove that no provider started, the source stopped, and the current files were saved. Keep the existing attempt budget and normal admission checks. **Steps to reproduce** 1. Close the actual API listener after a fresh native coordinator claim. 2. Persist the old executor's generic failure. Save the workspace and confirm source termination. 3. Restart recovery. The baseline schedules no successor. The repair schedules one successor after all receipts pass. **Paperclip version or commit** Baseline: `0252421d5c`. Old failure producer: `2f35eb1080`. **Deployment mode** Built from source. Local macOS arm64, Node 26.4.0, real embedded PostgreSQL, independent filesystem execution. No external AI service is used by the controlled acceptance. ## What Changed - Recognize the exact saved local-listener guard through one proof predicate shared by discovery and the locked transaction. - Require a fresh first attempt, matching execution bindings, no prior provider lineage, no provider events or tool calls, no checkpoint or admission receipt, and no cancellation. - Reuse current snapshot, exact stop, source block, attempt budget, and successor idempotency checks. - Wait for completed source file-recovery receipts. Retire only that same source's verified preservation hold in the successor transaction. - Bound discovery to 100 candidates per pass, keep completed history quiet, and wake through existing transactional producer notifications. - Retain retries after transient database and final ownership-proof errors; advance the bounded cursor and force a rescan so a failing page cannot hide later work. - Let only the replacement duty discharge a proved-empty notification during drain through its existing read-only readiness check, once per notification generation. Keep real work and concurrent changes outstanding without running replacement during drain. - Wake replacement at canonical local-stop proof writes and the existing dispatch eligibility boundary; these hints grant no new execution authority. - Add database regressions, scheduler quietness checks, real controller handoff coverage, and an opt-in controlled Runner continuation. ## Verification Final production head: `0df41b2fef2644195d1224d36984af3827da3e3b`. Local environment: macOS arm64, Node 26.4.0, TypeScript 7.0.2, Vitest 5.0.3, frozen-lockfile dependencies. Tests execute freshly; `--fsModuleCache` reuses only content-validated module transforms. - Native safe replacement: 62/62 PASS on isolated PostgreSQL 17 (63.18 seconds), including clean per-suite database disposal. All 62 assertions also passed on embedded PostgreSQL 18 beta, but its Darwin cleanup hook timed out; that invocation is not counted as a pass. Docker's disk was full, so the disposable PostgreSQL 17 data used a size-limited RAM mount, with no host mounts. The owned container was removed afterward. - Actual stopped-process producer suite: 31/31 PASS. A real exited child produces the stop receipt, and its replacement notification settles only after the outer transaction commits. - Workspace and separate-process drain regressions: 33/33 PASS (75.90 seconds). - Shared coordinator and execution-control lifecycle checks: 39/39 PASS (16.43 seconds). Pending work, concurrent changes, cold startup, shutdown, and passive drain boundaries remain covered. - Actual suppressed Cloud routine/drain workflow: PASS (34.64 seconds), using the unchanged test and timeouts. Its matching local RED reproduced CI's 462-query, zero-pending-query, one-idle-hold timeout after an admitted write committed during drain (69.46 seconds). - Actual historical controller handoff: PASS (48.40 seconds). A separate controller loads the original executor from `2f35eb1080`, writes its generic listener failure, and exits. Recovery at the final head admits exactly one successor. The real Rust Runner and controlled Codex process complete governed `paperclip_finish`; attempt two commits and the saved file remains intact. Only the child loads the old artifact, and both isolated artifacts are restored afterward. Rust source subtree `77961b70f177ec3c38d11abcaec92fd3d0e96ec0` matches the built binary. No external AI service is used. - Seven corrected existing fixtures pass: transactional stop-proof calls (5 cases) and notification inventory (2 cases). Their earlier CI failures exposed fixture calls outside the documented transaction and an outdated expected queue set. - Server typecheck: PASS. Red/green proofs also cover the original saved failure on baseline `0252421d5c` (zero successors instead of one), held-history quietness, bounded forward scanning through 100 parked rows and concurrent wakes, real snapshot/cleanup producer notifications, actual issue-row lock timeout, and task/history eligibility notifications. An obsolete prior-run question can be skipped to request another proof check; actual skip while the task is paused or the source owns provider tool activity schedules no successor. The failed-page regression starts with 25 persistent verifier errors and a healthy candidate 26. Baseline RED repeats the first page. Repair GREEN advances the healthy candidate, revisits the original failures after their fault clears, creates all 26 successors exactly once, and becomes quiet. Greptile is SUCCESS, 5/5 on the exact final head (2026-11-06 17:09:21 UTC). Final-head CI is GREEN: 56 successful checks, three intentional skips, one neutral result, and no pending or failing checks. [Attempt two and results](https://github.com/paperclipai/paperclip/actions/runs/42258352876/attempts/2) include passing build, typecheck, Rust, native server, package, browser E2E, serialized server, general server/chat tests, and final verification. Five jobs in the first attempt were interrupted together at 17:13:12 UTC; the native-server log records an explicit runner shutdown signal. The single infrastructure retry reran only those five jobs and dependent verification, preserving the 49 successful checks on the unchanged head. No timeout was weakened, and setup failures or interrupted invocations are not counted as passes. Focused commands from the repository root: ```sh ./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/services/remote-workspaces.integration.test.ts server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts -t 'native safe-replacement decision|reconciles the drained remote backend' --fsModuleCache --reporter=dot ./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/__tests__/execution-control-work.test.ts server/src/__tests__/delivery-work-coordinator.test.ts -t 'execution-control lifecycle|work coordinator' --fsModuleCache --reporter=dot node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit --pretty false ``` The historical acceptance sets `PAPERCLIP_TEST_HISTORICAL_LISTENER=1` and uses an original-executor child artifact with a `try/finally` restore. Normal regressions require no Rust binary. ## Risks - Compatibility is deliberately limited to positively proved fresh remote startup failures. Other old ambiguous failures remain held. - Existing provider work, a changed task block, incomplete preservation, or revoked authority must prevent execution. Normal retry admission remains in control. - This PR does not deploy the change or mutate a hosted task. Hosted recovery still requires the repaired revision and all current proof gates. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. The exact backend model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 11:49:28 -06:00
DottaandClaude Opus 5.5	a0cba70f18	fix(runner): continue restart-interrupted ACPX and OpenCode turns (#19501) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep their provider conversation when the control plane restarts > - A local Codex turn lost in a restart already continues once in the same conversation (#15297) > - ACPX (Claude and the other ACPX profiles) and OpenCode took a different path: the runner closed the lost turn as a provider failure > - The task then blocked with "Provider turn failed", and the user's queued follow-up stayed unread > - A restart must not need a user action when continuing is safe > - This pull request gives ACPX and OpenCode the same recoverable restart-loss cause and the same single continuation as Codex > - The benefit is that work interrupted by a controller restart finishes on its own ## Linked Issues or Issue Description Refs #15297 (Codex continuation), #19399 (restart-interrupted turn hook), #19471 (harness fix used for the live reruns). **What happened?** After a control-plane restart during an active native ACPX Claude turn, the task ended `blocked` with `native_provider_terminal_failed: Provider turn failed`. The queued follow-up comment was not consumed. No continuation started. The next run then failed with `native_session_cleanup_quarantined`. A native OpenCode run killed mid-turn ended the same way (`turn.failed` with `error: null`). **Expected behavior** When the old provider process is gone and the conversation can be restored, continue the task once in the same conversation. When that is not safe, record an honest restart cause instead of a generic provider failure. **Steps to reproduce** 1. Start a native ACPX Claude task on a local runner. 2. Submit a follow-up comment while the turn is active. 3. Restart the control plane while the turn is still running. 4. Run `pnpm test:e2e:runner -- --max-automatic-retries 0 --id everyday-workflows.runner-acpx-claude.local.recover-controller`. **Root cause** - `runner-core/src/acpx_provider_backend.rs` `restore_session_if_needed`: a saved `turn_active` state always closed the session. It recorded `turn.failed` with only a top-level `code: acpx_active_turn_recovery_closed` and no `error` object. It also emitted `run.terminal`. - The runnerd transport forwards only `error`, so the controller saw `error: null` and reported generic `provider_turn_failed`. - `runner-core/src/provider_backend.rs` gave the recoverable `provider_turn_lost_on_restore` cause only to `provider_label == "codex"`. OpenCode shares that backend. - `nativeRestartInterruptedTurnId` accepted only `codex_app_server`. The runtime continuation gate accepted only `provider.kind === "codex"`. - Two follow-on defects appeared once the continuation ran live: - The restored ACPX provider rejected the controller's result for the lost turn's open semantic call. That call never settled. - The restored provider's MCP request ids restart. The controller keys semantic calls by `hash(runId, callId)`. The continuation's first Paperclip tool call reused an earlier id, was treated as already answered, and hung. After the result was accepted, `runner_settlement_incomplete` failed the run. ## What Changed - ACPX runner: for an active turn lost to a runner restart, prove that the original provider lifetime lost its fence quorum. Then restore the same conversation and record `turn.failed` with `error: { code: "provider_turn_lost_on_restore", recoverable: true }`. Keep the run open for the controller's continuation. An accepted result, an unanswered runtime request, or an active goal keeps the old close behavior. So do an unproven lifetime fence and a failed restore. - ACPX runner fallback close: add a structured `error` (`acpx_active_turn_recovery_closed`, not recoverable, with an honest restart message), so the cause survives the transport. - ACPX runner: track open semantic calls durably. On restore, retire the lost turn's open calls. Acknowledge their controller results as `retired` and do not send them to the new provider. - ACPX sidecar: when the sidecar restores an existing session, prefix forwarded semantic `callId`s with a per-process namespace (`r<hex>:<id>`). Fresh sessions keep raw ids. - OpenCode runner: give OpenCode the same recoverable cause and open run as Codex (`restart_lost_turn_is_continuable`). Cursor Cloud stays excluded because its agent runs remotely. - Runner TS: `nativeRestartInterruptedTurnId` accepts `acpx_runtime` and `opencode_server`. The runtime continuation gate accepts `acpx` and `opencode`. All other Codex safety rules are unchanged: one continuation, admitted dead-runner restart only, no compaction, goal, semantic result, or pending runtime request. - Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`. ## Verification Red, then green, at each layer: - Rust `tests/acpx_warm_attachment.rs` `restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart`. Red: status `closed`. Then red on the lost call's result: "tool result does not match a pending provider call". Green after the fix. - Rust `tests/native_provider_backend.rs` `restart_lost_opencode_turn_records_a_recoverable_cause`, with the qualified OpenCode launch fixture. Red: `error` was `null`. Green after the fix. - Rust unit test `restart_lost_turns_continue_only_for_locally_restored_providers` (Codex, OpenCode yes; Cursor Cloud no). - TS `native-session-runtime.test.ts` "continues a restart-lost acpx_runtime / opencode_server turn once". Red: `NativeProviderTerminalFailure provider_turn_lost_on_restore`. Green: exactly one continuation, which does not resend the task. - TS `acpx-runtime-sidecar.test.ts`: a restored provider's call ids are namespaced and resolve. - Server `native-runner-restart-recovery.integration.test.ts`: a checkpointed `acpx_runtime` or `opencode_server` restart interruption is claimed as `resume_dead_runner`. Red without the driver gate. Green with it. - Two `native_provider_backend.rs` tests asserted the old close-on-restore result. They now assert the new contract. Their real invariant still holds: the delivered response is never replayed. Suites run locally on the final revision: - runner-core: `cargo test --no-fail-fast acpx`, `--test native_provider_backend` (38), `--test acpx_warm_attachment`, `--test codex_provider restart_lost` - runner TS: `native-session-runtime`, `runnerd-restart-continuation`, Codex driver recovery, contracts, and `acpx-runtime-sidecar` (all pass) - server: `native-runner-restart-recovery.integration.test.ts` (80 passed) - `pnpm --filter "@paperclipai/paperclip-runner..." build` (all generated-contract checks pass) Live runner E2E (local, real Claude via ACPX, macOS arm64): - `everyday-workflows.runner-acpx-claude.local.recover-controller`: PASS twice in a row on the final runner code. The restart crossed an active turn. The task finished `done`, the queued follow-up was consumed, the ZIP download was verified, and there were no failure codes. - `stock-harness.runner-acpx-claude.local.continuity-restart` regression: PASS. - Earlier failing reruns are kept as evidence of the follow-on defects: one harness `kill EPERM` while stopping the server (unrelated to this change), one hang on the colliding call id, and one `runner_settlement_incomplete`. ## Risks - A continuation can run more work in the restored conversation. Its prompt requires inspecting state before repeating an uncertain action, as for Codex. - The ACPX restore waits up to 5 s for the original provider lifetime fence before it falls back to the old close. - The ACPX provider state file gains two fields with defaults. Older saved states still load. A downgraded runner cannot read state that a newer runner wrote with these fields. - Restored ACPX sessions now forward prefixed call ids. Only fresh sessions keep raw ids. Live proof covers Claude. Codex-ACP, Pi, Cursor, Copilot, Grok, Kimi, and Antigravity share this path, but this PR proves them only by unit tests and code review. - OpenCode is covered by the runner fixture and gate tests. The live OpenCode restart-chaos cell was not rerun here. - `runner-core` `unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend` is flaky when run in parallel. It also fails with the new path disabled, so it predates this change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 11:39:29 -06:00
Nicky LeachandClaude Opus 5.5	98ebeefbc9	feat(environments): let managed config declare plugin environments (#19508) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A managed instance gets its instance-level environments from the `PAPERCLIP_MANAGED_CONFIG` document > - #19490 added the plugin environment rows, their marker index and write floor, but nothing creates them yet > - The managed config still accepts only one sandbox entry, so a managed instance cannot get a plugin-provided environment without manual setup > - This pull request lets the managed config declare plugin entries beside the sandbox entry, and provisions one locked row for each at boot > - The benefit is that a managed instance gets its plugin environments with no manual setup, and the rows recover by themselves when their plugin restarts ## Linked Issues or Issue Description Refs #19490 (the plugin environment rows this pull request provisions) **Subsystem affected** Server environments: the managed-config parser, the managed environment bootstrap, and the environment routes' write floor. **Problem or motivation** A managed instance cannot get a plugin environment from its managed config. The `environments` section accepts one entry, and that entry always provisions the managed sandbox. Company settings cannot create a plugin environment either. So each managed instance that needs one requires manual setup. **Proposed solution** Let an `environments` entry declare `driver: "plugin"`. Boot provisions one locked row for each plugin entry through the row layer from #19490. A later pull request adds a one-time instance default for one plugin entry, and advertises the new managed-config capability. **Alternatives considered** Manual setup on each instance, and a new control-plane endpoint. Both need a step outside the deploy. The managed config already delivers the sandbox entry. **Roadmap alignment** This extends the existing managed-config `environments` section, which already provisions the managed sandbox. It adds no new product surface. ## What Changed - `server/src/services/managed-config.ts`: an `environments` entry may set `driver` to `"sandbox"` or `"plugin"`. An entry without `driver` is still the sandbox entry, so today's documents parse unchanged. - A plugin entry has `name`, optional `description`, `pluginKey` (the plugin's manifest id), `driverKey` and optional `driverConfig`. - At most one sandbox entry. Names are unique across the section, and `pluginKey`–`driverKey` pairs are unique. Secret-like keys in `driverConfig` refuse startup, as for sandbox `config`. An unknown key or driver still refuses startup, and that includes `instanceDefault` until the follow-up pull request. - Plugin entries parse into a new `pluginEnvironments` list. The `environments` list keeps only the sandbox entry, so its consumers do not change. - `MANAGED_ENVIRONMENT_DRIVERS` lists the accepted drivers. - `server/src/services/managed-environments.ts` (`applyManagedEnvironments`): - Boot refuses to start unless an auto-installed catalog entry installs each plugin entry's `pluginKey`. `plugins.autoInstall` holds catalog keys, not manifest ids, so the parser cannot check this. Boot uses the manifest ids `createApp` already resolves from the bundled and distribution catalogs (`app.locals.managedBundledPluginKeys`, passed in by `server/src/index.ts`). Without that list it resolves the catalog itself. - Plugin entries run after the sandbox entry and reuse its readiness handling. Boot awaits `pluginsReady`, then requires the plugin `ready`, declaring the entry's driver, and running a worker. Otherwise boot archives the row and re-ensures it on the worker's `ready`. - Recovery re-ensures on every worker `ready` until one ensure succeeds or the plugin rejects the config (HTTP 422). - If the ensure fails because the worker died, boot archives the row and waits for `ready`. If the worker is still running (for example the call timed out), boot keeps the row as it is, retries once at once and on the next `ready`, and otherwise at the next boot. No timer is added. - Removing an entry never archives its row. - A plugin-only document does not conflict with `PAPERCLIP_EXECUTION_MODE`, because plugin rows do not use the sandbox slot. - Plugin entries never change the instance default. The sandbox stamp still fills only an unset or local default, so it never replaces a plugin default. - `server/src/routes/environments.ts`: `isPlatformSlotEnvironment` treats a marked plugin row as a live slot while the document declares its entry. So the marker-clear escape hatch is refused then, and allowed again after the entry leaves the document. - `doc/DEVELOPING.md`: new "Managed environments" section. ## Verification - `cd server && npx vitest run src/__tests__/managed-config.test.ts`: old documents parse unchanged. Sandbox plus plugin parses. Each of these refuses startup: two sandbox entries, `driver: null`, duplicate names, a duplicate pair, a secret-like key, an unknown key (including `instanceDefault`), an unknown driver, a malformed `driverKey` or `driverConfig`. - `npx vitest run src/services/managed-environments-plugin-entries.test.ts` (the sandbox-entry tests stay in `managed-environments.test.ts`; shared fixtures are in `managed-environments.test-support.ts`): - A plugin entry is ensured with the validator's worker manager, after the sandbox entry. - Boot refuses a plugin entry whose `pluginKey` no auto-installed catalog entry installs, both from the passed list and from its own catalog lookup. - A plugin that is down archives the row, and worker recovery re-ensures it. A missing plugin, a plugin without the driver, or a worker manager without `call` counts as unavailable. - Recovery stays armed after a failed attempt and stops after one success or a permanent config rejection (422). A `ready` during an attempt queues one more. - A worker that dies during the boot ensure archives the row and arms recovery. A failure while the worker keeps running leaves the row as it is and retries. - Removing an entry does not archive. Plugin entries and the sandbox stamp never change a plugin default. - An ensure failure is counted, not thrown. - `npx vitest run src/__tests__/managed-plugin-environments.test.ts` (embedded Postgres, real `applyManagedEnvironments`): the first boot creates the row and the second changes nothing. A plugin that is down archives the row, and recovery brings it back. Removing the entry does not archive the row. The instance default stays unset. The row tests from #19490 still pass. - `npx vitest run src/__tests__/environment-routes.test.ts`: the marker clear is refused while the document declares the entry, and allowed after it leaves. - `npx vitest run src/__tests__/instance-settings-service.test.ts src/__tests__/instance-settings-cloud-defaults.test.ts`: pass with the new `pluginEnvironments` field. - `pnpm --filter @paperclipai/server typecheck` passes. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. ## Risks - No schema change. - A managed-config document with a plugin entry now provisions a row at boot. A document without one behaves as before. - Adoption renames an unmarked row for the same plugin and driver to the declared name and resets its driver config to the declared one. The row keeps its id, agents and other metadata, and it is then locked like the managed sandbox. - Boot now refuses to start when a plugin entry names a plugin that no auto-installed catalog entry provides. This is the same fail-closed posture as an unknown sandbox provider. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (file edits, shell, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 17:35:13 +00:00
DottaandPaperclip	0d03610d2f	fix(opencode): export bounded tool argument diagnostics (#18699) ... ## Thinking Path > - Paperclip manages work across agent runtimes. > - Company observability sends allowed records to the operator's OTLP receiver. > - Native OpenCode exports tool results but omits submitted argument diagnostics. > - Exact argument values are excluded by the current capture policy. > - Safe type information can distinguish malformed envelopes without copying private values. > - This change exports a fixed diagnostic shape from authenticated dispatch and schema-rejection events. > - Operators can compare rejected and accepted calls under the existing capture controls. ## Linked Issues or Issue Description **What happened?** OpenCode tool spans omit argument diagnostics. The exporter reads native tool events. The authenticated MCP boundary has the submitted arguments, but its dispatch item events do not create tool spans. Calls rejected by its schema check do not emit dispatch items. **Expected behavior** With tool details and tool content enabled, Paperclip tool spans include the method, diagnostic schema version and allowed argument field types. Exact values and arbitrary field names stay excluded. **Steps to reproduce** 1. Configure a company OTLP receiver and enable native OpenCode export. 2. Enable tool details and tool content. 3. Submit a rejected Paperclip tool call and then a valid call. 4. Inspect the input diagnostics and outcome of each span. Related work: #17989 proposes metadata-only native export. #18098 added the normalized content exporter. This change repairs argument diagnostics in that exporter. ## What Changed - Export `paperclip.opencode.tool-diagnostics.v1` as `braintrust.input` for authenticated Paperclip tool calls. - Include `tools/call`, the argument type and types for a fixed list of envelope fields. - Use dispatch IDs for Paperclip tool spans. Keep separately named `opencode tool ...` provider records for failures before dispatch. Mark each span source. - Preserve schema-rejected calls as failed items without invoking the controller handler. Keep authentication and the closed tool catalog. - Bound schema-rejection error text in stored events and transcripts with the existing redaction and 8 KiB limit. Keep the provider response unchanged. - Preserve the controller opt-in, tool-detail and tool-content switches, redaction, limits and run identity. - Add MCP-to-receiver regression coverage and update the capture policy and capability maps. ### Hypothesis and thesis for a fix The exporter drops the dispatch diagnostic signal. Raw input omission is intentional. The repair belongs in the OpenCode driver and exporter. Shared protocol, tool permissions and task completion stay unchanged. The regression sends schema-rejected, handler-rejected and valid argument envelopes through a fixture OpenCode subprocess and the authenticated MCP bridge. It checks the downstream rejection and receipt, completion event and actual local OTLP request. The baseline has no dispatch spans. The candidate has separate spans with safe type diagnostics. Two warm runs and disabled capture use the same path. Source: [OpenCode diagnostic pattern](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/f172172e-9c65-453f-bb3a-7c6ba4f5a400). ## Verification - Baseline: `a92f616cd0a457c8ba85d994b24e1f92c2e4d1de`. The new integration regression fails because tool spans are absent. The same regression passes with this change. - Exporter and MCP bridge suites: 37 tests pass. Three final driver controls pass. Ten selected controls also passed before the final provider-record repair. Runner TypeScript typecheck and build pass after the review repair. - The review regression failed with two spans before the repair. It passes with three spans after the repair. Unauthenticated and unassigned calls do not reach the rejection observer. A broken observer cannot change the schema rejection. - The full OpenCode/exporter/MCP run passes 94 tests and fails one existing final-answer selection test. The same failure occurs in an untouched baseline checkout. It is not claimed as a pass. - A further filtered run reached the default five-second timeout in two existing continuation tests during concurrent repository checks. All five selected dispatch, warm-session and continuation controls pass with a 15-second test deadline. The earlier timeouts remain recorded. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Runner TypeScript checks and build pass. Full repository typecheck and build stop at Rust because `cargo` is not installed. The full repository test command was stopped before workspace handoff. No full-suite pass is claimed. Local source commits: `c48a8af1c186e415c802fa0ac95209b16b4a3122` `6744991f6`, and `a8ccf4693`. The remote branch is based on newer master `4814d1ab691c176170510a93a175dafdc9e021c7`. All initial eight changed files match between those bases before the patch. - The integration provider and controller are fixtures. The OTLP receiver is a real local HTTP server. A separate real OpenCode 1.18.34 probe opened a session. The remote free model returned `Free usage limit reached`. No tool call or receipt was produced. No live recovery is claimed. - Review follow-up: a real MCP request with 12,000 invalid array elements produced a 264,908-character event error before the repair. The same regression now stores 8,192 characters in both events and transcript. The MCP error receipt stays unchanged and no handler runs. All 37 exporter/MCP tests and 12 selected driver/exporter/MCP controls pass. Runner TypeScript checks and build pass. Local follow-up commit: `64d4fa583`. - The previous remote head passed Runner, typecheck, build and E2E. Its unchanged attention-retention shard failed twice with 15-second timeouts, then cleanup deadlock and duplicate fixture IDs. Both affected retention suites passed all 70 tests locally unchanged. The new head needs fresh CI and review; no full CI pass is claimed. - Current-master qualification (2026-11-06, merge `2959ecdb8d`): normally merge master `53645a0104`, including the existing test-only retention-fixture repair from #18774 reused by #18770. Keep the OpenCode diagnostics feature and upstream durable MCP and completion-feedback behavior. No test deadline or assertion changed. The three old CI timeouts did not reproduce locally: the four completed-blocker-history cases passed on the old head in 29.52 seconds, then passed on the merged head in 13.23 seconds. The fixture rollback/provenance regression also passed. All 116 exporter, MCP bridge and OpenCode driver fixture tests passed. Runner TypeScript checks and both TypeScript compilation targets passed; ACPX profiles match the release manifest and protocol-contract checks pass. These checks use local PostgreSQL, HTTP and process fixtures. No hosted provider call or deployed recovery is claimed. Fresh CI and review are pending for this merge. - Map-scope review follow-up (`61e662e179`): remove only the inherited diagnostics paragraphs from the company capability maps. This change adds logging detail within an existing capability. `node scripts/check-maps.mjs` and `git diff --check` pass. Production and test inputs are unchanged from the qualified merge; fresh CI and review are pending for the final head. - Test compiler follow-up (`6675b46377`): reproduce CI TS2722 with `tsc -p packages/paperclip-runner/tsconfig.test.json --noEmit`. The harness contract makes transcript retrieval optional. The OpenCode bounds regression now fails explicitly if that capability is absent, then inspects the narrowed method. The same complete Runner test compiler project passes, and the actual local schema-rejection fixture passes with all receipt and 8 KiB assertions retained. Production inputs are unchanged. Fresh final-head CI and review are pending. - Final-head qualification: `6675b4637743aa269e5214756b158e1e0a4561a5` passes all 56 applicable CI checks (three skipped, one neutral review). [CI result](https://github.com/paperclipai/paperclip/actions/runs/42253119577). Greptile gives 5/5 with no open findings. Hosted task recovery remains pending verified rollout. ## Risks - Authenticated spans use dispatch IDs. Provider records use separate names and a source attribute. Consumers must choose the source they need to avoid counting two observations of the same tool action. - Safe types cannot reconstruct nested arguments or explain every value-level validation failure. - Export remains best effort and bounded. Lost native frames no longer remove dispatch diagnostics, but abrupt process termination can still lose a turn. - Full local Rust checks and a deployed product journey are not yet verified. The pull request is unmerged and not deployed. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The exact runtime variant and context-window size are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 11:01:25 -06:00
DottaandPaperclip	c6573962f8	fix(workspaces): recover full sandboxes through read-only export (#19465) ... ## Thinking Path > - Paperclip preserves agent work across failed runs and server restarts. > - A remote workspace must have a durable save before its source can be released. > - Portable capture normally writes a Git mirror and a full archive inside the sandbox. > - A full sandbox can prevent capture before the provider's streaming download starts. > - Retrying that write cannot create the required free space. > - This change transfers a checked standalone source and seals its archive on the controller. > - The source remains retained unless the existing durable save completes. ## Linked Issues or Issue Description Related: #18644 reduced temporary capture space. #18172 added the provider's disk-free directory transfer. This repair connects portable capture to that transfer when the archive itself cannot be created. **What happened?** Workspace capture returns `remote_workspace_sandbox_disk_full` after provider work has finished. File recovery repeats the same sandbox write. **Expected behavior** Recover a stable standalone workspace without requiring another full archive on the source disk. Preserve staged changes, untracked work, confined links and the original source. **Steps to reproduce** 1. Create a standalone Git repository with a commit, staged changes and an untracked result. 2. Use the existing real-command transport fixture. Inject ENOSPC into the sandbox helper write or archive write. 3. Run portable capture. Before this change it rejects the helper failure. 4. With the change, restore the saved archive after removing the fixture source. Check the commit, staged content, untracked result and Git connectivity. ## What Changed - Attempt one controller-side capture after an exact sandbox disk-full failure. - Read a source inventory without installing a helper. Use the existing provider bulk transfer with confined links and runtime exclusions. - Compare the transferred inventory with source fingerprints before and after sealing. - Reject copied Git pointers, alternates and administrative symlinks. Parse config as data without includes; allow only portable keys before controller Git normalization. - Preserve socket exclusions. Retain the source and remove incomplete controller staging after failure. - Update execution semantics and focused regression coverage. No database migration or new execution authority. ## Verification - TDD red on base `2f35eb1080`: the new full-sandbox test failed with `remote_workspace_sandbox_disk_full` before production edits. - Green: 202 transport and archive tests passed before the additional Git configuration guards. At head `8541086cb5`, all 116 transport tests and the PostgreSQL durability acceptance pass. The final `e24e173ec9` exclusion guard passes those 116 cases plus its new Git-history regression across the full run and focused rerun. Environment: macOS arm64, Node 26.4.0; real Node/Git commands and filesystem transfer. - Review regressions were also red before their fixes: an internal Git config symlink bypassed the check, and `core.attributesFile` could select a controller path. Git administrative symlinks and nonportable config now fail before normalization. The focused safety selection passes 11 cases. - Cases cover helper and archive ENOSPC, commit/staged/untracked continuity after source removal, confined links, sockets, changed source, corrupt or extra copied files, unsafe Git metadata, retained diagnostic text and failed download. - Real PostgreSQL acceptance passes: normal workspace save recovers one sandbox ENOSPC, commits the encrypted snapshot, clears its writer, loses the original allocation, then independently reloads the saved result twice. The first acceptance attempt used an incorrect expected status (`available`); the documented state is `committed`. No production change was needed for that correction. - Review regressions for untracked Python/Node environments and excessive Git roots failed before their fixes. Dependency exclusions now match normal capture. Controller parsing starts only after whole-tree/repository budgets and layout checks; refspec/settings budgets and a shared two-minute verification/normalization deadline bound further work. The excessive-root test verifies zero Git subprocesses. - Real provider-boundary red/green: Daytona direct tar and capacity-fallback PTY transfers first dropped `._result.txt`. The new optional mapping contract preserves all user names and excludes only current transfer scratch. Kubernetes implements the same contract. Explicit caller exclusions and archive validation remain in force. - Standalone Linux qualification on Node 24.22.2/Debian bookworm: fresh shared/SDK build; Daytona and Kubernetes typecheck and build; Daytona file-sync/capacity/diagnostics 68/68 pass; Kubernetes file-sync 31/31 pass, including the actual `/proc` case. Qualification was refreshed again on the final `be6f7feacc` source tree: current shared/SDK rebuilt; all 815 staged source/config/manifest hashes match the delivered tree before and after testing. An initial container run as root bypassed a read-only file test; the same source passes as uid 1000. Installing zstd enabled eight initially skipped cases. - A further review regression proved `.git/pyvenv.cfg` could exclude Git history as a dependency. The inventory now handles `.git` before dependency filtering, matching normal capture. The full capture/restore test verifies the original commit and saved work. The first green attempt incorrectly compared command-result object identity; correcting the test to compare stdout passed with no production change. - Further red/green regressions cover read-only private-copy cleanup, a regular file replaced by a FIFO before open, case aliases of Git administrative metadata, and permission preservation under restrictive umask. Nonblocking no-follow file handles verify identity before and after reading. Cleanup changes only private controller directories. Provider extraction preserves ordinary permissions, refuses special mode bits before writing, and does not adopt source ownership, ACLs, attributes or flags. Final macOS Daytona tests pass 65/65; the final Linux qualification above was repeated on these exact provider sources. - Focused strict TypeScript compilation of the inventory and archive modules passes. - The first sandboxed archive run failed three Unix socket fixtures with EPERM. The permitted run outside that filesystem sandbox passed all tests. - Initial full local server typecheck could not resolve the unbuilt Runner package and produced cascading errors. Exact-head `e24e173ec9` CI typecheck, build, Runner and browser checks pass. That CI attempt failed two server shards: the unchanged retained-cleanup backlog test timed out at 15 seconds and passes locally unchanged; ten cache-reclaim fixture oracles failed because directory fallback consumed their sealed-archive counters/faults. Final commit `caed4e4f83` isolates the intended unavailable-directory boundary while preserving archive counts, recovery status, integrity and no-second-reclamation assertions. Those ten failures reproduced locally before the fixture edit. The full affected file passes 394/395; its sole other HTTP socket-hang-up case passes unchanged in an isolated rerun. This is combined coverage of all 395 cases, not a single complete green run. The final branch integrates master `53645a0104` at `98ea07225592930cbb0d18bcf250bc64f6bb13b4`, preserving upstream workspace phase timings. After integration: 117 transport cases, the PostgreSQL durability acceptance, the existing object-storage/timing regression, and 12 affected cache cases pass. The expanded recovery file now has 421 cases. Final-head `be6f7feacc45c1e32a26a2417e973f1e3752fd8d` CI passes all 55 applicable checks (three skipped, one neutral review), including the expanded recovery suite, typecheck, builds and Runner coverage. [CI result](https://github.com/paperclipai/paperclip/actions/runs/42251345554). - Provider exclusion regressions were red through four real Linux transfer paths: Daytona direct archive, direct safe-link repair, capacity PTY repair, and Kubernetes streaming. A root socket and nested regular file share a basename; literal glob-character files/subtrees also survive. Preserve mode now anchors tar exclusions as literal root-relative paths and escapes `find -path` pruning. Default callers keep glob semantics. Receipt-count regressions were also red before correcting preserved-output counting. All 31 Kubernetes and 68 Daytona tests, builds and typechecks pass on the final source with 815 matching inputs. - The final semantics documentation explicitly records special mode bits, the stricter config allowlist (including innocuous unlisted keys), and provider qualification limited to Daytona/Kubernetes; these address the optional current-head review observations without broadening controller Git authority. - No Daytona credential is available in this session. Provider transfer is a local filesystem boundary in these tests. Hosted task recovery and rollout remain unverified. ## Risks - Recovery reads the source repeatedly and transfers the standalone tree. It can take longer than normal capture. - Linked worktrees, initialized submodules, Git administrative symlinks, nonportable Git configuration, special mode bits and alternate object stores remain retained if the normal capture cannot finish. This fallback does not follow their pointers onto the controller. - A provider without usable directory export, a changed source, or insufficient controller space cannot produce a receipt. The original source stays retained. - The server, SDK and affected provider plugins must be built from this revision together. An older provider can fail inventory verification on names it omits; it cannot publish an incomplete save. No merge, publication or deployment is included. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, code editing, shell tools and automated tests. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 11:00:32 -06:00
DottaandClaude Opus 5.5	871a9b3180	test(restart-chaos): add Phase 1 control-plane restart chaos harness (#19477) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane owns task execution, recovery and Retry for every agent run > - Server restarts were the largest remaining source of stalled tasks in the staging soak campaign > - The soak could only observe restarts that staging happened to perform, so a failure could not be repeated > - This pull request adds a script harness that causes the restarts on purpose and grades each task on fixed rules > - The benefit is a repeatable restart failure with a replay command and an evidence bundle for each fixer ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart test coverage. Today the runner-e2e restart cells test one fixed restart point per cell. They cannot run many tasks at once, cannot choose a random fault time, and cannot replay a failure. **Subsystem affected** Test tooling (`tests/restart-chaos/`). It reuses the runner-e2e process code in `tests/runner-e2e/`. Product code does not change. **Current behavior** Restart robustness is checked through a small set of fixed cells. A random restart during a real task has no defined pass rule. **Proposed behavior** A harness starts an isolated control plane and seeds a "Chaos Lab" company. It launches seeded workloads, injects K1, K2 or K4 faults at seeded times, and grades each task on rules R1 to R6 plus a deterministic task oracle. `--replay <wave>/<task>` repeats one task. **Reason and benefit** Fixers get a replay command and an evidence bundle for each failure signature. The first smoke wave already found three product signatures (dead Retry after a restart, a false over-budget stop after restarts, and an OpenCode turn killed mid-turn that ends as a provider failure). **Breaking changes** None. The change adds new test files, one entry in the test-project typecheck list and a gitignore entry. ## What Changed - Add `tests/restart-chaos/`: - `supervisor.ts` owns one isolated server. It reuses `replaceController`, the process-tree stopper, the environment isolation helpers, the config seed, the port reservation, the wrapper-owned Postgres and `server-entry.ts` from `tests/runner-e2e/`. It does not copy them. Faults: K1 (SIGKILL, immediate restart), K2 (SIGTERM, restart) and K4 (SIGKILL, 2 to 5 minute hold). Postgres and the controller runtime cache stay across a fault. - `seed.ts` creates the company, encrypted provider secrets, native Codex, ACPX Claude and OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project. The project is a git repository with a Python CLI, one failing test and a four-minute `bench.sh`. - `scheduler.ts` makes a seeded plan of workload, adapter, fault, fire time, hold and nonce for each task. It saves the plan so that `--replay` can run one task again. - `observer.ts` polls each issue and its runs. When a task stops and offers Retry, it sends `POST /api/issues/:id/retry` one time, with the same body as `useTaskRetry.ts`. - `grader.ts` grades R1 to R6 as pure functions. `workloads.ts` defines workloads A1, B1, B5 and C1 and their oracles. - `evidence.ts` and `ledger.ts` write an evidence bundle for each failure, a JSONL ledger and `SUMMARY.md`. All results go to the gitignored `tests/restart-chaos/results/`. Known provider key values are redacted. - Unit tests: `scheduler.test.ts` (determinism, replay and CLI parsing) and `grader.test.ts` (R1 to R6 and the oracles on fixture timelines). These tests need no credentials. - Add `tests/restart-chaos` to the test-project typecheck list (`scripts/typecheck-test-projects.mjs`), with a `tsconfig.layers.json` like `tests/runner-e2e`. The harness runs through direct script invocations, not `package.json` scripts. - After the first Greptile review: - A failed fault or an unexpected server exit now gets one bounded recovery. If the recovery fails, the wave fails and does not hang. - Incomplete cleanup is now a harness error, and the harness keeps the instance. - After the C1 follow-up comment, the observer needs a fresh rest window before it grades the task. - `--replay` now fires the recorded faults that crossed the task, at the same offsets, including the faults that other tasks requested. - A failed workspace-operation read is now a sweep error. It cannot give a clean rest. - R2 now checks that workspace files from before the first fault still exist at rest. - B5 now needs the completion record that `bench.sh` writes for each task. - After the second round of reviews: - An idle `todo` or `in_progress` task with no live run and no named wait is no longer treated as resting. The observer keeps watching it until the task resumes or passes the R5 stall limit. - A failed injected fault now fails the wave with exit 2. - T-oracle commands run agent-editable code. They now run under a macOS Seatbelt sandbox with a scratch `HOME` and no network, and they cannot read the operator's home directory. On other platforms these commands are refused unless the operator explicitly allows them. - R4 now counts every active run. A wake comment may be served again only after a run that did not succeed. - Evidence redaction now covers every environment variable name that ends in `KEY`, `TOKEN`, `SECRET` or `PASSWORD`. - If the pre-fault snapshot fails, only R2 for that task is left unchecked. The wave continues. - B5 also needs bench progress that the harness observed while it polled. - Add `doc/plans/2026-11-06-control-plane-restart-chaos.md` and its campaign ledger. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 tests pass. The tests include observer tests with a fake API, a real `bench.sh` run, and a sandbox test that cannot read a canary secret. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - `server/scripts/typecheck-layers.test.mjs`: 45 tests pass, including "covers every project under tests/". - A supervisor check with no credentials passed. K1 was ready again in about 6 seconds. K2 with a 3-second hold was ready again in about 9 seconds. Company data stayed readable. - One paid smoke wave ran: `--env local --adapters codex,claude,opencode --workloads A1,B1,C1 --faults K1 --tasks 3 --concurrency 2 --seed 7`. The harness worked from start to end, and all three tasks crossed a restart. The 0/3 result comes from product failures, which the campaign ledger records. A scan found no provider key in the results. - To run a wave: ```sh pnpm --filter @paperclipai/paperclip-runner build:runner-binaries set -a; source ~/.secrets; set +a node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude,opencode --concurrency 4 --tasks 8 --faults K1,K2,K4 --seed 42 node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay <wave>/<task> ``` ## Risks - Low product risk. The change adds test tooling only and does not change product code. - A wave spends provider money. The company budget hard cap (`--budget-cents`, default 3000) is the only spend limit. The plan's per-wave and per-night refusal is not implemented yet. - Known gaps: - `--env daytona` is reserved. It fails with a TODO in `seed.ts`. The Daytona sandbox sweep for R6 is not implemented yet. - The project requests isolated git-worktree workspaces. A B1 task still ran in the shared fixture repository. Concurrent B1 and B5 tasks can share files. The oracles use marker files for each task. - Only K1, K2 and K4 at seeded random times are available. Boundary triggers (B1 to B16), other faults and the UI probe come in Phase 2. - The paid smoke did not exercise B5 or K4. - On macOS, B1 oracle commands need `python3` outside the operator's home directory, for example `/usr/bin` or Homebrew. On Linux they are refused until a sandbox is added. - The B5 proof is best-effort against a deliberately adversarial agent, because that agent can write to the workspace. - Startup phase timings come from polling `/api/ready`. The server records its own timings only as OpenTelemetry spans. - `pnpm verify` and the CI Vitest jobs do not select these unit tests. The root Vitest projects cover workspace packages only. CI typechecks the files, but the tests run only through the vitest command above. - After the smoke, rest detection and the review fixes changed. A recovery action that the board owns now counts as a rest, and R6 reports it. A paid wave did not run again after these changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code as an agent with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 10:45:52 -06:00
Nicky LeachandClaude Opus 5.5	e1332b997f	fix(plugins): align tool run context and interaction input types (#19493) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugins add tools and issue-thread interactions through `@paperclipai/plugin-sdk` > - The SDK types are the public contract between the host and plugin authors > - Two parts of that contract do not match what the host does at runtime > - First, `ToolRunContext.projectId` says it is always a project UUID. The tool gateway sends an empty string when a run has no project. The direct tool route rejects such runs with 400 > - Second, three interaction helpers ask for the parsed output type, so plugin authors must pass fields that the host fills in with defaults > - This pull request makes the types and both tool paths agree with the runtime > - The benefit is that agents with no project can run plugin tools, and plugin authors can omit the fields that have defaults ## Linked Issues or Issue Description Fixes #3273 Related: #3274 removes the `projectId` requirement from the route. That change alone is not enough today: the route now also looks up the project to check its company, and that lookup needs a value. This pull request also skips that lookup when the run has no project. It gives plugins one value for "no project" on both tool paths. The new test typecheck in #18739 and #18839 also listed both problems as possible product problems. ## What Changed - `server/src/routes/plugins.ts`: `POST /api/plugins/tools/execute` accepts a `runContext.projectId` that is missing, `null`, or an empty string. It passes an empty string to the plugin, the same value that the tool gateway sends. It checks the project's company only when a project ID is present. A `projectId` that is not a string returns 400. - `server/src/routes/openapi.ts`: the request schema marks `runContext.projectId` as optional and nullable. - `packages/plugins/sdk/src/types.ts`: - The `ToolRunContext.projectId` comment says the value is an empty string when the run has no project. - `suggestTasks`, `requestConfirmation` and `requestCheckboxConfirmation` take `CreateIssueThreadInteractionInput`, as `askUserQuestions` already does. The host parses this input with the schema, so the defaults apply. - `packages/plugins/examples/plugin-kitchen-sink-example/src/worker.ts`: the example passes `undefined` instead of an empty project ID to `issues.create`. - `doc/plugins/PLUGIN_SPEC.md` (section 13.10): documents the empty project ID. - Tests: - `server/src/__tests__/plugin-routes-authz.test.ts`: tests a run with an empty, missing, or `null` project ID, which skips the project lookup. Also tests that a project ID that is not a string returns 400. - `packages/plugins/sdk/tests/testing-actions.test.ts`: removes the cast that the output type required. ## Verification - `pnpm exec vitest run server/src/__tests__/plugin-routes-authz.test.ts server/src/__tests__/openapi-routes.test.ts packages/plugins/sdk/tests/testing-actions.test.ts`: 3 files, 81 tests pass. - `cd server && pnpm typecheck`: passes. This includes the source layers and the three test-tree shards. - `tsc --noEmit` passes for `packages/plugins/sdk` (`tsconfig.json` and `tsconfig.test.json`) and for the kitchen-sink example. - Not run locally: the full `pnpm test:run` and the root `pnpm build`. Required CI covers them. ## Risks - Low risk. The direct tool route now accepts calls that it rejected before. It still checks that the agent and the run belong to the company. With no project ID, there is no project to check. - A plugin that reads `runCtx.projectId` can get an empty string on both tool paths. The tool gateway already sent this value before this change. Only the documentation is new. - The SDK parameter types are wider. Existing plugin code still compiles. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 16:25:43 +00:00
Devin FoleyandPaperclip	7840d894fa	fix: retain files after accepted remote copy-back exhausts retries (#19491) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspaces keep unsaved files bound to the run that wrote them. > - An accepted result can exhaust its copy-back retries before those files are saved. > - Failed-workspace preservation rejects every accepted result, so a stopped writer can keep the workspace held after a newer setup attempt fails. > - This pull request lets the existing owner retain files from that exact stopped remote writer. > - File recovery can then release the hold without applying the old result or repeating its provider work. ## Linked Issues or Issue Description **What happened?** A failed native remote writer had an accepted result and exhausted copy-back retries. Its sandbox was stopped with a valid retained-source receipt. A newer attempt failed on the writer's unsaved workspace. Task Retry could not register the missing file-recovery obligation because preservation returned as soon as it found the old accepted result. **Expected behavior** Task Retry must let the existing recovery owner save the exact retained files. The accepted result, failed attempts and current task must stay intact. The old provider work must not run again. **Steps to reproduce** 1. Finish native work in a remote task workspace and accept its completion result. 2. Exhaust copy-back retries, stop and retain that exact sandbox, and leave the workspace dirty. 3. Let a newer setup attempt fail on the old writer's hold. 4. Use ordinary task Retry. Before this fix, the missing file obligation is never recorded. **Paperclip version or commit** Reproduced against master at `51a65af9b81f3c60f8210047e9a3eeaada7bc155` with isolated PostgreSQL and real files. Related: #19385 preserves accepted copy-back interrupted by Stop. #19422 admits saved input after verified recovery. #19464 and #19466 repair preparation and Retry continuation. This change covers an older remote writer whose accepted copy-back has exhausted its retries. ## What Changed - Admit only terminal, exhausted accepted copy-back on a failed remote writer with a finished run and no live controller or finalizer. - Require the exact stop-and-retain receipt, accepted completion contract and current dirty workspace binding. Keep native-sync sources on their existing export path. - Use the existing file-preservation recorder and worker. Add no new recovery worker, schema, polling or replay path. - Cover bound and orphaned entry points, reusable leases, changed proof and the ordinary HTTP Retry journey. Document the recovery behavior. ## Verification - Causal baseline: the corrected bound-writer and orphaned-lease tests both fail without the source change and pass with it. - Assembled acceptance: authenticated HTTP Retry records the old writer's missing obligation; the real recovery service saves its files; concurrent continuation calls queue exactly one successor for the newer failed setup. PostgreSQL, filesystem, tar, Git and the API/service path are real. Provider transport is controlled, and no model turn is dispatched. - `pnpm check:module-boundaries`, `pnpm -r typecheck`, `pnpm build` and `git diff --check` pass. - Four full affected recovery suites pass: **659 tests**. Command: `pnpm --dir server exec vitest run --config vitest.config.ts src/services/workspace-recovery.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts src/services/native-runtime/native-cancelled-workspace-preservation.test.ts src/services/native-runtime/native-workspace-export-retry.test.ts`. This includes the final 16 new regression, refusal and assembled journey cases. - `pnpm verify --explain` completed. It selected broad transitive coverage; this is planning evidence only. Hosted CI will supply the remaining broad coverage. - Dependency cost: extra indexed source/result reads run only for this exact terminal retained-result path, inside the existing bounded transaction. Existing notifications activate the existing recovery owner. No idle scan is added. - Fixture failures are accounted for: the first fixture used an active-only stop helper on a released lease. The corrected fixture produced the causal baseline above. One negative fixture tried to remove a contract while an accepted result referenced it and failed the database foreign key. That impossible fixture was removed; the positive journey passed in the same run. Neither fixture failure is reported as a product regression or a passing full run. - Hosted CI is fully green at the reviewed commit: 54 successful checks and 3 intentional skips, with no incomplete or failed checks. Greptile is 5/5 with no unresolved threads. - Hosted rollout and a live task continuation remain pending. Local API admission does not prove a deployed model run. ## Risks This expands file preservation for a narrow terminal failure. Wrong source identity could retain unrelated files; unchanged writer, epoch, location, provider lease, stop receipt and accepted contract checks fail closed. Active ownership, unsafe archives, rejected results and native-sync sources stay on their existing paths. Existing recovery owns physical source locking, copy verification and bounded retries. It preserves unknown action outcomes and leaves the accepted result uncommitted. There is no migration or configuration change. ## Model Used OpenAI Codex, `gpt-6.1-sol`, xhigh reasoning effort. Context window size is unavailable. Used repository tools, isolated tests and code execution. No delegated agents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 08:23:36 -08:00
3 people	51a65af9b8	fix(recovery): continue preserved preparation after process loss (#19464) ... ## Thinking Path > - Paperclip manages AI agents and their execution state. > - A server restart can stop workspace preparation before provider execution. > - Workspace recovery must preserve files before the task can continue. > - Recovery handled interrupted preparations but skipped failed process-loss records with the same no-provider certificate. > - A withheld first retry could also hide the remaining retry budget. > - This change resumes certified preparations through normal admission after file recovery. ## Linked Issues or Issue Description **What happened?** A process-loss attempt remained failed after workspace recovery completed. Its first retry had been cancelled before dispatch by the workspace hold. The task had no running attempt or automatic continuation. **Expected behavior** Continue once after the server proves that provider work did not start, file preservation completed, and the exact leases stopped. Retain the existing failure budget and all task admission gates. **Steps to reproduce** 1. Stop a Runner preparation before native runtime selection. Record the restart recovery bootstrap certificate. 2. Retain the workspace and withhold the first automatic retry before dispatch. 3. Complete file preservation and restart the recovery worker. 4. Observe that the old code schedules no successor. The regression test reproduces this with PostgreSQL and real filesystem operations. **Paperclip version or commit** Reproduced against master at `2f35eb1080`. Related prior work: #19422 continues saved user input after verified copy-back. This change handles automatic continuation of certified process-loss preparation. ## What Changed - Accept failed process-loss preparation only with the durable no-provider bootstrap certificate and existing execution, lease and file checks. - Discover and settle the recovered source through the existing preparation recovery path. - Continue from a verified withheld retry without rewriting its history or resetting its failure count. Adopt it only as the sole retry child; multiple-child history reuses the newest existing child deterministically. - Exercise automatic dispatch, preserved-file readback, one provider turn, accepted result and task completion in the local native-session fixture. - Keep a durable continuation receipt after file-hold settlement until a successor is recorded. Known gates wait for existing eligibility events. Unresolved delivery keeps its shared recovery deadline. - Document the recovery and retry-budget rules. ## Verification - Red: both new PostgreSQL/filesystem regressions failed with zero scheduled successors where one was required. - Green: 54 focused recovery, automatic-dispatch, interruption, multiple-child history and admission safety cases pass on `13c27c2c2f`. The full PostgreSQL admission adapter suite passes all 83 tests; server source declaration typecheck also passes. Local environment: Darwin arm64, Node 26.4.0, PostgreSQL 18.1, Vitest 5.0.3. - Additional red/green: interruption after committed settlement, pause/resume, failed enqueue after cursor advancement, and absence of an earlier retry all failed before their repairs. Restart before enqueue, restart after the successor claim, and pause/resume now preserve one continuation. A rolled-back eligibility change cannot start execution. Deferred and delivered records have no retry deadline. - Local acceptance uses the real heartbeat dispatcher, isolated files and PostgreSQL, with a simulated native provider. It reads the preserved file, writes continued work, accepts the structured result and completes the task once. - Negative cases retain no-certificate, provider-evidence, live-owner, Stop, missing-stop-receipt, wrong-snapshot, human-block, pause, reassignment, approval and exhaustion fences. - CI fixture regression: 4 error-propagation/concurrent-resolution cases pass after adding the new obligation read to their transaction stubs; existing assertions remain unchanged. - Existing idle-drain and notification contract regressions: 4 tests pass after restricting recovery wakes to actual durable obligations. Late approvals block scheduled, queued and final dispatch. - Additional existing retry-scheduling suite: 62 cases pass locally. The pre-provider quota case exceeds its existing 5-second terminal wait on this loaded Darwin host, including when tested alone and with the preceding retry implementation. It remains a local timing limitation; the assertion and timeout were not changed. All Linux CI server shards pass on the combined head, including this unchanged suite. - Existing run-control, legacy-recovery and shared execution-control lifecycle suites: 128 tests pass with the review repair. Required CI supplies final-revision broad coverage. - Real-Codex acceptance was not run: automatic approval review rejected authenticated provider execution because it could incur cost/external effects and required explicit approval. No provider call ran; the controlled-provider acceptance covers the causal restart and durable scheduling behavior. - Hosted acceptance and recovery of existing tasks require rollout. They have not been claimed from the local provider fixture. - Final-head review: Greptile 5/5 on `13c27c2c2f`, no outstanding findings. Another contributor added the multiple-child history safety repair in `13c27c2c2f` while CI was running. The delta was inspected and the 54 focused recovery cases rerun successfully. Required CI passes on that exact combined head: https://github.com/paperclipai/paperclip/actions/runs/42241794918. All 55 successful checks include aggregate `ci / verify`, typecheck, build, canary, all server and E2E shards, and Runner package/Rust/static checks. The separate Paperclip Review service check remains queued waiting for its assigned agent; no validated assessment has been returned. ## Risks A malformed preparation certificate or a false no-execution receipt could permit duplicate work. The existing checks still reject provider execution records, native coordinators, live controllers, unsettled leases and unverified files. The pending continuation must survive a restart without polling known admission blockers. A withheld retry retains its original row and consumes its remaining failure budget. Normal admission checks current ownership, approvals, pause and budgets. ## Model Used OpenAI GPT-6 family via Codex. The exact model ID and context window are unavailable in session metadata. Used reasoning, code editing and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal task id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing>	2026-11-06 09:36:46 -06:00
Nicky LeachandClaude Opus 5.5	83c6832ccb	fix(build): typecheck package, Playwright, and e2e test files in CI (#18839) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Runner, plugin, and Playwright tests verify how those agents run, connect, and appear in the board > - Most package tsconfigs exclude tests, and Vitest strips types, so a type error in those tests fails nothing > - The tests drifted to 803 real type errors, and some of those errors hid assertions that checked nothing > - paperclipai/paperclip#18739 and paperclipai/paperclip#18738 check the server test tree and four test projects, but not these files > - This pull request adds a required check for these test files, fixes every error, and fails CI if a test file is in no program > - The benefit is that a type error in any test file outside `server/src` now fails CI ## Linked Issues or Issue Description Refs #18739 Refs #18738 **What happened?** 517 test files were in no program that required CI checks. A type error in those files did not fail CI. **Expected behavior** CI fails on a type error in any test file. **Steps to reproduce** Add `export const value: number = "text";` to a test file under `packages/paperclip-runner/src` on master. CI stays green, because `tsconfig.json` excludes test files and Vitest strips types. **Measured gap** Each row is a program with its tests included, on one thread. File and error counts are for head `7d4b5e907a`. Memory peaks were measured before the two latest merges of master. | Group | Test files | Real errors | Peak | | --- | ---: | ---: | ---: | | `paperclip-runner` Node source | 290 | 644 | 1.09 GiB | | `paperclip-runner` surfaces | 9 | 12 | 0.42 GiB | | `paperclip-runner` devtools and examples | 14 | 0 | 0.68 GiB | | `sandbox-providers/kubernetes` | 21 | 65 | 0.44 GiB | | 8 other sandbox providers and `bridge-template` | 32 | 4 | 0.34 GiB | | `plugins/sdk` | 22 | 15 | 0.41 GiB | | `plugin-workspace-diff` | 5 | 0 | 0.37 GiB | | `plugin-orchestration-smoke-example` | 1 | 1 | 0.15 GiB | | `kv-demo-mcp-server` | 3 | 0 | 0.13 GiB | | `packages/db` scripts | 1 | 0 | 1.01 GiB | | 8 Playwright suites under `tests/` | 52 | 25 | 1.72 GiB | | `tests/runner-acceptance` | 3 | 0 | 0.58 GiB | | `tests/e2e`, through layer declarations | 63 | 37 | at most 1.59 GiB | | `ui` `AttentionInteractionResolver.test.tsx` | 1 | 0 | n/a | | **Total** | **517** | **803** | | 27 of the 803 errors came from master while this pull request was open. Each merge of master brought new test code that no check had read. Errors that were config errors, not real errors: - One TS2688 for each provider, because the root install does not install provider dependencies. - 9 in the Runner Node program (TS7016 and TS6059). - 2 in `tsconfig.browser.json`. TypeScript 7 removed `baseUrl`, so that config was invalid. - 213 in the Playwright suites (`@/*` aliases and Vite types). - About 60 in `tests/e2e` (import extensions and aliases). `packages/plugins/create-paperclip-plugin` has no gap. Its tsconfig excludes its test, but `cli/tsconfig.json` reaches that test through an import. `server/test-baselines` also has no gap, because `tests/lifecycle-baseline` reads it. ## What Changed - Added `scripts/typecheck-test-programs.mjs` and its tests. The `Typecheck + Release Registry` job runs it. It checks each program in `PROGRAMS` on one thread. - The script fails if a tracked `*.test.ts(x)` or `*.spec.ts(x)` file outside `server/src` is in no program. The compiler file list decides this, because `include` drops `foo.test.tsx` when `foo.test.ts` exists. - Added a `tsconfig.test.json` to each uncovered package, and `tests/tsconfig.playwright.json`. - Added `tests/e2e` to `PROJECTS` in `scripts/typecheck-test-projects.mjs`, with a `tsconfig.json` and a `tsconfig.layers.json`. - Added `scripts/typecheck-sandbox-providers`. It is a small workspace that installs provider SDKs from one lockfile. I did not add a lockfile inside each provider, because `scripts/build-standalone-public-packages.mjs` uses a frozen install when one exists, and that changes what a release resolves. - Corrected `packages/paperclip-runner/tsconfig.browser.json` for TypeScript 7. - Set `rootDir` to `..` in `packages/paperclip-runner/tsconfig.test.json`, because one Runner test imports from `packages/adapter-utils/src`. The program does not emit files, so this changes no build output. - Moved the Kubernetes provider from `vitest ^3.2.4` to `^4.1.8`, the version the other providers use. Vitest 3 requires `tinypool ^1.1.1`, which has two critical advisories (GHSA-85c8-ppgw-ccpr and GHSA-5gmw-xhrv-c9v3). The first patched version is 2.1.2. Its tests have the same result on both versions: 189 passed, and the same 12 `file-sync` tests failed. - Moved the Kubernetes provider from `zod ^3.24.2` to `^4.6.6`, and gave its two `z.record` calls a key schema. This is the only production change, and it stays in this pull request. See "Production change for review". - Renamed `AttentionInteractionResolver.test.ts` to `AttentionInteractionResolver.replace.test.ts`, so TypeScript reads the `.test.tsx` beside it. - Fixed all 803 errors, with changes to test files and test support only. The fix for `contracts/native-execution.test.ts` is the same as the fix in paperclipai/paperclip#18739, so those two changes merge cleanly. - Updated `doc/DEVELOPING.md`. This pull request does not edit `server/scripts/typecheck-layers.mjs`. paperclipai/paperclip#18739 and paperclipai/paperclip#18738 have both merged, and this branch includes them. ### Production change for review The new lockfile made dependency review flag `zod@3.25.76` as a high-severity advisory (GHSA-mrhx-c7xc-m3w5). The first patched version is 4.6.6, so 3.x has no fix. The published Kubernetes plugin uses zod to validate operator config. The zod 3 source does not compile against zod 4, so a typecheck-only override cannot fix this. The existing tests do not prove this change, because they also passed with the invalid one-argument `z.record`. So I parsed 45 inputs with the same schema on zod 3.25.76 and zod 4.6.7: | Result | Inputs | Detail | | --- | ---: | --- | | Identical | 23 | Includes defaults, valid config, and a `__proto__` key | | Same outcome, new error wording | 21 | For example, `Invalid url` is now `Invalid URL`. `validateConfig` shows this text to operators. | | Different outcome | 1 | `jobTtlSecondsAfterFinished: 2 ** 60` was valid and is now rejected. The new limit is `Number.MAX_SAFE_INTEGER`. | I did not test this against a real cluster. The decision is to keep this change in this pull request. Since then, paperclipai/paperclip#17364 has moved the rest of the repository to zod 4.6.x, so the Kubernetes plugin now matches it. ### Behavior change for review `reducedMotion` under `use` is not a Playwright option. It had no effect in 11 configs and 2 `test.use` calls, so those suites ran with `no-preference`. The options are now under `contextOptions`, so reduced motion now takes effect. `app-shell-design.spec.ts` asserts a real transition, so it now asks for `no-preference` itself. The baseline manifest has `snapshotCount: 0`, so this invalidates no published baseline. A frozen trial reference PNG can differ. ### Findings These are product questions. I did not change them, and I did not hide them with casts: 1. **`PrpEvent` and its schema disagree.** The TypeScript type in `src/protocol/replay-contract.ts` requires `normalizedSessionId`. The `required` list in `protocol/schemas/event.schema.json` does not include it. `validatePrpEvent` accepts events that the type forbids. 2. **`requestCheckboxConfirmation` is stricter than its runtime.** It uses the output type, so plugin authors must pass `continuationPolicy` and `allowDeclineReason`, although the schema gives them defaults. `z.input` would fix this. 3. **`RetainedRunnerdMaintenanceInput.backend.kind` is a free string.** The quarantine domain comes from `descriptor.kind`, but the release uses `proof.backend.kind`. If they differ, `completeRetainedNativeSessionCleanup` returns 0 and the quarantine stays. The production caller passes `"runner"` today, so this is not a live bug. 4. **`CodexHarnessSession.close(input?)` is optional, but the contract requires `{ reason }`.** These are tests that verified nothing. I corrected them, and they pass: - `backends/native-instruction-measurement.test.ts` filtered on `placement === "core"`, which is not a valid placement. It measured an empty tool list. It now measures 14 tools. - `drivers/acpx/codex-acpx-driver.test.ts` checked for `usage.updated`, which is not an event type. - The `hostFixture()` helper in `drivers/acpx/runtime-host.test.ts` ignored two overrides, so `expect(prepareSandbox).not.toHaveBeenCalled()` could not fail. - `live/live-session.test.ts` set `agent: "pi"` before a spread that overwrote it. - `native-distribution-integrity.test.ts` passed an AbortSignal where `openCommand` expects `{ signal }`. These are gaps that this pull request does not close: - **The root lockfile still has some old zod versions.** paperclipai/paperclip#17364 moved the direct zod dependencies to 4.6.x. `pnpm-lock.yaml` still resolves `zod@3.25.76`, `4.1.11`, and `4.4.3`, which are below the patched 4.6.6. `packages/paperclip-runner/scripts/pi-distribution/package.json` pins `zod` to `3.25.76`, and some agent SDKs resolve `4.4.3`. Dependency review does not flag them, because this pull request does not add them. - **Dead GitHub coverage.** `chat-adapters-ui-providers.spec.ts` returned early for GitHub, so four later `provider === "github"` branches never ran. I removed them. They included the private key file and paste match, and the reconnect flow. Nothing asserts those behaviors in that spec now. - **Provider tests do not run in CI.** The root `vitest.config.ts` lists only Daytona. The other 8 providers, `bridge-template` included, are typechecked now, but CI does not run their tests. - **One `packages/db` test never runs.** `scripts/clean-poisoned-claude-sessions.test.ts` is outside the `include` of `packages/db/vitest.config.ts`. - **`plugin-orchestration-smoke-example` does not collect.** `@paperclipai/shared` is not linked in its `node_modules`. It passes 3 of 3 with an alias. ## Verification Run these commands: ```sh node scripts/typecheck-test-programs.mjs --low-memory pnpm --filter @paperclipai/server typecheck node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations node --test scripts/typecheck-test-programs.test.mjs ``` Results on macOS (arm64), Node 26.12. CI uses Node 24, and its `Typecheck + Release Registry` job also passes. At head `7d4b5e907a`, the test programs, `tests/e2e`, and the script tests pass again with 0 errors. The times and peaks below come from the run before the two latest merges of master: | Check | Result | | --- | --- | | Test programs, `--low-memory` | Exit 0, 0 errors, 42 seconds, peak 1.78 GiB | | Server layers, low memory | Exit 0, 0 errors, 38 seconds, peak 3.41 GiB | | Test projects, including `tests/e2e` | Exit 0, 0 errors, 13 seconds, peak 1.59 GiB | | Script tests | 12 passed | | Injected type error in each of 23 places | Each one failed its program with TS2322 | | Coverage check | 0 unchecked files, 10 seconds | I did not run this in a 4 GiB container. The peaks above are `maximum resident set size` for the largest single process. **Runtime check of the fixes.** Types do not prove that a test still works, so I ran the suites before the two latest merges of master: | Suite | Result | | --- | --- | | Runner, full suite | 4,885 passed, 3 failed, 13 skipped | | Sandbox providers, 10 packages | 939 passed, 12 failed, 36 skipped | | Plugin SDK | 200 passed | | `kv-demo-mcp-server` | 12 passed | | Renamed UI test and its sibling | 4 passed | All 15 failures exist without these fixes or come from the local machine: - **3 Cursor Cloud timeouts.** `artifact-delivery.test.ts` has 1 and `cursor-cloud-driver.test.ts` has 2. They time out in a full local run, and all 133 tests in those files with `codex-security-config.test.ts` pass when run alone. They fail in the same way with those two files at their unedited versions. - **12 Kubernetes `file-sync.test.ts` failures.** They fail in the same way before and after the fixes. The first failure is that `/proc/self/fd` does not exist on macOS. After each of the two latest merges of master, I ran only the Runner test files that I changed: | Merge | Result | | --- | --- | | `885c249cb9`, 8 files | 534 passed | | `7d4b5e907a`, 7 files | 672 passed, 16 failed | The 16 failures are in the two `source-reconciliation` test files. They fail in the same way with master's unedited versions of those files. 15 of them fail with `ACPX sidecar exited at initialize`, so I think my local build of the Rust and sidecar binaries is out of date. I did not rebuild them to check. CI builds them from scratch, and all Runner jobs pass in CI at `7d4b5e907a`. **Not run:** - `tests/e2e` and the 8 Playwright suites need a server, so most of them have only compile checks. `playwright test --list` loads 308 tests from the main e2e config. - `packages/plugins/sandbox-providers/kubernetes/test/integration` needs a cluster, so it has only compile checks. - I did not run the full `pnpm test:run` or root `pnpm build`. Required CI runs them. **Reduced motion.** I ran 6 motion-sensitive Storybook specs in system Chrome with a temporary config, once with `reduce` and once with `no-preference`. Both runs had 45 passed and the same 5 failed. So reduced motion did not change those results. The 5 failures do not depend on motion. At least one comes from the browser, because system Chrome prints `oklch(55.6% 0 0)` where the test expects `oklch(0.556 0 0)`. I did not run the other Storybook specs. The `storybook-visual` label runs that workflow. ## Risks - **A new lockfile to maintain.** A change to a provider `package.json` now fails the typecheck job until someone refreshes `scripts/typecheck-sandbox-providers/pnpm-lock.yaml`. The failure message gives the command. - **The job has a new install.** The step downloads provider SDKs, so it depends on the npm registry. It uses `--frozen-lockfile` and `--ignore-scripts`. - **The local install replaces provider `node_modules`.** A contributor with a standalone provider install must reinstall that provider after running the script. - **Reduced motion can change Storybook results.** See "Behavior change for review". That workflow is not part of required CI. - **Kubernetes plugin validation changed.** A release of this plugin ships zod 4. Error wording changes, and very large integers are now rejected. See "Production change for review". - **Master moves fast.** Each merge of master brought new test code with type errors, 27 so far. After this merges, the check reports those errors in the pull request that adds them. ## Model Used - Provider: Anthropic Claude, through Claude Code - Model: Claude Opus 5.5 (`claude-opus-5-5`) - Capabilities: tool use, code execution, and subagents for the type fixes in separate files ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details. The app generated the branch name. - [ ] I have run tests locally and they pass. See the failures explained in Verification. - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 07:33:59 -08:00
Nicky LeachandClaude Opus 5.5	18ff71bca3	fix(ci): load the lockfile policy from the current base-branch tip (#19390) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Only trusted environments may change `pnpm-lock.yaml`. #19356 moved that rule into `.github/scripts/lockfile-trust.mjs`. > - The `pr-trusted.yml` policy job loads that script from trusted code, so a pull request cannot change the check that judges it. > - The job reads the trusted commit from `pull_request.base.sha`. GitHub does not move that value forward on every push. > - #17364 showed the result. Its recorded base was still the `master` commit from before #19356, so the job used the earlier rule. > - The earlier rule exempts any branch named `chore/refresh-lockfile`, including one from a fork. > - This pull request loads the policy from the first parent of GitHub's merge commit, which is the current base-branch tip. > - The benefit is that every pull request gets the current lockfile policy, including pull requests opened before #19356. ## Linked Issues or Issue Description Refs #19356, #17364. **What happened?** On #17364 at `482d76686e`, the "Block untrusted lockfile edits" step printed nothing and passed. `pull_request.base.sha` was `da3084a05d`, the `master` commit from before #19356. That commit has no `lockfile-trust.mjs`, so the step used the earlier rule. The merge commit that CI tested already contained #19356. **Expected behavior** The policy job runs the current `lockfile-trust.mjs` from `master` for every pull request. **Steps to reproduce** 1. Open a pull request before #19356 merged, or keep a pull request whose recorded base predates it. 2. Push a commit. GitHub keeps the old `pull_request.base.sha`. 3. The policy job uses the earlier rule. A fork branch named `chore/refresh-lockfile` can still change `pnpm-lock.yaml`. **Also found** In a local test, the script exited 0 without any check when it ran from a symlinked temporary directory. Its entry-point check compared a symlinked path with a real path. GitHub-hosted Linux runners do not use a symlinked `RUNNER_TEMP`, but the check must fail closed. ## What Changed - `pr-trusted.yml` "Block untrusted lockfile edits": - Load `lockfile-trust.mjs` from `$GITHUB_SHA^1`, the first parent of GitHub's merge commit. Do this only when `$GITHUB_SHA^2` is the PR head. Otherwise, keep `pull_request.base.sha`. - Give the script the same trusted base for its diff. - Fail the step if the script writes no `writer=` output. - `lockfile-trust.mjs`: compare real paths in the entry-point check. - `lockfile-trust.test.mjs`: run the real step script against a git history with a stale recorded base and a merge commit. Check that it blocks a contributor lockfile change on a `chore/refresh-lockfile` branch, allows a docs-only change, and falls back to the recorded base when no merge commit is present. - `doc/RELEASE-AUTOMATION-SETUP.md`: state where the job loads the policy from. ## Verification - `node --test '.github/scripts/tests/*.test.mjs'`: 709 passed. - The new stale-base test fails when the workflow change is removed. - The policy-step tests run with `RUNNER_TEMP` behind a symlink. They fail when the realpath fix is removed. - A test loads a policy that exits 0 without writing `writer=`. It fails when the fail-closed check is removed. - This pull request's own CI uses `pr-trusted.yml` from `master`. The new step takes effect after merge. ## Risks - If the checkout is not GitHub's merge commit, the step falls back to `pull_request.base.sha`, which is the current behavior. The `pull_request` event always checks out the merge commit by default. - The fail-closed check can fail a pull request if the script stops writing `writer=`. The script writes it on every path that returns. - Low risk otherwise. The change touches one workflow step, one script line, tests, and docs. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 07:25:41 -08:00
Nicky LeachandPaperclip	26071353d2	feat(plugins): define settled task release and resume (#19468) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Environment plugins provide Runner tasks through the public SDK. > - A settled turn can leave resources that a later run can reuse. > - Core needs a shared contract to end active use and request the same logical session later. > - The host must retain control of capacity and report when reuse is unavailable. > - This pull request adds an optional release and resume contract with strict validation and documentation. > - The benefit is a common boundary for later host and Core integration across Codex and ACPX. ## Linked Issues or Issue Description **Subsystem affected** packages/plugins — plugin SDK and its environment task documentation. **Problem or motivation** The SDK has stop and completion operations. It has no explicit contract to release active use after a settled turn and later request the same logical session without promising retention or the same process. **Proposed solution** Add a versioned optional capability, release for the identified session, and resume with fresh transient authentication. Define two explicit resume outcomes: resumed or unavailable. Keep authorization, safe retries and ordinary attachment requirements explicit. **Alternatives considered** A provider-specific command would couple Core to one host. Reusing stop, completion or archive would change their existing meaning. A new checkpoint format or Runner mode is unnecessary. **Roadmap alignment** This extends the existing plugin and sandbox boundaries in ROADMAP.md. It adds no Core caller, provider implementation or runtime activation. **Additional context** The duplicate search found #19371, which fixes Core failure cleanup for a session resume owner. Its scope is separate. Related recovery work in #19157 and admission work in #19196 remain unchanged. ## What Changed - Add strict release and resume operation/result schemas and optional releaseResumeVersion: 1. - Reuse the existing task identity, inferred SDK exports and worker RPC types. - Define settled release, host retention policy, atomic reuse reservation, fresh authentication, retries that cannot release a newer owner and ordinary recovery. - Add schema and serialized worker RPC tests, including legacy replies and failure cases. ## Verification - SDK typecheck: `node_modules/.bin/tsc -p packages/plugins/sdk/tsconfig.json --noEmit` passed. - From `packages/plugins/sdk`, `../../../node_modules/.bin/vitest run --config vitest.config.ts tests/environment-task-release-resume.test.ts tests/environment-tasks.test.ts tests/environment-task-error.test.ts tests/environment-task-error-code.test.ts tests/worker-rpc-host.test.ts` passed: 94 tests across five suites. - `git diff --check` passed. - [Full GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/42238743059) passed on `a55376801c91af29b68a13304d085706e4d7ed6f`. Standard Greptile scored that head 5/5 with no comments or unresolved review threads. - Full regressions run in GitHub CI, per the requested verification scope. No VM, deployment or live provider test was run. These tests prove the SDK boundary, not host retention or live attachment. ## Risks - This is an optional contract. Runtime behavior stays unchanged until later host and Core work uses it. - Older strict Core schemas reject the new description field. Providers must coordinate capability advertisement with a compatible Core. - Host implementations must enforce settlement, exact authority, exclusive claims and safe retries. Unknown outcomes remain errors. An unavailable result never proves physical termination. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, with repository inspection, code editing and local test tools. Context window size was not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 07:23:28 -08:00
Nicky LeachandClaude Opus 5.5	026dc6d135	fix(native): release a session-resume owner when the resumed run fails before its provider claim (#19371) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep a finalization row. Its lease owner tells the control plane that an execution may still be live. > - Pausing an agent waits for `agentExecutionsHaveStopped`. That check refuses to finish while any run of the agent has a lease owner, unless the run was cancelled and has stop proof. > - A same-run session resume claims that lease before it dispatches the run. If the run then fails before the native executor takes its own provider claim, nothing releases the resume owner. > - The leaked owner blocks pause and resume for good, and the task keeps looking like it has a live recovery path. > - This pull request settles those rows at the source, and it adds a stop-proof-gated release for rows that already leaked. > - The benefit is that agents can pause and resume after an ordinary resume failure, without database edits, and the cancelled-run stop-proof rule stays as it is. ## Linked Issues or Issue Description **What happened?** An agent stayed in `pausing` with `hostComplete=false` and climbing attempts. Its runs had `native_run_finalizations` rows with `phase: observed`, `recovery_state: resuming_session`, a `lease_owner` like `default:resume:<uuid>`, and no `result_id`. Each heartbeat run had ended `failed`: either `setup_failed` (`Failed to acquire lease for environment … Sandbox creation has failed`, `native_remote_recovery_lease_mismatch`) or `adapter_failed` (`runner_state_identity_mismatch: …`). The lease expired about 20 minutes after the run finished. Nothing settled the row. The only fix was to set `lease_owner` to null by hand. **Expected behavior** When a resumed run fails before any provider turn starts, its finalization row goes to `terminal_failure` and its lease owner is released, as the native executor does for failures it sees. A row that leaked before this fix is released in the background once its lease expires and the run has stop proof. Pause then completes. **Steps to reproduce** 1. Leave a native run in `retryable_failure` with a persisted checkpoint, on a task that already has an active recovery incident. 2. Let session resume claim it (`claimNativeSessionResumptions` sets `phase: observed` and a `…:resume:…` owner). 3. Make the resumed run fail before the executor claims it, for example when the environment driver cannot create the sandbox. 4. The heartbeat failure path marks the run `failed`. The issue release path does not settle the row, because it settles only when it opens a new incident. The owner stays set, and `agentExecutionsHaveStopped` returns false indefinitely. **Paperclip version or commit** `master` at 5ac05871b5. **Additional context** Root cause: two writers can end a resumed run without the executor. `failRun` and the setup-failure path in `heartbeat/run-completion.ts` never touch the coordinator. `recordNativeTerminalRecoveryIfNeeded` in the wake-queue adapter settles the coordinator only when no recovery incident exists yet. Agents that need their computer connection paired again usually already have an incident, so the owner leaks. Related, but different: #18905, #18866, #18072, and #17692 change how the stop check judges runs. This pull request does not change the stop check. ## What Changed - `server/src/services/native-runtime/native-failed-run-owner-release.ts` (new): one settlement for a result-less `observed` owner on a run that ended `failed`, `timed_out`, or `interrupted`. It writes `terminal_failure`, null lease fields, `recoveryState: "blocked"`, the run's failure code, and a `releasedOwner` record. It sets `nativePhase`, sends the same finalization and execution work notifications as the executor's terminal path, and logs `native.failed_run_owner_released` activity. Locks follow the issue, coordinator, run order. - `settleUnstartedNativeSessionResume`: runs after a dispatched session resume returns. It matches only the exact claim owner and the attempt the claim saw. The executor increments the attempt when it takes provider ownership, so an unchanged attempt proves that no provider turn started under this owner. The executor's own failures no longer match. - `releaseAbandonedNativeOwners`: the background release for rows that already leaked. It requires an expired lease and all of the stop proof that `agentExecutionsHaveStopped` checks: settled environment leases, no live run process or process group, no live controller lease, and no live controller process. This process answers from its in-memory execution registries. It skips cancelled runs, pending native Stops, and pinned workspace restores, because those have their own stop-proof owners. - `native-finalization-reconciler.ts`: `NativeSessionResumeClaim` now carries `attempt`. - `heartbeat.ts` and `heartbeat/recovery.ts`: both session-resume dispatch sites settle the claim after `executeRun` returns. This is one `.finally` per site. - `execution-control-reconciliation.ts` and `execution-control-work.ts`: the execution-finalization queue runs the background release. Its deadline includes the earliest lease expiry among eligible rows, so the release does not poll an idle instance. Rows with unreleased sandboxes are excluded from the deadline. - `agent-execution-stop.ts` is unchanged. The cancelled-only release keeps its stop-proof rule. ## Verification - `npx vitest run src/__tests__/heartbeat-native-resume-lease-failure.test.ts` drives the real heartbeat resume dispatch (`reapOrphanedRuns`). A sandbox driver's `acquireRunLease` throws `Sandbox creation has failed`, and the task has a recovery incident. The run fails with `Failed to acquire lease for environment "Managed computer" (sandbox): Sandbox creation has failed`, and the coordinator ends `terminal_failure` with no lease owner. I removed the `.finally` wiring and ran the test again. It then failed, and the row stayed `observed` with a `default:resume:` owner and a 20-minute expiry, which is the production state. - `npx vitest run src/services/native-runtime/native-failed-run-owner-release.test.ts` (10 tests): - A claimed resume that fails at lease acquisition is settled. `agentExecutionsHaveStopped` changes from false to true. A repeated settlement does nothing. - An owner the executor claimed (incremented attempt) and a run that is still running are not changed. - A leaked `runner_state_identity_mismatch` row is released by `reconcileAbandonedExecutionControl`, and then `agentExecutionsHaveStopped` returns true. - An unexpired owner still blocks the stop check. Its expiry becomes the finalization queue deadline. - An active environment lease and a live controller process block release. After both stop, the row is released. - Cancelled runs and pending native Stops are not changed. - Added during review: - Releasing an owner wakes the cost-accounting worker. - Releasing a run's environment lease wakes the finalization worker, and the owner is then released. - A recorded PID and group that now belong to an unrelated process do not hold the owner. - Bounded passes rotate past a held row (`[0, 1, 1]` with page size 1), then wrap back to it. - The immediate settle keeps the owner while the run's recorded process is alive. - Suites that cover the changed files pass together: the two new files, `native-session-resumption.test.ts` (with the claim shape updated), `execution-control-reconciliation.test.ts`, `execution-control-work.test.ts`, and `agent-lifecycle/__tests__/lifecycle.test.ts`. With `environment-cleanup-work.test.ts` added, that is 7 files and 156 tests. I did not run the full `heartbeat-process-recovery.test.ts` suite locally. CI runs it. - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Behaviour change: a resumed run that fails during setup now ends with a terminal coordinator in every case. Before, this happened only when the issue release path opened a new incident. After a restart, restart recovery no longer sees such a row as `observed`. This matches what the release path already does when no incident exists. - The background release trusts an expired lease on a run that is already terminal, together with the same process and lease evidence that the stop check uses. A controller on another host cannot be probed by PID. The stop check has the same limit today. A live owner still renews its lease, and lease-gated writes from an expired owner are already refused. - If an expired row's process or controller still looks live, the release retries at the shared 15-second finalization cadence until that process exits. Each pass handles one bounded page, and a rotating cursor moves to the next page on each pass. - A run-scoped environment cleanup notification now also wakes the execution-finalization queue. Passes are coalesced, and a pass is a cheap deadline query when nothing is eligible. - No schema change and no migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 07:19:09 -08:00
DottaandPaperclip	06c0566a08	fix: make task Retry work after a refused automatic attempt (#19466) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task Retry stores a durable request before it starts a new run. > - An automatic attempt can stop at the execution gate before a provider starts. > - That cancelled attempt becomes the source of the next Retry click. > - Admission rejected that source, or chose an older run that did not match the stored Retry authority. > - This pull request verifies the refused attempt and keeps it as the exact authority source. > - The user can start one new turn after the older execution holds pass their existing checks. ## Linked Issues or Issue Description **What happened?** Retry can remain deferred when the latest run was cancelled with `execution_reconciliation_required` before dispatch. An older failed preparation has an execution recovery hold. The Retry receipt names the cancelled child, but admission only accepts a failed preparation as an extra source. It can also choose the older run as `previousRunId`, which fails the exact source check at dispatch. **Expected behavior** Retry starts one new turn when the latest attempt has a verified pre-dispatch refusal and every older execution hold is ready. Provider activity, retained execution resources, missing receipts, or a live earlier process keep the hold. **Steps to reproduce** 1. Seed a failed native preparation and its recovery hold. 2. Let a later automatic attempt be cancelled by the stale queued execution gate before dispatch. 3. Click Retry against that latest cancelled attempt. 4. Before this change, continuation admission returns no receipt. After this change, one successor starts with authority bound to the clicked attempt. **Paperclip version or commit** Reproduced on `2f35eb1080`. Rebased onto `6cafcaf89b` for submission. **Deployment mode** Local source checkout with embedded PostgreSQL and the native Runner on macOS arm64. Refs #15383 and #16939. Those changes cover legacy conversation cancellation and queue promotion. This change covers the exact source of a task Retry receipt after a gate refused its automatic child. ## What Changed - Reuse the existing database-backed pre-dispatch refusal proof when admitting the exact Retry source. - Keep the clicked source as `previousRunId`. Continue to check every older execution hold. - Add database regressions for safe refusal, unsafe evidence, concurrent dispatch, and duplicate clicks. - Add an opt-in acceptance test that drives the authenticated Retry route through a real Codex turn and task finalization. - Document the admission contract. ## Verification - TDD: the new database regression returned `null` on the original code. It passes with the repair. - Final head `c43a84b1bf`: 487 tests pass across task Retry, explicit native continuation, and execution continuation. Server TypeScript and module boundary checks pass. - Live acceptance: `PAPERCLIP_LIVE_REFUSED_RETRY=true PAPERCLIP_RUNNER_BINARY=<current-source-runnerd> pnpm --dir server exec vitest run --config vitest.config.ts src/services/native-runtime/refused-task-retry.live.test.ts`. - The real HTTP request returned 202 on the final rebased head `c43a84b1bf`. Codex `gpt-6.1-sol` with high effort produced the requested file. The task became done, both task locks cleared, and finalization committed. A repeated click reused the same receipt and did not create another run. The source file stayed unchanged. - Server TypeScript check and module boundary check. - `pnpm verify --explain` returned 2 because the execution contract doc requires broad CI coverage. The required CI checks supply that coverage. ## Risks This changes execution admission. The existing proof requires the stored refusal receipt and no provider work or retained execution resources. Earlier recovery holds, process stop checks, decisions, actor authority, and task ownership still apply. No schema change or migration. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The host does not expose an exact author model ID or context window. The live acceptance provider was `gpt-6.1-sol` with high reasoning effort. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — all GitHub Actions checks pass on `c43a84b1bf`; Paperclip Review completed at 5/5 with no findings (neutral check conclusion). - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — reviewed `c43a84b1bf` with no actionable findings - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 07:03:54 -08:00
DottaandPaperclip	6ec4180ea1	fix: preserve ordinary diagnostic text during environment secret redaction (#19461) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Run diagnostics help operators find the cause of a failed agent run. > - The failure reporter treated unknown environment settings as credentials. > - It replaced every use of those values in diagnostic text. > - A flag set to `1`, `2`, or `safe` removed parts of dates, model names, versions, and normal words. > - This pull request preserves explicit ordinary settings and keeps unknown values private. > - Operators can read the failure details while real credentials stay masked. ## Linked Issues or Issue Description **What happened?** Run-failure contexts contained redaction markers inside timestamps, model names, driver versions, and the word "safely". Provider failure diagnostics used the same broad environment-value rule. **Expected behavior** Ordinary flags, counts, paths, regions, and mode labels leave diagnostic text readable. API keys, URL passwords, and declared secret bindings remain masked. **Steps to reproduce** 1. Set ordinary environment settings to `1`, `2`, and `safe`. 2. Report a failed native run with a timestamp, `gpt-6.1-sol`, `phase6-v2`, and "safely" in its details. 3. Inspect the run failure report. Before this fix, unrelated fragments are replaced. Related work: #13668 addresses a separate model-ID false positive. #11311 and #15524 address credential capture at other boundaries. This change keeps the existing command scanner and registered-secret replacement. ## What Changed - Add a shared environment selector in adapter-utils. Keep unknown values private. Exempt explicit ordinary process settings, typed flags and counts, mode labels, and key selectors. - Use the same selector for run-failure reports and inherited provider failure environments. Keep one policy for both consumers. - Keep configured provider bindings private because that API does not carry separate secret-reference metadata. - Preserve short declared secrets, unknown opaque values, encoded credentials, and inherited Bedrock tokens. Add regressions for ordinary `SORT_KEY` values and both Bedrock consumers. - Add screenshot-shaped regressions, real database reporter coverage, and credential tests. - Document the policy in `doc/observability.md`. The optional monitoring gate stays in place. ## Verification Linux, Node v25.9.0, pnpm 9.15.4. The published tree matches the tested local tree `cb236f072cca195e4c2cfb4034c7cc06ea7adbc5`. - Before the fix, four diagnostic regressions failed. The output reproduced the damaged timestamp, model, version, and word. - Review regressions: six tests failed on the first revision. They proved the missed Bedrock token, opaque inherited-value exposure, and key-selector false positive. All pass after the correction. - 555 focused tests pass on the final source tree. - `cd server && pnpm exec vitest run src/services/__tests__/run-failure-diagnostics.test.ts src/__tests__/redaction.test.ts src/__tests__/redact-sensitive.test.ts src/__tests__/run-secret-redaction.test.ts`: 206 passed. - `cd server && pnpm exec vitest run src/services/__tests__/run-failure-report.test.ts`: 289 passed, with real embedded PostgreSQL. - `pnpm exec vitest run packages/adapter-utils/src/command-redaction.test.ts packages/adapter-utils/src/acpx-engine/terminal-session-failure.test.ts`: 60 passed. - `pnpm --filter @paperclipai/adapter-utils typecheck`: passed. - `git diff --check`: passed. - The first reporter attempt could not load the unbuilt plugin SDK. It passed after `pnpm --filter @paperclipai/plugin-sdk ensure-build-deps`. - The filtered install omitted the CLI tsx loader, so `pnpm verify --explain` could not start. The equivalent verifier entry points ran with the installed server loader and returned exit 2: broader CI coverage is required for the observability document. - Full local build/typecheck and browser suites were not duplicated. [Final-revision CI](https://github.com/paperclipai/paperclip/actions/runs/42231412547) passed on `013960a4c6fa6279e1f28b0f6af5f036d7cad5a7`, attempt 1. This includes typecheck, build, general/server/chat tests, serialized suites, Runner checks, eight E2E shards, and the canary dry run. - [Greptile review](https://github.com/paperclipai/paperclip/pull/19461#issuecomment-6686592783) scored this head 5/5 with zero new findings. All four inline threads are resolved. The security rescan and Snyk status passed. The PR has no merge conflicts. ## Risks - Ordinary-setting exemptions need a known name or a name/value type rule. Unknown opaque values remain private. Declared secret bindings and recognizable credentials override these exemptions. - A real short secret still masks matching fragments. Removing that protection would expose the secret. - Existing stored redaction markers cannot restore their original text. This fix applies when new diagnostics are produced. - No migration, dependency, workflow, authorization, or default monitoring changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. This change is a focused bug fix. ## Model Used OpenAI GPT-6, through Codex, with shell execution and file tools. The host does not expose the exact runtime variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 09:01:55 -06:00
DottaandClaude Opus 5.5	d87f6fa4c2	test(runner-e2e): add staging-soak reliability cases to everyday workflows (#19455) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The paid runner E2E suite (`tests/runner-e2e`) drives the real browser, server, native runner, provider and Daytona > - A manual staging soak found about nine reliability bugs (S47–S55, H2, U9). Each fix got unit or integration tests > - The end-to-end journeys that exposed the bugs existed only as ad-hoc browser scripts, so a regression could return without a signal > - These journeys are expensive, so they must not run per PR or in `--all`. They must run on demand before a release > - This pull request adds the journeys to the existing `everyday-workflows` suite as an explicit `reliability` group > - The benefit is one command that runs the whole reliability set before a release or after a risky runner change ## Linked Issues or Issue Description **What is the problem?** The staging-soak scenarios that found S47–S55, H2 and U9 are not repeatable. They were ad-hoc browser scripts on staging. **Why does it matter?** Without a repeatable end-to-end case, a reliability regression (stalled follow-ups, duplicate execution, dead Retry, hangs after silent commands) can return without a signal. **What is the proposed solution?** Add the scenarios to the paid `everyday-workflows` suite as explicit-only cases with independent graders and a selectable `reliability` group. Refs #19370, #19355, #19354, #19407, #19419, #19394, #19421, #19431 (the related fixes). ## What Changed - `everyday-reliability-cases.ts`: seven new cases (`interrupt-redirect`, `reassign-midrun`, `three-turn-revise`, `long-silent-command`, `tree-pause-release`, `queue-steer`, `queued-across-restart`), their soak signatures, prompts, bounds and the explicit matrix. - `everyday-reliability-flow.ts`: browser and public-API drivers for each journey. A shared watcher records `executionBlocker`, `interruption` and execution projections while it waits. It uses an offered Retry at most once (never while an automatic retry is scheduled). - `everyday-reliability.ts`: pure graders. They encode the product rule: interrupted work resumes automatically or offers a Retry that works on first use. A send-a-message blocker, a dead or repeated Retry, a silent stall, or duplicate execution fails. - `types.ts` / `selectors.ts`: new optional `explicitGroup` on a task. Such a task is selected only by its group, `--case`, or an exact `--id`. `--all`, the plain suite and generic selectors skip it. `--list` still shows it. - `catalog.ts`: the suite gets the reliability tasks, the released Grok/Kimi/Pi facades (only for `long-silent-command`), a per-turn Daytona override for `three-turn-revise`, the `reliability` and `slow` groups, definition version 11, and a computed `expectedMatrixSize` (52 → 62). - `everyday-flow.ts`: a narrow hook calls the scenario driver after task creation. Riley is created for the reassign and tree cases. The reliability checks are added after settlement. - `everyday-observations.ts`: an allowed interrupted run may end `operator_interrupted` or `issue_reassigned`. Lifecycle checks accept a declared reassigned parent owner. - `live-fixtures.ts`: reliability cells get the existing 1,000-cent company and lead-agent hard stop. - Tests: new `everyday-reliability.test.ts` (selection exclusion, matrix, credentials, and calibrated pass/fail cases for every grader). Updated counts in `catalog.test.ts`, `everyday.test.ts` and `installed-cli.test.ts`. - Docs: new "Staging-soak reliability group" section in `EVERYDAY-WORKFLOWS.md` (cases, signatures, run commands, cadence, coverage boundary) and a link from `doc/evals.md`. `everyday-flow.ts` was already over 1,000 lines. This PR adds only about 20 lines of wiring there. The scenario logic is in the new modules. Matrix (10 cells, all Daytona, single attempt): one `runner-codex` (gpt-5.6-sol) cell per case, plus `runner-acpx-grok`, `runner-acpx-kimi` and `runner-acpx-pi` for `long-silent-command`. Run command: ```sh pnpm test:e2e:runner -- --suite everyday-workflows --group reliability --max-parallel 2 ``` ## Verification - `pnpm test:e2e:runner:typecheck` passes. - `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts everyday catalog installed-cli selectors support.test`: 15 files, 362 tests pass. - The full runner-e2e Vitest run has the same 14 failures with and without this change (`controller-outage`, `operating-receipts`, `pi-controls-evidence`, `pi-file-evidence`, `workflow-security`). These fail on the base commit in this local environment too. - `pnpm test:e2e:runner -- --list --suite everyday-workflows --group reliability` lists exactly the ten cells. - `--all`, `--suite everyday-workflows` (still 52 cells), profile/environment selectors and the daily/weekly scheduled selections contain no reliability cell. A test asserts this. - Not run: the paid cells. They need provider and Daytona credentials. The first paid run of the group is the acceptance step. ## Risks - The drivers have not run against a live stack. A first paid run can expose harness timing or selector issues (for example, how fast the queued Interrupt control appears). The failure then names the stalled step and keeps `reliability-observations.json`. - `long-silent-command` does not inject a runner connection loss. The named-cause contract is checked only if a loss happens naturally. - The Grok/Kimi/Pi facades run as ordinary released profiles. If one returns to pending qualification, it drops out of the group and the matrix size follows. - No product code changes. Existing everyday cells keep their selection and checks. The suite definition hash changes because the suite definition changed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, code execution). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 08:37:22 -06:00
DottaandPaperclip	6cafcaf89b	fix(workspaces): recover unpublished archived Git uploads (#19459) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspace imports preserve existing files before the server changes workspace authority. > - Archived Git imports reconstruct saved files and Git state in a private repository. > - A failed upload can leave a pending snapshot journal with no object in storage. > - A later reconstruction can have different Git administrative bytes for the same saved logical source. > - This pull request lets that import record a new unpublished capture after it proves that prior objects are absent. > - The benefit is safe retry through the original import journal while source checks and atomic publication remain in place. ## Linked Issues or Issue Description Refs #19396. Refs #19409. A snapshot upload can fail before object persistence. The caller then removes its private archive. A retry reconstructs the same saved Git workspace. If its administrative bytes differ, the snapshot owner rejects the retry with `remote_workspace_snapshot_operation_content_changed`. The pending journal remains blocked even though no snapshot object or remote authority was published. ## What Changed - Add an optional unpublished recapture capability to snapshot storage. Bind it to the original source operation. Existing callers keep the exact manifest check. - Enable this capability only for archived Git reconstruction. The migration owner keeps the source recipe, task binding, ownership, and publication checks. - Verify every prior object before new capture admission. Unknown or corrupt objects stop the retry. Reuse an available prior object, including an object that appears while the replacement upload runs. - Compare the observed upload journal under the snapshot row lock before appending an attempt. Keep network I/O outside that lock. Preserve all prior attempt metadata. - Keep the first atomic publication as the winner. A losing uploader verifies and returns that saved reference. - Add real PostgreSQL, filesystem, and Git regression tests. Document the retry behavior. ## Verification - Baseline `70d23612dd9176cae8ee6b2b6dcf2f6eade9a42a`: the new real Git fixture fails with `remote_workspace_snapshot_operation_content_changed`. It fails the first upload before persistence, removes the old source cache, and advances unrelated remote refs. The saved task commit and restore point stay fixed. - Final frozen source: 20 tests pass in a fresh process. This includes the real Git regression and all snapshot storage cases. Tests cover prior object recovery, corrupt and unknown objects, normal caller mismatch rejection, operation binding, stale journal admission, and late upload publication in both orders. - Reproduce the final focused check from `server/`: ```sh node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/services/remote-workspace-snapshot-storage.test.ts src/services/remote-workspace-migration.test.ts -t 'failed-upload-retry|snapshot storage' --reporter=dot ``` - An earlier six-suite run had 201 passing cases and one failing late-object case. That run started before the final prior-object recheck was added. The failing case passes in the fresh final run above. This earlier run is not a clean full-family result for the final source. - That run skipped 16 lifecycle cases because `PAPERCLIP_REMOTE_WORKSPACE_LIFECYCLE_TEST_DATABASE_URL` was unset. Migration, storage, reservation, and concurrency tests used real disposable embedded PostgreSQL. - `git diff --check` passes. Full repository typecheck passed before rebasing. Full test, build, and current-head CI results are pending. ## Risks - The optional capability permits different physical captures. The migration owner must prove that the logical source and current ownership are unchanged. Ordinary mutable workspace uploads keep the previous mismatch guard. - An old upload can complete after the last object check. The first authenticated atomic publication wins. A late upload cannot replace its object or snapshot reference. Every candidate remains in the retention journal. - Unknown storage results and journal changes fail closed. They can require another canonical retry. This change adds no retry loop and changes no database schema. ## Model Used OpenAI Codex, GPT-6 family. The worker used reasoning, tool use, code editing, and local test execution. The worker runtime did not expose an exact model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 08:04:35 -06:00
DottaandPaperclip	4696ff4990	fix: reserve agent In Progress display for active runs (#19453) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task status must tell operators whether an agent is working or waiting. > - A task can retain its in-progress lifecycle status after its run stops. > - The shared read projection only handled scheduled monitoring waits. > - The UI also ignored other projected statuses for in-progress tasks. > - This pull request requires an active task run for the In Progress display. > - Operators can see waits without changing execution or recovery authority. ## Linked Issues or Issue Description **What happened?** An agent task still showed In Progress after its run ended with an access failure and left a pending connection request. Stale execution locks and saved lifecycle state could show the same false activity. **Expected behavior** An agent task shows In Progress only with a task-linked queued or running run. Otherwise, it shows Blocked for recovery, In Review for a pending response or review, Monitoring for a scheduled check, or Todo for remaining work. Human ownership keeps its selected status. **Steps to reproduce** 1. Create an agent-owned task with an in-progress lifecycle status. 2. End its run and retain a pending human request. 3. Read the task, its summary, and search results. Before this change, they still show In Progress. ## What Changed - Derive visible activity from company-scoped, task-linked runs in the existing shared SQL projection. - Project inactive tasks to existing waiting or todo states. Preserve lifecycle status, locks, admission, and recovery. - Make the shared UI resolver use the projection while preserving explicit and optimistic status changes. - Cover stopped runs, stale locks, human waits, company isolation, monitoring, filters, counts, search, and UI labels. - Document the presentation rule in the product and execution contracts. ## Verification - Red before the server fix: 7 new regression cases failed because they returned in_progress. The 10 existing cases passed. - Green after the server fix: 20 cases passed in issue-display-status.test.ts, using embedded Postgres and real task APIs/search. Seven nearby active-question cases also passed. - Review regression: dismissed requests, historical questions, and returned review work each failed before the follow-up fix. All three now pass using the existing active-interaction rule and pending review state. - Red before the UI fix: 4 label cases failed. Five additional safety cases caught stale projections hiding explicit transitions. - Green after the UI fix: 43 tests passed across task-status-presentation, issue-filters, and issue-detail-subissues. - pnpm -r typecheck passed. All UI token gates and synchronization checks passed. - pnpm build passed. Server typecheck and build passed again after the review fixes. - Full CI passed on 45c4ababc5, including typecheck, build, general/serialized tests, Runner tests, and browser checks. The final rollup has 55 successful checks, one neutral check, and three configured skips. - The local pnpm test:run command passed all 1,931 chat integration tests. Its remaining duplicate full-suite run was stopped once full CI passed on the same commit. This is not a claim that the complete local command finished. - Greptile reviewed the final commit at 5/5. Both prior findings have regression coverage and all review threads are resolved. - Live acceptance at commit 45c4ababc5: started a fresh isolated local instance, created an inactive agent task, and saved a pending human request. API reads changed from Todo to In Review. An initial readiness probe ran before startup finished; the doctor then verified the exact serving revision before acceptance. Chrome showed In Review in the task header, breadcrumb, and sidebar. Stored lifecycle status remained in_progress. The instance was stopped after verification. - The reported stopped state was inspected in the live UI. The fix is not deployed there. Local API tests reproduce the failure shape without replaying the real deployment task. ## Risks - This changes visible status and display-status filter membership for inactive agent tasks. Stored lifecycle filters keep their existing meaning. - The projection uses existing indexes on task runs, recovery actions, and interactions. It adds no worker, timer, schema change, or execution action. - A queued run counts as active admission. Human-owned tasks retain explicit ownership status. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. A more specific model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 07:59:38 -06:00
DottaandPaperclip	2f35eb1080	feat(dev): add incremental local verification and pass reuse (#19345) ... ## Thinking Path > - Paperclip coordinates agents that make and verify repository changes. > - Local verification currently defaults to the complete Vitest suite. > - Contributor instructions also require broad local checks at each PR handoff. > - CI already owns conservative affected-test selection and broad final-revision checks. > - This pull request adds local selection and pass reuse from those existing contracts. > - Agents can get focused feedback and use CI to close the remaining coverage gaps. ## Linked Issues or Issue Description **Subsystem affected** Developer verification scripts and contributor guidance. Related selection work: #18398 and #18464. **Problem or motivation** An agent that repeats an unchanged local verification step starts the complete suite again. The handoff guidance also duplicates broad CI coverage locally. **Proposed solution** Add `pnpm verify` for the actual working tree. Select affected Vitest specifications with the CI task policy and dependency graph. Reuse successful checks whose inputs are modeled. Run opaque checks fresh. Report exit 2 when broader verification is required. Update the contributor contract to accept applicable final-revision CI coverage. **Alternatives considered** Plain `vitest related` omits filesystem and opaque dependencies. Commit-only cache keys miss dirty files. Running full local suites at every handoff repeats CI work. **Roadmap alignment** This is repository development tooling. It does not change the product roadmap. ## What Changed - Add `pnpm verify`, `--explain`, `--base`, `--full`, and `--no-cache`. - Reuse CI selection and exact runner ownership for dirty, staged, committed, deleted, renamed, and untracked inputs. - Keep native server suites and serialized suites in their local owners. Keep six sequential chat shards. - Cache selected passes within one worktree. Hash source, fixtures, installed package contents, dependency bindings and installation metadata, dotenv files, child environment, toolchains, platform, and selection. - Run opaque consumers fresh. Preserve failed, interrupted, mutated-input, and partial-coverage outcomes. - Update AGENTS.md, CONTRIBUTING.md, and DEVELOPING.md guidance. Record the shared `verify` command as the sole exception to the package-script rule. Register the new tests in the existing CI contract-test command. - Add optional root/environment inputs to shared graph helpers. Existing CI defaults stay the same. ## Verification - Fourteen new integration tests pass locally across focused runs with real Vitest selection and execution; CI colors are covered. - 51 existing targeted selector/input, project-isolation, and stable-runner checks passed. Four relevant runner contracts also pass after the workspace change. - Repeated unchanged execution skips reusable tests. Source, environment, installed code, dependency bindings, and new untracked tests invalidate results. - Installed-code mutation with a restored modification time still invalidates a cached pass. Internal skill-directory links preserve reuse. - Workspace B/C tests prove discovery, reusable/fresh partition execution, pass reuse, and external-input failure. Chat selection produces exactly six single-worker partitions. - A deliberate unrelated failing test is omitted by narrow selection and caught by `--full`. - Red/green regressions prove fingerprints preserve file boundaries and reject edits made between diff discovery and snapshotting. - Opaque external-file changes still fail on a subsequent run. Source mutation and process interruption cannot publish reusable passes. - `pnpm verify --explain` works in the checkout and requests broad CI coverage for this tooling change. No duplicate full local suite was started. - [Final-revision CI](https://github.com/paperclipai/paperclip/actions/runs/42117291789) passed on `ed85d96ddff82c05c325948680b593b0f48c473d` (attempt 2). The scoped Canary retry preserved all passing test/build results. - [Greptile review](https://github.com/paperclipai/paperclip/pull/19345#issuecomment-6670050175) is 5/5; all four review threads are resolved. - The additional Paperclip review concluded neutral because its execution stopped; it produced no assessment. ## Risks - Pass reuse depends on conservative dependency classification. Opaque consumers, external database/live settings, and non-isolated projects retain fresh or broad coverage. - Local passes cover selected Vitest checks. Required CI, build/typecheck, release, browser, and platform acceptance remain separate obligations. - Cross-worktree caching and cross-agent scheduling are outside this change. - The initial installation scan reads dependency contents; stat-validated, worktree-local digest memoization avoids rereading unchanged contents. Subsequent scans still inspect file identity and nanosecond change/modification times. - No dependency lockfile or workflow changes. ## Model Used OpenAI Codex, `gpt-6.1-sol`, max reasoning, with local shell and file tools. Context window size is not exposed by this host. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 07:29:10 -06:00
DottaandPaperclip	70d23612dd	fix: offer Interrupt when queued-message steering is unavailable (#19450) ... ## Thinking Path > - Paperclip lets people manage work performed by AI agents. > - Task chat saves follow-up messages while an agent runs. > - A native runner can accept Steer only after an active provider turn exists. > - During preparation, or with an unsupported provider, the queue showed a disabled Steer button and no delivery action. > - The server also rejected Interrupt for ordinary native queues. > - This change offers an explicit Interrupt fallback and uses the existing cancellation and admission path. > - People can send their saved messages while Stop confirmation, cleanup, and execution gates remain enforced. ## Linked Issues or Issue Description Related: #15865 covers mixed legacy queues. #15030 covers steering acknowledgements. This change covers native queues without available steering. **What happened?** A queued message had a disabled Steer button while the native runner was still preparing. Clicking it did nothing. The server returned HTTP 409 for an explicit Interrupt request against the same native queue. **Expected behavior** Offer Steer when the provider can accept same-turn input. Otherwise offer Interrupt, labeled with its actual effect. Preserve saved messages until the previous execution stops and normal admission accepts the successor. **Steps to reproduce** 1. Start a task with the Paperclip Runner adapter. 2. Submit a follow-up while the run is preparing, before its provider turn opens. 3. Observe the disabled Steer button above the composer. 4. With this change, click Interrupt. The existing Stop path cancels the previous execution and submits the saved queue through normal admission. **Paperclip version or commit** Reproduced locally on master at 523818ea9b. The branch is rebased on 2f9d3d43c3. ## What Changed - Offer Interrupt for unavailable or unsupported native steering when the parent supplies an interrupt callback. - Keep Steer for an available provider turn. - Accept explicit Interrupt for a deferred native queue and record its run target. - Bind native continuation to the exact Interrupt queue, run, actor, and cancellation receipt. This lets one click send the messages after Stop completes. - Add component, task-thread, real PostgreSQL route, and continuation authorization regressions. - Close the route fixture database pool before disposing its PostgreSQL instance. - Document the fallback and its Stop and admission requirements. ## Verification - Red: two component cases failed because Interrupt was absent. Two route cases returned HTTP 409, `This queue does not support interruption`, before the repair. - Red: stronger delivery assertions also caught `send_predates_stop` after the first repair. The admission fix covers both the freshness exception and later eligibility check. - Green: both route regressions now wait for preparation settlement, then deliver both saved messages exactly once without a second click. The complete route suite passes 119 tests. The complete continuation suite passes 376 tests, including wrong actor, queue, target, and missing-receipt cases. - UI: task-thread and queued-message suites pass 394 tests, with one existing skip. The issue-page suite also passes (the earlier three-suite run passed 644 tests before the final three UI cases). - Token gates and `git diff --check` pass. - Full workspace build and low-memory typecheck pass. The initial typecheck caught a missing `membershipRole` in an unrelated fixture; the rebase picked up the correction already on master, and the complete typecheck then passed. - The full local `pnpm test:run` passed all six chat shards (1,931 tests). It was stopped during general-server validation after the final commit passed all PR CI gates. The complete local suite is not claimed as a pass. - Earlier route-suite teardown timed out with an open database pool. Closing that pool fixed teardown; the complete route suite then passed. - All final-head CI gates pass: 52 successful checks and two intentional skips. Greptile is 5/5 on the same head with no unresolved review threads. ## Risks Interrupt stops the previous execution and starts a successor; it is a different action from same-turn steering. The UI labels that action explicitly. Cleanup or execution gates can keep the messages queued. Existing company access, board permissions, queue revision, run target, cancellation proof, and admission checks remain in place. No schema migration is needed. ## Model Used OpenAI Codex (GPT-6), with reasoning, tool use, and code execution. The exact model ID and context window are not exposed by this client. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 07:11:38 -06:00
DottaandPaperclip	275b334fc5	fix(workspaces): skip redundant repository admission snapshots (#19444) ... ## Thinking Path > - Paperclip manages AI agents and their execution state. > - Remote workspace admission must preserve files before execution starts. > - Admission saved the entire workspace once per registered repository on every run. > - Existing snapshot receipts can prove that validated repositories are already saved. > - This change uses those receipts and saves new repositories once per admission batch. > - It also preserves fresh configuration metadata and reports actual workspace reuse. ## Linked Issues or Issue Description **What happened** Remote follow-up runs uploaded an unchanged workspace for each registered repository before starting the provider. Configuration persistence also overwrote new evaluation metadata, and freshness logs could claim replacement when the same workspace was reused. **Expected behavior** Validate existing repositories without uploads when a committed snapshot covers their descriptors. Save acquired repositories before publishing ready receipts. Retain current evaluation metadata and report the actual workspace outcome. **Steps to reproduce** Use a remote workspace with two registered repositories. Prepare it, then admit a follow-up run. Observe repository snapshot operations despite unchanged repositories. The regression tests exercise this with real Postgres, Git, and separate sandbox directories. **Environment** Server remote workspace admission at commit c0214826b3. Related work: #19374 handles interrupted clone staging; this change handles snapshot admission cost. ## What Changed - Skip repository saves when the committed snapshot covers the validated descriptor. - Save all repositories in one admission snapshot before publishing ready receipts. - Retain descriptor coverage only when the new archive contains the checkout. Drop deleted checkout coverage so admission can acquire it again. - Restore `needs_repair` status for failed batch saves and retry failed uploads. - Let fresh preparation metadata override stale reserved metadata while preserving storage authority. - Report actual reuse when freshness policy requested replacement. - Add regression coverage and fix an existing routine user fixture missing its membership role. ## Verification - Red: original code fails unnecessary-snapshot, evaluation-metadata, and reuse-message regressions. - Green: 517 affected tests across six suites before the final coverage correction. After that correction, all 102 remote workspace/concurrency tests and 96 archive tests pass. Imported-repository, deleted-checkout, and upload-retry cases pass. - Additional red/green: deleted checkout coverage fails before the fix and passes after it. All 96 archive tests pass, including manifest digest and size bounds. - `pnpm -r typecheck`, `pnpm build`, and `pnpm check:token-gates` pass locally. - All final-head CI verification jobs pass, including the aggregate `ci / verify` check. Greptile gives the final head 5/5. - The monolithic local `pnpm test:run` passed all six chat shards and continued through general server tests. It was stopped after final-head CI passed; it did not complete locally. The targeted local regressions, affected suites, typecheck, and build completed successfully. ## Risks Snapshot receipts must refer to committed bytes. Legacy receipts require one conservative save. Failed uploads must not publish repository readiness. Replacement-class drift still retains the original configuration for an explicitly reused workspace. Existing directories with damaged or missing Git metadata still require repair. Master and this PR both validate those directories without cloning over them, preserving any surviving user files. Clearing a snapshot descriptor would not change that path. Complete directory deletion is different: this PR clears coverage so normal acquisition can clone it again. ## Model Used OpenAI gpt-6.1-sol through Codex. Used reasoning, repository inspection, code editing, and local execution. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 07:08:59 -06:00
DottaandPaperclip	c0214826b3	fix(workspaces): preserve pending repository requests during import (#19439) ... ## Thinking Path > - Paperclip manages agent work and its saved files. > - An ephemeral web server needs remote workspace authority before the old volume can detach. > - A task can have a repository request that has never prepared a checkout. > - The offline importer currently treats that pending request as unfinished file publication. > - This blocks migration even when the target does not exist and there is no saved commit, branch or failure. > - This pull request preserves that request in the database and imports the existing files separately. > - The next authorized sandbox preparation can acquire the repository through its existing owner. ## Linked Issues or Issue Description **What happened?** On Core `253bf071f0a496554b61fff897ad3f11afaf8766`, offline migration of a volume-backed workspace fails with `repository_preparation_unresolved` for an untouched pending repository request. The requested directory has never existed. The request has no pinned commit, branch or failure. **Expected behavior** Preserve the pending request and all existing files. Do not label the absent checkout as part of a saved archive. Keep partial publication and active preparation blocked. **Steps to reproduce** Create a local task workspace. Add an unmaterialized pending repository request. Run the canonical offline importer after stopping writers. The unchanged importer rejects the request before preserving the workspace. Related migration work: #19429. A GitHub search found no duplicate pending-repository import fix or matching public issue. ## What Changed - Admit an untouched pending request only at the existing offline writer fence and after proving its target is absent. - Keep the database request unchanged and exclude it from materialized snapshot descriptors. - Recheck file absence and source identity before publication. Reject a request that becomes ready during capture. - Cover cold filesystem loss, real sandbox-side Git acquisition, and eleven blocked states. Document the preservation contract. ## Verification - Red: the new real PostgreSQL/filesystem regression fails on the unchanged importer with `repository_preparation_unresolved`. - Green: both affected migration/lifecycle suites pass, **236 tests**. The cold test deletes the predecessor directory, restores saved bytes, and acquires the pending repository through the full admission owner, checking ready state and the pinned commit. Real Git runs against a local HTTP origin through an exact-URL redirect at the sandbox transport boundary. It does not claim live Daytona or GitHub qualification. - Full workspace typecheck and build pass at the final head. Both affected suites pass **236 tests**, including the full admission owner and eleven blocked states. All final-head CI checks pass: **54 success, 2 skipped, 1 neutral**. Greptile is **5/5**, with no unresolved review threads. - The full local test command failed in `recovers lost inline and check responses without duplicate GitHub publications` in the chat integration suite. That exact case passes in a fresh isolated invocation. Its cause remains unclassified; this is not a full-local-suite pass claim. The original failure is retained alongside the isolated result. - No candidate code has been applied to the serving instance. ## Risks - The exception applies only to offline migration and a pending request with no target, commit, branch or failure. Unreadable paths, symlinks, partial directories and active/repair states retain their blockers. - Requests retain normal repository authorization and acquisition behavior. This change neither fetches repositories during migration nor marks requests ready. - A changed materialized source aborts publication and leaves original files intact. ## Model Used OpenAI GPT-6 through Codex, with reasoning and tool use. The exact deployment variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 05:49:16 -06:00
DottaandPaperclip	21d2a577bb	feat: show and change routine responsible users (#19128) ... ## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Routines create tasks for repeated work. > - Each routine has a responsible person. > - The Overview did not show this person or permit a change. > - People need to see who will be responsible for automatic tasks. > - This pull request adds a small name picker beside Default agent. > - Authorized people can adopt a routine or transfer it to an active company member. ## Linked Issues or Issue Description **What existing behavior does this improve?** The routine Overview and the routine update API. **Subsystem affected** UI, server, and shared contracts. **Current behavior** The routine stores a responsible user. The Overview does not show that person. The update API cannot change that person. **Proposed behavior** Show Responsible user beside Default agent. Click the name to search active company members. Save only the new person and the current revision ID. **Reason and benefit** People can see who owns future automatic tasks. An authorized person can adopt a routine without making a new routine. **Breaking changes** The detail response adds an optional actor-specific field. The update API adds an optional responsible user ID. Existing requests keep their current behavior. Related: #17368 covers local Board ownership migration. This change adds the routine control and explicit transfer. ## What Changed - Show a quiet, searchable person picker beside Default agent. - Explain that automatic tasks use this person. Manual runs use the person who starts them. - Permit human operators with `tasks:assign`, instance administrators, and local operators to transfer responsibility. - Check active company membership inside the update transaction. - Save a revision and record the old and new person in the activity log. - Keep existing task responsibility and ownership during revision restore. - Show errors for failed saves and stale revisions. - Add tests for permissions, company boundaries, transfers, inheritance, and picker states. - Keep the picker disabled until saved data is refreshed. Refresh the activity list after a transfer. - Capture the actual previous person under the update row lock for the activity record. - Use shared query error handling and retain cached people during a connection failure. ## Verification - Focused checks cover 145 tests in 5 files. The first post-fix run passed 144 tests. One existing Fireflies route test timed out during the first import while the UI typecheck also ran. The same test passed in the pre-fix run. The isolated route suite then passed all 21 tests with no source change. - The new tests reproduce slow detail refreshes, activity refresh, and two concurrent transfers without a base revision. The concurrency test uses PostgreSQL row locks. - UI and server TypeScript checks passed at commit `1df9587eb`. - The UI production build passed for the updated head. - The branch includes master through `26391707f`. - Design token gates, query-error policy, and Git whitespace checks passed. The query-error policy reports zero findings. - Local browser checks passed on desktop and mobile for the initial feature commit. Search, selection, reload persistence, and revision creation passed. No page errors or horizontal overflow were found. The follow-up does not change the normal layout. - Earlier full local checks could not finish: Cargo was absent, and concurrent full test/build checks exceeded the 8 GiB memory limit. These attempts are not passes. - The first CI run failed on two error-rendering policy findings. That failure is fixed locally. The three findings from the first Greptile review are fixed and their threads are resolved. - Greptile reviewed `1df9587eb2d3e034c2769b1f6955ac52323c95bb` and scored it 5/5. It confirms all three prior findings are addressed. There are no open review threads. - [CI attempt 1](https://github.com/paperclipai/paperclip/actions/runs/42046369415/attempts/1) passed 53 checks. The server 8/12 shard failed only because the existing `history-http` runtime fixture did not become ready within 10 seconds (99 refused connections). The fixture and runtime source are unchanged from the merged master. That case passed locally on the same PR head with no source change. Its full 75-test suite also passed locally. The cause of the CI failure remains unknown. - [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/42046369415/attempts/2) passed on `1df9587eb`: 54 successful checks, 3 skipped, and 1 neutral. No failed or pending checks remained. The original fixture-readiness failure remains recorded above. - A later review found that viewers with an explicit `tasks:assign` grant could see an editable picker even though writes were denied. Commit `8b95ee707` now mirrors active-membership and viewer restrictions in the GET capability flag. Instance-admin and local-operator behavior is preserved. - Three membership regressions failed on the incorrect editable flag before the fix. After the fix, all 43 routine-route and company-access cases passed (`--testTimeout 60000` accommodated this sandbox's cold route imports; no repository timeout was changed). The first pre-fix run timed out during the cold import in one case; the second pre-fix run reproduced all three assertion failures. - Layered server typechecking and whitespace checks passed after the permission fix. - [Fresh CI](https://github.com/paperclipai/paperclip/actions/runs/42057752313) passed on `8b95ee7075417527687b4a86cbaf63ba87408364` without a retry. Greptile reviewed this exact head, scored it 5/5, and added no new findings. All four review threads are resolved. The security status is also successful. - The separate Paperclip Review check is still queued because its assigned review worker requires runtime recovery. This is the remaining check before completion; the CI workflow itself passed. ## Risks - A transfer changes the person used for future automatic tasks. Existing tasks keep their current person. - Only people can transfer responsibility. The server checks the permission again on every write. - A stale revision rejects the save. The user must reload before retrying. - No database migration is required. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The exact runtime model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (CI passed; separate Paperclip Review check pending) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 05:47:30 -06:00
DottaandPaperclip	253bf071f0	fix(workspaces): preserve resolved pending merges in snapshots (#19429) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task workspaces must survive replacement of the web server. > - Workspace snapshots therefore preserve both files and portable Git state. > - A resolved merge can still await its merge commit. > - The archive rejected this state, so an existing workspace could not migrate. > - This pull request saves and checks the complete supported merge state. > - The benefit is that migration preserves the user's choice to continue or abort the merge. ## Linked Issues or Issue Description **What happened?** A legacy workspace contained a resolved merge with `MERGE_HEAD`, `MERGE_MODE`, `MERGE_MSG`, `ORIG_HEAD`, and `AUTO_MERGE`. Its index had no unresolved entries. The snapshot owner refused the operation. This kept the original safe but blocked migration. **Expected behavior** Save this supported state without changing the source. Restore it without the original repository, borrowed object store, or linked worktree. Retain the merge parents, selected index, later edits, and Git's normal safety checks. **Steps to reproduce** 1. Merge two branches that conflict. 2. Resolve and stage the conflict. Leave the merge uncommitted. 3. Capture the workspace. The base implementation rejects the merge marker. Related prior work: #19310 preserves an inactive rebase bookmark. No duplicate open merge-preservation PR was found. This is a workspace preservation fix, not a new roadmap feature. ## What Changed - Admit only resolved pending merges with valid, bounded metadata and available objects. - Retain and verify the merge mode, message, parents, original head, automatic merge tree, and index. - Verify copied metadata and object closure before publication. Keep the before/after source checks. - Keep unresolved conflicts, autostash, rerere, mixed operations, malformed state, and unsafe links blocked. - Test real ordinary, linked, and borrowed repositories through local and serialized capture. Delete all original sources before continuing or aborting the restored merge. - Document the supported state and the remaining refusal cases. ## Verification - Six new continuation cases fail against the unchanged base at the pending-operation guard. - The first candidate restored and continued the merges, but six abort assertions failed. The fixture rewrote the worktree without updating Git's index cache. It now first verifies that abort protects later edits, uses normal `git restore` to discard those fixture edits, then aborts. No archive assertion was removed. - The final archive and environment suites pass all 98 tests at `3c87d13d1b`, including linked-metadata protections and pending merges in ordinary, linked, borrowed, shallow, partial, and SHA-256 repositories. All 12 local/serialized continuation cases pass after source removal. - Full repository build and typecheck pass at `1544cc4a90`. The build first found a nullability error; an explicit rejection return fixed it. - The first full local test attempt stopped before tests ran because omitted package postinstall steps left PostgreSQL library links absent. Direct `initdb --version` reproduced the missing-library failure. Restoring the package links made that probe pass; the full test rerun is active. - The initial typecheck also found missing excluded-plugin SDK links after the same install. Running the repository postinstall link helper corrected this, and the full typecheck now passes. - All current-head CI gates pass at `3c87d13d1b`: 54 success, 2 skipped, 1 neutral. Fresh Greptile is 5/5 and no review threads remain open. - Review requested shallow and SHA-256 coverage. The current commit adds both plus partial-clone coverage; all 98 archive tests and the full server typecheck pass. - All six local chat shards pass. The monolithic local general-suite run reports one viewer-permission test failure and remains active. That exact test passes alone, and its full 505-test file also passes alone. The original failure cause is unresolved; this is not a full local-suite PASS. The complete current-head CI suite is green. ## Risks - This widens capture eligibility only for the checked resolved-merge state. No migration commits or aborts the source operation. - A missing object or changed metadata still blocks publication and retains the source. - Unresolved merges and other operation formats remain unsupported. Their preservation guard is unchanged. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, and code execution. The runtime does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with the available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task id - [x] I have run the affected tests locally and they pass; full-suite limitations are recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented the risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 04:20:32 -06:00
DottaandClaude Opus 5.5	8c578668b8	fix(heartbeat): start new queued runs promptly after restart and name the wait (#19431) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts queued agent turns. Restarts and deploys happen often. > - After a restart, new queued turns waited minutes before they started. The task showed only "Queued", with no reason. > - The cause is startup order. The queued-run dispatch worker registers only after many unrelated first passes finish one after another: tool reviews, delivery workers, the terminal workspace sweep, native restart recovery, and residual and issue recovery. > - The server binds its listener before this recovery, and `/api/health` answers during it. A deployment can route traffic to the new server for minutes before queued work starts. > - This pull request starts queued turns that never executed as soon as plugin workers and adapters are loaded. It also names the wait while the server starts, recovers, or shuts down. > - The benefit is that tasks resume quickly after a restart and show a visible reason while they wait. ## Linked Issues or Issue Description **What happened?** On a hosted instance, the server restarted twice for deploys. Both times, queued runs on idle agents started about 3.5 to 4 minutes after the restart. Before the restart, the runs also waited about 5 minutes while the deploy drained the old server. During the wait after the restart, the execution presentation showed `phase: queued`, `label: "Queued"`, `cause: null`. **Expected behavior** Queued work resumes automatically and quickly after a restart. While it waits, the task shows a reason. **Steps to reproduce** 1. Start a task drain (`POST /api/instance/task-drain`). Assign issues to idle agents. The runs stay queued. 2. Stop the server (SIGTERM) and start it again. 3. Make any startup first pass slow (a large instance does this naturally). Queued runs start only after every earlier startup pass finishes. **Paperclip version or commit** master at 7e3caa7bac. **Deployment mode** Hosted Railway deployment with fleet deploy drain. Reproduced locally with embedded PostgreSQL and an injected 20-second startup step. ## What Changed - `heartbeat/queue.ts`: add `resumeFreshQueuedRunsAtStartup`. It starts queued runs on agents with no run left `running` and no queued retry or continuation (`started_at`, `retry_of_run_id`, `process_loss_retry_count`, `continuation_attempt`). Every run still passes the ordinary per-agent claim and dispatch gates, and scheduling suppression (including the task drain) still applies. - `heartbeat.ts`: expose it. Count it as a pending wake, like `resumeQueuedRunsAfterDrain`, so drain status and upgrade readiness see it. - `index.ts`: run that pass at startup after plugin workers and external adapters load. It does not wait for the unrelated startup first passes. It skips while a drain, warm standby, or shutdown is active. Log when it completes and when the full queued-run pass completes, with process uptime, so a hosted restart shows both timings. - `execution-projection.ts`: while the server is `starting`, `recovering`, or `draining` (shutdown), a queued run shows **Waiting for server restart** (cause `server_restarting`) with a next action. A task drain keeps its **Waiting for maintenance** reason. - `doc/execution-semantics.md`: describe the startup fast path and the new presentation. - The full queued-run pass, its position after native restart recovery, and the drain behavior are unchanged. ## Verification - Red, then green, local restart with a 20-second injected startup step (embedded PostgreSQL, three process agents, runs queued under a task drain, SIGTERM, restart): - Before: the listener bound at 09:41:50.6. Runs started at 09:42:10.8, 20.2 s later, after the slow step ended. - After: runs started 0.2 to 0.3 s after the listener bound, while the slow step still ran. Each original run executed once. - During the before run, `GET /api/issues/:id/execution` showed `Waiting for server restart` / `server_restarting` until dispatch. - Red, then green: `execution-projection.test.ts` restart-phase cases failed on the old code (`label: "Queued"`, `cause: null`) and pass now. - New `heartbeat-startup-fresh-dispatch.test.ts` (real PostgreSQL): a fresh run starts; an agent with a `running` row from the previous process and an agent with a queued retry stay queued; nothing starts while a task drain is active. - `npx vitest run` for the new test, `heartbeat-task-drain*.test.ts`, `execution-projection*.test.ts`, `startup-recovery-timing.test.ts`, `heartbeat/queue.test.ts`, `liveness.test.ts`: 132 passed. - `npx tsc --noEmit -p server/tsconfig.json` passed. ## Risks - New queued runs can start before native restart recovery finishes. The fast path excludes any agent with a `running` row or a queued retry or continuation, and the claim checks still enforce issue ownership, predecessor cleanup, budgets, pauses, and concurrency. A new wakeup during startup already starts runs at this point, so this adds no new class of early execution. - The drain is unchanged. It still holds new runs while the deploy waits for in-flight runs (the Cloud wait budget is at most 5 minutes). This PR does not change Cloud. - The new label depends on the process-local startup phase. It shows only on the process that is restarting. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local server runs). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 04:00:42 -06:00
DottaandClaude Opus 5.5	140fcf98ed	fix(retry): name a declined task Retry and let a new click replace it (#19426) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task interrupted by a restart or deploy must resume automatically, or offer a Retry that works on first use > - A board Retry saves an intent first. Dispatch then rechecks the requester's authority and saved source before it starts a run > - When that recheck refused the intent, dispatch returned a bare "deferred". The intent stayed saved with no reason, and the task still showed Retry > - Each later click with a new key joined that same intent, so Retry returned 202 and never started a run > - This pull request names the refusal on the intent and lets a new click replace an intent that can never start > - The benefit is that a Retry button that is shown starts one run, or explains why it cannot ## Linked Issues or Issue Description **What happened?** A native task in a managed sandbox lost its runner twice. Saved-work recovery recovered the files. The task showed "Saved work recovered. Retry or send a message to continue." and `canRetry: true`. A Retry click returned 202 and logged `issue.retry_requested`, but no run started. A second click with a new idempotency key returned the same request ID and `state: failed`. No run started. **Expected behavior** If Retry is offered, a click starts exactly one run. If the Retry cannot start, the task names the reason, and a later click is a fresh attempt. A Retry accepted during the pre-deploy drain starts after the restart. **Steps to reproduce** 1. Recover the files of a failed native run so the hold offers Retry. 2. Save a Retry intent whose dispatch-time authority recheck fails (for example, the requester lost access). 3. Dispatch it: no run starts, and the intent has no wait reason. 4. Click Retry again with a new key: the response returns the first request ID, and no run starts. **Paperclip version or commit** Reproduced on `b400914345` (staging served `4d49a31286`; the affected code is the same). Related: #19399 (automatic Retry after a lost runner), #19422. ## What Changed - `heartbeat/queue.ts`: when dispatch declines a task Retry because the requester can no longer change the task, or its saved board message changed, record a named wait (`retry_authority_denied`, `retry_source_changed`) and log it. Before, it returned "deferred" with no record. - `task-retry.ts`: a new click with a new key replaces an unadmitted intent that dispatch declined for its requester's access (`retry_authority_denied`). The old intent becomes `skipped` / `task_retry_replaced`, and activity records `replacedRequestId`. Waits on shared state (cleanup, recovery, an active run, preservation) still coalesce. The restart hook never replaces a user's click. Chat-owned intents keep their own owner. - `task-interruption-presentation.ts`: fixed public messages for the two new wait reasons. `retry_authority_denied` keeps Retry, because a new click is a fresh attempt. `retry_source_changed` is terminal: an edited saved message cannot be replayed, so Retry is hidden and the user sends a new message. - `doc/execution-semantics.md`: document the rule. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` use the real recovery, Retry and dispatch services on embedded Postgres. The fixture runs the automatic Retry from a lost runner, then that Retry's own run loses its runner, and recovery saves its files: - `starts exactly one run when Retry is pressed after the automatic Retry's own run lost its runner` (also checks that a replayed dispatch and a second click start no second run). - `does not let a Retry that could not start swallow a later Retry`: red before the fix in two steps. With no fix, the declined intent has no wait reason. With only the wait fix, the second click returned the first request ID. Green after the fix: one run, bound to the second click. - `dispatches a Retry accepted during the pre-deploy drain once the restarted server resumes`: green before and after. The drain does not lose a saved Retry. - `npx vitest run src/services/workspace-recovery.test.ts src/services/task-retry.test.ts src/services/task-interruption-presentation.test.ts src/services/quarantined-task-retry.test.ts src/services/explicit-native-continuation.test.ts`: 892 passed. - Not verified on staging. After deploy, an intent that dispatch refuses shows a named `retryWait` reason. ## Risks - Low risk. Only intents that dispatch declined for their requester's access are replaced. All other coalescing stays. - A replaced intent's late dispatch finds it `skipped` and starts nothing, so a double click still starts at most one run. - The exact gate that held the staging intent is not confirmed from records. If it was a shared-state wait, the new code names it only if it is one of the two reasons above. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 03:52:21 -06:00
DottaandClaude Opus 5.5	7e3caa7bac	fix(tree-control): resume work a pause gate refused after release (#19421) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The board can pause an issue tree. A pause stops work. Resume or release must continue that work once. > - PR #19394 made release re-dispatch a run that a pause Stop cancelled. > - That rule covers only a run that an operator Stop ended and that the hold snapshot named. > - A scheduled retry that comes due during a pause is not stopped. The dispatch gate cancels it with `issue_paused`. The release did not see it as owed work, so the task stalled with no run. > - This pull request makes every gate that refuses work for a pause record the covering hold. Release then re-dispatches that work exactly once. > - The benefit is that a paused task with a parked retry or a queued run continues after release, with no manual wake. ## Linked Issues or Issue Description Refs #19394 **What happened?** A child task had a `workspace_busy` scheduled retry. The board paused the parent tree. The retry came due during the pause. The scheduled-retry gate cancelled it with `issue_paused` ("Scheduled retry suppressed because the issue is held by an active subtree pause hold"). The board released the hold. Four minutes later the child was still `in_progress` with no run, no execution blocker, and no new activity. **Expected behavior** Release (or `mode: "resume"`) re-dispatches the refused work exactly once. No reconciliation and no user message are necessary. Stranded-issue recovery does not add a second run. **Steps to reproduce** 1. Give an assigned child task a parked `workspace_busy` scheduled retry. 2. Pause the parent tree. 3. Let the retry come due while the pause is active. The gate cancels it with `issue_paused`. 4. Release the pause. No run starts for the child. **Paperclip version or commit** master at 4d49a31286 (includes #19394). **Agent adapter(s) involved** Not adapter-specific (core bug). ## What Changed - `issue-tree-control.ts`: add `ISSUE_PAUSED_ERROR_CODE` and `pauseSuppressionResult()`. A gate records `treeHoldSuppression: { holdId, rootIssueId }` on a run that it refused for a pause. - `findOwedPauseResume` now treats the latest run as owed when it is an `issue_paused` cancellation under a pause hold. It uses the recorded hold. For an unmarked run (a scheduled retry cancelled before this change), it uses a snapshot pause that was active at cancellation. The operator Stop rule does not change. - Scheduled-retry gate (`run-dispatch/adapters/postgres.ts`, `cancelSuppressedRetryInTx`): locks the pause row `FOR SHARE` and records the covering hold. A concurrent release either sees the refusal, or the retry sees no pause and runs normally. - Queued-run claim gate (`heartbeat/queue.ts`): cancels with `issue_paused` and the hold marker, after the agent start lock is released. Before, the cancel re-entered that lock from inside it, and the agent's queue stalled until the 30-second stale-lock timeout. After the refusal is saved, the gate re-checks the hold and resumes the task itself if release already finished (`resumeIfPauseAlreadyReleased`). - Release (`issue-tree-resume.ts`, route): revisits the snapshot members and every task with a run refused for that hold, also when `wakeAgents` is set. This covers tasks created under the tree after the snapshot and tasks moved out before release. - `packages/db`: partial index `heartbeat_runs_tree_hold_suppression_idx` (migration 0436, `CREATE INDEX CONCURRENTLY`, registered as a top-level migration) for that lookup. - Docs: `doc/execution-semantics.md` (gate-refused work, race handling, non-assignee and pre-fix scope) and `doc/DATABASE.md` (index). ## Verification - Red before the fix, green after: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/issue-tree-pause-resume-run.test.ts` (21 tests). - "resumes a parked retry the pause suppressed exactly once after release / resume / legacy": before, `expected [] to have a length of 1` (no resumed run). After, one `issue_tree_resumed` run with `interruptedRunId` = the suppressed retry, no execution blocker, and the task finished. `reconcileStrandedAssignedIssues` added no run during the hold or after release. - "resumes a queued run the claim gate refused during the hold exactly once": before, the test timed out at 15 s (start-lock re-entry). After, the run is cancelled with `issue_paused` and the hold marker, and release re-dispatches it once. - "resumes a refused retry on a task moved out of the paused tree before release" and "... on a task outside the snapshot when release also wakes members": review findings, one resume each. - "leaves a due retry parked when the pause is released while the gate refuses it": the test waits until the gate blocks on an uncommitted release. The retry is not cancelled and runs once. Without the `FOR SHARE` lock the gate does not wait and the test fails. - "resumes work a gate refused after the release that should have resumed it had finished": the late-refusal re-check resumes once, and a repeated check does not dispatch again. - "does not resume a paused-code cancellation that no pause hold covered": negative case. - Related suites pass locally: `src/__tests__/issue-tree*`, `src/modules/run-dispatch`, `src/services/heartbeat/queue.test.ts`, `heartbeat-dependency-scheduling`, `heartbeat-workspace-busy`, and `packages/db` `client.test.ts`. - I did not run a live staging check here. Staging APIs are not reachable from this environment. ## Risks - Migration 0436 adds a partial index on `heartbeat_runs`. It is built concurrently and covers only cancelled runs with the marker, so it is small. - The scheduled-retry gate now takes a `FOR SHARE` row lock on the pause hold before it cancels. Release updates that row, so the two serialize briefly. Release does not lock issues or runs, so the lock order cannot deadlock with the gate's issue-then-run locks. - The queued-run claim gate now records `issue_paused` in place of the generic `cancelled`. Stranded recovery already classifies `issue_paused` as non-retryable. The pause resume owns that work, so recovery does not add a second run. - Scope limits (documented): a non-assignee run refused by a pause is not replayed, the same as non-assignee wakes that admission drops during a hold. Among pre-fix rows, only scheduled retries carry `issue_paused`, so only those fall back to the snapshot. A pre-fix row whose hold is already released does not self-recover; the staging task that showed this bug needs one manual wake, or a new pause and release. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 03:03:15 -06:00
Devin FoleyandPaperclip	bf53060904	fix(recovery): continue saved input after verified workspace copy-back (#19422) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution preserves accepted results and task files after a failure. > - A later user message can request a new turn after the old execution stops. > - File recovery can finish by copying files back or by retaining a verified archive. > - The retained-result gate accepted the archive proof but missed a verified copy-back receipt. > - This change accepts that existing receipt for terminal failures and preserves execution authority and prior results. ## Linked Issues or Issue Description **What happened?** A task keeps a user message queued with `finalizing_saved_result` after its old native coordinator reached `terminal_failure`. The accepted result and a successful, exact `workspace_recovery` copy-back receipt are saved. The gate only checks the alternate `workspace_recovery_snapshot` receipt. An older failed result can also keep a newer failed turn's saved input blocked. **Expected behavior** A new user message or deliberate task Retry can continue from verified recovered files after the old execution stops. Preserve the old result and unknown action outcomes. Keep an active finalizer or missing stop evidence blocked. **Steps to reproduce** 1. Retain an accepted result when a native run reaches terminal failure. 2. Complete host copy-back with the exact run, allocation, descriptor and final file digest. 3. Save a new user message or request task Retry. 4. Observe that admission still reports `finalizing_saved_result`. **Paperclip version or commit** Reproduced against `4d49a3128613b282fb344272ed7ca15f98dea2ac`. **Deployment mode** Authenticated native execution with retained workspace recovery. Local regressions use disposable PostgreSQL and the real heartbeat queue. They start no model provider or sandbox. Related work: #19355 covers saved input after Stop. #18267 concerns manual reconciliation of duplicate legacy holds; it excludes workspace and system recovery and does not fix this receipt gate. I searched open PRs and issues before starting this change. ## What Changed - Reuse the existing host-verified repair receipt when an accepted result belongs to a coordinator in `terminal_failure`. - Retain the separate checks for active controllers, environment termination, live executions, questions, approvals and current user authority. - Add message, saved-message and task Retry regressions, plus concurrent queue delivery and invalid-receipt cases. - Document this continuation path. Add no schema, API or background worker. ## Verification - All 483 tests pass across explicit continuation, task Retry, quarantined Retry and finalized recovery migration suites. - The final focused fixture has 17 cases. All pass with the change. With the original production gate restored, all four positive cases fail and all 13 safety cases pass. - Negative cases cover missing, unfinished and mismatched receipts, another task or source, unrecovered files, a live finalizer, an active controller, missing remote stop and active execution. - The real heartbeat path delivers one saved message under concurrent recovery calls. Later recovery calls create no second turn. The old accepted result and recovery budget stay unchanged; unknown outcomes stay unknown. - The first delivery fixture omitted the newer failed source needed by the recovery scanner. I corrected the fixture to preserve that history and repeated the unchanged-code comparison. Earlier failure logs remain in the verification record. - `pnpm -r typecheck`: passed. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - `pnpm build`: passed. - Full local `pnpm test:run`: running with a 20-minute bound; no full-suite pass is claimed. All hosted CI checks passed. Greptile scored the exact head `d04fb98892fd77853ad03e6b44cd612e805abc13` 5/5 with no findings or review threads. - Hosted delivery of the already saved input still requires a qualified staging build containing this change. ## Risks The new path requires the existing exact repair receipt and a terminal failure. A coordinator still accepting or committing a result keeps the previous hold. Stop, owner and live-run checks still run before admission. The old result is neither applied nor replayed. Receipt verification adds bounded indexed reads only on this retained-result failure path. No migration or provider configuration changes are required. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex in T3 Code, with xhigh reasoning. The context-window size is not exposed. Used code editing, tool calls and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, including the available runtime details - [x] I have checked ROADMAP.md and confirmed this is a scoped bug fix - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in this PR using the bug-report fields - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task id - [x] I have run the affected tests locally and they pass - [x] I have added regression tests and updated relevant documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all review findings before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 00:15:30 -08:00
Devin FoleyandPaperclip	11bc490cda	fix(recovery): keep a stopped copy-back's files instead of holding the workspace (#19385) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can have its result accepted before the copy-back of its sandbox files finishes. If the copy-back fails, it retries. > - A board Stop during those retries ends the run. It sets the run's finalization to `terminal_failure` / `native_retry_cancelled`, and the result stays accepted. > - After that, nothing exports the files. The finalization worker skips terminal rows, and cancelled-workspace preservation skipped any run with an accepted result. > - The task workspace stays dirty and bound to the stopped run. Every later run on the task waits for the workspace, then fails setup with the unfinished-writer error. > - This pull request lets cancelled-workspace preservation also admit an accepted result whose copy-back a Stop ended. > - The benefit is that stopping a stuck copy-back saves its files and frees the task, like stopping any other run. ## Linked Issues or Issue Description Refs #16700, #17677, #18820, #19304 and #19305 (cancelled-workspace preservation). This pull request adds the accepted-result case to that path and keeps its other checks unchanged. **What happened?** On a hosted instance, a native run's result was accepted, but its copy-back kept failing and retrying. A board operator pressed Stop to unblock the task. The run became `cancelled`, its sandbox was stopped and kept, and its finalization moved to `terminal_failure` / `native_retry_cancelled` with the result still accepted. The next run on the task was cancelled with "Shared execution workspace is busy: run … ended, but its workspace save has not settled". About 24 automatic retries followed over an hour. The last one failed with "The workspace still has an unfinished writer or save operation." After that nothing ran: the task stayed `in_progress` with no run and no recovery action. A Retry or a new comment would hit the same block. The workspace stayed `dirty: true` with the stopped run as its writer, and no `workspaceRestoreRecovery` was ever recorded. The reason: `preserveCancelledNativeWorkspaces` requires `resultId IS NULL`, and `registerCancelledWorkspacePreservationWork` and `preserveNativeFailedWorkspace` both return early when `resultId` is set. **Expected behavior** Stopping a run during its copy-back retries keeps its files, as stopping any other run does. Paperclip's workspace recovery saves them and releases the workspace. Nothing commits or replays the stopped result. **Steps to reproduce** 1. Accept a native result for a run whose copy-back is in `retryable_failure`. 2. Stop the run with the board Stop. The finalization becomes `terminal_failure` / `native_retry_cancelled`. 3. Start another run on the task. It waits for the workspace and then fails setup. No save of the stopped run's files is ever recorded. The new database tests run steps 1 and 2 against PostgreSQL and check that the save is recorded. **Paperclip version or commit** `master` at `9ceaec6f97e`. **Deployment mode** Any deployment that uses remote (sandbox) task workspaces. Seen on a cloud-managed instance. ## What Changed - `server/src/services/native-runtime/native-failed-workspace-recovery.ts`: cancelled-workspace preservation now also admits an accepted result when the finalization is `terminal_failure` with `native_retry_cancelled`. That is the state a Stop leaves after ending copy-back retries. All three entry points use the same rule: - the startup scan `preserveCancelledNativeWorkspaces`; - the worker `registerCancelledWorkspacePreservationWork`; - `preserveNativeFailedWorkspace`, only when it is called with `requireCancelledStop`. Preservation still requires everything it required before: a cancelled run, an acknowledged Stop, no live finalization owner or controller lease, a prepared workspace reference, and the exact retained source. - An accepted result with any other failure code is still skipped; those runs keep their own export repair path. Failed (not cancelled) runs are unchanged. - The accepted result stays recorded and uncommitted. Preservation does not change the task's status or queue a run. - `doc/execution-semantics.md`: describe the stopped copy-back case in the paragraph on writers stopped mid-run. - `native-cancelled-workspace-preservation.test.ts`: move the accepted-result fixture into an `acceptResult` helper, and add a test with three cases. ## Verification - Failing first. The new test "preserves files of an accepted result whose copy-back was stopped" has three cases: worker, startup scan, and a sandbox that the Stop already stopped and kept, through the real `prepareSandboxStopAndRetain` and `settleStopOnlyCleanup`. All three fail on `master`, with no `workspaceRestoreFailure` recorded, and pass with this change. - Each case checks that: - the save obligation and recovery action are recorded for the exact lease; - `inspectWorkspaceRecoverySource` finds one source; - the finalization keeps its `resultId`, `terminal_failure` and `native_retry_cancelled`; - the workspace stays held by the stopped writer until the save publishes; - no run or wake is created, and the task status is unchanged. - The existing "does not admit preservation for accepted-result" case still passes. It uses an accepted result with a different failure code. - `cd server && npx vitest run src/services/native-runtime/native-cancelled-workspace-preservation.test.ts`: 44 passed. - Related suites, 10 files and 854 passed: - `native-failed-workspace-recovery`, `workspace-recovery`, `heartbeat/run-control` - `native-cancellation-request` (unit and integration) - `heartbeat-remote-workspace-busy`, `native-workspace-export-retry`, `native-workspace-reconstruction` - `native-stop-accounting`, `task-workspace-binding` - `cd server && npx tsc --noEmit -p tsconfig.json`: pass. - Not run end to end on a live instance. Diagnosed from that instance's run, finalization, lease, workspace and recovery-action rows. ## Risks - On deploy, the startup scan and the worker will pick up existing cancelled runs in this state and record their saves. That is the intended repair, and it uses the same retention checks as other stopped writers. - A copy-back interrupted by a server restart can leave a `remote_workspace_snapshot` operation marked `running` after its process is gone. This change does not settle such rows. If one blocks the later save, the save follows its existing retry and board hand-off. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs, read-only database queries). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 00:03:41 -08:00
DottaandPaperclip	c1d567ff6a	fix(workspaces): isolate archived nested Git reconstruction (#19409) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace import must preserve archived task files and Git history. > - A saved nested checkout can be a linked worktree whose branch belongs to its old absolute path. > - Archived import reconstructs that checkout in a new scratch directory. > - Ordinary linked recovery rejects the old branch at that new path. It can also write into a cache used by other import workers. > - This pull request reconstructs archived nested checkouts as private standalone repositories. > - Each import preserves saved files and extra history without changing shared worktree registrations or branches. ## Linked Issues or Issue Description **What happened?** Archived workspace import fails when a saved nested linked checkout has a branch owned by the old path. The new scratch path fails the linked-worktree ownership check. The existing cache lock is process-local, so separate import processes also share mutable Git state without a common lock. **Expected behavior** Archived import must create an independent portable copy of the saved checkout. It must retain the saved commit, branch name, staged edits, working files, ignored recovery files, settings, and extra saved commits. It must keep the source cache unchanged and enforce source authorization. **Steps to reproduce** 1. Create a managed nested linked checkout with an unpushed commit and staged and unstaged edits. 2. Save an encrypted restore point, including ignored files and an extra history tip. 3. Restore its saved contents under a different absolute workspace path. 4. Before this repair, the new path cannot adopt the branch owned by the old path. **Paperclip version or commit** The real Git regression fails before the repair at `fc1f5f3052`. The final branch includes the merged interrupted-upload and nested-source authorization fix from #19396. **Deployment mode** Archived remote workspace migration with real Git repositories, disposable PostgreSQL, and encrypted object storage. Local integration tests use the filesystem storage provider. Builds on merged #19402 and #19396. Related #19374 concerns interrupted clone staging; it does not implement private archived reconstruction. No duplicate open repair was found. ## What Changed - Enable private repository reconstruction only for archived migration. Ordinary linked recovery keeps its path ownership guard. - Add an object-only source to the managed checkout owner. A verified cache supplies only the saved commit and authenticated bundle prerequisites through scoped fetches. The private copy includes no unrelated task refs or objects, hard links, object alternates, or current cache files. - Fetch missing authenticated bundle prerequisites before importing a saved bundle in both private reconstruction and ordinary linked recovery. A pruned cache can use the approved remote; unavailable history still blocks restore. - Keep every authenticated restore point's extra commit under a private ref in root and nested private copies so portable archive capture and garbage collection retain it. Ordinary recovery keeps its existing history and expiration rules. - Share the verified, bounded ignore-file copy between root and nested reconstruction. Sources remain unchanged. - Retain source authorization before and after reconstruction and the final source-row locks from #19396. - Test concurrent private copies, source rejection, foreign task refs and objects, ordinary linked prerequisite recovery, a real migration, and cold restore after removing all original repositories. Test hard links, oversized excludes, unsafe info paths, and replacement during a descriptor read. ## Verification - Final affected run: 292 tests pass across ten suites. These cover real Git repositories, encrypted restore points, disposable PostgreSQL migration, portable archives, the restore-point ledger, source rejection, and concurrent copies. - Full repository typecheck and build pass after the scoped-object repair. Final server typecheck and full build also pass after restricting preserved refs to archive mode. - RED then GREEN: a linked checkout at a different path; a pruned bundle prerequisite still available from the approved remote; foreign task refs and objects entering the archive; root extra history disappearing after cold restore; and archive-only refs blocking ordinary standalone expiration. - Two concurrent private copies retain the saved head, branch, staged and unstaged edits, ignored recovery bytes, settings, and modes. Source refs, index, worktree registrations, and working files remain unchanged. - Delete the original checkout, cache, remote, and private copies after portable capture. Cold restore retains the saved files and extra history after garbage collection. No alternates or unrelated task objects remain. - Ordinary linked recovery retains its path ownership check and recovers pruned prerequisites. New and existing ordinary standalone destinations have no archive-only refs and remain eligible for expiration after normal object cleanup. - Unsafe exclude tests reject hard links, oversized files, nonregular paths, info-directory substitution, and replacement during the descriptor read. They preserve the target and replacement source bytes. - Preserve the failed attempts: an initial migration fixture changed its home after constructing the source path. Later expiration fixture attempts used a filtered current-head tip, omitted the local-source callback, or used cleanup that generated Git server-info files or left packed unreachable objects. Correct those setup assumptions before the actual regressions fail. Final affected suites pass. - Two earlier local full-suite attempts were interrupted before editing valid review findings. Neither is reported as a full-suite PASS. Final-head full CI passes; review is 5/5 with no open findings. ## Risks Archived imports use additional private disk space for independent Git objects. Missing, changed, or unsafe sources still block import. The repair does not weaken the default linked-worktree ownership check, change stored formats, or update live provider state. The shared checkout addition is optional and has no effect for existing callers that do not supply an object source. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 00:56:06 -06:00
DottaandClaude Opus 5.5	90b8190c33	fix(native-runtime): detect half-open runner ingress and project Stop for live runs (#19407) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs in a managed sandbox talk to the server through a provider-ingress WebSocket. The server opens this socket to the runner (`paperclip-runnerd`) through the sandbox preview proxy. > - A staging run ran a 20-minute foreground shell command that printed nothing. After the first second, no run events arrived. The run stayed "Working" for over 35 minutes. No watchdog acted, and the run projection said `canStop: false`. > - The server only detected a lost runner connection from a `close` or `error` event. It never sent pings. A half-open route, for example a proxy that drops the connection without a FIN, never emits `close`. The run then waits forever and the 7-minute reconnect grace never starts. > - A long silent tool call is legitimate. The fix must tell "quiet but alive" apart from "nothing alive". A protocol ping does that: runnerd answers every ping with a pong while its provider waits on the tool. > - This pull request pings authenticated runner connections and terminates the socket after 90 seconds of total silence. The existing reconnect grace and transport-failure path then recover the run. It also makes the execution projection report `canStop: true` for every queued or running run. > - The benefit is that a run with a dead runner connection now ends in a bounded time instead of hanging. API readers also see the Stop that the board already offers. ## Linked Issues or Issue Description **What happened?** A native-runner run (managed sandbox, OpenCode with an OpenRouter model) ran a foreground shell command for about 20 minutes with no output. The last run event came at about T+1s. The command finished on disk, but the run never got another event and stayed `running` / "Working" for over 35 minutes. Its projection had `canStop: false` and `permittedActions: ["inspect_run"]`. A board cancel through the API worked. **Expected behavior** The server detects a dead runner connection and recovers or fails the run in a bounded time. A runner that is alive but quiet stays attached. The projection reports that a live run can be stopped. **Steps to reproduce** 1. Start a native run in a managed sandbox. 2. Make the agent run a long foreground command that prints nothing. 3. Let the preview route drop the WebSocket without a close frame. 4. Before this fix, the run stays "Working" forever. ## What Changed - `server/src/realtime/runner-prp-outbound.ts`: after PRP authentication, `WsJsonWireConnection.monitorLiveness` sends a WebSocket ping every 20 s. Any inbound frame or pong counts as liveness. After 90 s with nothing inbound, it terminates the socket. That emits an ordinary `close`, so the PRP authority and the reconnect loop use the existing 7-minute grace and then fail the transport. If the server's own event loop stalls, the window restarts instead of blaming the runner. The 90 s bound is longer than runnerd's own 60 s connection lease, which already requires runnerd to read the socket. - `server/src/services/execution-projection.ts`: `canStop` is now `true` for every `queued` or `running` run. The old failed-native-retry rule is unchanged. - `server/src/services/execution-projection.ts`: a queued run now shows "Waiting for previous run to stop" (cause `previous_run_settling`) when a predecessor that admission checks (the task execution owner, the retried run, or the agent's latest settled run on the task) still has pending cleanup by `hasPendingRunCleanup`. Before, a turn queued after a Stop showed only "Queued" with no reason while the stopped run's controller still waited on the dead runner. That in-process wait (`claimQueuedRun` settling-owner gate) also made Retry a no-op. The liveness bound above makes that controller settle. Its existing cleanup then calls `startNextQueuedRunForAgent`, which starts the queued turn. - `server/src/services/run-cancellation.ts` and `server/src/modules/run-dispatch/adapters/postgres.ts`: a user-stopped native run whose runner was unreachable can later settle as `failed` (`native_provider_terminal_failed`) when its transport fails closed. The dispatch gate's message deferral required a `cancelled` source and compared the message with `finishedAt`. The user's follow-up message was therefore marked `skipped`, and the task asked for a second message. The new `recordedUserStop` helper honors the recorded user Stop on such a failed source, so the gate saves the message for the deferred-receipt worker. - `packages/shared/src/types/execution-projection.ts`: documents the `canStop` meaning. - `doc/execution-semantics.md`: documents the ingress liveness bound and the new `canStop` meaning. ## Verification - Red then green: `runner-prp-outbound.test.ts` "detects a half-open authenticated ingress and fails the run within a bounded window". On the old code it failed because no ping was sent and the socket was never terminated. It now passes and the active-run `failure` rejects with `runner_ingress_unavailable`. - Safety: "keeps a quiet but live runner attached through a long silent tool call" checks that a runner answering pings stays attached for 20 simulated minutes with no termination and no failure. - Red then green: `execution-projection.test.ts` "offers Stop for every live run, including a long-silent native turn". On the old code it failed with `canStop: false`. - Red then green: `execution-projection-recovery.test.ts` "names the wait when a stopped predecessor still holds the task's runner" (real Postgres). On the old code the successor projected a plain "Queued". It also checks that the wait clears on lease release and ignores another agent's run. - Red then green: `workspace-recovery.test.ts` "continues a message sent after Stop when the stopped run later failed closed". On the old code it failed 3 of 3 runs with the receipt `skipped`. It now passes, and the full file passes twice (392 tests). The test now pins the message after the recorded Stop. Before, the fixture could write both in the same millisecond, which made the sibling variants flaky. - Passed: run-cancellation, explicit-native-continuation, run-dispatch and run-control suites (622 tests). - Passed: `npx vitest run src/realtime/ src/services/execution-projection*.test.ts src/__tests__/heartbeat-run-privacy-routes.test.ts`, the native-runner-execution-driver tests, and the plugin-runner-task tests. - Passed: `pnpm --filter @paperclipai/server typecheck`. - I read runnerd (`runner-core/src/durable/transport.rs`) and confirmed it answers `Ping` with `Pong` in its non-blocking read loop. - Not run: a live managed-sandbox reproduction. Staging and Daytona were not reachable from this environment. `plugin-runner-probe.test.ts` needs a local cargo build of runnerd, and it did not start in this worktree. ## Risks - A false termination is possible if a runner fails to answer pings for 90 s while it is still alive. runnerd already loses its own connection lease after 60 s without reading, so this bound does not add a stricter requirement. A terminated live runner reconnects within the existing grace. - If the preview proxy drops WebSocket control frames, ping alone does not prove liveness. Inbound PRP frames (runnerd's lease renewals, about every 30 s) still count, so a live runner stays attached. - `canStop` now means "the board can Stop now" for live runs. The UI already shows Stop for running and queued runs without reading this hint, so the UI does not change. - Out of scope: the inbound runner WebSocket server (`runner-prp-ws.ts`), and a lost OpenCode SSE event while the runner is alive. A stopped run whose controller hangs for a reason other than a dead transport still holds the queued successor. The projection now names that wait, but this PR does not force-release it, because the controller has no termination evidence. A queued reason for the agent concurrency cap is also not covered. The wait label calls the admission predicate (`hasPendingRunCleanup`) for up to three predecessor runs per queued run. This adds a few reads to projections of queued runs only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 00:52:01 -06:00
Dotta	92452524f4	fix(server): recover authenticated empty retention checkpoints (#19417) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Ephemeral server enrollment must preserve existing task files. > - Archived task directories can depend on encrypted retention checkpoints. > - A missing object blocks migration even when its authenticated contents were exactly empty. > - Metadata counters alone cannot prove that no saved files existed. > - This change verifies the exact candidate against the original encrypted key envelope before normal migration publication. > - The benefit is lossless migration of this bounded case without resetting task history or weakening ordinary restore. ## Linked Issues or Issue Description **What happened?** Migration stops at an archived empty directory when its retained encrypted object is missing. The original encrypted key envelope still authenticates the payload hash, byte length, company, workspace, checkpoint ID, and generation. **Expected behavior** Administrative migration can reconstruct those exact original bytes when storage confirms absence and every authenticated check passes. It must keep missing nonempty, expired, corrupt, or ambiguous sources blocked. **Steps to reproduce** 1. Capture and encrypt an empty task directory. 2. Retain its reference and encrypted key envelope. 3. Remove the old directory and storage object. 4. Run the normal remote workspace migration. **Paperclip version or commit** The regression fails before this change at `205ffcb606`. Related preservation work is in #19396, #19402, and #19409. No duplicate open fix was found. **Deployment mode** Administrative migration to external workspace storage. Tests use real encryption, real filesystem storage, and disposable PostgreSQL. ## What Changed - Let the encryption owner verify a candidate against the existing scoped encrypted key envelope. Normal ciphertext validation keeps its previous order and checks. - Share the task-directory encoder with an exact empty payload constructor. - Add a default-off loader option for confirmed missing empty checkpoints. Only the administrative migration caller enables it. - Keep the original checkpoint reference and object unchanged. Publish the recovered bytes through the existing snapshot and source-fence owners. - Reject forged counters, wrong identities, missing nonempty data, present corruption, unreadable present objects, unknown absence, and expiry. ## Verification - Final affected run: all 238 tests pass across five crypto, snapshot, retention, and migration suites. Full repository typecheck and build pass; server typecheck and full build also pass after the cancellation repair. - RED then GREEN: exact missing-empty recovery and cancellation of a stalled HEAD through the real storage-service boundary. Cancellation retains the caller's original abort reason. - Disposable PostgreSQL migration publishes a cold-restorable empty archive. Archived status, task status, task binding, original reference, and the absent original source stay unchanged. Replay keeps one canonical snapshot. - Negative cases reject forged empty counters, an authenticated directory or file, every foreign identity field, expiry, present corruption, unreadable present objects, unknown absence, and missing nonempty data. Ordinary restore keeps its previous failure behavior. - Initial verification retained one new fixture-path failure after 145 passes, and a typecheck rejected the fixture's widened schema literal. Correct the fixture to use the returned staging path and its explicit reference type. The final complete affected run passes. - The new HEAD request runs only after a failed read with the administrative option enabled. It uses the existing cancellation contract and adds no polling or background work. - Final-head full CI passes; review is 5/5 with no open findings. This PR does not claim deployed enrollment or a live migration result. ## Risks Recovery requires the original valid encrypted key envelope and an exact format match. A format change or unavailable key continues to block migration. Existing ciphertext hash, size, authentication, and ordinary restore behavior remain enforced. This change has no schema, provider-session, API, or UI contract changes. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-06 00:50:28 -06:00
DottaandClaude Opus 5.5	a4bb059f59	fix(heartbeat): retry transient sandbox provider lease refusals and explain the block (#19412) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task runs in an execution environment. A managed sandbox environment gets a lease from a sandbox provider (for example Daytona) before the agent starts. > - When the provider answered sandbox create with a temporary error (HTTP 502), the run failed as the generic `setup_failed`. > - `setup_failed` is the fail-closed code for unknown setup errors. It gets no automatic retry, so the task went to `blocked` at once. The task showed only "Couldn't finish this attempt". > - A user had to find and press Retry. The Retry then worked, which shows that the failure was temporary. > - This pull request classifies temporary provider refusals during lease acquisition from structured fields. It retries them within the existing bounded bootstrap budget, and it explains the block when the retries are used up. > - The benefit is that a short provider outage no longer stops a task. When the outage continues, the task shows the cause and a Retry that works. ## Linked Issues or Issue Description **What happened?** A task was assigned to an agent that uses a managed Daytona sandbox environment. Sandbox creation returned `HTTP 502`, and the cleanup of the provisional sandbox could not be confirmed. The run failed with `setup_failed` and the error `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona sandbox creation failed (HTTP 502); cleanup could not be confirmed for paperclip-create-…`. Three seconds later the task was `blocked`. No automatic retry was scheduled. The task showed no specific reason. A manual Retry started a new run that completed normally. **Expected behavior** A temporary provider error (HTTP 5xx, 429, timeout) before the agent starts must retry automatically with bounded backoff. The task must show why it waits. If the retries are used up, the task must show the cause, a next action, and a Retry that works on the first click. **Steps to reproduce** 1. Configure an agent with a sandbox environment. 2. Make the provider return HTTP 502 on sandbox create. 3. Assign a task to the agent. 4. See that the run fails with `setup_failed` and the task goes to `blocked` with no retry. **Paperclip version or commit** `master` at `612da1e8e0`. ## What Changed - Plugin SDK: an `environmentAcquireLease` error now carries a closed diagnostic envelope (HTTP status or a closed errno code) across the worker RPC. It takes the status of the create call only, not of the compensating cleanup lookup. The existing creation-cleanup ownership envelope is unchanged. New module: `environment-acquire-error.ts`. The resume diagnostic helpers are shared. - Server: new `EnvironmentProviderUnavailableError` and code `environment_provider_unavailable`. The environment run orchestrator sets it as the cause of `lease_acquire_failed` only when structured evidence shows a transient status (408, 425, 429, 500, 502, 503, 504, or a closed connection errno). It never parses message text. Saved-sandbox resume and stopped-worker refusals keep their own codes. - Setup failure (`failRunSetup`): a fresh, pre-dispatch run records `environment_provider_unavailable` with a `providerFailure` (phase `startup`, retryability `transient`). The run uses the bounded bootstrap retry (30 s, then 60 s). An accepted interaction continuation keeps its own continuation retry, as for `plugin_worker_unavailable`. - Recovery: the new code is registered with the other bootstrap codes. It is not sent to the generic continuation retry. It does not create an execution-reconciliation hold, because no provider work started. On exhaustion the stranded notice has the title "Sandbox provider was temporarily unavailable" and a specific next action. - UI: `ProviderFailureNotice` labels the `sandbox` provider as "Sandbox provider". The task notice shows the provider failure, "Paperclip will retry automatically" while a retry is scheduled, and Retry after exhaustion. - Docs: `doc/execution-semantics.md` describes the policy and the sandbox ownership rule. ## Verification Red before the fix, green after the fix: - `server/src/__tests__/heartbeat-process-recovery.test.ts` › "transient sandbox provider refusal during lease acquisition": - "retries automatically and runs exactly one successful successor without user action": the real heartbeat. The lease runtime returns a Daytona 502 once, then delegates. Before the fix: one `failed` run and no successor. After the fix: one `scheduled_retry` successor with no hold and a `retry_scheduled` presentation. After promotion, exactly one successor succeeds. - "explains the block with a working Retry once automatic retries are used up": three refusals. The task is `blocked` with a "Sandbox provider was temporarily unavailable" notice. The presentation is `{ state: "failed", canRetry: true, providerFailure: { httpStatus: 502 } }`, and no execution-reconciliation hold exists. One `requestTaskRetry` is accepted, and its run succeeds. - `server/src/services/heartbeat/run-completion.test.ts` › "routes a … sandbox provider lease refusal through bounded bootstrap recovery": the real orchestrator wrapping. Only the fresh, structured 502 is retried. A message-only 502, a 400, a run after provider dispatch, and a retained native run stay `setup_failed`. - `packages/plugins/sdk/tests/environment-acquire-error.test.ts`: the real worker RPC. The test checks the create status (not the 404 cleanup lookup), the errno, no provider text, the unchanged cleanup envelope, and that forged envelopes are rejected. Commands: ```sh pnpm --filter @paperclipai/plugin-sdk exec vitest run tests/environment-acquire-error.test.ts tests/environment-creation-cleanup.test.ts tests/environment-resume-error.test.ts cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts src/services/recovery/ src/services/heartbeat/run-completion.test.ts src/services/heartbeat/retries.test.ts src/services/legacy-execution-recovery.test.ts src/services/task-interruption-presentation.test.ts src/services/task-retry.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts cd packages/plugins/sandbox-providers/daytona && npx vitest run pnpm --filter @paperclipai/plugin-sdk typecheck && pnpm --filter @paperclipai/server typecheck ``` I did not run a live Daytona 502 on staging. Re-check on staging after deploy (see Risks). ## Risks - Sandbox leaks or double creation: each retry is a new run with a new durable creation record and a new provider name (`paperclip-create-<operationId>`). The failed attempt stays owned by its own record, and the existing creation reconciler settles it. A retry never reuses or abandons it. While a provisional sandbox waits for reconciliation, up to two more sandboxes can be created for the same task. This is the same as a manual Retry today. - Plugins built with an older SDK do not send the envelope. Their refusals stay `setup_failed`, which is the current behavior. - An in-process error with `status`/`statusCode` 500–504 or a connection errno during lease acquisition is now retried twice. The budget is bounded and shared with the other bootstrap codes. - An RPC timeout on the host side is not classified by this change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code as an agent with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 00:08:44 -06:00
DottaandPaperclip	66d3fa97a9	fix(workspaces): preserve ignored files in archived Git imports (#19402) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace migration must preserve archived files and saved recovery data. > - An archived Git workspace can contain ignored recovery receipts and bundles in its encrypted restore point. > - The importer reconstructs a private repository from a verified source before it applies that restore point. > - Reconstruction omitted the source repository's local ignore rules. The restored files then changed classification, and the exact restore-point check failed. > - This pull request copies the verified local ignore rules into the private repository before restoration. > - The source stays unchanged, and the original restore-point integrity checks still apply. ## Linked Issues or Issue Description **What happened?** An archived workspace with saved files under an ignored recovery directory failed migration with `restored_contents_mismatch`. The task commit and staged index matched. The ignored files were restored, but the private repository treated them as untracked work because its `info/exclude` rules were missing. **Expected behavior** Reconstruction from an existing verified source must preserve its local ignore policy. Saved ignored files must survive migration and a cold object-store restore with their original classification. Import must not follow a symlink to an unrelated ignore file or edit the source repository. **Steps to reproduce** 1. Exclude a recovery directory in the managed source repository's `info/exclude` file. 2. Save staged edits, unstaged edits, an ignored recovery receipt, and an ignored binary bundle in an encrypted restore point. 3. Archive and remove the task worktree. 4. Import the archived workspace. Before this change, restore-point validation fails because the reconstructed repository lacks the local exclusion rule. **Paperclip version or commit** Reproduced on master at `879140cbbd`. **Deployment mode** Cloud workspace migration with a retained verified Git source, PostgreSQL, and encrypted durable object storage. The tests use real disposable PostgreSQL, Git repositories, and encrypted filesystem-backed object storage. Related migration work: #18826, #19138, and #19396. No duplicate open fix was found. ## What Changed - Copy the verified source's `info/exclude` file into the private reconstructed repository before applying its restore point. - Bound the read to one MiB. Reject symlinked, non-regular, hard-linked, or changing source files. - Keep the original source bytes unchanged. - Add regressions that check staged and unstaged edits, ignored recovery files, cold restore, and rejection of a symlinked ignore file. - Document the reconstruction behavior. ## Verification - Before the repair, the saved-ignored-files regression fails with `restored_contents_mismatch`. - After the repair, both focused cases pass. - At rebased head `81a0b1c041`, all 206 tests across migration, snapshot storage, archive, and migration concurrency suites pass. The merged interrupted-upload and nested-source approval regressions remain intact. - Server package typecheck passes. - Full repository typecheck and build pass at the rebased head. The full local test suite is running. The prior local full-test run was interrupted for rebase, and is not a passing result. - Initial CI hit a 15-second timeout in unchanged `activity-routes.test.ts`; the exact case passed in a focused local run. The browser shard lost its self-hosted runner, as confirmed by its check annotation; GitHub did not retain its full log. Those failures remain recorded. An unchanged retry was superseded by the required conflict rebase. At exact head `81a0b1c041`, all 53 CI checks and both required aggregate gates pass; Greptile is 5/5. The full local suite remains in progress, so no local full-suite PASS is claimed. A second automated review noted the existing nested-checkout counterpart. This PR fixes root reconstruction. The nested linked-checkout repair and its saved-ignored-files/cold-restore regression are implemented in #19409, which is required before the migration rollout uses nested imports. This PR leaves that sibling recovery path unchanged. The two identical scope warnings are addressed through that linked repair. ## Risks The copy applies only when an archived Git restore-point source is still present and verified. It changes only the private reconstruction directory. Missing sources retain the existing restoration path. Unsafe or unstable local ignore files block import and retain the original migration obligation. No database migration or saved format changes are included. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-05 23:11:28 -06:00
DottaandClaude Opus 5.5	9668e73b4b	fix(native): continue a turn automatically when reattachment loses its runner (#19399) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs in managed sandboxes recover through reattachment: after a server restart, and for a same-run retry when a turn ends without a semantic result > - Reattachment fails closed when the original runner or sandbox is gone (`native_remote_recovery_lease_mismatch`, `runner_remote_process_identity_unavailable`) > - Saved-work recovery then saves the files, but the task stays blocked with "Saved work recovered. Send a message to continue." The recovered hold offers no Retry > - A restart, deploy or maintenance window must not need a user message to continue work > - This pull request requests the task Retry automatically, once, when it is safe, and offers Retry on the recovered hold when it is not > - The benefit is that interrupted work continues after infrastructure events, and a user who must act gets a Retry that works on first use ## Linked Issues or Issue Description **What happened?** A native task in a managed sandbox was interrupted. In one case a server restart claimed a queued user-comment turn for reattachment. In another case a turn completed without a semantic result, and the same-run retry claimed the run for reattachment with no restart. In both cases the original runner or sandbox lease was gone. The run failed with `native_remote_recovery_lease_mismatch` or `runner_remote_process_identity_unavailable`. Recovery saved the work. The task stayed `blocked` with "Saved work recovered. Send a message to continue." The execution blocker had `canRetry: false`. A manual `POST /api/issues/:id/retry` started a run that succeeded. **Expected behavior** After an infrastructure interruption, the task continues automatically when the interrupted turn's outcome is known. Otherwise the task offers a Retry that works on first use. **Steps to reproduce** 1. Start a native task in a managed sandbox from a user comment. 2. Lose the sandbox lease or the remote runner process (for example, restart the server during a maintenance window). 3. Let reattachment fail and saved-work recovery complete. 4. See that the task waits for a new message and offers no Retry on the execution blocker. **Root cause** Reattachment (`reattach_remote_runner`) correctly refuses to replace a missing runner or allocation. Nothing then continued the user's request. The automatic continuation rule only allows a new user turn after reconciliation. `getExecutionBlocker` hid Retry whenever a workspace restore hold existed, also after the files were verifiably recovered. The same-run retry path also re-enters the restart-recovery owner, so its event said "after server restart" with no restart. ## What Changed - New `server/src/services/heartbeat/restart-interrupted-turn.ts`: after a reattachment claim executes, if the run failed because its runner or allocation was lost, request the same task Retry that `POST /issues/:id/retry` creates. It runs once per run (idempotency key `restart-interrupted-turn:<runId>`), under the authority of the user who sent the message, assigned the task, or clicked Retry. It requires that the run recorded no provider turn or that its last provider turn is settled. System wakes, in-flight turns, and a Retry that this hook already requested do not continue automatically. - `heartbeat/recovery.ts`: call the hook after each reattachment claim. Label same-run retry reattachment as "Reattaching to the existing remote runner to resume this run", and add `trigger: "same_run_retry"` to the event payload. - `execution-blocker.ts`: a failed run whose saved work is verifiably recovered now offers Retry (`canRetry: true`) with "Saved work recovered. Retry or send a message to continue." An intentional Stop keeps "send a message" only. - `task-retry.ts`: optional `initiatedBy: "restart_recovery"` records the activity as actor `system`/`restart-recovery` with `onBehalfOfUserId`. - `doc/execution-semantics.md`: document the rule. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` use the real recovery, Retry and dispatch services on embedded Postgres: - `continues a user turn once ... after $title and recovery saved the work` (3 cases: restart lease loss before the turn, same-run retry with the runner gone after a settled turn, same-run retry with the lease gone after a settled turn). Each asserts one queued successor bound to the Retry and the user's authority after recovery, idempotent hook calls, system activity attribution, and no second automatic attempt for the automatic Retry's own failure. - `does not continue automatically when $title, and offers a Retry that works on first use` (in-flight turn, system wake, unrelated error). Each asserts no successor, `canRetry: true` on the blocker and the interruption presentation, and one successor after one Retry click. - Red before the fix: all 6 tests fail (no successor, blocker `canRetry: false`). Green after the fix. - `npx vitest run src/services/workspace-recovery.test.ts src/services/task-retry.test.ts src/services/task-interruption-presentation.test.ts src/services/execution-blocker-read.test.ts src/services/heartbeat/recovery.test.ts src/services/explicit-native-continuation.test.ts src/services/quarantined-task-retry.test.ts src/__tests__/workspace-restore-repair-route.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts`: 1098 passed. - Not verified on staging. After the next deploy, check a task interrupted by a restart: it should continue without a message, or show Retry. ## Risks - The automatic Retry uses the original user's authority. Admission rechecks that user's current permission, stop proof, saved-work recovery, approvals, pauses, budget and ownership. A refused request leaves the visible Retry. - A settled turn that ended without a result is continued with a fresh Retry turn from saved work. A turn whose outcome is unknown (in flight) is not. - The hook runs after the claim's execution returns. If the server stops between the failure and the request, the task shows Retry instead. - Not in scope: same-run disposition recovery still cannot start a new runner in the existing sandbox after the original remote runner exits. This change continues the task with one fresh Retry turn instead. If that Retry turn also ends without a result and loses its runner, the task shows Retry. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #19355 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 22:29:46 -06:00
DottaandPaperclip	da34bcd48d	fix(workspaces): resume archived imports from their saved upload (#19396) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspaces let a replacement web server restore saved task files. > - Legacy workspace imports keep a journal and an encrypted upload receipt across interruption. > - An archived Git import can rebuild a missing source from its saved task commit and edits. > - A fresh clone can also contain newer remote refs. Those refs change the archive manifest even when the task files are unchanged. > - This pull request recovers the exact authenticated upload for an interrupted archived Git import. > - The original journal, source binding, task binding, and execution checks still control publication. ## Linked Issues or Issue Description **What happened?** An archived Git workspace import stopped after its upload persisted. A retry rebuilt the source from the remote repository. New remote refs changed the archive manifest, so the original upload receipt rejected the retry with `remote_workspace_snapshot_operation_content_changed`. **Expected behavior** The retry must recover its original saved upload and complete the same import. It must keep the saved task commit, staged edits, unstaged edits, lifecycle state, and task binding. A changed source or task binding must still block publication. **Steps to reproduce** 1. Archive a Git task with staged and unstaged edits, save its restoration recipe, and remove its local repository. 2. Start the workspace import and lose the upload reply after object persistence, or interrupt its independent restore after the upload is acknowledged. 3. Advance the remote default branch and add an unrelated branch. 4. Retry the original import. Before this change, both interruption cases fail with the same content-change error. **Paperclip version or commit** Reproduced on master at `d8b0eae68772a3bd2bb4a8163b5ce95764b97cff`. **Deployment mode** Cloud workspace migration with external PostgreSQL and durable encrypted object storage. The regression tests use real disposable PostgreSQL, an HTTP Git remote, and encrypted filesystem-backed object storage. Related merged work: #18826 and #19138. No duplicate open fix was found. ## What Changed - Add scoped recovery of an acknowledged snapshot or a pending upload whose object persisted. Authenticate the archive before returning it. Recovery alone does not publish the snapshot. - Resume archived Git imports from those exact saved bytes. Keep the original operation and all publication checks. - Lock and recheck each saved nested source’s company, project, and repository identity in the final publication transaction. - Keep mutable local sources on the existing capture and content-comparison path. - Add regressions for both interruption points, unrelated remote changes, source revocation during recovery, and local edits after interruption. - Document the retry behavior. ## Verification - Before the repair: both new archived retry regressions fail with `remote_workspace_snapshot_operation_content_changed`; the mutable-source control passes. - After the repair: all 204 tests in the migration, snapshot storage, archive, and archive environment suites pass. The suites use real PostgreSQL and encrypted storage. - Server package typecheck and full repository typecheck pass on `bffebbab60edbdf50b42fc37a9e58beb01ceb94d`. - An added source-revocation test initially expected a later expiry error. The existing publication fingerprint rejected the change first. The test now requires that earlier `remote_workspace_migration_publication_changed` error and still checks that authority was not published. - The review finding was reproduced before the safety repair: changing a saved nested repository URL after upload still allowed publication. The new regression requires rejection, and unchanged approved nested sources still restore. - Full repository typecheck and build pass on the final head. The full local test command stopped at one Slack manager authorization fixture with `socket hang up`; the exact case then passed in isolation on the same head, and all three CI chat shards passed. The original local failure is retained. - The first CI server shard run had 4,724 passes and one runtime-process readiness failure in an unchanged fixture. The exact fixture passed in isolation. One unchanged rerun of that shard passed. The first failure remains recorded in the workflow history. - The earlier candidate’s full local test run was stopped after the confirmed review finding; that interrupted run is not reported as a pass. - All CI gates, including `ci / verify` and `ci / e2e`, pass on `bffebbab60edbdf50b42fc37a9e58beb01ceb94d`. Greptile is 5/5 on that head, with no remaining findings or unresolved review threads. ## Risks Recovery applies to archived Git sources, whose saved recipe and lifecycle checks still run before publication. A source change during recovery must keep the snapshot unpublished. Missing or corrupt saved objects still fail authentication or retain the existing import obligation. Mutable local sources still reject changed contents. This change adds no database migration and changes no saved archive format. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-05 22:19:20 -06:00
DottaandClaude Opus 5.5	5c185db77c	fix(tree-control): resume work a subtree pause interrupted (#19394) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task tree control lets the board pause a task tree and then resume it. > - A pause stops the runs that are active in the tree. The release must then continue the work that was pending. > - A child task whose run was active at pause time did not start again after release. The child became blocked and asked for a user message. > - A pause is an operator hold, not a failure. It must not strand work, and it must not require reconciliation for a run that never started. > - This pull request re-dispatches interrupted work one time on release, and it settles a legacy Stop that never entered the adapter. > - The benefit is that pause and resume keep a task tree moving without manual recovery, and real safety gates stay in force. ## Linked Issues or Issue Description **What happened?** A parent task delegated two children. One child's run was active (in host workspace preparation, before the adapter started) when the board paused the parent's tree. The pause cancelled that run. After the board released the hold, no new run started for that child. The child became `blocked` with the execution blocker cause `legacy_execution_requires_reconciliation`. Later the next action became "Send a message to continue." A sibling child that had no active run at pause time resumed normally. **Expected behavior** A run that a subtree pause cancels is an operator hold. On release, the pending work starts again one time, automatically. It does not go through execution reconciliation when the run never entered its adapter. It does not wait for a user message. **Steps to reproduce** 1. Create a parent task and a child task that is assigned to an agent. 2. Wake the child. While its run is in host preparation (or queued), call `POST /api/issues/:parentId/tree-holds` with `{ "mode": "pause" }`. 3. Call `POST /api/issues/:parentId/tree-holds/:holdId/release` without `wakeAgents`. 4. Before this change, no new run starts, and `getExecutionBlocker` reports `legacy_execution_requires_reconciliation` for the child. **Root cause** - (a) The pause records each interrupted run in the hold member snapshot (`activeRunId`). Nothing used that record on release. - (b) Release woke tasks only when `metadata.wakeAgents === true`. Without it, release relied on stranded-issue recovery. That recovery stands down when the latest run was cancelled by an operator (`isOperatorCancelledRun`). So the interrupted child had no owner. - (c) The pause Stop of a legacy run in preparation is written directly as `cancelled`. Legacy terminalization classifies a cancelled run with no bootstrap evidence as an unknown outcome. It records `legacy_execution_requires_reconciliation`, although the gated dispatch never entered the adapter. - Native runs: the verified native Stop does not create the reconciliation hold, but gap (b) applies to native runs too. The release change is runtime-independent. ## What Changed - `issueTreeControlService.findOwedPauseResume` decides if a task is owed one resume. The task's latest run must be an operator Stop that a pause hold member snapshot names, recorded while that hold was active. The task must still be executable by the same agent, and no pause may still cover it. The obligation belongs to the run, so overlapping pauses keep it until the last covering release. - New `services/issue-tree-resume.ts` re-dispatches owed work with wake reason `issue_tree_resumed` (`holdId` and `interruptedRunId` in the context). A compare-and-set claim (`treeHoldResume`) on the interrupted run makes the resume happen at most once. A failed wake returns the claim. - Pause release and `mode: "resume"` call it for every member of the released hold, also without `wakeAgents`. When `wakeAgents` is true, the existing loop wakes every executable member. - A task that still has an execution blocker is not replayed. The release writes `issue.tree_resume_wake_skipped`, the blocker stays visible, and the obligation stays unclaimed. - `releaseHold` updates only an `active` hold. A repeated release gets `409`. - `settleCancelledLegacyPreparation`: when the executor that owned a cancelled legacy run proves that dispatch never entered the adapter, it records bootstrap evidence and resolves that run's `legacy-execution:<runId>` hold as `false_positive`. It also corrects an automatic no-replay disposition and restores a `blocked` status only when that disposition made the change. Then it re-dispatches owed work, for a release that happened while the Stop was unwinding. It requires a requested Stop with no existing recovery receipt, and no process, session, native finalization, unreleased lease, or execution-start event. - `settleUnrecoverableExecutions` waits while a legacy executor still holds its controller lease, and records a `legacyCancellationBlock` receipt when it blocks a task for a cancelled legacy run. - Docs: `doc/execution-semantics.md` and `doc/composer-stop.md` describe the resume rule. ## Verification - New regression `server/src/__tests__/issue-tree-pause-resume-run.test.ts` uses embedded Postgres, the real heartbeat, and the real tree-control routes: - A run paused during preparation: nothing runs during the hold, there is no blocker, and release creates exactly one `issue_tree_resumed` run that succeeds. A second release returns `409` and creates no run. - Release before the stopped preparation finishes unwinding (with the periodic disposition run in between): exactly one resumed run after the executor settles. - An automatic no-replay disposition that ran first: the blocker clears, the status returns to `in_progress`, and release resumes once. - Overlapping child and parent pauses: no resume while the parent pause remains, then exactly one resume after it is released. - A queued run cancelled by the pause: release creates exactly one resumed run. - A task reassigned to a user during the hold: no resume. - A mid-turn legacy Stop with unknown outcome: no replay, the reconciliation blocker remains, and `issue.tree_resume_wake_skipped` is logged. - Red before each fix: the first 4 cases failed on master; the 3 review cases failed on the first PR commit. All 7 pass now. - Also passed locally: `issue-tree-control-routes`, `issue-tree-control-service`, `heartbeat-workspace-busy`, `heartbeat-undispatched-execution-wait`, `execution-control-work`, `legacy-workspace-restore-recovery`, `legacy-continuation-authority`, `task-retry`, `legacy-execution-recovery`, `agent-conversations`, `heartbeat-retry-scheduling`, `satisfied-connection-intents`. - Not verified here: a live staging pause/resume. Re-check on staging after deploy. ## Risks - Behavior change: release now starts the work that the pause interrupted, also without `wakeAgents`. Before, that work stayed idle. Other idle tasks still wake only with `wakeAgents`. - The settlement trusts the executor's adapter-entry flag. The flag is set only inside the atomic dispatch gate, which refuses a cancelled run. The settlement also requires no process, session, lease, or execution event. - A native run whose Stop has not finalized at release time is not a target. Stranded recovery and explicit wake still apply. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 22:08:01 -06:00
DottaandClaude Opus 5.5	958f9aa9f3	feat(issues): limit agent-created task trees to parent and child (#19312) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents split work into tasks. A task can have a parent task. > - Today agents can nest tasks to any depth: parent, child, grandchild, and deeper. > - Deep agent-made trees are hard to follow, and an agent that works on a child task makes grandchildren by default. > - This pull request limits agent-created trees to two levels: a top-level task and its children. > - A request for a grandchild does not fail. The server puts the new task beside the requester and tells the agent. > - Board users keep full control. They can still nest to any depth. Existing deep trees stay valid. > - The pull request also records which task created each new task, so the audit trail stays complete when the parent changes. ## Linked Issues or Issue Description No public issue exists. This is the feature description. Related: #16853 (budgets that include descendant work) also reasons about task trees; this change does not alter its traversal. **Subsystem affected** Issues (tasks): create, child create and move in the issue service and routes. Agent tools, chat suggestions and creation provenance use these paths. **Problem or motivation** Agents create subtasks as deep as they want. An agent that runs on a child task gets that child as the default parent, so its follow-ups become grandchildren. The trees become deep and hard to read. The parent link is also the only record of which task created a follow-up, so that record is lost when the parent changes. **Proposed solution** Limit agent-created trees to a parent and its children. When an agent asks for a grandchild, put the new task under the requester's parent and return `parentAdjusted`. Do not reject the request. Keep board users unrestricted. Record the originating task in a new server-set `origin_issue_id` column. Keep `origin_run_id` as the originating run. **Alternatives considered** Reject grandchild requests with an error. We did not do this, because agents then fail mid-task. Add a database constraint. We did not do this, because existing deep trees must stay valid and nesting can return later. ## What Changed - Add `MAX_AGENT_TASK_HIERARCHY_DEPTH` (2) and the `IssueParentAdjustment` type in `packages/shared`. - Add `resolveNewTaskParent` and `planIssueParentChange` in `server/src/services/issues.ts`. `create`, `createChild` and `update` use the agent limit by default (`parentDepthPolicy: "flatten"`). - The issue create, `/children` and PATCH routes pass `allow_nested` for board users. For agents, the routes check authorization against the parent that the server actually writes. - The requester keeps its behavior when the server moves a task beside it: `blockParentUntilDone` blocks the requester, and the new task inherits the requester's workspace and privacy. - The server never moves a task above a low-trust boundary root that contains the requester. - These subsystems keep any depth, because they find their work by parent: watchdog reviews, monitor recovery, email-channel placement, and routines that a person configured. - An agent move of a task that has children skips only the parent change and returns `reason: "has_children"`. - Add `issues.origin_issue_id` (nullable, `on delete set null`, indexed) in migration `0434`. Only the server sets it. The `createdFrom` detail field, the `createdFromIssueId` filter and task-creation provenance read it first. - Chat prompts now let a chat agent create one task, a task with children, or separate tasks. The conversation is never a parent. - The runner `create_task` tool accepts an optional `parentTaskId`. A chat agent can create a top-level task and then put subtasks under it. The run must be able to read and change the parent. A conversation is never a parent. The depth limit still applies. Project tools map the field to `parentId` through the issue route. The protocol manifest, semantic contracts and eval fixture are regenerated. - Watchdog runs that have an active watchdog keep their requested parent on all create paths, because their access is limited to the watched subtree. - Fix a bug that the test drive found: the chat (MCP) `create_task` ignored the requested `status`, so a `backlog` task became `todo` and started to run. It now uses the same rule as the native tool. - Update `skills/paperclip`, `doc/SPEC-implementation.md`, and add the plan `doc/plans/2026-11-05-two-level-task-hierarchy.md`. ## Verification - `cd server && npx vitest run src/__tests__/agent-task-hierarchy-depth.test.ts src/__tests__/agent-task-hierarchy-depth-routes.test.ts`: 19 new tests for service and route behavior, all pass. - Related suites (64 files: issues service, thread interactions, privacy, watchdogs, provenance, runner tool authority, decisions, routines, project tools): 1208 tests pass. Two tests changed because of the intended behavior and pass now. Two `email-channels.integration` tests fail locally because the test cannot find a bundled skill file from this worktree path. This change does not touch that code. - `packages/db`: migration safety, snapshot drift and journal tests pass. - Typecheck passes for `server`, `packages/shared`, `packages/db`, `cli` and `ui` (local `ui` errors come only from generated Storybook catalog files that are absent). - Local test drive on an isolated instance with real Claude agents: 29 of 29 use cases pass, including an agent on a child task (follow-ups placed beside it, with the notice) and a chat that creates a top-level task with three subtasks. - I did not run the full `pnpm test:run` or `pnpm build` locally. CI runs them. ## Risks - Behavior change for agents: some follow-ups that were grandchildren are now siblings. The parent of the requester is woken when all its children finish, which now includes these tasks. - The runner `create_task` input contract adds one optional field, so the protocol manifest and the pack contract change. The tool output does not change: it shows the actual `parentId`, and the activity log records `parentAdjusted`. - Migration `0434` adds a nullable column, a foreign key and an index. It does not backfill rows. - `IssueCreationSource.run` is now nullable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 21:22:33 -06:00