PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-10 01:50:13 +01:00

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	2ce5f03cd8	fix(workspaces): name remote restore failures and leave a board recovery action (#19207 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task restores its saved workspace into a sandbox before the agent starts. > - When that restore fails, the run records only `remote_workspace_operation_failed: operation=restore exit=1`. > - The run then fails as `setup_failed`. No retry starts, and a `blocked` task gets no recovery action. > - An operator cannot tell which step failed, and nobody owns the next step. > - This pull request makes the sandbox scripts name their failed step in a closed grammar, and it gives a restore failure a bounded retry or a named board recovery action. > - The benefit is a visible cause and a clear owner. Saved work stays protected. ## Linked Issues or Issue Description Refs #19152 and #19187. Those changes name copy-back (save) failures. Refs #18852 and #19195. Those changes added the bounded bootstrap retry that this change reuses. What happened? A board reply woke an agent on a `blocked` parent task. The run failed during setup 13 seconds later. The only run event was `remote_workspace_operation_failed: operation=restore exit=1`. The run had no workspace operation row. The task stayed `blocked`, and its recovery actions showed `active: null`. The same opaque family also appeared for workspace preparation (`exit=1` only). Expected behavior The run names which restore step failed, without command output or paths. A failure before provider work uses a bounded retry when a retry can help. Otherwise, and when the retry budget is spent, the board gets a named recovery action. The saved version is never changed or overwritten. Steps to reproduce 1. Save a remote workspace, then start a new run on a different allocation. 2. Put a folder at the published workspace path in the sandbox before restore. 3. Observe that the base code records only `operation=restore exit=1`, then `setup_failed`, with no recovery action for a `blocked` task. Paperclip version or commit Base: `87dfba6818`. Deployment mode Legacy runtime with sandbox (Daytona) remote workspace storage. ## What Changed - The prepare, repository, and restore sandbox scripts track their current stage. On failure they print `remote_workspace_<op>_failed: stage=<stage>` plus closed fields: `reason=` (the script's own snake_case refusal), `archive=` (the archive helper's fixed reason), `code=` (errno), `disk=full`, `git=` (a fixed label chosen from Git output), and the existing `exit`/`signal`/`program` fields. - The server accepts that text only when it matches a strict grammar for the same operation. Other output is dropped, as before. - Provider refusals during the restore upload and restore command are recorded by closed labels, like copy-back. - A failed saved-workspace restore raises `RemoteWorkspaceRestoreError` with the server step (`snapshot_lookup`, `snapshot_download`, `sandbox_restore`). It is retryable only when the owner re-reads the same writer, epoch, and saved version with no unsaved work, and the sandbox did not name a refusal reason. - Setup failure handling classifies a pre-dispatch legacy restore failure as `remote_workspace_restore_failed`. A retryable failure uses the existing bounded bootstrap retry. A non-retryable or exhausted failure records a board-owned recovery action with the redacted cause, when stranded-work recovery does not scan the task status (for example `blocked`). - Stranded-work recovery gets a named notice for this code. Generic continuation recovery does not grant it a second budget, and legacy reconciliation does not invent provider actions for it. - `doc/execution-semantics.md` documents the contract. ## Verification - Red: the new transport tests failed on the base code with `remote_workspace_operation_failed: operation=restore exit=1` (and the same for prepare and repository). The new completion tests failed because the run became `setup_failed` with no retry and no recovery action. - Green: `npx vitest run src/services/remote-workspace-transport.test.ts src/services/heartbeat/run-completion.test.ts src/services/remote-workspaces.integration.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery src/services/remote-workspace-save-concurrency.test.ts src/services/native-runtime/native-workspace-finalizer-failure.test.ts src/__tests__/native-workspace-finalization-work.test.ts` passes (563 tests). - The transport tests use real sandbox commands. They prove an occupied folder reports `stage=inspect_root reason=occupied_root` and keeps its files. They prove a wrong manifest reports `stage=extract archive=manifest_mismatch`. They prove a missing branch reports `stage=checkout_branch`, and a changed origin reports `stage=verify_origin`. Forged output with a path is dropped. - The real-PostgreSQL cold-restore test proves an occupied root gives a non-retryable `sandbox_restore` failure and keeps the unknown files. - The completion tests cover retried, exhausted, refused, post-dispatch, and `in_progress` cases. - `tsc --noEmit` for the server reports no errors in the changed files. ## Risks - A retry keeps the same restore path. If the cause is deterministic but unnamed, the run uses its two retries before the board action. Named refusals skip the retries. - The board action is recorded only for statuses that stranded-work recovery does not scan, and only when no other action is active. This prevents a second recovery identity for one failure. - Script stderr grammar is new. Old sandboxes run the script that the server sends, so no runner update is necessary. - No migration, deployment, or credential change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use, code execution, and browser tools for read-only staging evidence. Context window and reasoning mode are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 02:40:01 -06:00
DottaandClaude Opus 5.5	c656b95b6b	fix(environments): retry a refused sandbox resume with the bounded bootstrap budget (#19195 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm remote agent keeps its sandbox after a turn, so a follow-up turn resumes the same sandbox and its files > - The sandbox provider auto-stops an idle sandbox (Daytona: 15 minutes by default). The warm native idle window is also 15 minutes > - A follow-up that reaches the sandbox during that stop gets a provider refusal. The host keeps the sandbox ("the lease was preserved and no replacement was created"), which is correct > - But setup then records a generic `setup_failed`. No retry is scheduled, no cause is recorded, and the reply that woke the agent is effectively dropped > - This pull request gives that refusal its own failure code and routes it through the existing bounded bootstrap retry. The retry resumes the same recorded sandbox > - The benefit is that a follow-up continues a few seconds later on the same saved sandbox, instead of stopping the task with no next action ## Linked Issues or Issue Description Refs #18852 (the bounded bootstrap retry that this change reuses). Related: #19152 and #19187 (copy-back causes), #19178 (Daytona errored-sandbox recovery before stop-only cleanup), #19182 and #19180 (recovery holds). This change does not overlap with them: they cover cleanup, copy-back, and recovery after a run. This change covers a resume refusal before provider work starts. What happened? On a staging soak, two follow-up turns failed at setup after a successful first turn: - A board reply woke the agent. The run failed after 4.9 seconds with `setup_failed`: `Reusable sandbox lease "…" could not be resumed; the lease was preserved and no replacement was created.` The issue stayed in review with no recovery action. The reply was not delivered. - An earlier run on another task failed with the same message. Both resume attempts started 15 minutes 8–25 seconds after the previous turn released the sandbox. The first turns used `keep_running` with a warm idle timeout of 900,000 ms. The Daytona default `autoStopInterval` is also 15 minutes. So the resume reached the sandbox while the provider was stopping it. The provider refused within seconds; this was not a long resume timeout. The provider cause was not recorded, so this timing is the strongest evidence. The exact provider state remains an inference. Expected behavior A refused resume before provider dispatch is a temporary infrastructure condition. The host retries it with a small, bounded budget. Each retry resumes the same recorded sandbox. A refusal does not prove that the sandbox is lost, so the host never destroys it or creates a replacement. Only an explicit provider not-found or a workspace identity mismatch with a verified backup permits replacement (unchanged). The run records why the provider refused, without provider text. Steps to reproduce 1. Run a warm remote turn that keeps its sandbox. 2. Make the provider's `environmentResumeLease` throw a non-transient error (for example, a 400/409 while the sandbox stops). 3. Wake the agent with a reply. 4. Before this change: the run fails with `setup_failed`, no retry is scheduled, and the error has no cause. Paperclip version or commit Seen on staging at `5b99bdae`. The touched code is unchanged on current `master` (`f7a4c5678f`). ## What Changed - Move `ReusableSandboxResumeError` to `server/src/services/reusable-sandbox-resume-error.ts` (re-exported from `environment-runtime.ts`). Give it the code `reusable_sandbox_resume_failed`. - Add the provider refusal to the error message by closed labels only: `(cause=<label> status=<http> code=<errno>)`. The labels come from the existing transport classifier, now exported as `sandboxProviderFailureLabel`. Provider text, paths, and IDs are never copied. - Carry the provider status across the plugin worker RPC. A JSON-RPC error has only a message and a numeric RPC code. The worker host now attaches a validated `paperclip/environment-resume-error/v1` envelope (HTTP status and an allowlisted errno only) to `environmentResumeLease` failures. The host re-validates it and adds it to the recorded cause. - In setup-failure settlement, find this error through the orchestrator wrapper. Record `reusable_sandbox_resume_failed` only when provider dispatch did not start and the run has no native session or runner identity (the same guard as `remote_workspace_prepare_timeout`). - Add the code to `isTransientWorkspacePreparationCode`. The existing scheduler then gives two retries (30 s, 60 s) with its issue gate, Stop, pause, and dependency checks. - Exclude the code from generic continuation recovery and from legacy reconciliation (no unknown provider actions; bootstrap evidence proves none started), like the template. - Add a specific stranded-recovery notice: "Saved sandbox could not be resumed". - Document the contract in `doc/execution-semantics.md`. ## Verification - Red (before the fix, same tests): 6 failures. The `fresh` case recorded `setup_failed` and did not schedule a retry. The error had no cause. Legacy reconciliation held the run. Generic recovery granted another budget. - Green: `npx vitest run src/services/heartbeat/run-completion.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery/ src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/environment-runtime.test.ts src/services/remote-workspace-transport.test.ts src/services/heartbeat/retries.test.ts` (in `server/`): 619 tests pass. Also `src/services/recovery/stranded-notice` and `src/services/remote-workspace*`: 411 pass. - Safety regressions: - `run-completion.test.ts`: only a pre-dispatch legacy failure with no native identity gets the code and a retry. `started`, `retained` (native), and an untyped copy of the message stay `setup_failed` with no retry. The recorded error contains the closed cause and not the provider path. - `heartbeat-retry-scheduling.test.ts` (real PostgreSQL): two concurrent schedulers across a restart create exactly one successor per attempt. The budget ends after two attempts. Stop, pause, and a new dependency cancel the scheduled retry. Generic stranded recovery creates no extra run after exhaustion. - `packages/plugins/sdk/tests/environment-resume-error.test.ts`: through the real worker RPC, the `JsonRpcCallError` has no status, and the envelope gives 409 and the errno. Provider text does not cross. Other methods get no envelope. Malformed envelopes are rejected. - `environment-runtime.test.ts`: a throwing resume still preserves the exact lease. The runtime does not call destroy or acquire. The resume throws a `JsonRpcCallError` whose message has no status. The thrown message names `cause=provider_error status=409` from the envelope, and not the provider text. - `node scripts/typecheck-low-memory.mjs` passes. `git diff --check` passes. - Not verified: a live staging run at this revision. The original tasks were not retried. This PR is not deployed. ## Risks - Low. The change affects only runs that fail with this typed error before provider dispatch. Each retry resumes the same lease and inherits the source wake context, so a reply is delivered once and no second executor starts. - If the provider cannot start the sandbox for longer than about 90 seconds, the two retries end and the task gets the specific recovery notice. The sandbox and its saved files are kept. - The error message now has a parenthetical cause before the semicolon. No code parses this message. - An issue in review does not get the immediate stranded notice (unchanged existing behavior). The retry, not the notice, delivers the reply in that case. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, browser reads of staging diagnostics). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 00:20:21 -06:00
Nicky LeachandClaude Opus 5.5	11ac90e31f	fix(build): typecheck server/src/__tests__ and fix every type error (#18739 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server has about 1,100 test and helper files in `server/src/__tests__` > - `server/tsconfig.json` excludes that directory, Vitest does not check types, and no script runs the tsconfig in that directory > - So a test can call a function with the wrong arguments and still pass, and no tool reports it > - This gap hid 2,168 real type errors, and some of them hid tests that did not test what they claim > - This pull request fixes all of these errors and adds the directory to the server layers from paperclipai/paperclip#18737 > - The benefit is that a type error in a server test now fails the typecheck, with no baseline and no backlog ## Linked Issues or Issue Description Refs paperclipai/paperclip#18737, which is merged and added the server layers. Refs paperclipai/paperclip#18738, which is merged and checks the test projects outside `pnpm -r`, so it is not a duplicate. Refs paperclipai/paperclip#18137, which records that this directory has no check. What happened? No tool checked the types of 1,074 of the 1,101 files in `server/src/__tests__`. I searched `package.json`, the workflows, and all tsconfig files. 27 files are in the program of another tsconfig, and these 27 have 0 errors. 11 of the 27 are in `helpers/maintenance/tsconfig.json`. No script ran that project when I measured this. paperclipai/paperclip#18738 now runs it. Expected behavior The server typecheck checks these files, and a type error fails the check. Steps to reproduce 1. On the base commit, run `tsc -p server/src/__tests__/tsconfig.json --noEmit`. It reports 6,536 errors and needs 12.3 GiB. 2. Add a type error to any file in `server/src/__tests__`. 3. Run `pnpm --filter @paperclipai/server typecheck`. It passes. Paperclip version or commit I measured the error breakdown below on commit `8dfd632452`, with TypeScript 7.0.2. The branch now includes master up to paperclipai/paperclip#18865. Deployment mode Not applicable. This issue applies to development and CI. No production source file in the server changes. ### Error breakdown Each row adds one change to the row above. \| Step \| Total errors \| Change \| Cause \| \| --- \| --- \| --- \| --- \| \| Config as committed \| 6,536 \| \| \| \| Add `src/types/.d.ts` \| 4,447 \| -2,089 \| Config. The Express request types were missing. \| \| Set `rootDir` to the repository root \| 2,436 \| -2,011 \| Config. \| \| Set `allowImportingTsExtensions` \| 2,181 \| -255 \| Config. Vitest accepts `.ts` imports. \| \| Add `ES2024.Promise` to `lib` \| 2,138 \| -43 \| Config. Node 24 has `Promise.withResolvers`. \| \| Set `allowJs` for `.mjs` imports \| 2,148 \| +10 \| Config. It removes 8 `TS7016` errors and shows new real errors. \| \| Set `declaration` to false \| 2,142 \| -6 \| Config. Tests emit nothing. \| \| Count only files in `server/src` \| 2,058 \| -84 \| Other packages: 53 Daytona, 29 Runner test, 2 UI. \| So 4,478 of the 6,536 errors (69%) came from config or scope, and 2,058 are real type errors in 314 files. The 29 Runner test errors are also real, which gives 2,087 errors. Tests that master added after this measurement had 81 more errors, so this pull request fixes 2,168 errors. This table classifies the first 2,058 errors. \| Kind of real error \| Errors \| \| --- \| --- \| \| Wrong argument or assignment type \| 1,023 \| \| Stale or partial fixture objects \| 525 \| \| Property that the type does not have \| 317 \| \| Missing narrowing of null or unknown \| 101 \| \| Implicit any \| 60 \| \| Missing name or module \| 16 \| \| Other \| 16 \| ### What the type errors hid Most errors were stale fixtures and loose mocks. These errors hid real test problems, and this pull request fixes them: - A test that never ran.* `instrumentation.test.ts` required `sdk-trace-base`, which is not installed. So 3 span tests were skipped. They now run and pass. - Assertions that could not fail. - `issue-thread-interactions-service` used a company ID as an issue ID. - `status-card-evaluation-work` compared `[]` with `[]` in a rollback check. - `slack-conversation-lifecycle` read `item.issueId`, which does not exist. - `email-channels` used a permission key that does not exist. - `heartbeat-project-env` imported `LOW_TRUST_REVIEW_PRESET`, which is not exported. The value was undefined. - Postgres processes that did not stop. 11 test files called `stop` on the test database helper. That method does not exist, and the helper method is `cleanup`. - Arguments that the service ignored. - `methodKey` is now `connectionMethodKey` in the Daytona, Unikraft, 1Password, and Railway tests. - 108 calls passed `"owner-user"` to a parameter that expects an options object. - 9 chat wizard fixtures passed `managementContext` as `source`, so they ran `existing_app` and not `manifest`. - 2 tests passed a `{ userId }` actor that the function ignores. - Inserts into `environments` passed a `companyId` column that the table does not have. - Problems in tests that master added during this work. - `sandbox-creation-operations` read `database.connectionString` when an external test database was set. In that mode `database` is undefined, so the fixture would stop with a TypeError. It now uses the external URL. - `idle-completed-history` set a Dot operation to `dispatched`. That is a Muse status, and the Dot type does not have it. The fixture now uses `admitted`, which is a real in-flight Dot status. ### Possible product problems, not changed These may be real problems in production code or types. I did not change them, because this pull request does not change production behavior: - `decisions-service` `service.create` does not validate the lenient `cancel_issue_tree` input. - `createConnection` ignores `input.connectionPurpose`. - The `createConnection` route schema default stores `{}` for `transportConfig`. - `prepareManagedAiRuntime` accepted the invalid value `mode: "connection"`. - `createBinding` accepts invalid `projectionClass` strings outside class 3. - `ensureManagedProjectWorkspace` forwards an untyped `repoRef` by spread. - `routineVariableSchema.label` is optional, but `RoutineVariable.label` is required. - `ToolRunContext.projectId` may be too strict. - `withdrawInteraction` reads `issue.status`, which its type omits. - `issueService.list` and `reconcileNativeFinalizations` omit fields that they return at runtime. - `createGitHubExternalObjectProvider` handles `tokenProvider: null`, but its type forbids this value. ## What Changed - Fixed the 2,168 type errors. The diff changes 322 test and helper files. - Corrected `server/src/__tests__/tsconfig.json`. Editors and the typecheck now read the same options and report the same errors. - Added three `test-tree` layers to `server/scripts/typecheck-layers.mjs`. They read the server layers as declarations, and any error fails them. A hash of the path selects the shard, so a new file does not move other files. - Added `server/scripts/typecheck-stubs/daytona-sdk.d.ts`. Server tests reach Daytona plugin source, and the root install does not have `@daytonaio/sdk`. - Changed two imports in `ui/src/lib/issue-execution-policy.ts` to use `.js` extensions, because server tests import this file. - Made `declarationsConfig` and `--declarations-only` skip the `test-tree` layers, because these layers emit nothing. Other projects read only the source layers. Added a test for this. - Updated `doc/DEVELOPING.md`. ## Verification Run these commands: ```sh pnpm --filter @paperclipai/server typecheck pnpm exec vitest run server/scripts/typecheck-layers.test.mjs ``` Results on macOS (arm64), Node 24, 16 cores. Each check ran after the merge of paperclipai/paperclip#18738 unless the row says otherwise: \| Check \| Commit \| Result \| \| --- \| --- \| --- \| \| Server typecheck, `PAPERCLIP_TYPECHECK_LOW_MEMORY=1`, all 7 layers \| `6e45a08573` \| Exit 0, 0 errors, 62 seconds, peak 3.52 GiB \| \| `node scripts/typecheck-test-projects.mjs --low-memory` \| `a3306bd13e` \| Exit 0, 0 errors, 42 seconds, peak 3.25 GiB \| \| The same script with `--reuse-declarations` \| `1b878aabc0` \| Exit 0, 0 errors \| \| `typecheck-layers.test.mjs` and `live-recovery-work.test.ts` \| `01db5634e4` \| 90 passed. These include real compiler runs for a pass, a type error, and an invalid tsconfig. \| \| `tsconfig.declarations.json` after a full server typecheck \| `0fb1e4fb0a` \| It references only `below`, `cycle`, and `above`. No generated config names a `tsconfig.layers.json` file. \| \| Owned runtime test in `idle-completed-history` \| `e8f2f448ac` \| Passed, with the fix that master later replaced \| \| `idle-incoming-wake`, `plugin-idle-manager`, and the changed `chat-channels` test \| `6e45a08573` \| 30 passed \| \| Runner `native-execution.test.ts` \| `eab4dcc1f4` \| 114 passed \| \| The 19 server test files that changed since `eab4dcc1f4`, and `typecheck-layers.test.mjs` \| `259737b700` \| 4,275 of 4,277 passed. The 2 failures are in `chat-channels` and are in the list of 7 below. \| The peak is the largest single process, from `/usr/bin/time -l`. Peaks for the server typecheck were between 3.42 GiB and 3.56 GiB across 8 runs. Peaks for the test projects were 3.25 GiB and 3.37 GiB across 2 runs. I did not measure a cgroup. Runtime check of the fixes. Types do not prove that a test still works. So I ran all 309 changed server test files, 11,929 tests, before the rebase onto paperclipai/paperclip#18737: \| Result \| Tests \| Detail \| \| --- \| --- \| --- \| \| Passed \| 11,913 \| Includes the 2 tests and the 54 `dot-runner` tests that passed in a later run, as described below. \| \| Failed \| 7 \| The same 7 tests also fail on the commit before the fixes. \| \| Skipped \| 9 \| These tests have their own skip conditions: `sentry` 5, and 1 each in `bundled-plugin-registration`, `customer-success-cloud`, `onepassword-connection`, and `workspace-runtime-https-live-exercise`. \| The 7 tests that fail with and without this change, on my machine: - 3 tests in `chat-channels` and `email-channels` stop with `ENOENT` on a `SKILL.md` file. The path is in the parent checkout, above the git worktree that I used, so I expect that the worktree is the cause. I did not prove this. - `workspace-instance-cleanup` compares `/private/var/...` with `/var/...`. This is a macOS symlink difference. - 2 tests in `git-reproducible-managed-checkouts` get `unknown_base_git_files` and expect `unpushed_managed_checkout`. I did not find the cause. - 1 more test in `chat-channels`, `materializes direct external-chat finals once`, gets `undefined`. I did not find the cause. CI is the check for these 7 tests. `ci / verify` passed on `eab4dcc1f4`. Failures during this work: - My fix to `agent-runtime-connector-access` changed an invalid permission key in the setup, but not in the assertion. The runtime check found this. I corrected the assertion, and the file passes. - `heartbeat-plugin-environment` failed one wait in the full run. It passed three times alone, so I think that load caused the failure. It can fail again under load. - `dot-runner` skipped all 54 tests in the full run because `cargo` was not on my `PATH`. It then passed 54 of 54 with the toolchain on the `PATH`. - My first `lib` value removed the default DOM types, and the error count went up by 375. I added the DOM libraries. - My Daytona stub with bare `any` left 48 errors, because generic helpers then inferred `unknown`. I added shapes that return promises. - One fix to `agent-secrets-routes` used `class_3_static_lease`, and the test failed. That class allows only specific paths. I reverted to `unclassified`. - `ci / General tests (server (8/12))` failed on `259737b700` because of the flaky port in `idle-completed-history`. paperclipai/paperclip#18853 fixed this test on master, so this pull request keeps that fix. Audit of the diff. I counted these patterns in added and removed lines, to find fixes that only hide an error: \| Pattern \| Added \| Removed \| \| --- \| --- \| --- \| \| `@ts-ignore` \| 0 \| 0 \| \| `@ts-expect-error` \| 0 \| 1 \| \| `.skip` \| 0 \| 0 \| \| `as any` \| 9 \| 9 \| \| `: any` \| 11 \| 2 \| \| `as never` \| 8 \| 2 \| \| `as unknown as` \| 35 \| 4 \| \| `Promise<unknown>` \| 16 \| 5 \| \| `expect(` \| 128 \| 113 \| No hunk removes more `expect(` calls than it adds. Three removals of test content were intentional, and I checked each one: - `muse-control` removed `signalToken`. Line 1012 already asserts that it is absent. - `merged-pr-confirmation-sweep` removed legacy aliases. Other tests cover them. - `heartbeat-workspace-branch-containment` removed a `finalize` branch that no test row can reach. Some partial fixtures stay unchecked through `as unknown as` and `Promise<unknown>`, mostly in `issue-agent-mutation-ownership-routes` and `issue-comment-reopen-routes`. Accuracy of the check. One program over the tree and the three shards reported the same errors at the same positions. Memory, as the peak of the largest process on one thread, was 3.93 GiB for one program and 2.43 GiB for the largest shard. I measured these two values before the fixes. Not run or not proven: - I did not run the full `pnpm test:run`, or a root `pnpm build`. I expect CI to run the full suites. - I did not run the 4 GiB container check, or the full `scripts/typecheck-low-memory.mjs`. The peaks above come from macOS, so they are not a Linux cgroup measurement. - I did not run the UI typecheck after the merges. - `workspace-runtime-https-live-exercise` is skipped, so its `cleanup` fix did not run. ## Risks - A large diff in test files. 328 files change. A pull request that edits the same tests can get merge conflicts. New tests on master can also add type errors before this merges. Any such error fails this check and needs a fix here. - A higher cost for changes to source types. A change to a type can add errors in test files that the author did not touch. The author must then fix them. This is the purpose of the check, but it is new work. - Changed test behavior. Some tests now run paths that they did not run before, such as the 9 chat wizard fixtures and the 3 span tests. They pass here, but they can be more sensitive to timing or to the environment. - The Daytona stub can go stale. The stub is written by hand. If the plugin uses a new SDK member, the typecheck reports an error until someone updates the stub. - The release path. `release-verify.yml` and `scripts/release.sh` run `pnpm -r typecheck`, so they now run this check. - The tsconfig change. The test transform reads `server/src/__tests__/tsconfig.json`. The options that I changed are for type checks, and the runs above passed. - A gap that remains. Tsconfigs in `packages/paperclip-runner` and in most sandbox provider plugins exclude `*.test.ts`. This check does not read most of those test files, so they can still drift. That work is separate from this pull request. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with reasoning, tool use, code execution, and subagents. The runtime does not show the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (7 tests fail locally, with and without this change. See Verification.) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 15:27:38 -08:00
DottaandPaperclip	0a1e4c3d0f	fix(workspaces): retry pristine remote setup timeouts (#18852 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task needs a workspace before its model can start. > - Fresh workspace preparation can reach its transport deadline. > - Failure handling records that timeout as a generic setup failure and stops the task. > - An unpublished workspace has no saved task work to repeat. > - This change routes that narrow failure through the existing bounded retry scheduler. > - Saved work, retained executions, and normal admission gates keep their existing protection. ## Linked Issues or Issue Description Refs #18827, which preserves remote timeout diagnostics. This change builds on that merged repair. Related #16727 concerns controller-managed clones and optional partial clones; this change keeps remote clone contents and deadlines unchanged. What happened? A fresh remote workspace reached its 300-second command limit before model execution. The task then stopped with `setup_failed` and no scheduled continuation. A separate full clone completed in 137 seconds on a disposable Daytona sandbox. The original delay remains unexplained. This change repairs the missing recovery path; it does not claim to remove the underlying delay. Expected behavior A fresh, unpublished workspace preparation timeout can use the existing two automatic retries. Retry scheduling remains durable and company scoped. Stop, pause, dependencies, budgets, task ownership, saved work, and retained native execution must still prevent unsafe work. Steps to reproduce 1. Start preparation for a fresh remote workspace with no saved version or published allocation. 2. Make its initial directory/clone command return a transport timeout. 3. Observe that the base code records a generic setup failure without bounded recovery. 4. With this change, the owner validates current workspace authority and preserves a specific bootstrap failure for the normal retry scheduler. Paperclip version or commit Base: `e0276e0166`, including #18827. Deployment mode Native Paperclip Runner with sandbox workspace storage. Verification uses real PostgreSQL, filesystem commands, and a separate live Daytona transport probe. ## What Changed - Recognize only the initial preparation timeout while the same company, writer, and epoch still own a pristine workspace. - Keep restore, additional repository, provisioning, save, changed ownership, dirty workspace, and retained native execution failures outside this recovery policy. - Persist the typed failure before using the existing retry scheduler, admission gates, and attempt budget. - After successful terminal lease cleanup, reclaim an unpublished bootstrap-only reservation even when a provider returns no stop receipt. Preserve receipt requirements for saved/published work, unknown execution, and native identities; do not synthesize termination evidence. - Prevent generic recovery from granting another budget after exhaustion. Keep a specific recovery notice. - Add regressions for persistence, concurrent scheduling, restart, exhaustion, Stop, pause, dependencies, fresh allocation, lost replies, and saved output after cold restoration. - Document the recovery contract. ## Verification - Red: the real-PostgreSQL workspace regression first failed because the timeout had no recovery classification. The completion regression then failed because it became `setup_failed`. A retained-native negative failed before the admission guard. The review follow-up reproduced a failed successor reservation after production lease release returned void. - Green: all 127 transport/archive/real-PostgreSQL workspace cases pass after the review repair. All 12 receipt/void handoff cases pass against the final query. The 141 complete completion/retry/recovery cases and the fresh-allocation selection case pass. Safety cases cover concurrent scheduling across restart, the two-attempt cap, Stop/pause/dependency gates, native identities, saved/published files, active leases, failed cleanup, and cold-restored output. - `node scripts/typecheck-low-memory.mjs` and `pnpm build` pass at `ba96cff771707415e4a5e893c8b3a97499ead25b`. `git diff --check` and the module-boundary check pass. The full local test wrapper completed all 1,835 chat cases across six partitions, then was interrupted during its serial general-server phase after the complete current-head CI suite passed. Its interrupted run is not a full local-suite pass; its log is retained. The owned process tree and database children exited. - The [complete GitHub Actions run](https://github.com/paperclipai/paperclip/actions/runs/41718108375) is successful at this exact head, including all server/workspace/chat partitions, all nine serialized groups, all eight browser shards, build, typecheck, native Runner, package, Rust, and release-policy checks. The first Rust worker shut down during cache restore before tests; one retry passed without source changes. Expected Storybook skips remain skips. - Earlier failures remain accounted for: the first typecheck found two incomplete process-result fixtures, now corrected. An intermediate completion run exposed a test double that returned no promise and a teardown timeout; the complete 141-case suites subsequently passed. The first-head CI had four worker shutdowns and one worker communication loss. A three-second monitor settlement wait failed in CI and locally, then all 21 cases passed unchanged at base and head. An earlier CI retry found a real-process history adoption assertion failure; that exact case passed locally at base and head. Both server groups pass in final-head CI. These timing causes remain unknown. [Earlier CI evidence](https://github.com/paperclipai/paperclip/actions/runs/41713712522). - Live supporting evidence: unchanged production preparation code completed a full repository clone in 137 seconds using Daytona SDK 0.203.0, Node 24.22.2, Git 2.39.5, and Linux x86_64, with 1 CPU, 1 GiB memory, and 1 GiB disk. The approved diagnostic account could not access the original fleet snapshot, so the probe used public `node:24-bookworm`. Both probe allocations were deleted and absence verified. Full task acceptance at the fixed revision remains unverified: the original fleet snapshot and an explicitly configured disposable Codex subscription fixture were unavailable. No CLI login or hosted subscription was copied. - Greptile is 5/5 at the final head with no actionable findings. The Paperclip review finding is repaired, replied to, and resolved. The separate Paperclip Review check remains queued. Its task failed workspace export after publishing the first assessment; the staging service was then observed showing a workspace recovery page after an update failure. Subsequent browser rechecks timed out, including a fresh tab, so the normal recovery control could not be used. Reconcile that existing export after service access returns, then let the saved current-head review request run. That external review is the remaining check; this PR is not yet fully green. - The original hosted task was last verified blocked. This PR has not been deployed and the task has not been retried by this work. Reviewers can run `pnpm test:run`, `node scripts/typecheck-low-memory.mjs`, and `pnpm build`. The remote-workspace integration suite additionally needs an owned disposable PostgreSQL database named `remote_workspace_*test` through `PAPERCLIP_REMOTE_WORKSPACE_TEST_DATABASE_URL`; otherwise that suite is skipped. ## Risks Each retry keeps the existing preparation deadline, so persistent slow clones can consume the bounded budget before intervention. A retry receives fresh allocation admission; it does not replay the command against the old allocation. The workspace owner checks persisted authority before classifying the failure, and normal admission still rechecks that authority. Historical generic setup failures do not gain automatic replay. No database migration, provider-pack update, model fallback, credential change, or deployment is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser tools. The host does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 15:36:07 -06:00
DottaandPaperclip	982ff62949	test(shared): consolidate ZIP conversion observations (#16118 ) Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-15 00:06:50 -05:00
DottaandPaperclip	b67db12d90	feat: add durable storage for private tasks (#14717 ) Add private task and project ownership, downward access grants, and immutable run/workspace provenance. Apply migration 0313 with bounded batch commits and concurrent indexes. Keep the enforcement and sharing changes in their dependent PRs. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-07 06:53:20 -05:00
Devin FoleyandPaperclip	3ebd7bc0c9	test: isolate accounting fixtures and include all adapter suites (#14989 ) Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-06 18:13:46 -07:00
DottaandPaperclip	7a52dcdc74	fix: repair MCP validation and cancelled execution recovery (#14951 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The tool gateway gives agents access to connected services. Recovery controls what happens when a run stops. > - Generated tool names can exceed the provider limit after the MCP client adds its prefix. > - The same invalid definition can fail each automatic retry. A cancelled run can also hold saved messages without showing its cause. > - This pull request bounds tool names, stops configuration retries, and retains cancellation evidence. > - It shows the stopped run and admits saved input only after the existing safety checks pass. > - The benefit is a clear recovery path that preserves operator Stop and prevents duplicate message delivery. ## Linked Issues or Issue Description What happened? A long connected MCP tool name makes the provider reject the entire request. Automatic recovery repeats the invalid request. Separately, unexpected legacy cancellations can leave saved input behind a recovery hold. The notice does not identify the stopped run or its cause. Expected behavior Complete MCP names fit the provider limit. Tool-definition errors require configuration repair. Cancelled runs retain their source and reason. The recovery notice shows the cause and saved-message count. Verified unexpected cancellations can start a fresh turn through the existing admission checks. Steps to reproduce 1. Assign an App gallery connection with a long application key and tool name to a Claude agent. 2. Start a run. The provider rejects a name over 128 characters, including its MCP prefix. 3. For cancellation recovery, stop a legacy provider turn without an operator Stop request and send a user message while the recovery hold is active. 4. Inspect the recovery notice and the deferred message queue. Paperclip version or commit Rebased onto master at `cf8ad63c806685bfd7c48e3ed4a919d61a7c55f1`. Deployment mode Hosted or self-hosted server with legacy Claude or Codex execution. Related public work: - Refs #14017. That PR caps name segments. This PR preserves existing short names and uses stable hash aliases for long complete names. It also covers classification and recovery. - Refs #4510. That PR adds a cancellation-source column. This PR records bounded evidence in the existing run result, without a migration. - Refs #12552 and #4506. Those PRs suppress recovery after operator cancellation. This PR preserves operator intent and uses the existing continuation gates. ## What Changed - Bound gateway names with the full provider prefix in the 128-character budget. Retain the original upstream tool name for dispatch and permissions. - Classify invalid tool definitions as configuration failures before diagnostic redaction. Stop automatic retries and continuation attempts for that error code. - Persist cancellation source, expectedness, initiator, reason, and time. Preserve recorded Stop intent when adapter results arrive. Report unexpected started cancellations with closed diagnostic labels. - Show the run cause, saved-message count, and Inspect run link. Offer Continue for eligible unexpected cancellations. Require verified provider stop, empty tool inventory, ownership, and the existing pause, budget, approval, and dependency gates. Use the existing queue for single delivery. - Add regression coverage and update the execution, MCP gateway, and run-log documentation. ## Verification - `pnpm -r typecheck` and `pnpm build` passed. - `pnpm check:token-gates` passed. - Ran `pnpm test:run` and completed its workspace and serialized groups. Initial resource and timing failures passed on isolated reruns. All 149 serialized route suites passed. - Reran the changed server, adapter, and UI suites after the rebase. Coverage includes long-name upstream dispatch, configuration retry suppression, cancellation evidence retention, privacy labels, oversized run projection, and concurrent saved-message delivery. - `pnpm test:e2e tests/e2e/legacy-failure-continuation.spec.ts` passed all six browser scenarios. The recovery notice shows the run cause and inspection link, and each recovery entry point reaches one new response. - Added database-backed checks for active, removed, paused, unavailable, and disabled chat connections. The final continuation and recovery-notice suites passed 167 tests. Externally bound chats hide board Continue and show a usable next action. - All 55 GitHub checks passed on `42afbf1371dcaeb72646e3d8f65c19ff7cddf8de`. Two unrelated Storybook jobs were skipped by their normal conditions. Greptile reviewed that commit at 5/5 with no findings and no open review threads. ## Risks - Long tool names change to aliases. Existing short names stay compatible. The original connection and upstream name remain the dispatch authority. - Invalid tool definitions no longer get automatic retries. An operator must repair the configuration before a new attempt. - Continuation changes apply only to positively identified unexpected legacy cancellations with complete empty tool inventory. Operator Stop, unknown historical cancellations, outstanding tools, and unverified provider termination keep their holds. - No database migration. The added projection fields are optional. Cancellation reason and initiator IDs remain local run evidence; Sentry receives only closed source and initiator-type labels and expectedness. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository editing, shell execution, and GitHub tool use. The runtime does not expose the exact model variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-02 13:47:59 -05:00
DottaandPaperclip	96bf004a79	fix: use persisted state for lifecycle continuation and retry budgets (#13888 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its control plane decides when a task can continue, wait, stop, or complete. > - Legacy continuation could change when an agent changed its wording without changing task state. > - Shared attempt counts also let repair and infrastructure retries affect each other's limits. > - This pull request uses persisted state and separate, bounded allowances for these decisions. > - If automatic repair stops, the task explains what happened and offers a guarded retry. > - Paired tests and real-provider evaluations verify that Stop, approvals, ownership, and spending limits remain authoritative. ## Linked Issues or Issue Description Related work: Refs #13761, Refs #11126, Refs #13610. These cover obsolete continuation dispatch and retry storms. Open and closed issues and PRs were searched for related lifecycle, continuation, and retry work. What happened? Legacy continuation depended on English wording and progress heuristics. Repair, failure retry, and productive continuation could consume shared counts. When bounded repair stopped, the task showed a technical recovery message without a clear next action. Expected behavior Persisted disposition and owned execution paths determine the next action. Missing disposition prompts bounded agent repair. Explicit work mode determines planning mode. Narrative changes and raw activity counts cannot replenish allowances. An exhausted repair shows a readable notice. An explicit retry checks current controls and preserves the assigned agent. Steps to reproduce Run `pnpm test:lifecycle-baseline`. The paired probes keep structured state constant while varying completion, planning, blocker, and progress prose. Run the explicit `lifecycle-baseline` and `continuation-accounting` Product E2E suites for real-provider coverage. In Storybook, open Design previews / Recovery notice to inspect the production component's normal, pending, acknowledged, unavailable, failure, and mobile states. ## What Changed - Hide the image attachment button, icon, and drop/paste hint in answer composers. Image paste and drop support remains available. - Merge current master and retain both browser regression sets. Use a production-stamped service worker in the offline recovery browser fixture. - Share one state-based legacy continuation decision across immediate, delayed, and recovered dispatch. Bind bounded repairs to their source run and episode. - Remove title and description wording from work-mode authority. Agents can still write requested plans in execution mode. - Persist separate failure-retry and productive-continuation counters. Disposition repair and resource waits cannot consume or reset those allowances. - Validate delayed repair identity, then recheck current gates before provider dispatch. Fence native startup cancellation. - Show Agent needs attention, a plain-language explanation, Retry agent, and expandable details in both task interfaces. Report request progress, acknowledgement, and errors inline. - Store typed recovery notice metadata. Recognize older active notices only through exact stored action and run IDs. Notice text never grants retry authority. - Use the existing recovery-action endpoint for retry. Recheck current action, status, owner, agent availability, dependencies, active runs, pending questions and confirmations, approvals, pause controls, and budget. Duplicate requests do not wake twice. - Add component, page, route, database, contract, and Storybook coverage. Keep the scenario inventory and executable evals here. Historical reports and snapshots live in the [commit-pinned paperclip-evals archive](https://github.com/paperclipai/paperclip-evals/blob/ce3e5afcd4a1184650f586a2b5b8be5874c66c8b/experiments/2026-09-lifecycle-authority/README.md). - Preserve unsaved project fields while the same project URL changes to its canonical alias. Do not reuse data across projects or companies. This separate fix addresses the repeated repository-editor browser failure without changing the browser test. - Keep the development service worker from intercepting Vite module reloads. Update the connection-intent browser fixture to record progress and completion through the agent API. ## Verification Merge preparation on September 25, commit `c1e8e4b7ddd9fbc4913ed55ce21b8e12906c2f97`: - Merged master `bd2030932` and resolved the browser test-list conflict by keeping both sets of regressions. - Deterministic lifecycle baseline: 1,090/1,090 assertions passed; no failures, skips, or missing selected evidence. Unit 423, runner 184, database integration 397, grading 86. - Browser support: 17/17 passed. The offline recovery test first failed with an unstamped development worker, then passed with the production stamp. Its assertions are unchanged. - Focused interaction UI and offline fallback tests: 19/19 passed. Verified the custom-answer composer in Storybook: no attachment controls or hint; entering an answer enables Next. - Recursive typecheck, production build, token gates, and diff checks passed. The worktree is clean. No new real-provider campaign was run. - Current CI and review: [Current PR CI passed](https://github.com/paperclipai/paperclip/actions/runs/36166011243): 55 successful checks and two optional Storybook skips. Greptile scored this exact commit 5/5. Hiding the question attachment controls is an intentional UI change; paste/drop remains available. Earlier recovery UI verification, commit `21be0fec0e90e86b6d662b8ee4831847cd041cdb`: - Recursive typecheck, production build, token gates, and diff checks passed. - Focused UI coverage: 338 tests passed across six suites (336 before the interaction guard, with the two affected suites rerun at 149 passed after it). Covers both task interfaces, the real page mutation, pending/error acknowledgement, stale state, and unavailable controls. - Recovery database integration: 352 tests passed before the interaction guard. The complete recovery-action and mutation-route suites passed 181 tests after it. The two new pending question/confirmation regressions failed before the fix and passed afterward, including resolved-interaction controls. Shared validator suite: 31 passed. E2E catalog suites: 34 passed. - Browser inspection passed for light/dark themes, mobile layout, expandable details, pending retry, acknowledgement, failure, and disabled retry. Storybook renders the production component; its request is simulated. - The broad local run hit two chat callback-order wait failures and was stopped after all CI unit/database/runner shards passed. Both local failures passed when rerun without the competing full-suite process. - CI exposed a repeated project-repository draft-loss race during canonical redirects. A new unit regression failed before the fix; all nine project-page tests now pass, including controls for other projects and companies. Both unchanged repository browser tests passed against a fresh local server. UI typecheck, production UI build, and token gates passed after this fix. - [Earlier PR CI passed](https://github.com/paperclipai/paperclip/actions/runs/36072486798) on `21be0fec0e90e86b6d662b8ee4831847cd041cdb`: 55 successful checks, two optional Storybook skips, and no failed or pending checks. The repository browser shard passed with the production fix. Greptile is 5/5 on this exact commit with no unresolved review threads. The PR is mergeable. Historical, source-qualified lifecycle evidence: - Lifecycle baseline: 1,074 assertions. Native session coverage: 447 tests. Product E2E support: 515 tests. Browser support: 11 tests. Full earlier verification is retained in the archive. - [Real-provider campaign: 8/8 passed, zero retries](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35881382080-1/index.html), source `e88d210417280140b44a36449027290adcb1aeaa`. Evidence and cleanup checks passed. This includes deliberately exhausted repair cases that correctly remain blocked; it does not mean every task finished Done. This campaign predates the recovery UI change. - Archive migration verified all 16 original JSON files byte-for-byte and all 24 checksum entries. App tests do not need private archive access. [Archive PR #27](https://github.com/paperclipai/paperclip-evals/pull/27) is merged. ## Risks - Agents that omit durable disposition receive at most two repair attempts by default. Prose-only completion exposes missing state rather than silently changing scheduling. - A retry is an explicit board action. The server rechecks current controls. A successful response confirms the task returned to To do; it does not claim that the provider has already started. - Existing notice metadata remains valid. Only older active notices with matching structured evidence receive the new UI. Historical notices without that evidence keep their existing rendering. No schema migration is required. - Old run records require conservative retry accounting. Tests cover old counters, alternating retry lanes, restarts, and exhausted repairs. - Historical snapshots require private `paperclip-evals` access. The app index retains public campaign links. Live campaigns qualify specific sources and scenarios; no new real-provider campaign has run for the recovery UI commit. > This fixes existing lifecycle and recovery behavior and does not duplicate planned core work. ## Model Used OpenAI GPT-6 through Codex assisted implementation, reasoning, code execution, and review. The exact serving model ID and context window are not exposed in this task. Historical real-provider evaluations used Codex model `gpt-5.6-sol`, separately from the implementation assistant. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-25 15:28:11 -07:00
Devin FoleyandPaperclip	bd6caf51bb	fix: preserve restore failure results and stop unsafe retries (#14035 ) Preserve agent output and earlier execution errors when workspace restore fails. Report the restore phase and confirmed saved-plan links. Require verified repair before retrying unsafe archives, while preserving approval states and the retry budget. Verified with full CI, 506 focused regression tests, and Greptile 5/5 with all review threads resolved. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-09-25 11:34:31 -07:00
Devin FoleyandPaperclip	667c79ded2	fix: prevent retry-exhaustion events from exhausting attention-feed memory (#13451 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Its attention feed shows failed runs whose retry budget is exhausted. > - Startup retention builds this feed before startup completes. > - The feed joins every exhaustion event to the full run context, then removes duplicate runs in JavaScript. > - Recovery can revisit an exhausted run and append the same event again. This multiplies the data loaded into memory. > - This pull request selects one small row per run in PostgreSQL and makes exhaustion writes idempotent. > - Existing duplicate events can stay in the database without multiplying run contexts in server memory. ## Linked Issues or Issue Description Refs #13367. This fixes the repeated-event allocation path in the retention feed. Other full-feed sources and sweep cadence remain separate concerns. What happened? The attention query loaded one full run context for every matching exhaustion event. Deduplication ran only after the driver had loaded those rows. A run with thousands of exhaustion events therefore produced thousands of context copies. The startup retention sweep can exhaust the server heap while reading this result. Expected behavior The query should return one row per exhausted run and only the context fields that the feed needs. Repeated checks of the same exhausted retry budget should reuse the original event. Steps to reproduce 1. Create a failed run with a 32 KB context and 2,500 matching exhaustion events. 2. Build the attention feed, including dismissed items, as startup retention does. 3. Inspect the database result before JavaScript feed processing. The old join returns 2,500 copies of the run context. 4. Call bounded retry scheduling repeatedly for a run at its retry limit. The old writer appends another exhaustion event on every call. Paperclip version or commit The attention query was introduced by #9380 and is present in stable `v2026.831.1`. The retention caller is also present in that stable release. The later recovery callback added by #13075 provides a repeated path into the exhausted-budget writer. This change is based on `c0fda8fac` after rebasing onto current master. Related PR: [#12162](https://github.com/paperclipai/paperclip/pull/12162) changes retention cadence and newer-run suppression queries. This change addresses the exhaustion-event join and duplicate event writes. ## What Changed - Select the newest company-scoped exhaustion event per run with a PostgreSQL `DISTINCT ON` subquery before joining run data. - Project only `issueId` and `taskId` from run context. Preserve JSON types, fallback behavior, run ordering, company filters, and run/agent status filters. - Reuse an exhaustion event for the same run, reason, attempt, and retry limit under the existing run-row lock. Recognize historical events without a migration. - Skip sequence allocation and live publication when a receipt already exists. - Document the run-log behavior and add PostgreSQL regression tests. ## Verification - Focused attention, retry-scheduling, and event-sequencing suites: 72 tests pass on the rebased head `39ce77158`. - The large-history fixture verifies two database result rows under 4 KB, newest-message selection, task-ID fallback, and company/status filtering. This assertion runs before feed deduplication. - Concurrent event tests verify one receipt across two database clients, reuse of historical receipts, distinct reason/attempt/budget keys, and stable event sequences. - Repeated scheduling through a new service instance produces no extra run-log or live events. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: passes. - `pnpm -r typecheck` and `pnpm build`: pass on the rebased head, including native runner checks. - Greptile: 5/5 on `39ce77158`, with no review threads. - [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/34929002304): all 25 workflow jobs pass, including all server/workspace test shards, serialized server suites, browser tests, typecheck, build, native-runner verification, and the release dry run. Security checks also pass. The branch has no merge conflicts. - `pnpm test:run`: stopped with unrelated failures. Two chat integration cases passed when rerun separately (2 passed, 993 unselected). Three skill-cache cases failed on both this branch and the unmodified parent commit, with `EACCES` during directory rename. The full suite is not reported as green. ## Risks - No schema migration or data cleanup is required. The query still scans matching event history in PostgreSQL; its result size now scales with exhausted runs. - This does not bound every source in the attention feed or change retention scheduling. - An exhaustion receipt is emitted once per retry decision. Consumers that observed repeated copies will now receive one event. - Native source-event replay handling stays on its existing path. - A deployed application boot has not been verified. ## Model Used OpenAI GPT-6, used through Codex with reasoning, repository inspection, code editing, and test execution. The runtime does not expose a more specific model snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the focused tests locally and they pass (full-suite limitations are documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-14 21:58:34 -07:00
DottaandPaperclip	0e9b24c821	fix: resume subscription comment wakes and preserve retry status (#13433 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Concurrent task runs can share a managed AI subscription with one credential lease. > - #13438 added durable retries for busy subscriptions and task-lock checks. > - A comment wake can run for an agent who is not the task assignee. Such a run never owns the task lock, so the new checks suppress its retry. > - This pull request preserves those comment wakes while keeping the lock checks for assignee runs. > - It also shows the subscription wait in task status and preserves the failure count through the database projection. ## Linked Issues or Issue Description Refs #13438. What happened? When a subscription is busy, a non-assignee comment wake is cancelled without a successor. Repeated subscription waits also display an inflated attempt count because the execution query omits the preserved failure count. Expected behavior An eligible comment wake waits and resumes without claiming the assignee's task lock. The task shows “Waiting for AI subscription”. Waiting does not consume provider-failure retries. Assignee retries still stop when the task lock is cleared or transferred. Steps to reproduce 1. Configure an agent with a managed subscription and concurrent runs. 2. Hold its credential lease in one run. 3. Mention the agent on a task assigned to another actor. 4. Release the lease and inspect whether the comment wake has a scheduled retry. The test uses a real embedded Postgres database and a real credential lease. Provider execution uses a fixture. The reassignment test applies a database mutation after the real checkout. No live provider account is needed. Paperclip version or commit Based on `f912ecaac` from #13438. Before the reconciliation fix, the rebased regression at `2063cfd13` failed because the comment wake had no scheduled retry. The original configuration failure was reproduced on `5282cabde` before #13438 merged. Deployment mode Built from source with embedded Postgres for local verification. ## What Changed - Record non-assignee comment-wake authority at admission while holding the task and run locks. A later reassignment cannot grant this exception. Preserve it through repeated subscription waits. - Keep master’s lock checks for assignee retries, including the check inside the scheduling transaction. - Show the subscription wait and preserve its failure count in the execution projection query. - Limit pre-provider wait receipts to fresh executions. A persisted native execution input must retain its recovery path. - Add lease, retry, projection, cancellation, reassignment, pause, revocation, service-recreation, and ownership regression coverage. - Use master’s 60–120 second retry interval and document the resulting behavior. ## Verification - Red: after rebase, the comment-wake test failed with no scheduled retry; the other 50 contention and retry-scheduling tests passed. - Red/green: reassigning the task immediately after real checkout reproduced an unwanted successor for both assignment and comment wakes. Both tests pass after recording authority at admission. - Green: all 285 targeted tests pass across 12 suites, including #13438’s four cancellation-race phases, its hiring/connection suite, run-dispatch integration tests, and both reassignment regressions. - `pnpm -r typecheck` passes after rebase. - `pnpm build` passes on the final revision. - The full general and serialized test suites pass in [CI run 34899514681](https://github.com/paperclipai/paperclip/actions/runs/34899514681) on `99f1e4c1d`. Full-suite verification ran in CI; local verification used the 285 targeted tests. - All 32 active PR checks pass, including build, typecheck, native runner verification, canary dry run, and all browser shards. Two Storybook checks are skipped by path filters. - Greptile reviewed `99f1e4c1d` at 5/5 with no outstanding findings. All review threads are resolved. - `git diff --check` and `node scripts/check-module-boundaries.mjs` pass. ## Risks - The non-assignee exception requires authority recorded under admission locks or its server-created subscription retry. Tests cover caller-supplied flags, assignment and comment reassignment races, repeated comment waits, and assignee lock protections. - A wait uses master’s 60–120 second interval. A long-held lease can produce multiple wait records. - Existing native sessions can have provider effects. They must not receive a fresh-execution receipt that permits replay. - No schema migration or credential permission change is required. ## Model Used OpenAI Codex, model `gpt-6-astra`, with repository search, code editing, shell execution, and test tools. The session does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-14 16:58:37 -05:00
DottaandPaperclip	f912ecaacf	fix: carry AI connections through hiring and unblock task execution (#13438 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents hire other agents and assign tasks to them. > - Managed AI connections must follow those hires across legacy and native runners. > - Missing accounts should pause task execution and let the user connect from the task. > - Subscription contention must wait without asking for new credentials. > - This pull request fixes these paths and the native tool and Daytona staging failures found during live tests. > - The result is a working hire, subtask, and connection setup flow on local and remote runners. ## Linked Issues or Issue Description What happened? A managed Claude or Codex agent could hire a teammate without a usable AI binding. Cross-provider hiring could fail before the user had a chance to connect the new provider. First-time task setup did not show the existing AI credential form inline. A busy subscription could request a new connection. Native API replies could stop the parent after a hire had already committed. Fresh Daytona sandboxes could fail to extract read-only skill directories created on macOS. Expected behavior Compatible hires inherit the managed connection choice. A hire for another provider uses the responsible user's default. If that account is missing, the hire succeeds and the task asks for a connection. Completing setup in the task resumes work automatically. Explicit child auth settings and existing unmanaged login paths keep precedence. Shared-account access checks remain in force. Steps to reproduce 1. Connect a Claude or Codex parent with a managed AI account. 2. Ask it to hire one agent of each provider and create a self-assigned subtask. 3. Assign work to both hires without connecting the second provider first. 4. Connect the missing provider from its task card. 5. Check that all tasks finish and same-provider work uses the original account. 6. Repeat with native runners and fresh Daytona sandboxes. The opt-in browser suite in `tests/hiring-ai-connections/README.md` performs these steps. Paperclip version or commit The live failures were reproduced from `f2c5e54dc`. The branch is rebased onto `5282cabde`. Deployment mode Isolated local development instance. Legacy CLI and native runners. Local execution and ephemeral Daytona sandboxes. Related work: Refs #13247 for managed AI connections. Refs #13268 for legacy credential-reference inheritance, which this branch preserves. Refs #13432 for a concurrent managed-inheritance fix. This PR also covers cross-provider task setup, subscription waits, native API replies, and Daytona extraction. It permits missing responsible-user defaults at hire time; restricted shared selections still fail. ## What Changed - Apply managed connection defaults to both agent creation routes. Preserve explicit auth choices and legacy credential-reference inheritance. - Allow hires before their responsible user connects the provider. Keep approval gates, company boundaries, and shared-account access checks. - Reuse the production AI credential form inside the pending task card. Resume the task after setup. - Retry subscription lease contention without consuming the provider-failure allowance or creating a connection request. - Require task execution-lock ownership when scheduling, promoting, and dispatching subscription retries. Recheck ownership under the issue row lock. - Rename the HTTP operation identity at the native tool boundary so it cannot override the runner's operation identity. - Delay directory permission restoration during Daytona extraction. Preserve the final read-only modes. - Add database-backed regressions, real browser acceptance tests, and Storybook states. Document setup and run-log behavior. ## Verification - Six real browser scenarios passed: both parent providers on legacy local and legacy Daytona; native Codex locally; native Claude on Daytona. Each scenario hires both providers, completes a self-subtask and assigned work, and connects the missing provider inline with automatic continuation. - Successful runs verify the account, responsible user, runner mode, and Daytona lease. All 18 test sandboxes were deleted. - Live authentication used API keys. Subscription inheritance, lease contention, and retry have integration coverage. Fresh subscription OAuth sign-in was not automated. - Red/green tests reproduced missing bindings, missing inline forms, subscription contention, native API reply failure, and GNU tar permission failure. - Seven Storybook browser checks passed. They cover both providers, method selection, narrow layout, completion, cancellation, and invalid credentials. - Full local suite coverage completed before rebase. Initial timing and fixture startup failures passed unchanged on isolated reruns. The first full command did not exit cleanly; the remaining workspace and serialized groups were completed separately. - After rebase, 107 hiring/auth/retry tests and 59 native API, task-card, and Daytona tests passed. The full workspace typecheck, production build, and token gates passed again. Storybook build passed before rebase. - Review fixes: 169 hiring/retry/dispatch tests, 37 adjacent tests, and four explicit cancellation-race cases passed. Eight cross-provider cases cover stale auth keys on both creation routes and both runner types. Server typecheck and build passed. - Final CI on `ee4890837a8a4913e07453392b9a75969580dae1`: 32 checks passed. Two optional Storybook jobs were skipped. The full server, workspace, browser, native runner, build, typecheck, and release checks passed. - Three unchanged tests initially failed on a busy port, a chat row-lock race, and preview-server readiness. Each affected job passed after one CI rerun. Isolated local checks also passed: 41 credential tests, the chat-concurrency case, and 25 preview-runtime tests. - Greptile reviewed the final commit at 5/5. Both review threads are resolved. GitHub reports no merge conflicts. ## Risks - A missing personal account now defers authentication to the first task. Explicit incompatible bindings and restricted shared accounts still fail at hire time. - An inherited personal default uses the responsible user's existing authorization to install access for the new agent. It never copies credentials or another user's identity. - Subscription contention retries after a delay and rechecks task eligibility. It does not consume the provider-failure budget. - Native hiring uses the existing managed API-tools opt-in. Remote native runners require a matching Linux binary and provider pack, as documented in the acceptance README. - No schema changes. Live tests make paid provider calls and remain opt-in. ## Model Used OpenAI GPT-6 (`gpt-6-astra`) in Codex, with reasoning, repository inspection, code execution, browser automation, and API tools. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-14 16:15:48 -05:00
DottaandPaperclip	b1efd65edc	fix: continue interrupted task conversations with bounded retries (#13237 ) ## Thinking Path > - Paperclip manages AI agents and their tasks. > - A task can outlive a provider process or a server restart. > - Legacy recovery treated unknown tool outcomes as a permanent execution hold. > - That hold could also reject a later user message. > - A conversation turn can use prior history without replaying prior tool calls. > - This pull request lets supported conversation adapters continue within the existing retry budget. > - Users can send a new message after automatic attempts stop. ## Linked Issues or Issue Description What happened? A server restart could interrupt a local ACP run and leave its task behind a permanent recovery hold. A later user message could be cancelled before the provider answered. The immediate recovery path could also create a successor outside the durable failure counter. Expected behavior Continue with a bounded new conversation turn. Preserve a compatible provider session or use full task context when it is unavailable. Do not replay recorded tools. When automatic attempts stop, allow a new user request through the normal execution gates. Steps to reproduce 1. Start a task with a local conversation adapter. 2. Restart the server while the provider is working. 3. Let the previous run become interrupted. 4. Send a follow-up message and observe the recovery hold on the old behavior. Related work: Refs #13075 for durable task recovery. Refs #12946 for retry-limit and checkout-lock handling. This change routes conversation recovery through the existing bounded scheduler. ## What Changed - Mark supported local conversation failures for continuation. Keep native-runner and non-conversation recovery rules. - Carry an interruption notice into the next turn. Retain stopped ACP session history even when a write outcome is unknown. - Clear unavailable ACP sessions so the next bounded attempt can use full task context. - Route immediate failure recovery through the same durable scheduler as process-loss recovery. Release only the predecessor checkout when its retry takes ownership. - Retire obsolete conversation holds using immutable run evidence, in bounded batches with an activity record. Preserve outcome evidence and do not wake historical tasks. - Block actual admission and Resume while a predecessor process or environment lease is still active. Keep the original interruption notice after a rejected wake. Preserve the upstream blocked-wake waiting contract: bounded retry planning can happen during cleanup, while deferred messages and execution remain gated. - Add subprocess and database regression tests. Update the execution contract. - Add the current thread-status field to the native recovery provider fixture so its damaged-journal test reaches the intended boundary. Tolerate an already-exited fixture process during test cleanup while still asserting both processes terminate. ## Verification - Workspace typecheck passed: `pnpm -r typecheck`. - Build passed: `pnpm build`. - Module boundaries passed: `pnpm check:module-boundaries`. - Focused tests passed: 293 recovery/session/dispatch tests, 66 retry and response-gate tests, and 37 native-session tests. Some suites overlap. - Tests cover interrupted writes, missing sessions, concurrent retries, restart persistence, pending questions and approvals, execution gates, and historical holds. - Built the Rust test executables with `pnpm --filter @paperclipai/paperclip-runner build:rust` for native-runner verification. - Full Vitest coverage verified locally using the repository’s general and serialized shards, with focused reruns for failures and files not reached after a shard stopped. The ownership-gate regression is fixed and the complete affected server shard passes (1,390 tests). Local parallel runs also hit temporary-directory, resource, and timing failures; those suites pass with canonical temporary paths and sequential reruns. No test timeouts were increased. - Final merged-branch regression run: 577 tests pass across process recovery, retry scheduling, liveness, durable chat, wake-queue application/adapter, dispatch, continuation, native sessions, and task chat. Earlier focused verification also passed 19 native control tests. Token gates and whitespace validation pass. - Browser verification passed all three ACP Stop/continue/pause scenarios, including a rerun after merging the upstream waiting behavior: `PAPERCLIP_E2E_PORT=3397 pnpm test:e2e tests/e2e/acp-stop-continuation.spec.ts`. The interrupted-write case verifies that follow-up completes without a repeated write. - Final-head [CI run 34625037394](https://github.com/paperclipai/paperclip/actions/runs/34625037394) passed on `06ac4bd9d150f8b209a96e5fd609c696958794a0`: all 31 reported checks are green, including server/workspace suites, all browser shards, native runner verification, build, typecheck, release dry run, and aggregate gates. The two conditional Storybook checks were skipped. Greptile reviewed this exact commit at 5/5; all review threads are resolved. ## Risks - A new model turn can choose to repeat an action. Paperclip does not replay recorded tool calls and does not certify unknown action outcomes. - Conversation adapters now stop after their retry budget instead of requiring action reconciliation. Explicit Stop, pause, dependency, approval, budget, and ownership gates remain in force. - No schema migration or dependency changes. Historical holds are folded without changing task status or waking work. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and test execution. The session does not expose a more specific model build ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-11 12:16:04 -05:00
DottaandPaperclip	35fdc0c66b	fix: make task recovery durable and preserve current requests (#13075 ) Make task recovery durable and preserve the latest user request across native and legacy continuations. Keep routine recovery quiet and prevent replay when action outcomes are uncertain. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-09-09 09:14:25 -05:00
Nicky LeachandPaperclip	f65991a5f1	refactor(server): move scheduled-retry and queued-run dispatch into a run-dispatch module (#12920 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server heartbeat service dispatches scheduled retries and queued runs. > - The service kept policy decisions and database writes in one large file. > - This layout made policy branches harder to test and transaction boundaries harder to inspect. > - This pull request moves the policy rules and database transactions into a run-dispatch module. > - The benefit is a smaller service, pure policy tests, and clear transaction ownership. ## Linked Issues or Issue Description What existing behavior does this improve? The server heartbeat service promotes scheduled retries and cancels stale queued runs. Subsystem affected server/ — REST API and orchestration services. Current behavior The heartbeat service contains the policy rules and the database writes for these dispatch paths. Proposed behavior A run-dispatch module owns pure policy functions and semantic database transactions. The public service contracts stay unchanged. Reason and benefit The new layout separates branch rules from database effects. It makes each policy branch easier to test and keeps each operation’s row writes in one transaction. Breaking changes None. The public service contracts stay unchanged. ## What Changed - Move scheduled-retry promotion and queued-run staleness rules into pure functions. - Add table-driven unit tests for each policy branch. - Move promotion and cancellation writes into semantic transactions. - Keep row locking, company isolation, and post-commit effects unchanged. ## Verification - `node scripts/check-module-boundaries.mjs` passes. - `tsc --noEmit` from `server/` reports no errors. - The focused server test command passes 228 tests in six files. - Full pull request CI passes. - Greptile reports 5/5, and all review threads are resolved. ## Risks The main risk concerns changed transaction boundaries in scheduled-retry promotion and queued-run cancellation. The focused tests retain coverage for locking, transactionality, company isolation, and transport contracts. The public service contracts do not change. ## Model Used Codex, GPT-5, with code execution and tool use. The context window is not provided by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I have addressed all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-08 11:00:55 -07:00
Nicky LeachandPaperclip	184b014c25	feat(telemetry): add the agent.task_run event and emit it at every terminal run transition (#12809 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip records agent run outcomes through telemetry and run lifecycle services > - Terminal run transitions need one consistent event for outcome analysis > - The current paths do not report every terminal transition through one event > - This pull request adds the agent.task_run event and emits it at each terminal transition > - The benefit is complete run outcome data without exposing raw task identifiers ## Linked Issues or Issue Description What existing behavior does this improve? Paperclip telemetry reports agent activity, but it does not report every terminal task run through one event. Subsystem affected Cross-cutting (multiple of the above): packages/shared telemetry and server run lifecycle services. Current behavior Several run paths write a terminal status without a matching agent.task_run telemetry event. Proposed behavior Each terminal run transition emits one agent.task_run event. The event records the terminal state and uses the existing pseudonym helper for the optional task identifier. Reason and benefit Complete terminal-run data helps operators measure agent outcomes. The pseudonym helper prevents the raw task identifier from leaving the installation. Breaking changes None. The change adds an event and keeps existing event behavior compatible. ## What Changed - Add the agent.task_run telemetry contract and client helper. - Reuse the existing pseudonym helper for the task identifier. The helper hashes the identifier with a per-installation salt and returns 16 hexadecimal characters. The raw identifier never leaves the installation. Existing identifiers do not move. - Emit one event from each legacy, native, recovery, and issue terminal transition. - Keep emissions outside database transactions and make delivery best-effort. - Add regression tests for event shape, hashing, terminal transitions, and emission failures. - Document the event and its privacy rule in the telemetry data contract. ## Verification - `npx tsc --noEmit` in `server/` passes at the submitted commit. - The pull-request CI suite must pass. CI is the authority because local Vitest has a known dependency artifact. - The added regression tests cover event output shape, per-installation hash divergence, raw identifier handoff, omitted identifiers, and non-throwing emits. ## Risks - A missed terminal path could reduce event coverage. - Telemetry delivery remains best-effort and cannot change run finalization. - The pseudonym helper uses installation-specific state, so identifiers differ between installations. ## Model Used OpenAI Codex, GPT-5, tool use and code execution. Context window details were not provided. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I have addressed all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-04 08:21:32 -07:00
Nicky LeachandPaperclip	d51f42ed64	test(server): drain heartbeat runs to quiescence in sibling suite teardown (#10464 ) ## Thinking Path > - Paperclip uses heartbeats to run work. > - Test suites share heartbeat run state during teardown. > - Late heartbeat work can race shared table deletes. > - That race can deadlock or fail foreign key checks. > - The primary suite already uses a drain helper to wait for quiescence. > - This pull request reuses that helper in the sibling suites that share the race. > - The benefit is stable teardown and fewer flake failures. ## Linked Issues or Issue Description No public GitHub issue exists for this change. Refs: #10450 This pull request reuses the quiescence drain from the primary suite. ## What Changed - Added `server/src/__tests__/helpers/drain-heartbeat-runs.ts`. - Reused the shared helper in `low-trust-red-team-routes.test.ts`. - Applied the drain to the eight sibling suites that share the race. - Kept the existing test intent unchanged. ## Verification - `git log --oneline origin/master..origin/test/heartbeat-teardown-quiescence-drain-sweep` - `git diff --stat origin/master...origin/test/heartbeat-teardown-quiescence-drain-sweep` - Existing local test evidence in the handoff shows the primary suite and the guarded suites pass. - The handoff also records a stress loop with no `40P01` or `23503` errors. ## Risks - Low risk. The change touches test teardown only. - The helper waits for active runs to drain. A new real background execution path may need the same guard. ## Model Used OpenAI GPT-5, tool-use and code execution enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues with `Fixes: #` / `Closes #` / `Refs #` or described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes, or no docs update was required for this test-only change - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-07-29 14:23:12 -07:00
DottaandPaperclip	665408c6d0	fix(codex): classify mid-turn harness crashes structurally as retriable infra (#10210 ) ## Thinking Path > - Paperclip is the open source control plane people use to manage AI agents and their work > - Heartbeat execution relies on adapters distinguishing agent failures from failures in the harness running beneath the agent > - Codex MCP transport crashes can kill the CLI after the JSONL protocol has started but before it emits a protocol-terminal event > - Those interrupted streams were left unclassified, so the control plane terminalized the heartbeat as `heartbeat_failed` / `agent_failure` with no continuation > - Agent-level failure is already expressible through the JSONL protocol via an `error` event, `turn.failed`, or `turn.completed`, so an interrupted nonzero exit can be classified structurally without inspecting unstable error strings > - This pull request reports that shape as `codex_harness_crash` in the `transient_upstream` family and routes it through Paperclip's existing bounded retry and recovery-continuation paths > - The benefit is that transient Codex harness failures recover safely without misclassifying quoted agent output or depending on transport-specific wording ## Linked Issues or Issue Description - What happened: Codex MCP transport failures, including rmcp worker death, could terminate the CLI mid-turn after protocol output began but before any terminal JSONL event. The run then became an unclassified terminal heartbeat failure with `continuationCount: 0`; this occurred in 3 of 44 L3 Codex-lane trials during the associated benchmark investigation. - Expected behavior: a nonzero Codex exit after the protocol starts but before an `error`, `turn.failed`, or `turn.completed` event should be treated as a harness/infrastructure crash and enter the existing bounded retry policy. - Why structural classification: transport error strings vary, and stdout may quote agent output that merely discusses network failures. The protocol boundary identifies whether the agent itself produced a terminal result without regex matching. - Recovery behavior: `codex_harness_crash` maps to `errorFamily: transient_upstream`, using the existing `same_session` → `safer_invocation` → `fresh_session` ladder plus the recovery-continuation transient-infrastructure path. - Supersedes the regex-based approach in #10150, which is closed. ## What Changed - Added protocol-state tracking that identifies a nonzero exit after protocol start and before any protocol-terminal event as `codex_harness_crash`. - Propagated the structural classification as `transient_upstream` through the Codex adapter. - Added parse unit coverage, including a faithful crash-shaped stream, without matching stderr transport strings. - Added adapter execution coverage using a fake Codex process that emits a protocol prefix and then dies with the observed rmcp stderr line. - Added heartbeat bounded-retry coverage, including the `errorCode`-only fallback, and recovery-continuation classification coverage. ## Verification - `parse.test.ts` — 16 passed. - `codex-local-execute.test.ts` — 16 passed. - `heartbeat-retry-scheduling.test.ts` — 30 passed. - `service.pause-durability.test.ts` — 6 passed. - Server and Codex adapter TypeScript checks passed. - The branch commit is unchanged from the tested and pushed `88f5464d40` handoff. ## Risks - Low risk: the classification requires a nonzero exit after protocol start and before any protocol-terminal event, so normal agent-declared failures and completed turns keep their existing behavior. - The change intentionally broadens recovery for structurally interrupted Codex runs; bounded retry limits still prevent indefinite continuation loops. - No schema, migration, public API, UI, lockfile, or workflow changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex coding agent. The exact runtime model ID and context-window size were not exposed by the execution environment; capabilities used for the implementation included repository analysis, reasoning, code editing, and terminal-based test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details — the pre-existing, already-pushed branch name was explicitly prescribed for this replacement PR - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-07-24 16:40:47 -05:00
Dotta Paperclip Claude Opus 4.8	0b496c9c03	feat(secrets): add run-bound agent secret access (#9921 ) ## Thinking Path > - Paperclip is the open source control plane people use to manage AI-agent companies. > - Agents already receive selected company secrets through `env.` bindings at run launch, but environment injection is ambient, long-lived, and not suitable for every secret consumer. > - The existing binding and secret-access-event models already provide company-scoped authorization and per-resolution audit seams. > - Agents need an explicit way to discover only the secrets granted to them and fetch a value on demand without exposing the wider company catalog. > - That capability must remain run-bound, preserve low-trust token carve-outs, and make every value read visible in both security and operator audit trails. > - This pull request adds an `access.` delivery namespace, two run-bound agent routes, dual audit logging, documentation, and an operator grants editor. > - The benefit is least-privilege, revocable, auditable secret access while preserving existing env injection behavior. ## Linked Issues or Issue Description No pre-existing public issue. Related work: - Refs #9797 — existing in-sheet agent access UI that this PR extends to distinguish env and API delivery. - Refs #9918 — complementary searchable-agent picker improvement for the same secrets sheet. - Refs #9530 — related company-wide metadata catalog proposal; this PR intentionally exposes only the authenticated run's granted aliases and values. Problem / motivation: Agents can currently consume secrets only through process environment injection. This keeps values resident for the run, does not support on-demand consumers, and cannot provide a discrete operator-visible activity event for each agent-initiated read. Proposed solution: Treat `company_secret_bindings` as the source of truth for agent secret grants. Keep `env.KEY` as env delivery and add `access.ALIAS` for API-only delivery; an env binding also implies read access because the value is already present in the agent process. Add run-bound list/fetch endpoints that derive scope from the authenticated heartbeat run and never accept caller-selected overlays. Alternatives considered: A company-wide agent-readable catalog was rejected for this value path because it increases reconnaissance and does not prove a per-secret grant. Reusing the ephemeral environment-probe resolver was rejected because it lacks binding enforcement. Approval-gated reads and user-scoped secrets remain deferred beyond v1. Roadmap alignment: This extends the completed Secrets Manager with per-agent access roadmap capability from launch-time env injection to explicit run-bound API delivery without duplicating a separate planned initiative. ## What Changed - Added `access.` agent binding validation and a dedicated run-bound resolver that combines `secrets:read` authorization with binding-context enforcement. - Added `GET /api/agents/me/secrets` for minimal granted metadata and `POST /api/agents/me/secrets/:key/value` for on-demand value fetches with `Cache-Control: no-store`. - Preserved the existing denials for low-trust review agents, task-bridge credentials, and skill-test tokens; standard long-lived agent API keys cannot call the run-bound routes. - Added dual audit behavior: value attempts write `secret_access_events` and `activity_log` (`secret.value.read`), while metadata listing writes the lighter `secret.access.listed` activity event. - Kept env compatibility: `env.` remains injected at launch and also implies API read for the same bound agent; `access.` never becomes an environment variable. - Added the agent-settings Secret access* editor plus delivery-mode/alias surfacing on the Secrets page, with focused UI tests and tokenized layout styles. - Updated OpenAPI, shared types, agent-facing skill documentation, and API reference documentation. ### UI Screenshots P3 produced and reviewed three screenshots using mock data; images are intentionally not committed to the repository: - `secret-access-editor.png` — agent settings grant editor. - `secret-access-light.png` — Secrets-page delivery surfacing in light mode. - `secret-access-dark.png` — Secrets-page delivery surfacing in dark mode. The source attachments are retained with the implementation task and linked in the internal handoff; the public page publisher was unavailable in the PR-prep runtime. ## Verification - `pnpm exec vitest run server/src/__tests__/agent-secrets-routes.test.ts server/src/__tests__/secrets-service.test.ts server/src/__tests__/secrets-routes.test.ts ui/src/lib/secret-delivery.test.ts ui/src/components/AgentSecretAccessEditor.test.tsx` — 5 files, 122 tests passed. - Security follow-up: `pnpm exec vitest run server/src/__tests__/agent-secrets-routes.test.ts server/src/__tests__/secrets-service.test.ts` — 2 files, 73 tests passed after active-run and version-consistency fixes. - Final-head CI: all feature, typecheck, build, e2e, security, and review gates pass; `General tests (server (1/3))` remains red after one rerun because unrelated `heartbeat-retry-scheduling.test.ts` cleanup deletes `heartbeat_runs` before referenced `activity_log` rows. - `pnpm --filter @paperclipai/server typecheck` — passed. - `pnpm --filter @paperclipai/shared typecheck` — passed. - `pnpm --filter @paperclipai/ui typecheck` — passed. - `pnpm check:token-gates` — feature-local arbitrary-value violations fixed; command still reports five unchanged `#9627` literals outside this PR. - End-to-end QA passed all eight acceptance criteria: grant/list, fetch, dual audit, env-implies-read, denial matrix, revocation, UI rendering, and env-injection regression. Evidence: https://github.com/paperclipai/paperclip/pull/9921#issuecomment-5027455492 - Security review returned PASS-with-required-changes; the implementation uses the required dedicated binding-enforcing resolver, run-bound JWT restriction, run-derived overlays, minimal metadata, and a resolver redaction-registration hook. Evidence: https://github.com/paperclipai/paperclip/pull/9921#issuecomment-5027455382 ## Risks - A compromised agent can exfiltrate any secret explicitly granted to it; explicit company-scoped/run-scoped grants, revocation, and audit reduce but cannot remove that inherent capability risk. - The resolver invokes a redaction-registration hook before returning values, but the current route has no persistent cross-request per-run redaction registry. Paperclip-owned later comments/events therefore cannot yet guarantee automatic scrubbing of a deliberately copied fetched value; QA classified this as non-blocking residual hardening. - Audit-event insertion currently fails open if the security-event insert itself fails; the operator activity event provides partial redundancy, but a future hardening change should define fail-closed behavior for value delivery. - This PR overlaps `ui/src/pages/Secrets.tsx` with #9918 and may require a straightforward rebase after that PR moves. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, `gpt-5.3-codex`, with reasoning, repository tool use, terminal execution, Paperclip API access, and GitHub CLI capabilities. Context-window size is not exposed by the runtime. - Anthropic Claude Opus 4.8 with 1M context and tool use assisted with the UI implementation commit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-22 08:04:39 -05:00
Nicky LeachandPaperclip	1d2b6af5ac	fix(tests): stabilize heartbeat cleanup for tsx update (#9573 ) ## Thinking Path > - Paperclip is an open-source platform for orchestrating AI agents, built on an embedded-Postgres server running a heartbeat loop to advance agent work. > - The server test suite exercises heartbeat liveness escalation and retry scheduling logic against a real embedded database; tests create and tear down full database state across every case. > - Dependabot PR #9480 bumps `tsx` from 4.22.4 to 4.23.1. The new version exposed two fragile teardown patterns in the heartbeat tests that caused failures. > - The first problem: `TRUNCATE TABLE "companies" CASCADE` in the liveness-escalation teardown clashes with FK constraints when child tables (e.g. `heartbeat_run_events`, `issue_tree_hold_members`) hold rows that tsx 4.23.1's changed execution order materialises before the CASCADE runs. > - The second problem: the retry-scheduling test duplicated a 10-line delete block inline at two mid-test reset points; one copy deleted `heartbeat_run_events` after `heartbeat_runs` (wrong FK order) and `activityLog` was deleted twice. > - A third concern was identified during review: several `GET /tool-connections/:connectionId` routes called `assertCompanyAccess` before checking whether the actor has access at all, leaking 403 (existence oracle) instead of 404. This is fixed in this PR. > - This PR updates the three `tsx` version pins to `^4.23.1`, replaces the TRUNCATE with explicit child-to-parent deletes, centralises the retry cleanup into a shared `cleanupRetryFixture()` helper, and adds `hasCompanyAccess` pre-checks before the four affected `assertCompanyAccess` calls in `tool-access.ts`. > - The benefit is CI green on tsx 4.23.1, cleaner non-duplicated teardown code across both test files, and no cross-tenant existence leakage on tool-connection routes. ## Linked Issues or Issue Description Refs #9480 (`tsx` 4.22.4 → 4.23.1 dependabot bump whose CI failures this fixes) ## What Changed - cli/package.json, packages/db/package.json, server/package.json: bump `tsx` dev-dependency range from `^4.22.4` to `^4.23.1` so package manifests agree with the lockfile update landing in #9480. `pnpm-lock.yaml` is left untouched — GitHub Actions owns lockfile regeneration. - heartbeat-issue-liveness-escalation.test.ts: replace `TRUNCATE TABLE "companies" CASCADE` with explicit FK-ordered deletes. The new chain adds `heartbeatRunEvents`, `issueTreeHoldMembers`, `agentRuntimeState`, and `companySkills` before their respective parent tables. - heartbeat-retry-scheduling.test.ts: extract the repeated teardown block into a `cleanupRetryFixture()` helper; call it from `afterEach` and the two mid-test resets; fix `heartbeatRunEvents` deleted before `heartbeatRuns` (parent-child FK order); remove the duplicate `activityLog` delete. - server/src/routes/tool-access.ts: add `hasCompanyAccess` pre-checks before `assertCompanyAccess` on four `GET /tool-connections/:connectionId` and `GET /tool-profiles/:profileId/new-tools` routes. Returns 404 instead of 403 when the actor cannot access the resource, closing the cross-tenant existence oracle. ## Verification ```sh # Focused test run (49 tests, all pass) pnpm exec vitest run \ server/src/__tests__/heartbeat-retry-scheduling.test.ts \ server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts pnpm --filter @paperclipai/server typecheck # pass pnpm -r typecheck # pass pnpm build # pass ``` Full `pnpm test:run` was also attempted: server suite (242 files, 2 243 tests) and UI suite (310 files, 2 536 tests) both passed. A backup-dir assertion in `src/__tests__/onboard.test.ts` failed but is unrelated to this diff — it expects a temp `PAPERCLIP_HOME` but receives the global instance path. ## Risks Low risk. Changes are limited to test teardown logic, dev-dependency version pins, and existence-oracle guard additions on read-only tool-connection routes. No new business logic or production data paths are introduced. ## Model Used Claude Sonnet 4.6 (`claude-sonnet-4-6`, 200 k context, tool use, agentic coding) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-07-16 14:02:14 -05:00
Nicky LeachandPaperclip	7947308276	fix(codex): classify refresh auth failures (#9598 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip supports the Codex local adapter, which runs OpenAI Codex CLI sessions on behalf of agents > - Codex uses OAuth refresh tokens to maintain long-running authenticated sessions > - When a refresh fails, the failure has distinct root causes: a refresh token was already reused in a parallel request, the token expired by TTL, or the token was invalidated/revoked by the provider > - Without classifying these failure modes, all refresh auth errors surface identically — operators cannot distinguish retryable transient collisions from permanent invalidations, and run logs carry no actionable diagnosis > - This pull request adds structured classification (`refresh_token_reused`, `refresh_token_expired`, `refresh_token_invalidated`) of Codex refresh-token auth failures across the CLI quota-probe, ACP auth path, and execute path > - The benefit is that these distinct failure modes can be surfaced in run logs and acted on appropriately — transient reuse can be retried; true invalidations require re-auth ## Linked Issues or Issue Description <!-- Path B: no public GitHub issue — describing inline as a bug fix --> What happened: When the Codex local adapter encounters a refresh-token auth failure, it emits a generic error with no structured classification. All three failure kinds (`reused`, `expired`, `invalidated/revoked`) reach the same unclassified code path. Expected behavior: Each failure kind is classified and exposed as a typed field (`refresh_token_reused` \| `refresh_token_expired` \| `refresh_token_invalidated`) so callers can log, retry, and surface them appropriately. Steps to reproduce: 1. Run a Codex agent session with a reused or expired OAuth refresh token. 2. Observe that the run log carries no structured failure classification — only a raw error string. Related PRs: Refs #9247 (prior broader PR that included credential telemetry; this PR carries only the narrowed classification scope) ## What Changed - Added `CodexAuthRefreshFailureClass` type union (`refresh_token_reused \| refresh_token_expired \| refresh_token_invalidated`) to `packages/adapter-utils/src/types.ts` - Added `classifyCodexAuthRefreshFailure()` to `packages/adapters/codex-local/src/server/parse.ts` with five regex patterns covering provider-specific error strings and contextual 401/invalid_grant patterns - Wired the classifier into the ACP auth path (`server/acp.ts`), execute path (`server/execute.ts`), and CLI quota-probe (`cli/quota-probe.ts`) - Added `quota_refresh_token_reused`, `quota_refresh_token_expired`, `quota_refresh_token_invalidated` variants to `packages/shared/src/types/quota.ts` - Added classification unit tests (`parse.test.ts`, `quota-spawn-error.test.ts`, `acp.test.ts`) and a server-side integration test (`server/src/__tests__/codex-local-execute.test.ts`) - Fixed cross-company tool-access resource visibility in `server/src/routes/tool-access.ts` - Stabilized `heartbeat-retry-scheduling.test.ts` (CASCADE cleanup), `heartbeat-run-log.test.ts`, and `quota-windows.test.ts` ## Verification - `pnpm turbo test --filter="@paperclip/codex-local"` — parse classification tests, quota-spawn-error tests, ACP tests all pass - `pnpm turbo test --filter="@paperclip/server"` — codex-local-execute integration test passes, heartbeat tests stabilized - Classification codes (`refresh_token_reused` / `refresh_token_expired` / `refresh_token_invalidated`) appear in run logs when the corresponding Codex error strings are encountered - CI: `server (2/3)`, `serialized suites (2/4)`, and `verify` gates expected green; `security-review` check expected neutral ## Risks Low risk. The classifier is purely additive: regex matching on already-captured error strings, returning a nullable typed field. Callers that do not inspect the classification field are unaffected. No execution paths, retry logic, or existing error surfaces changed. ## Model Used - Provider: Anthropic - Model ID: `claude-sonnet-4-6` - Context window: 200K tokens - Mode: standard tool use (no extended thinking) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-07-14 22:40:38 -07:00
DottaandPaperclip	cc81eefb60	Make plan-approval continuations durable after failed wakes (#9331 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents often work from reviewed plans that are approved through issue-thread interactions. > - Accepting a plan is not just a UI decision; it must reliably resume the assignee so approved work continues. > - A failed continuation wake could leave an approved plan stranded in review with no durable retry or visible recovery path. > - This pull request makes approved plan continuations retryable, recoverable, and visible when resume fails. > - The benefit is that operators can trust plan approval to either resume the agent or produce an explicit actionable failure instead of silent limbo. ## Linked Issues or Issue Description No public GitHub issue exists. Inline bug report follows the repository bug template. ### Pre-submission checklist - [x] I have searched existing open and closed issues and this is not a duplicate. - [x] I am on the latest released version of Paperclip or can reproduce on `master`. - [x] I have confirmed the error originates in Paperclip itself, not in my agent adapter, API provider, or local configuration. ### What happened? When a plan-confirmation interaction was accepted, the assignee continuation wake could fail before useful agent execution. In that case the issue could remain in review even though the plan had been approved, because the failed wake was fire-and-forget and there was no durable retry or recovery path for accepted continuations. ### Expected behavior Accepted plan continuations should either wake the assignee successfully, retry bounded infrastructure failures, recover dropped wakes, or surface an explicit failure state that operators can act on. ### Steps to reproduce 1. Create an issue with an assignee and a plan confirmation that wakes the assignee on accept. 2. Accept the confirmation. 3. Simulate a pre-flight continuation failure, such as process loss before agent start or workspace validation failure. 4. Observe that the approved issue can remain in review without an active assignee wake or visible retry/failure state. ### Paperclip version or commit Reproducible on `master` before this PR's retry/recovery changes. ### Deployment mode Local dev (`pnpm dev`) and server-side recovery paths. ### Installation method Built from source (`pnpm install`, `pnpm dev`, test runner). ### Agent adapter(s) involved Not adapter-specific; this is a core continuation/recovery bug. The tests cover local-agent failure shapes without relying on a provider-specific API. ### Database mode Embedded Postgres test database for verification. The affected logic is database-backed and applies to normal Postgres deployments as well. ### Access context Board accepts the interaction; agent execution resumes through the assignee wake path. ### Relevant logs or output No sensitive logs are needed. The regression tests simulate the failed wake and recovery states directly. ### Relevant config No special config is required beyond an assignee with wake-on-demand enabled. ### Additional context This PR also prevents a stale workspace-validation payload from quarantining another issue's active workspace and prevents unrelated successful runs from masking a continuation that never resumed. ### Privacy checklist - [x] I have reviewed all pasted output for PII, usernames, file paths, API keys, tokens, and company names, and redacted where necessary. ## What Changed - Added bounded infrastructure retries for failed accepted-interaction continuation wakes. - Extended stranded issue recovery so dropped accepted-plan continuation wakes are requeued. - Recorded and rendered explicit resume-failure state on accepted confirmation cards. - Added clean-workspace fallback for workspace-validation failures while preventing cross-issue workspace quarantine. - Tightened recovery so unrelated successful runs do not mask an accepted continuation that never resumed. - Added focused server/UI coverage for retry scheduling, recovery, visible failure state, and interaction card rendering. ## Verification - `pnpm vitest run server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts` — 106 tests passed. - Earlier branch verification also covered the issue-thread interaction card tests for the visible resume-failure UI. - GitHub CI is green on the replacement PR head, and Greptile reports 5/5 with no blocking issues. ## Risks - Medium behavioral risk: this changes recovery behavior for accepted continuation interactions and workspace-validation retries. - Mitigation: retries are bounded, scoped to same-company issue context, and workspace quarantine now requires ownership by the issue being retried. - Existing stored confirmation results remain compatible because the new resume-failure field is optional. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5 coding agent, tool-enabled terminal workflow. The runtime does not expose an exact context-window value to the agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-07-10 08:12:16 -05:00
DottaandPaperclip	8b6a06ee25	[codex] Add built-in agents and Reflection Coach bundle (#9206 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators need first-party agent capabilities for repeatable company work, not just manually created one-off agents. > - Built-in agents need to behave like normal company-scoped agents while preserving approval gates, permissions, budgets, and audit trails. > - Reflection and coaching work also needs bundled instructions, skill content, and a routine so the feature can be installed and reset predictably. > - The API, database, UI, portability, and tests all need to agree on the built-in lifecycle from not provisioned through setup, approval, ready, paused, and reset. > - This pull request adds built-in agent provisioning and the Reflection Coach bundle end-to-end. > - The benefit is a safer first-party path for Paperclip-managed agents without bypassing the same governance model used for operator-created agents. ## Linked Issues or Issue Description No public GitHub issue was found for this exact built-in agent and Reflection Coach bundle work. Problem/motivation: - Paperclip did not have a first-party built-in agent lifecycle for product-owned agents. - Bundled agent resources such as default instructions, skills, and routines needed managed ownership and reset semantics. - Approval-gated companies needed built-in setup to preserve requested adapter, budget, manager, and permission state through board approval. - The board UI needed clear built-in badges, setup affordances, readiness state, and bundle status without exposing secrets. Proposed solution: - Add a company-scoped built-in agent registry, provisioning/reset/reconcile/status APIs, and Reflection Coach bundled resources. - Track bundled managed resources in the database with idempotent migration behavior. - Reuse existing agent approval, authorization, budget, and activity-log paths instead of creating a bypass. - Add UI setup, badges, gates, bundle panels, and route coverage for built-in agents. Duplicate search: - Searched GitHub PRs for `built-in agents Reflection Coach repo:paperclipai/paperclip`; only this PR was returned. - Searched GitHub issues for the same query; no public issues were returned. ## What Changed - Added built-in agent definitions, lifecycle state derivation, provisioning, reset, reconcile, status, and routine-control routes. - Added the `built_in_managed_resources` migration and schema exports for bundled instructions, skill, and routine ownership. - Added the Reflection Coach built-in bundle with default instructions, skill catalog content, routine template, default permissions, and managed-resource drift handling. - Added approval-aware provisioning behavior that preserves requested adapter config, budgets, manager assignment, and built-in permissions through hire approval. - Added authorization and mutation gates for built-in agent and skill changes, including consented Reflection Coach change paths. - Added UI surfaces for built-in agent setup, roster/detail badges, readiness gates, bundle status, routine controls, and route filtering. - Added company import/export and validator coverage for built-in managed resources and low-trust/red-team presets. - Addressed Greptile follow-ups for pending approval reconciliation, consent-gate error propagation, config-read authorization fallback, approval-path manager preservation, and non-model adapter provisioning. ## Verification Local verification: - `git diff --check public/master..HEAD` passed. - `pnpm check:token-gates` passed with all gates clean. - `pnpm exec vitest run ui/src/components/ConfigureBuiltInAgentModal.test.tsx` passed: 1 file, 4 tests. - `pnpm exec vitest run ui/src/components/EntityRow.test.tsx ui/src/pages/Agents.test.tsx ui/src/components/BuiltInAgentGate.test.tsx ui/src/components/ConfigureBuiltInAgentModal.test.tsx ui/src/components/BuiltInBundlePanel.test.tsx ui/src/pages/InstanceExperimentalSettings.test.tsx ui/src/pages/Routines.test.tsx` passed: 7 files, 64 tests. - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/built-in-agents.test.ts src/__tests__/authorization-service.test.ts src/__tests__/company-skills-routes.test.ts` passed: 3 files, 91 tests. - `pnpm --filter @paperclipai/db check:migrations` passed. - `pnpm -r typecheck` passed after the rebase; `pnpm --filter ui typecheck` passed after the final UI review fix. Remote verification on latest head `1c61f693a4ec881d739022b0e75a8ca8bf8c2cd8`: - Merge state: `CLEAN`. - Greptile: `5/5`, zero unresolved Greptile threads. - PR check rollup: all checks successful, neutral, or skipped as expected. - Passing gates include Build, Typecheck + Release Registry, all server shards, all workspace shards, all serialized server suites, e2e, Canary Dry Run, policy, review, verify, Socket, Superagent, and Snyk. ## Risks - This adds a new managed-resource table and migration; the migration uses idempotent create/add/index guards and passed migration safety checks. - Built-in agent provisioning touches approval and authorization paths; tests cover pending approval preservation, stale retry rejection, consent gates, and config-read fallback behavior. - Reflection Coach creates managed instructions, skill, and routine resources; drift/reset behavior is covered by service tests and redacted API responses. - Non-model adapter setup now provisions a `needs_setup` built-in row before command/endpoint fields are complete; this matches the server lifecycle and is covered by the setup modal regression test. ## Model Used OpenAI Codex coding agent based on GPT-5. Exact hosted model ID, context-window size, and reasoning-mode labels are not exposed in this runtime; tool use, shell execution, GitHub CLI/API access, and local code editing were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-07-09 16:29:30 -05:00
DottaandPaperclip	555391fed7	fix: run restart recovery, workspace self-heal, quota-aware retries, failed-run metrics (#9183 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in heartbeat runs orchestrated by the server; run lifecycle, retry scheduling, and the dashboard's run-activity metrics are the subsystems involved > - A spike in "failed" tasks traced to three causes: server restarts killing in-flight runs and mislabeling them as failures, deterministic workspace-validation loops when a worktree's branch diverged, and provider quota/usage-limit errors being classified as generic transient failures (putting agents into error state and polluting metrics) > - Killed-then-recovered runs and quota waits are not product failures, so both the runtime behavior and the reporting needed to distinguish them > - This pull request drains runs gracefully on shutdown with idempotent restart retries, self-heals workspace branch mismatches, adds a quota-aware failure class with reset-time retry, separates recovered restart kills from true failures on the dashboard, and documents restart hygiene for operators > - The benefit is fewer spurious failures, automatic recovery instead of manual repair, and dashboard metrics that reflect real failure rates ## Linked Issues or Issue Description No public GitHub issue exists; describing the bug inline per the bug-report template: What happened? In-flight heartbeat runs are marked `failed` when the server restarts, even though a retry later succeeds. Worktrees whose checked-out branch diverges from the issue branch fail workspace validation on every subsequent run with no recovery path. Provider quota/usage-limit responses are treated as generic transient upstream errors, putting agents into an error state and retrying before the quota window resets. The dashboard counts all of these as true failures, inflating failure metrics. Expected behavior Graceful shutdown should interrupt (not fail) running runs and chain exactly one recovery retry. Workspace validation should repair recoverable branch mismatches automatically. Quota errors should get their own error class with the retry scheduled at the provider reset time and the agent left idle. The dashboard should report recovered restart kills separately from true failures. Steps to reproduce 1. Start a heartbeat run, then restart the server (SIGTERM) while it is in flight — the run lands as `failed` with a process-loss error code even when its retry succeeds 2. Check out an issue whose worktree branch has diverged (e.g. after a force-moved branch) — every subsequent run fails `workspace_validation_failed` deterministically 3. Drive an agent into a provider usage-limit window — the run fails as a generic transient upstream error and the agent enters an error state instead of idling until the reset time Paperclip version or commit master (base `c07e650cd`) Deployment mode Self-hosted dev plane (Linux, node server + embedded Postgres) ## What Changed - Graceful shutdown (SIGTERM hook) now marks in-flight runs `interrupted` instead of `failed` and enqueues an idempotent process-loss retry (pre-insert existence check on `retryOfRunId` prevents duplicates; bursts chain exactly one retry per interrupted run) - Run-liveness classification routes `interrupted` to `needs_followup` rather than `failed` - Workspace validation self-heals branch mismatch / missing-branch states instead of failing deterministically on every run - New `provider_quota` error class: session/usage-limit responses schedule the retry at the provider reset time and leave the agent idle (not errored); fixes a case where a quota-terminated run with subtype `success` was misclassified as failed; HTTP 529 remains transient - Dashboard run-activity query separates recovered restart kills from true failures via a recursive CTE over `retry_of_run_id` (ancestors of a succeeded retry count as recovered), adds a per-day failed-by-error-code breakdown, and binds the window start as a timestamptz string - Activity charts UI: amber "Recovered" segment with legend and per-day error-code tooltip; success-rate chart counts recovered runs as successes - New ops runbook: `docs/deploy/dev-plane-restart-hygiene.md` ## Verification - Greptile follow-up fixes on `992705edd`: `pnpm exec vitest run server/src/__tests__/server-startup-feedback-export.test.ts server/src/__tests__/heartbeat-workspace-branch-containment.test.ts server/src/__tests__/heartbeat-workspace-session.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts` (209 tests); `pnpm --filter @paperclipai/server typecheck`; `git diff --check` - Post-rebase CI fixes: `pnpm exec vitest run server/src/__tests__/heartbeat-retry-scheduling.test.ts`; `pnpm exec vitest run server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/workspace-runtime.test.ts server/src/__tests__/heartbeat-workspace-branch-containment.test.ts server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts server/src/__tests__/heartbeat-workspace-session.test.ts` (227 tests); `pnpm --filter @paperclipai/server typecheck` - `npm test` server suites covering the changes: `heartbeat-process-recovery`, `heartbeat-stop-metadata`, `heartbeat-retry-scheduling` (95 tests), quota parse + execute/retry-scheduling suites (100 tests), workspace self-heal suites (200 tests), dashboard run-activity tests (3 tests) — all green, typecheck exit 0 - Dashboard CTE cross-checked against a real development database: two restart-burst days moved from 17 to 8 and 19 to 6 true failures once recovered kills were separated, matching manual retry-chain inspection - Screenshot verification of the real ActivityCharts component (recovered segment + tooltip) during QA ## Risks - Behavioral shift: runs killed by a restart no longer surface as `failed`; anyone consuming raw run statuses will see `interrupted` (new status value) — dashboards/queries in this repo were updated accordingly - Retry chaining on repeated restarts is bounded (one chained retry per interruption) but a pathological restart loop still delays work rather than failing it; the runbook covers operator hygiene for that case - Dashboard query adds a recursive CTE; cost is bounded by the day-window row count and was verified against production-sized data - No schema migrations; low migration risk ## Model Used - Claude (Anthropic) — claude-fable-5 via Claude Code / Paperclip agent harness, extended thinking with tool use; implementation commits also produced with Codex CLI (GPT-5 class) agents under the same harness ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-07-07 20:30:09 -05:00
Dotta Paperclip Claude Opus 4.8	ad961227f5	feat(secrets): add user-specific runtime secrets (#8825 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent runs often need provider credentials, API tokens, and other environment-bound secrets. > - Company-level secrets work for shared credentials, but they do not model values that should differ by human operator. > - Without a user-scoped model, a run can dispatch without knowing whether the responsible human has supplied the needed value. > - Paperclip also needs run attribution to make those user-scoped runtime checks deterministic and auditable. > - This pull request adds user-specific secret definitions, per-user values, environment bindings, responsible-user attribution, and runtime resolution gates. > - The benefit is that teams can define the secret once, let each user provide their own value, and block runs before dispatch when required user secrets or active definitions are unavailable. ## Linked Issues or Issue Description Refs #224 Refs #6057 This PR implements user-specific secret support as a core secret-management capability rather than a one-off adapter setting. It is related to existing public work on company secrets UI and runtime secret refs, but is distinct because the value is owned by the responsible user and resolved at run dispatch time. Related PR search before opening found existing secrets work such as #1550, #8256, #8614, #8634, and #8647; none of those add the full user-secret definition/value/runtime gate covered here. ## What Changed - Added user-secret definitions and per-user "My secrets" values, keeping stored values out of access metadata. - Added `user_secret_ref` environment bindings and UI affordances to pick them alongside existing secret refs. - Added responsible-user runtime resolution so user-secret refs resolve against the human responsible for the run. - Added pre-dispatch missing-secret gates so runs fail before adapter dispatch when required user values are absent or definitions are inactive. - Added low-trust allowlist hardening for user-secret runtime access. - Added issue, routine, run, and agent API key responsible-user attribution and fail-closed dispatch behavior when attribution cannot be resolved. - Added denial-copy mapping so responsible-user authorization failures surface as actionable run outcomes instead of opaque setup failures. - Added OpenAPI documentation for the user-secret routes. - Rebases cleanly on current `master`; migrations were renumbered incrementally as `0128_user_specific_secrets`, `0129_agent_api_key_responsible_user`, and `0130_run_responsible_user_invariant` after upstream `0126`/`0127` migrations. - Removed previously committed local design screenshots so the PR contains code/docs/tests only. ## Verification - PASS: PR head `2527febd106bcf3ca264ca0da7fca491084192d6` is based on `paperclipai/paperclip:master`. - PASS: `git diff --check` - PASS: `git diff --name-only public/master...HEAD \| rg '^(pnpm-lock\\.yaml\|\\.github/workflows/\|screenshots/)' \|\| true` produced no files. - PASS: migration journal audit confirmed unique indexes through `130` with tail entries `0126_issue_comment_derived_attribution`, `0127_environment_custom_images_instance_scoped`, `0128_user_specific_secrets`, `0129_agent_api_key_responsible_user`, and `0130_run_responsible_user_invariant`. - PASS: `pnpm --filter @paperclipai/ui typecheck` - PASS: `pnpm --filter @paperclipai/server typecheck` - PASS: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-responsible-user-invariant.test.ts` - PASS: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-active-run-output-watchdog.test.ts src/__tests__/heartbeat-stale-queue-invalidation.test.ts src/__tests__/heartbeat-workspace-finalize-branch.test.ts src/__tests__/issue-monitor-scheduler.test.ts` - PASS: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-comment-wake-batching.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts src/__tests__/heartbeat-plugin-environment.test.ts` - PASS: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/low-trust-red-team-routes.test.ts` - PASS: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/secrets-service.test.ts` (55 tests) - PASS: `pnpm vitest run server/src/__tests__/secrets-routes.test.ts server/src/__tests__/secrets-service.test.ts` (89 tests after final Greptile cleanup fixes) - PASS: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-issue-liveness-escalation.test.ts` (17 tests after the final rebase CI fix) - PASS: focused server Vitest batches covering heartbeat recovery, project env, plugin env, routines, low-trust, pipelines, monitors, watchdog, and stale queue paths. - PASS: GitHub checks are green on `2527febd106bcf3ca264ca0da7fca491084192d6`, including Typecheck + Release Registry, Build, General tests, serialized server suites, e2e, Canary Dry Run, verify, security checks, and Greptile Review. - PASS: Greptile Review completed successfully on `2527febd106bcf3ca264ca0da7fca491084192d6` with Confidence Score 5/5, and GraphQL review-thread audit returned zero unresolved non-outdated threads. ## Risks - Runtime behavior now depends on a run having a correct responsible user; missing or incorrect responsibility assignment can block runs before adapter dispatch. - `user_secret_ref` bindings intentionally expose metadata without values, but UI/API callers may need to handle the new binding kind explicitly. - External secret providers and IAM policies are not automatically provisioned by this PR; operators still need to configure provider-side access for non-local vaults. - The PR is broad across db/shared/server/UI/runtime paths, so release validation should include both API and UI secret workflows before merge. - The migration renumbering is intentionally incremental after upstream migrations; the branch migrations use guarded column/table/index/constraint creation so users who tested the older draft numbering should not hit duplicate DDL for the existing objects. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5-based coding agent (`gpt-5`), Codex local adapter with shell/tool use and code execution. Context window and internal reasoning mode are not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-05 05:58:20 -05:00
Dotta Paperclip Claude Opus 4.8	67b22d872f	[codex] Clarify interrupt handoffs and scoped wake semantics (#7855 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The issue thread is the operator surface where comments, assignee changes, pauses, resumes, and wakeups turn human intent into agent execution. > - Interrupting a live run and handing work to another assignee needs clear semantics so the product does not accidentally keep work alive, wake the wrong participant, or hide why an agent stopped. > - Comment-driven wakes also need strict boundaries so closed, blocked, and dependency-driven work only resumes when there is real actionable input. > - This pull request codifies the interrupt handoff contract, implements backend scheduling behavior, and gives the UI clearer handoff/pause language. > - The benefit is a more inspectable and predictable task lifecycle for both operators and agents. ## Linked Issues or Issue Description Paperclip issue: `PAP-10664` / `PAP-10751`. Problem: interrupting or reassigning live agent work could be ambiguous in the UI and backend. Operators needed clearer feedback about whether a handoff wakes an agent, what pause/cancel affects, and when comments should revive execution. The backend also needed stronger tests around comment wake boundaries, retry supersession, and structured agent mention dispatch. Related GitHub PR search found broad workflow-adjacent PRs #5082, #6359, and #4083, but no exact duplicate for this head branch or interrupt-handoff scope. ## What Changed - Added an interrupt handoff semantics document covering destination behavior, wake expectations, and live-run interruption states. - Implemented backend interrupt handoff behavior and comment wake/reopen handling in issue routes/services and heartbeat scheduling. - Hardened structured agent mention dispatch so mentions resolve through the intended dispatch path. - Added UI helpers and components for handoff chips, wake rows, interrupt banners, pause-affects summaries, and composer guidance. - Updated the issue properties assignee picker and issue chat/composer surfaces to make interrupt/reassign behavior clearer. - Added backend, UI utility, component, and Storybook coverage for the new behavior. - Stabilized the new UI component tests with a local `flushSync`-backed act helper matching existing repo practice in this dependency set. - Addressed Greptile feedback by threading historical run `errorCode` through issue-run data and operator-interrupted chat labels. - Addressed Greptile's cancel ordering concern by terminating/deleting in-memory heartbeat processes before cancellation status persistence, with regression coverage for DB update failure. ## Verification - `git diff --check $(git merge-base HEAD origin/master)..HEAD` - `pnpm --filter @paperclipai/ui exec vitest run src/lib/interrupt-handoff.test.ts src/lib/issue-chat-messages.test.ts src/components/IssueProperties.test.tsx src/components/interrupt-handoff/InterruptHandoffViews.test.tsx --no-file-parallelism --maxWorkers=1` — 4 files / 91 tests passed before the Greptile follow-ups. - `pnpm run preflight:workspace-links && pnpm exec vitest run server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/issue-comment-reopen-routes.test.ts server/src/__tests__/issue-tree-control-service.test.ts server/src/__tests__/issue-update-comment-wakeup-routes.test.ts server/src/__tests__/issues-service.test.ts --no-file-parallelism --maxWorkers=1` — 6 files / 191 tests passed before the Greptile follow-ups. - `pnpm --filter @paperclipai/ui exec vitest run src/lib/issue-chat-messages.test.ts --no-file-parallelism --maxWorkers=1` — 1 file / 24 tests passed after the historical `errorCode` follow-up. - `pnpm exec vitest run server/src/__tests__/activity-service.test.ts server/src/__tests__/activity-routes.test.ts --no-file-parallelism --maxWorkers=1` — 2 files / 11 tests passed after the historical `errorCode` follow-up. - `pnpm exec vitest run server/src/__tests__/heartbeat-process-recovery.test.ts --no-file-parallelism --maxWorkers=1` — 1 file / 52 tests passed after the cancel ordering follow-up. - Greptile is green for head `272647636287d034bab8d981eaf5305865aa0f96`; the old inline P2 is resolved/outdated. - GitHub Actions, Socket, security-review, and Greptile checks are green for head `272647636287d034bab8d981eaf5305865aa0f96`. The external `security/snyk (cryppadotta)` status was still pending at `https://app.snyk.io/org/cryppadotta/pr-checks/85b3e8f4-04e1-4f8e-9362-899c8148c23c` after a bounded wait. ## Risks - Medium: changes touch issue comments, wake scheduling, and live-run interruption semantics, so regressions could affect when agents resume or stay stopped. - Medium: UI copy and state grouping for assignee changes may need reviewer tuning after product review. - Low migration risk: no database schema migration is included. - The branch was created before the latest `origin/master` commits; reviewers should confirm CI merge-base behavior and resolve any merge conflicts if GitHub reports them. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5-based coding agent, tool use and local command execution enabled. Exact hosted model build and context window were not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Screenshot note: this PR includes Storybook coverage for the new interrupt handoff UI states rather than captured before/after browser screenshots in this PR-creation heartbeat. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-09 21:57:21 -05:00
Dotta	bfe6369ef5	Guard cheap recovery model usage (#6371 ) ## Thinking Path > - Paperclip is the control plane that coordinates AI-agent work through issues, heartbeats, comments, approvals, and auditable recovery paths. > - The affected subsystem is heartbeat/recovery orchestration, especially the optional cheap model profile used for operational recovery overhead. > - Cheap recovery should repair status and liveness, but it must not become the worker lane that writes deliverables, continues source work, or propagates cheap execution hints into downstream retries. > - The gap was that cheap-profile hints could follow recovery wake contexts and assignment overrides farther than intended, making real work eligible to run on the cheap model. > - This pull request separates status-only cheap recovery from normal source-work continuations, adds route guards for deliverable mutations during cheap status-only runs, and documents the invariant. > - The benefit is safer retry/recovery behavior: cheap runs can clean up control-plane state, while any remaining source work resumes through a normal/original model path. ## What Changed - Added recovery model-profile work classes so status-only recovery carries explicit guard context and normal-model continuations scrub cheap hints. - Updated heartbeat, productivity review, liveness continuation, and recovery service wakeups to request cheap only for bounded status-only recovery work. - Blocked cheap status-only recovery runs from writing issue documents, plans, attachments, work products, or assigning downstream work back to `modelProfile: "cheap"`. - Added/updated server tests for cheap profile propagation, artifact/document guards, route authorization, retry scheduling, and successful-run handoff behavior. - Documented the recovery model-profile lane in `doc/SPEC-implementation.md` and `doc/execution-semantics.md`. - After rebasing onto current `public-gh/master`, stabilized the new `InstanceSidebar` plugin-filter tests so the PR check lane stays green. ## Verification - Local: `pnpm exec vitest run --config vitest.config.ts src/services/recovery/model-profile-hint.test.ts src/__tests__/issue-agent-mutation-ownership-routes.test.ts src/__tests__/issue-document-restore-routes.test.ts` from `server/` - 3 files, 37 tests passed after final edits. - Local: `pnpm exec vitest run --config vitest.config.ts src/__tests__/heartbeat-process-recovery.test.ts` from `server/` - 44 tests passed after rerunning the cleanup-sensitive file alone. - Local: `pnpm --filter @paperclipai/ui exec vitest run src/components/InstanceSidebar.test.tsx` - 4 tests passed. - Local: `pnpm --filter @paperclipai/server typecheck` - passed. - Local: `pnpm --filter @paperclipai/ui typecheck` - passed. - PR checks on latest head `6f8c3b1380f5bd872c6f49f6f7188ecf3bb6d263` - all green, including `verify`, build, typecheck, server/general/serialized tests, e2e, Snyk, and policy. - Greptile: pass 3 returned Confidence Score 5/5 with zero unresolved Greptile review threads. ## Risks - Medium risk: recovery behavior is intentionally stricter, so any path that incorrectly relies on cheap recovery to keep doing source work will now need to hand back to a normal-model run. - Low migration risk: no schema changes. - No product UI changes; the UI file touched is a test-only stabilization after rebasing onto current `master`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex coding agent, GPT-5 model family (`gpt-5`), tool use and local code execution enabled; context window not exposed in this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots (N/A: no product UI changes) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge	2026-05-19 13:46:02 -05:00
DottaandPaperclip	454edfe81e	Add recovery handoff system notices (#5289 ) ## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies. > - Agent runs can end productively while the source issue still lacks a durable final disposition. > - That leaves the control plane unsure whether to resume, escalate, or close the work. > - Issue comments also need a presentation contract so system-authored recovery notices can render as first-class thread messages without overloading normal comments. > - This pull request adds successful-run handoff recovery, comment presentation metadata, and system notice rendering. > - The benefit is stricter task liveness with clearer operator-facing recovery state. ## What Changed - Added successful-run handoff decisions, wake payloads, escalation behavior, and recovery tests. - Added issue comment presentation metadata with migration `0078_white_darwin.sql` and shared/server/company portability support. - Rendered recovery/system notices in issue chat with dedicated UI components, fixtures, tests, and storybook/lab coverage. - Included the current recovery model-profile hint patch so automatic recovery follow-ups use the cheap profile. ## Verification - `pnpm install --frozen-lockfile` - `pnpm exec vitest run server/src/services/recovery/successful-run-handoff.test.ts ui/src/components/SystemNotice.test.tsx ui/src/lib/system-notice-comment.test.ts ui/src/components/IssueChatThreadSystemNotice.test.tsx` ## Risks - Migration-bearing PR: merge this before any other branch that might later add a migration. - The branch touches both recovery services and issue-thread rendering, so review should pay attention to recovery wake idempotency and comment metadata compatibility. ## Model Used - OpenAI GPT-5 Codex via Paperclip `codex_local` adapter, with shell/git/GitHub CLI tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-05-06 06:05:58 -05:00
DottaandPaperclip	15eac43b43	[codex] Retry max-turn exhausted heartbeats (#5096 ) ## Thinking Path > - Paperclip orchestrates AI agents for autonomous companies, and heartbeat execution is the control-plane loop that keeps assigned work moving. > - Max-turn exhaustion is a recoverable local-adapter stop condition for Claude and Gemini agents when a run needs another heartbeat to continue safely. > - The previous behavior could leave max-turn continuation details hard to inspect, and duplicate/stale continuation wakes could keep running after issue state changed. > - The adapter layer also needed to avoid trusting arbitrary stdout/stderr text as scheduler control metadata. > - This pull request adds bounded max-turn continuation scheduling, visible retry state, structured stop metadata handling, and stale/duplicate continuation guards. > - The benefit is safer automatic continuation after max-turn stops, clearer operator visibility, and fewer duplicate or stale agent runs. ## What Changed - Replaces closed PR #4952, whose head repository was deleted. - Rebases the recovered max-turn continuation branch onto current `paperclipai/paperclip:master`. - Adds max-turn continuation scheduling and retry-state plumbing for heartbeat runs. - Adds stale/duplicate continuation suppression when issue status, ownership, or execution locks change. - Normalizes Claude/Gemini max-turn detection around structured stop metadata instead of unstructured stdout/stderr text. - Surfaces max-turn continuation settings and retry visibility in the board UI. - Adds focused server, adapter, and UI tests for max-turn stop metadata, retry scheduling, stale queued-run invalidation, adapter parsing/execution, run ledger display, and agent config patching. ## Verification - `pnpm install --no-frozen-lockfile` to refresh local dependencies after rebasing onto current `master`. - `pnpm run preflight:workspace-links && pnpm exec vitest run server/src/__tests__/claude-local-adapter.test.ts server/src/__tests__/claude-local-execute.test.ts server/src/__tests__/gemini-local-adapter.test.ts server/src/__tests__/gemini-local-execute.test.ts server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts server/src/services/heartbeat-stop-metadata.test.ts ui/src/components/IssueRunLedger.test.tsx ui/src/lib/agent-config-patch.test.ts ui/src/lib/runRetryState.test.ts --testTimeout=20000` - `pnpm --filter @paperclipai/adapter-claude-local typecheck && pnpm --filter @paperclipai/adapter-gemini-local typecheck && pnpm --filter @paperclipai/server typecheck && pnpm --filter @paperclipai/ui typecheck` - UI screenshot note: the UI changes are limited to config/ledger state rendering rather than layout changes; component/unit coverage above verifies the rendered behavior. ## Risks - Medium behavior risk: heartbeat retry gating now suppresses max-turn continuations when issue state or execution locks drift, so any callers that relied on stale continuations running will now see cancellation instead. - Low adapter risk: Claude/Gemini unstructured text no longer triggers max-turn scheduler metadata, so only structured stop signals and Gemini exit code 53 are trusted. - No database migrations. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex coding agent, GPT-5-class model, tool-enabled local repository editing and command execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots (not applicable: state/default rendering only; covered by component/unit tests) - [x] I have updated relevant documentation to reflect my changes (not applicable: no user-facing command or docs contract changed) - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-05-03 11:30:48 -05:00
DottaandPaperclip	6916e30f8e	Cancel stale retries when issue ownership changes (#4445 ) ## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies > - Issue execution is guarded by run locks and bounded retry scheduling > - A failed run can schedule a retry, but the issue may be reassigned before that retry becomes due > - The old assignee's scheduled retry should not continue to hold or reclaim execution for the issue > - This pull request cancels stale scheduled retries when ownership changes and cancels live work when an issue is explicitly cancelled > - The benefit is cleaner issue handoff semantics and fewer stranded or incorrect execution locks ## What Changed - Cancel scheduled retry runs when their issue has been reassigned before the retry is promoted. - Clear stale issue execution locks and cancel the associated wakeup request when a stale retry is cancelled. - Avoid deferring a new assignee behind a previous assignee's scheduled retry. - Cancel an active run when an issue status is explicitly changed to `cancelled`, while leaving `done` transitions alone. - Added route and heartbeat regressions for reassignment and cancellation behavior. ## Verification - `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/issue-comment-reopen-routes.test.ts --pool=forks --poolOptions.forks.isolate=true` - `issue-comment-reopen-routes.test.ts`: 28 passed. - `heartbeat-retry-scheduling.test.ts`: skipped by the existing embedded Postgres host guard (`Postgres init script exited with code null`). - `pnpm --filter @paperclipai/server typecheck` ## Risks - Medium risk because this changes heartbeat retry lifecycle behavior. - The cancellation path is scoped to scheduled retries whose issue assignee no longer matches the retrying agent, and logs a lifecycle event for auditability. - No migrations. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, GPT-5 coding agent, tool-enabled with shell/GitHub/Paperclip API access. Context window was not reported by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-04-24 19:24:13 -05:00
DottaandPaperclip	8f1cd0474f	[codex] Improve transient recovery and Codex model refresh (#4383 ) ## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies > - Adapter execution and retry classification decide whether agent work pauses, retries, or recovers automatically > - Transient provider failures need to be classified precisely so Paperclip does not convert retryable upstream conditions into false hard failures > - At the same time, operators need an up-to-date model list for Codex-backed agents and prompts should nudge agents toward targeted verification instead of repo-wide sweeps > - This pull request tightens transient recovery classification for Claude and Codex, updates the agent prompt guidance, and adds Codex model refresh support end-to-end > - The benefit is better automatic retry behavior plus fresher operator-facing model configuration ## What Changed - added Codex usage-limit retry-window parsing and Claude extra-usage transient classification - normalized the heartbeat transient-recovery contract across adapter executions and heartbeat scheduling - documented that deferred comment wakes only reopen completed issues for human/comment-reopen interactions, while system follow-ups leave closed work closed - updated adapter-utils prompt guidance to prefer targeted verification - added Codex model refresh support in the server route, registry, shared types, and agent config form - added adapter/server tests covering the new parsing, retry scheduling, and model-refresh behavior ## Verification - `pnpm exec vitest run --project @paperclipai/adapter-utils packages/adapter-utils/src/server-utils.test.ts` - `pnpm exec vitest run --project @paperclipai/adapter-claude-local packages/adapters/claude-local/src/server/parse.test.ts` - `pnpm exec vitest run --project @paperclipai/adapter-codex-local packages/adapters/codex-local/src/server/parse.test.ts` - `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/adapter-model-refresh-routes.test.ts server/src/__tests__/adapter-models.test.ts server/src/__tests__/claude-local-execute.test.ts server/src/__tests__/codex-local-execute.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/heartbeat-retry-scheduling.test.ts` ## Risks - Moderate behavior risk: retry classification affects whether runs auto-recover or block, so mistakes here could either suppress needed retries or over-retry real failures - Low workflow risk: deferred comment wake reopening is intentionally scoped to human/comment-reopen interactions so system follow-ups do not revive completed issues unexpectedly > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex GPT-5-based coding agent with tool use and code execution in the Codex CLI environment ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-04-24 09:40:40 -05:00
Dotta	09d0678840	[codex] Harden heartbeat scheduling and runtime controls (#4223 ) ## Thinking Path > - Paperclip orchestrates AI agents through issue checkout, heartbeat runs, routines, and auditable control-plane state > - The runtime path has to recover from lost local processes, transient adapter failures, blocked dependencies, and routine coalescing without stranding work > - The existing branch carried several reliability fixes across heartbeat scheduling, issue runtime controls, routine dispatch, and operator-facing run state > - These changes belong together because they share backend contracts, migrations, and runtime status semantics > - This pull request groups the control-plane/runtime slice so it can merge independently from board UI polish and adapter sandbox work > - The benefit is safer heartbeat recovery, clearer runtime controls, and more predictable recurring execution behavior ## What Changed - Adds bounded heartbeat retry scheduling, scheduled retry state, and Codex transient failure recovery handling. - Tightens heartbeat process recovery, blocker wake behavior, issue comment wake handling, routine dispatch coalescing, and activity/dashboard bounds. - Adds runtime-control MCP tools and Paperclip skill docs for issue workspace runtime management. - Adds migrations `0061_lively_thor_girl.sql` and `0062_routine_run_dispatch_fingerprint.sql`. - Surfaces retry state in run ledger/agent UI and keeps related shared types synchronized. ## Verification - `pnpm exec vitest run server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/routines-service.test.ts` - `pnpm exec vitest run src/tools.test.ts` from `packages/mcp-server` ## Risks - Medium risk: this touches heartbeat recovery and routine dispatch, which are central execution paths. - Migration order matters if split branches land out of order: merge this PR before branches that assume the new runtime/routine fields. - Runtime retry behavior should be watched in CI and in local operator smoke tests because it changes how transient failures are resumed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, GPT-5-based coding agent runtime, shell/git tool use enabled. Exact hosted model build and context window are not exposed in this Paperclip heartbeat environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge	2026-04-21 12:24:11 -05:00