mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 21:59:38 +01:00
31e71ccd0aa0e0f61d222eac75c1e080c1390c42
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
555391fed7 |
fix: run restart recovery, workspace self-heal, quota-aware retries, failed-run metrics (#9183)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in heartbeat runs orchestrated by the server; run
lifecycle, retry scheduling, and the dashboard's run-activity metrics
are the subsystems involved
> - A spike in "failed" tasks traced to three causes: server restarts
killing in-flight runs and mislabeling them as failures, deterministic
workspace-validation loops when a worktree's branch diverged, and
provider quota/usage-limit errors being classified as generic transient
failures (putting agents into error state and polluting metrics)
> - Killed-then-recovered runs and quota waits are not product failures,
so both the runtime behavior and the reporting needed to distinguish
them
> - This pull request drains runs gracefully on shutdown with idempotent
restart retries, self-heals workspace branch mismatches, adds a
quota-aware failure class with reset-time retry, separates recovered
restart kills from true failures on the dashboard, and documents restart
hygiene for operators
> - The benefit is fewer spurious failures, automatic recovery instead
of manual repair, and dashboard metrics that reflect real failure rates
## Linked Issues or Issue Description
No public GitHub issue exists; describing the bug inline per the
bug-report template:
**What happened?**
In-flight heartbeat runs are marked `failed` when the server restarts,
even though a retry later succeeds. Worktrees whose checked-out branch
diverges from the issue branch fail workspace validation on every
subsequent run with no recovery path. Provider quota/usage-limit
responses are treated as generic transient upstream errors, putting
agents into an error state and retrying before the quota window resets.
The dashboard counts all of these as true failures, inflating failure
metrics.
**Expected behavior**
Graceful shutdown should interrupt (not fail) running runs and chain
exactly one recovery retry. Workspace validation should repair
recoverable branch mismatches automatically. Quota errors should get
their own error class with the retry scheduled at the provider reset
time and the agent left idle. The dashboard should report recovered
restart kills separately from true failures.
**Steps to reproduce**
1. Start a heartbeat run, then restart the server (SIGTERM) while it is
in flight — the run lands as `failed` with a process-loss error code
even when its retry succeeds
2. Check out an issue whose worktree branch has diverged (e.g. after a
force-moved branch) — every subsequent run fails
`workspace_validation_failed` deterministically
3. Drive an agent into a provider usage-limit window — the run fails as
a generic transient upstream error and the agent enters an error state
instead of idling until the reset time
**Paperclip version or commit**
master (base
|
||
|
|
c297ba2a80 |
fix(codex-local): replace stale auth.json copy with symlink on prepare (#5028) (#5240)
## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies > - codex_local runs Codex CLI under a per-company "managed home" so multiple companies don't trample on each other's session state > - For `auth.json` specifically, the managed home keeps a SYMLINK to the user's real `~/.codex/auth.json` rather than a copy — Codex refresh tokens rotate and are single-use, so any copy goes stale the moment the source rotates and every subsequent run dies with `401 refresh_token_reused` > - Older Paperclip versions copied `auth.json` instead. After upgrading, `ensureSymlink()` saw a regular file at the target, hit `if (!existing.isSymbolicLink()) return;`, and silently kept the stale copy > - This pull request makes the upgrade path self-healing inside `ensureSymlink()` itself: when the target is a regular file, unlink it and create the symlink, since the target lives under the Paperclip-managed home and is safe to delete. Directories are skipped to avoid `EISDIR` on Unix (and inconsistent behavior on Windows) > - The benefit is operators who upgraded from a copy-based version stop getting refresh-token-reused failures without having to manually purge `companies/<id>/codex-home/auth.json`, and the healing is defense-in-depth even outside the `prepareManagedCodexHome` cleanup path ## What Changed - `packages/adapters/codex-local/src/server/codex-home.ts` — `ensureSymlink()` previously bailed out of the `!existing.isSymbolicLink()` branch, leaving any pre-existing regular file untouched. Now unlinks and recreates the symlink in that branch via the existing `createExpectedSymlink()` helper (preserves the EEXIST race-tolerance behavior added in #5119). A guard skips directories so the call never throws `EISDIR` and aborts `prepareManagedCodexHome`. Inline comment explains the safety: target is always under the company-scoped managed home (`<paperclipHome>/instances/<id>/companies/<companyId>/codex-home/`), never the user's real `~/.codex`. - `packages/adapters/codex-local/src/server/codex-home.test.ts` — adds a regression test for #5028: pre-seed a stale copy at the target, run `prepareManagedCodexHome`, assert the target is now a symlink and reads through to the fresh source. The existing concurrent-symlink test is preserved. ## Verification ``` pnpm --filter @paperclipai/adapter-codex-local exec vitest run # Test Files 8 passed (8) # Tests 26 passed (26) pnpm --filter @paperclipai/adapter-codex-local exec tsc --noEmit # clean ``` Manual repro flow that the regression test mirrors: 1. Create a stale copy: `echo '{"token":"old"}' > <managedHome>/auth.json`. 2. Rotate source: `echo '{"token":"new"}' > ~/.codex/auth.json`. 3. Trigger any codex_local run — `prepareManagedCodexHome` is called from the execute path, the managed file is now a symlink to the source, and the CLI sees the fresh token. ## Risks - **Low risk.** The new branch only fires when the target file is a regular file (the upgrade path) — a pure copy that Codex couldn't have written, since Codex never writes into the managed home. Operators in steady-state on the symlink-based version are unaffected. - The `fs.unlink` only runs against the per-company managed-home path, never the user's real `~/.codex`. Inline comment makes this guarantee explicit. - A directory at the auth.json path is left in place (no silent `EISDIR` crash) — this requires operator inspection rather than autonomous deletion. - The healing uses `createExpectedSymlink()` so it remains tolerant of EEXIST races with concurrent prepare calls (the concurrent-symlink test still passes). - No DB / migration / schema impact. ## Model Used - Anthropic Claude Opus 4.7 (claude-opus-4-7), via Claude Code CLI with extended tool use (Read / Edit / Bash / Grep). No extended-thinking budget consumed beyond default. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots — N/A, adapter-only - [x] I have updated relevant documentation to reflect my changes — inline comment explains the why and the safety of the unlink - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge - [x] I searched the GitHub PR list for similar PRs and confirmed this is not a duplicate Fixes #5028. --------- Co-authored-by: Devin Foley <devin@paperclip.ing> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8f1cd0474f |
[codex] Improve transient recovery and Codex model refresh (#4383)
## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies > - Adapter execution and retry classification decide whether agent work pauses, retries, or recovers automatically > - Transient provider failures need to be classified precisely so Paperclip does not convert retryable upstream conditions into false hard failures > - At the same time, operators need an up-to-date model list for Codex-backed agents and prompts should nudge agents toward targeted verification instead of repo-wide sweeps > - This pull request tightens transient recovery classification for Claude and Codex, updates the agent prompt guidance, and adds Codex model refresh support end-to-end > - The benefit is better automatic retry behavior plus fresher operator-facing model configuration ## What Changed - added Codex usage-limit retry-window parsing and Claude extra-usage transient classification - normalized the heartbeat transient-recovery contract across adapter executions and heartbeat scheduling - documented that deferred comment wakes only reopen completed issues for human/comment-reopen interactions, while system follow-ups leave closed work closed - updated adapter-utils prompt guidance to prefer targeted verification - added Codex model refresh support in the server route, registry, shared types, and agent config form - added adapter/server tests covering the new parsing, retry scheduling, and model-refresh behavior ## Verification - `pnpm exec vitest run --project @paperclipai/adapter-utils packages/adapter-utils/src/server-utils.test.ts` - `pnpm exec vitest run --project @paperclipai/adapter-claude-local packages/adapters/claude-local/src/server/parse.test.ts` - `pnpm exec vitest run --project @paperclipai/adapter-codex-local packages/adapters/codex-local/src/server/parse.test.ts` - `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/adapter-model-refresh-routes.test.ts server/src/__tests__/adapter-models.test.ts server/src/__tests__/claude-local-execute.test.ts server/src/__tests__/codex-local-execute.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/heartbeat-retry-scheduling.test.ts` ## Risks - Moderate behavior risk: retry classification affects whether runs auto-recover or block, so mistakes here could either suppress needed retries or over-retry real failures - Low workflow risk: deferred comment wake reopening is intentionally scoped to human/comment-reopen interactions so system follow-ups do not revive completed issues unexpectedly > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex GPT-5-based coding agent with tool use and code execution in the Codex CLI environment ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
09d0678840 |
[codex] Harden heartbeat scheduling and runtime controls (#4223)
## Thinking Path > - Paperclip orchestrates AI agents through issue checkout, heartbeat runs, routines, and auditable control-plane state > - The runtime path has to recover from lost local processes, transient adapter failures, blocked dependencies, and routine coalescing without stranding work > - The existing branch carried several reliability fixes across heartbeat scheduling, issue runtime controls, routine dispatch, and operator-facing run state > - These changes belong together because they share backend contracts, migrations, and runtime status semantics > - This pull request groups the control-plane/runtime slice so it can merge independently from board UI polish and adapter sandbox work > - The benefit is safer heartbeat recovery, clearer runtime controls, and more predictable recurring execution behavior ## What Changed - Adds bounded heartbeat retry scheduling, scheduled retry state, and Codex transient failure recovery handling. - Tightens heartbeat process recovery, blocker wake behavior, issue comment wake handling, routine dispatch coalescing, and activity/dashboard bounds. - Adds runtime-control MCP tools and Paperclip skill docs for issue workspace runtime management. - Adds migrations `0061_lively_thor_girl.sql` and `0062_routine_run_dispatch_fingerprint.sql`. - Surfaces retry state in run ledger/agent UI and keeps related shared types synchronized. ## Verification - `pnpm exec vitest run server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/routines-service.test.ts` - `pnpm exec vitest run src/tools.test.ts` from `packages/mcp-server` ## Risks - Medium risk: this touches heartbeat recovery and routine dispatch, which are central execution paths. - Migration order matters if split branches land out of order: merge this PR before branches that assume the new runtime/routine fields. - Runtime retry behavior should be watched in CI and in local operator smoke tests because it changes how transient failures are resumed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, GPT-5-based coding agent runtime, shell/git tool use enabled. Exact hosted model build and context window are not exposed in this Paperclip heartbeat environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Dotta
|
c566a9236c | fix: harden heartbeat and adapter runtime workflows | ||
|
Brandon WooandPaperclip
|
15bd2ef349 |
fix: recognize missing-rollout Codex resume error as stale session
The Codex CLI can return "no rollout found for thread id ..." when resuming a heartbeat thread whose rollout has been garbage-collected. Extend isCodexUnknownSessionError() to match this wording so the existing single-retry path in execute.ts activates correctly. Add parse.test.ts covering the new pattern, existing stale-session wordings, parseCodexJsonl, and a negative case. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 4.6
|
f60c1001ec |
refactor: rename packages to @paperclipai and CLI binary to paperclipai
Rename all workspace packages from @paperclip/* to @paperclipai/* and the CLI binary from `paperclip` to `paperclipai` in preparation for npm publishing. Bump CLI version to 0.1.0 and add package metadata (description, keywords, license, repository, files). Update all imports, documentation, user-facing messages, and tests accordingly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
ForgottenandClaude Opus 4.6
|
6e335b3fd0 |
Improve codex-local adapter: skill injection, stdin piping, and error parsing
Codex adapter now auto-injects Paperclip skills into ~/.codex/skills, pipes prompts via stdin instead of passing as CLI args, filters out noisy rollout stderr warnings, and extracts error/turn.failed messages from JSONL output. Also broadens stale session detection for rollout path errors. Claude-local adapter gets the same template vars (agentId, companyId, runId) that codex-local already had. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
ForgottenandClaude Opus 4.6
|
d56618e9fe |
Add adapter session codecs with cwd-aware resume and unknown-session retry
Introduce AdapterSessionCodec interface for structured session serialization, deserialization, and display ID extraction. Implement codecs for claude_local and codex_local adapters with cwd validation — sessions saved for a different working directory are not resumed. Both adapters now return sessionParams and sessionDisplayId alongside legacy sessionId. Add isCodexUnknownSessionError detection and automatic retry with fresh session for codex_local (matching existing claude_local behavior). Inject approval context env vars (PAPERCLIP_APPROVAL_ID, PAPERCLIP_APPROVAL_STATUS, PAPERCLIP_LINKED_ISSUE_IDS) into adapter environments. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
ForgottenandClaude Opus 4.6
|
631c859b89 |
Move adapter implementations into shared workspace packages
Extract claude-local and codex-local adapter code from cli/server/ui into packages/adapters/ and packages/adapter-utils/. CLI, server, and UI now import shared adapter logic instead of duplicating it. Removes ~1100 lines of duplicated code across packages. Register new packages in pnpm workspace. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |