mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
d5c4f7dd02d408a261cedd9a1926b9900a4aedc4
1974
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
d5c4f7dd02 |
fix: preserve shared Git sources during online workspace migration (#19688)
## Thinking Path > - Paperclip preserves the work produced by its agents. > - Remote workspace migration must preserve legacy Git history and local edits. > - Linked checkouts can share administration and objects with other tasks. > - A task lock alone cannot exclude writers of a shared or borrowed store. > - The online source owner must derive and retain the full physical ownership closure. > - This change reuses task, project, lifecycle and finalizer locks for that closure. > - Idle legacy Git work can then migrate while unrelated remote turns continue. ## Linked Issues or Issue Description Refs: #19642. **What happened?** Online migration keeps legacy Git sources blocked because several task checkouts can share a project base or a private Git cache. A read-only source observation found 160 pending Git records, 120 distinct paths and nine common stores. Its largest project cohort contained 126 records and 110 task owners. An ordinary-directory fence cannot preserve these sources by itself. **Expected behavior** Derive the physical source owners from stable Git metadata. Hold their existing source and lifecycle keys through publication. Retain all history and recovery obligations. Reject foreign aliases, unknown stores and changed sources. Let unrelated object-backed work continue. **Steps to reproduce** 1. Create a managed project checkout and linked task worktrees. 2. Keep a settled prepared recovery source on a sibling task. 3. Keep unrelated remote work running. 4. Request an online import of an idle leaf checkout. 5. The existing owner keeps the shared source blocked. **Paperclip version or commit** This branch builds on merged #19642 (`7ac99718a4`) and fresh master `fb3cf1a072`. It adds the Git closure. It does not add a writers-stopped shortcut. **Deployment mode** Hosted native Runner with remote workspace storage. Local qualification uses owned synthetic Git repositories, disposable PostgreSQL databases and the actual archive and storage owners. ## What Changed - Read bounded stable Git pointers, common-directory metadata and local alternates. Derive the complete physical source ownership cohort. Preserve all object stores. - Retain common and borrowed store owners in preparation, lifecycle, runtime, expiration and actual reopen paths. Keep company boundaries and the existing 128-key bound. - Hold exact sibling native recovery proofs under their existing physical finalizer keys. Recheck descriptors, allocations, recovery actions and source identities before capture and publication. Preserve original states and warnings. - Require leaf-first nested imports. Reuse the existing immutable child download, restore and source comparison before parent exclusion. - Bind the self-contained archive owner to the held Git namespaces. Keep its three walks, history normalization, fsck, bytes and digest format. Reject bare, detached, network, unknown or changed Git storage. - Check both foreign path fields. Keep foreign metadata private. Document the proof, bounds, source observation and remaining eligibility limits. ## Verification - Final revision `2934e41c923b93e16508b978c849b747c86be645`: all checks are terminal, with 56 successful and four skipped, zero failures and zero pending. The [required CI aggregate](https://github.com/paperclipai/paperclip/actions/runs/42326464408/job/128266279210) passed. Build, types, all server/Runner/serialized suites, end-to-end aggregate and Snyk passed. [Fresh exact-head Greptile review](https://github.com/paperclipai/paperclip/pull/19688#issuecomment-6699760104) is 5/5 with no open threads. GitHub reports MERGEABLE and CLEAN. - Review then found two correctness gaps: an ordinary managed directory inside an unrelated host repository was blocked, and a Git parent could miss a completed nested operation after its child snapshot became missing or deleted. Three actual-owner regressions reproduced both gaps in 9.70 seconds. The corrected owner passes 12 focused cases in 17.08 seconds. It imports and restores only the ordinary root bytes, preserves the outer repository, rejects both missing/deleted child receipts before a parent journal, and retains valid/changed child coverage and private-cache contention. Its wrapper exited 0 and cleanup completed. The first bootstrap attempt failed before tests ran because the host exhausted its 32 shared-memory segment limit; no shared services were stopped and no pass is claimed for that attempt. The final qualified source/test type graph, module boundaries and diff checks pass. - Receipt scans keep canonical path resolution. A lexical SQL prefix can omit children reached through symlinked ancestors. The observed full metadata closure completed in 1.17 seconds; no measured bottleneck justifies weakening that alias proof. Writer admission still rejects uncertain Git ownership before mutation. Expiration returns its typed skip, and reopen returns its structured failure. Runtime and lifecycle do not bypass an unknown source. No broad error-contract change was added. - Full CI at `720104ea975e` exposed five existing boundary cases across three server shards. Four exact cases reproduced locally as failures in 25.98 seconds. Reopen now preserves its structured failure when source inspection rejects the path, without reconstructing it. Runtime recognizes only the exact regular ignore-only `.git/info/exclude` fragment; migration still rejects that fragment. The original arbitrary-path and file-URL import/delete-host/resume fixtures keep both supported input forms and use the genuine offline setup path after their sole writer finishes. The production mode gate is unchanged. A new real-owner control proves the offline flag still rejects a running legacy database writer. - The final focused run passed all ten cases across four suites in 22.75 seconds, including all five CI failures, invalid-pointer admission, fragment rejection, offline writer exclusion and actual nested reopen lock ordering. Its wrapper exited 0 and cleanup completed. The known invalid-pointer case remains fail-closed; the separate unexpected-I/O diagnostic retains safe error metadata without command text or credentials. Existing Git stderr redaction coverage remains intact. The final metadata reader suite passed 20 cases in 2.68 seconds. The qualified server/test source type graph, module boundaries and diff checks pass. No production assertions or timeout defaults were weakened. Fresh CI and review are required for the repair revision. - A later review found that the ordinary early return still used only one foreign path field. The actual ordinary providerRef-only regression failed as incorrectly eligible. One source-path owner now enumerates both local locators for company/foreign ordinary and Git scans, layout checks, early admission, lifecycle/reopen/runtime and expiration/move/preparation retention. Capture and display keep their original preferred path. The actual owner canary passed 49 cases across two suites in 109.15 seconds. It covers cwd-only/ref-only/different/symlink locators, same-company mismatches, foreign owners, lifecycle ordering, Git/recovery imports and expiration/reopen. Three selected-locator fallback cases also failed before their guard; the final eight-case control passed in 31.05 seconds, including the configured alias and valid import/private-child controls. All wrappers exited 0 and completed cleanup. The metadata/locator suite passed 19 cases. The final source/test type graph, module boundaries and diff checks pass. - Review found two P1 gaps. Both configured-directory symlink cases failed before canonicalization. The actual parent import lost an already-imported child's unique private cache and failed its archive fence instead of retaining that owner. The corrected real-owner canary passed four cases in 20.78 seconds: configured symlink runtime provisioning, private-cache contention before the parent journal, successful parent publication after release, and unchanged/changed child coverage. The metadata suite passed 15 cases in 4.14 seconds. Both wrappers exited 0 and cleanup completed. - A fresh source read-only observation found 160 pending Git records and nine common stores. An additional exact receipt query found 1,243 completed migration receipts and 557 nested source paths under pending Git roots. All 126 records in the largest project were present in the captured Git metadata set. For all 110 present candidate workspace records, the derived closure fits the unchanged 128-key cap. The worst case needs 110 issue keys and one project key, with 17 keys of headroom. Its 173 imported children introduce no physical issue owner outside those 110 logical task owners. This checks the observed key bound only. It does not claim migration eligibility or a coherent snapshot. - The initial stacked CI did not run four Runner test lanes: trusted master passed new --package-lane/--native-server options to the old base lane script. The branch is now rebased onto actual merged master and retargeted before synchronization. It includes the canonical current lane parser without an unrelated Runner patch. Fresh new-head CI is required. - The actual importer/source owner run passed 36 cases across three suites in 187.75 seconds. It covers sibling prepared recovery, unrelated running remote work, common-cache contention, changed coordinator/allocation/path state during upload, exact nested coverage, dirty unpushed work, and existing expiration/reopen owners. Wrapper exit 0 and cleanup completed. - A separate previously hanging lifecycle-worker restore passed in 17.33 seconds with the unchanged 15-second test-body limit. An earlier run timed out in three expiration/reopen cases and blocked cleanup; its wrapper later exited 1. These failures remain recorded. They are not counted as passes. - Two boundary controls first failed: a foreign providerRef-only alias was incorrectly eligible, and actual reopen did not admit its distinct cache owner at the initial source boundary. The narrow guards now inspect both foreign locators and derive physical Git keys before the raw lifecycle transaction. The foreign alias and existing real nested reopen contention passed. The private-cache control first retained contradictory recorded source provenance and stayed fail-closed after release. The corrected present legacy-worktree fixture passed in 12.61 seconds. These failed attempts remain recorded; they are not passes. - The metadata reader and serialized/local archive controls use actual Git repositories. They compare archive bytes and digests, remove every original source, restore staged and unstaged work, and run fsck. The metadata reader passed 13 cases in 2.85 seconds. The local/serialized archive controls passed nine cases in 9.27 seconds. All processes exited 0. An additional actual importer control changed a relative commondir pointer to the absolute name of the same store during upload. Publication was rejected, the uploaded journal remained available, and original files stayed intact (one case, 5.86 seconds; wrapper exit 0 and cleanup complete). - The final rebased full local server/test type graph passes with db/shared/Runner source aliases because reused dependency declarations predate new master exports. The first rebased attempt reported those stale declaration fields/tables, and is not counted as a pass. Module boundaries and diff checks pass. Standard package builds, types and broad regression checks remain required in CI. - No hosted import, admission hold, cancellation, provider mutation, source filesystem write or shared-cache Git command was performed for this change. The source observation is read-only and is not a locked eligibility proof. ## Risks - A missing writer owner could make physical source exclusion incomplete. Existing mutation owners retain the same company/task/project keys, and uncertain metadata fails closed. - A project can contain a large cohort. Limits remain finite. A bound failure keeps the source blocked. - Bare, detached, unowned or network Git stores remain blocked online. Missing archived sources retain their existing authenticated reconstruction path. - Exact recovery proofs preserve authority and obligations. They do not infer provider termination or mark a prepared descriptor finalized. - This change does not claim all observed Git records are immediately eligible. Existing preparation, provider, recovery, runtime and source-change blockers remain. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
30857db55e |
fix(native): recover a run killed between runner launch and its identity record (#19694)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd` process. The server records the request to start it, then records its process identity after it spawns. > - If the server is killed between those two records, there is no process ID to check after the restart > - Restart recovery then restarted the bootstrap, but the executor failed with `durable_identity_unreadable` because the runner never wrote its state > - The user's Retry then waited forever with "Retry is waiting for Paperclip to verify that the previous execution stopped", because the stop check needs a process ID > - This pull request restarts that bootstrap and proves on the host that no runner for the run exists > - The benefit is that a restart in the runner launch window no longer loses the task or leaves Retry permanently stuck ## Linked Issues or Issue Description **What happened?** In restart-chaos testing, an ACPX Claude run was created and the server was SIGKILLed 1.3 s later, right after `native.process_start_requested` and before the runner identity was recorded. On restart, recovery claimed `bootstrap_incomplete` (`dead_runner_before_provider_identity`). The executor then threw `runner_state_identity_mismatch: durable_identity_unreadable`: the state root existed, but `control-plane/control-plane-state.json` had never been written. The run failed with `native_continuation_requires_reconciliation`. A Retry was accepted (HTTP 200), but no run was ever created. The issue stayed blocked with `retryWait.reason=process_identity_missing` for over 20 minutes and through another restart. **Expected behavior** When the runner never wrote state, no provider started, so the bootstrap can start again. A Retry should be admitted once the host proves that no runner for the run exists. **Paperclip version or commit** `master` at `fb3cf1a072`. **Steps to reproduce** Create a native local task and SIGKILL the server about 1 s after the run starts, after the runner launch is requested and before its identity is recorded. Restart the server, then click Retry. ## What Changed - `migrateRunnerdStateRootForExecution` (`native-session-executor.ts`): for a `bootstrap_incomplete` restart on a local target, a root is quarantined (kept for inspection) and the bootstrap starts again only when both of these hold: - the root has no `control-plane-state.json`; - no live runner for the run exists on this host. A state file that exists but cannot be read, or a runner that started but has not connected yet, still fails closed with `durable_identity_unreadable`. - `unidentifiedLocalRunnerProvenAbsent` (`native-session-executor.ts`) proves that a runner is absent when both of these hold: - no host process carries the run's `--run-id <runId>` argument (`paperclip-runnerd` is always started with it, detached and unreferenced); - no saved control-plane state for this run exists, in the canonical root or in quarantine. A provider starts only after its runner connects and writes that state. - `native.process_start_requested` now records the launching `hostname`. - `recordUnidentifiedLocalRunnerStop` (`native-local-process-stop.ts`) applies the proof only when all of these hold: - the run is native; - the run has no process ID; - its latest launch event is `native.process_start_requested`, from this host; - all of its leases are local. A launch recorded without a host stays blocked. - The process listing runs asynchronously with a 5 s deadline. A failed or slow listing proves nothing. It then writes `native.local_process_stopped` with null process IDs and `basis: "unidentified_runner_absent"`. A later launch request invalidates it as usual. - The continuation and Retry gate (`explicit-native-continuation.ts`) uses this proof before returning `process_identity_missing`. A dry run checks the proof but records nothing. - `doc/run-log-events.md` documents the new receipt basis. ## Verification - `native-session-executor.test.ts`: - The bootstrap restart with absent state was red before the fix (`durable_identity_unreadable`) and is green after it (root quarantined, backend admitted). - Unreadable state, and a live runner that has not connected, still reject. - Proof cases: absent → true; a live process with `--run-id <runId>` → false; this run's saved state → false; another run ID → false. - `explicit-native-continuation.test.ts` (embedded Postgres, real admission): - The `absent` and `dry_run` cases were red before the fix (blocked) and are green after it (admitted). Only the real admission records the receipt. - The `live_runner`, `saved_state`, `foreign_host` and `unbound_launch` cases stay blocked. - Broader local run after the review fixes: `explicit-native-continuation`, `native-session-executor`, `native-recovered-owner-followup`, `native-runner-restart-recovery.integration` and `task-retry` gave 1836 passed, 0 failed. - A chaos replay of the original task (`--replay w-20261106T232854-s7103/t26`) passed (R1 PASS). Its kill landed after the runner identity was recorded (`resume_dead_runner`), not in the launch window, so the tests above are what reproduce that window. - `tsc --noEmit` for `server` passes. ## Risks - The proof depends on `ps -axww` listing every process on the launching host. If `ps` fails or times out, the proof fails closed (still blocked). Remote runners and other hosts are excluded. - A runner that was spawned but has not yet written its state could be mid-start when admission runs. It would still carry `--run-id` in its arguments, so the proof refuses. - Only the bootstrap path with no state file changes. Ownership, checkpoint and reattach checks are unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c37bc749ac |
fix(native): keep restart continuations alive through early results and repeat restarts (#19699)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs continue a turn lost to a control-plane restart once, in the same conversation (#15297, #19501) > - In restart storm 3, two Codex runs each crossed one restart and still needed a Retry > - The continuation was attempted, but the driver failed it with `turn_binding_mismatch`, and the run then reported "runnerd exited before provider turn startup" > - A second restart that loses the continuation turn itself also needed a Retry, because the one-shot marker was already used > - A restart should not cost a click when continuing is safe, and a failure should name its real cause > - This pull request fixes the start race, reports the driver's own failure, and gives each admitted restart its own continuation > - The benefit is that interrupted tasks finish without a Retry, within the existing recovery budget ## Linked Issues or Issue Description Refs #19501, #15297, #19658, #19539 **What happened?** Restart storm 3 (seed 7103, RC06 mask on, load 9–12). Codex B5 Retry runs crossed one K1 restart mid-turn. After the restart, recovery resumed the dead runner. The runner restored the conversation and recorded `turn.failed` with `provider_turn_lost_on_restore` (`recoverable: true`). The control plane started the same-conversation continuation (`resumeInterruptedTurn` → `startTurn`). About 1 s later the transport closed with `protocol_failure:turn_binding_mismatch`. The run failed as `native_runner_process_exited: runnerd exited before provider turn startup`. The task needed another Retry. I reproduced it with the focused cell (`--workloads B5 --faults K1 --triggers retry`) and a temporary diagnostic. The failure is "Provider semantic result did not name the active turn": a `paperclip/runResult` reached the Codex driver while the continuation's `turn/start` was still pending and no turn was bound. The driver waits for a pending start before it handles `turn/completed`, but not before it handles `paperclip/runResult`. **Expected behavior** After a restart that loses a turn, the run continues automatically when that is safe. Each admitted restart attempt can continue once, within the provider-attempt budget. A failure names its real cause. **Steps to reproduce** 1. Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads B5 --faults K1 --triggers retry --tasks 2 --concurrency 2 --seed 4040 --rc06-masked`. 2. When a lost turn's continuation receives a semantic result before its start settles, the run fails with `turn_binding_mismatch` and the task needs a Retry. The race is intermittent: 1 of about 14 lost turns in my runs. **Root cause** - `packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts`: the `paperclip/runResult` branch checked the active-turn binding without waiting for `state.turnStartSettled`. The `turn/completed` branch already waits. - `codex-harness-session.ts` `startTurn`: the driver's own protocol failure closed the transport. The pending start then reported the runner exit, which hid the cause. - `codex-session-terminal.ts` `mapTerminalTurn`: a consumed one-shot marker blocked every later continuation in the run, even when the newly lost turn was that continuation itself. The one-continuation allowance is per run, not per issue or session lineage: attaching a new run resets the marker. The storm runs did attempt their continuation; the start race made it fail. ## What Changed - Codex driver: a `paperclip/runResult` waits for a pending turn start to settle before it is bound, like a terminal. - Codex driver: when this session's own protocol failure ends a pending turn start, `startTurn` throws `codex_turn_start_protocol_failure: <code>: <message>` instead of the runner exit. CodexRpc errors and protocol-integrity errors keep their own types. The server now classifies this as `native_session_interrupted` instead of `native_runner_process_exited`; both are retryable. - A later restart that loses the continuation turn itself re-arms the one-shot marker: the driver resets it as it was before the first continuation. It does this when it maps the lost terminal, including during recovery from a checkpoint taken while the continuation ran. The checkpoint gate `nativeRestartInterruptedTurnId` stays strict: a consumed marker still means the allowance is used. Each continuation still needs an admitted dead-runner restart. The existing provider-attempt budget bounds the repeats, and an exhausted budget still blocks with Retry. - `doc/execution-semantics.md`: an additive paragraph. ## Verification Red before the fix, green after: - `codex-app-server-driver.events.test.ts` "binds a semantic result that arrives before its pending turn start settles". Red: `session.failed`. Green: the result binds to `turn-1` and the turn completes. - `codex-app-server-driver.events.test.ts` "names its own protocol failure when that failure ends a pending turn start". Red: "runnerd exited before provider turn startup". Green: `codex_turn_start_protocol_failure: turn_binding_mismatch`. - `codex-app-server-driver.recovery.test.ts` "offers another restart continuation when a later restart loses the continuation turn itself". Red: `null`. Green: `turn-2`. A checkpoint with a consumed marker still returns `null`. - `codex-app-server-driver.recovery.test.ts` "re-arms the continuation when recovery finds the checkpointed continuation turn lost". This recovers from a checkpoint taken during the continuation, then starts the next continuation. Red without the re-arm (the marker stays consumed); green with it. - `native-runner-restart-recovery.integration.test.ts`: the same checkpoint (continuation active, consumed marker naming it) is claimed as `resume_dead_runner`. This is regression coverage. Suites run locally on the final code: - Runner: Codex driver, `native-session-runtime`, `runnerd-restart-continuation` and contracts (745 tests). - Server: `native-runner-restart-recovery.integration.test.ts`. - `tsc` for the runner tests and the server. - A few real-process tests timed out at about 6 s while six live agents ran. Each passed when rerun alone. The settlement suite passed both with and without this change at the same load. Live (local, real Codex, macOS arm64), focused restart-chaos cell `--adapters codex --workloads B5 --faults K1 --triggers retry,time --tasks 6 --seed 4042 --rc06-masked`: 6/6 crossed a restart and passed. R1 passed for all 6, so each continued without a Retry. ## Risks - A run can now continue more than once in the same conversation, once per admitted restart. Each continuation asks the agent to reconcile unfinished actions before repeating them. The provider-attempt budget still caps the repeats. - A semantic result that arrives during a pending start now waits for that start. If the start fails, the result is still rejected by the existing binding check. - The start race was intermittent (1 of about 14 lost turns), so the live cell shows absence of the failure, not a forced repro. The driver regression reproduces it deterministically. - No schema, API or migration change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
11c07c622c |
fix(native): restart a remote bootstrap retry that shutdown cut before runner launch (#19684)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. Restart recovery decides what to do with each live or due run
after the control plane restarts.
> - #19616 added the `remote_runner_never_launched` proof. A claimed
remote run with no runner process request and no controller checkpoint
starts its bootstrap again.
> - A graceful shutdown closes the local runtime listener before its
drain. A remote run that is still starting then fails with
`local_runtime_api_listener_unavailable`. The executor settles it as a
same-run `bootstrap_retry` and releases its lease. #19596 treats this as
a restart detach only for local runners.
> - When the server restarted with that retry due, restart recovery
claimed the row. The proof accepted only rows that still held a lease,
so recovery blocked the task as ambiguous.
> - This pull request lets the same proof also admit that settled
`bootstrap_retry` row.
> - The benefit is that a graceful restart during Daytona startup
continues automatically.
## Linked Issues or Issue Description
**What happened?**
A chaos wave restarted the control plane gracefully (SIGTERM) while an
ACPX Claude run on Daytona was starting. At 2 s after `stage.workspace`,
the run failed with `local_runtime_api_listener_unavailable`, and the
executor persisted "same-run resume persisted" with `recoveryMode:
bootstrap_retry`. After the restart, restart recovery returned
`ambiguous_remote_recovery_evidence` with disposition `blocked`. The
task rested blocked.
Evidence in the saved database at that point:
- Coordinator `phase: retryable_failure`, `attempt: 1`,
`controllerGeneration: 1`, controller identity recorded, and no lease
owner.
- `failureDetail` recorded `recoveryMode: bootstrap_retry`,
`checkpointExists: false`, `providerEventsExist: false` and
`providerSessionEstablished: false`.
- The run had no runner checkpoint rows, no
`native.process_start_requested`, no process identity, and no
runner-sourced event. It also had no PID and no provider session.
**Expected behavior**
Restart recovery starts the bootstrap again, as it does for a claimed
run that holds a lease (#19616) and for a local run.
**Steps to reproduce**
1. Start a native run on a Daytona environment.
2. Send SIGTERM to the control plane after workspace staging and before
the runner is requested. The run settles as a same-run
`bootstrap_retry`.
3. Restart the control plane while that retry is due.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.
Root cause: in
`server/src/services/native-runtime/native-restart-recovery.ts`, the
`remoteRunnerNeverLaunched` condition required `phase === "observed"`
and a lease owner. The settled `retryable_failure` / `bootstrap_retry`
row failed that check. The remote classifier then returned
`ambiguous_remote_recovery_evidence`.
Refs #19616, #19596, #19583
## What Changed
- `native-restart-recovery.ts`: `remoteRunnerNeverLaunched` now accepts
one of two coordinator shapes:
- an `observed` claim that holds a lease (unchanged);
- a `retryable_failure` with `recoveryMode: bootstrap_retry`, run status
`failed`, and no lease owner.
- All other proof conditions are unchanged. The run must still have no
process start, process identity, adapter, tool or runner-sourced event.
It must still have no process fields, no provider session, no provider
evidence and no checkpoint. The same three-attempt budget applies.
- An `exact_checkpoint_resume` retry keeps its own admission path.
- Admission: before claiming a settled retry, restart recovery applies
the same admission as the resumption reconciler
(`readNativeSessionResumptionAgent`): task pause hold, agent status,
dependencies, ownership, and competing runs. An ineligible retry is
deferred with `native_retry_admission_ineligible` and keeps waiting.
- `doc/run-log-events.md`: note the settled `bootstrap_retry` case.
## Verification
- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "bootstrap retry"`. Before the fix, the proven case and the budget
case returned `blocked` / `ambiguous_remote_recovery_evidence`.
- New cases: the proven case gets `bootstrap_incomplete`. The
coordinator returns to `observed`, `nextAttemptAt` is cleared, the run
returns to `running`, and no recovery action is created. A requested
process start or a committed runner event keeps the hold. A spent budget
blocks with `execution_recovery_budget_exhausted`. An
`exact_checkpoint_resume` retry does not take this path.
- Regression: `native-runner-restart-recovery.integration`,
`heartbeat/recovery`, `native-restart-recovery`,
`native-bootstrap-retry.integration` and
`native-remote-retry-admission`: 337 passed, 14 skipped.
- Admission: a paused agent's due `bootstrap_retry` is deferred with
`native_retry_admission_ineligible`. It is not started (red before the
follow-up commit, green after).
- Live Daytona replay of `w-20261106T234719-s12/t02`, 3 runs, ACPX
Claude B1, using the recorded K4, K2 and K1 sequence. Image: the wave's
own `paperclip-daytona-runner@sha256:2767e91c…`, with runner inputs
pinned to its revision
|
||
|
DottaandClaude Opus 5.5
|
9a52a2dfeb |
fix(native): retry ACPX session.open startup timeouts and give cold Claude a 60 s bound (#19661)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs start an ACPX provider through Runner. `session.open` opens the provider session before any prompt. > - On a loaded host, a cold ACPX Claude `session.open` took longer than its 30-second bound. The controller reported `runner_local_connect_failed: PRP command session.open timed out`. > - The server classified this as the generic `native_session_interrupted` with `ambiguous_state`, so the task was blocked at once. A person had to click Retry, sometimes several times. In one restart-chaos storm, one task needed three Retry clicks, and on another task both Retry runs timed out too. > - The server already has a bounded same-run bootstrap retry for failures that come before provider work (#18854). It did not recognize this timeout. The 30-second bound also cut off normal slow startups. > - This pull request gives cold ACPX Claude a 60-second `session.open` bound, like Pi and OpenCode (#19487). It also retries a startup `session.open` timeout automatically on the same run, from clean state. > - The benefit is that a slow cold start on a busy host completes, or retries itself, instead of waiting for a person. ## Linked Issues or Issue Description Refs #18854 (bounded bootstrap retry for definite `session.open` failures), #19487 (60-second cold OpenCode admission), #19665 (the bootstrap proof that this change reuses). **What happened?** ACPX Claude runs failed with `runner_local_connect_failed: PRP command session.open timed out` about 35 to 40 seconds after they started. The coordinator recorded `native_session_interrupted` with `recoveryMode: "ambiguous_state"` and `providerSessionEstablished: false`. The task was blocked with a board-owned action. No provider event and no checkpoint existed. The host 1-minute load was 17 to 44, sometimes with no restart nearby. **Expected behavior** A normal slow cold start completes. A `session.open` that still times out during runner startup, with no provider event, no checkpoint and no turn, retries on the same run with backoff. At most two automatic attempts follow the first. A run where provider work may have started is never retried automatically. **Steps to reproduce** 1. Run ACPX Claude tasks on a host with a 1-minute load of about 25 or more. 2. Observe a run fail after about 35 to 40 seconds with `PRP command session.open timed out`. 3. Observe the task blocked with `ambiguous_state` and a manual Retry. **Paperclip version or commit** `master` at `6c7cc2c5d6`. ## What Changed The bound (evidence below): - `coldAdmissionTimeoutMs` gives cold ACPX Claude 60 seconds (live adoption keeps 30 seconds). The runner's Rust sidecar `session_open_timeout` and the sidecar's provider handshake (`acpxSessionHandshakeTimeoutMs`) give Claude and Pi 60 seconds too. All three bounds must agree, or the shortest one fails first. Claude's warm run attachment uses the same cold budget, like Pi. The automatic retry: - `nativeSessionFailureSourceCode` returns `provider_session_open_timeout` only for exactly `<startup failure code>: PRP command session.open timed out`. The startup failure codes are `runner_local_connect_failed`, `runner_direct_wss_failed` and `runner_ingress_unavailable`. The transport uses them only before startup completes. After startup the code is `provider_transport_failed`, which is unchanged. - `nativeProviderRecoveryEvidence` treats the cause as pre-session. Durable evidence still decides: a bootstrap retry needs no provider event (`session.started`, `turn.started` and the others) and no checkpoint. The existing budget applies: three attempts with backoff of about 30 seconds, then about 60 seconds. When the budget is used, the task is blocked with the existing "Provider startup failed on 3 attempts before any provider work began" action. - A timeout is indeterminate: the runner can open a provider session after the controller stops waiting. Live runs showed both leftovers, so the retry now starts from clean state: - `prepareNativeSessionOpenTimeoutRetry` (server) runs when the failure settles. It quarantines a local runner root only when the restart-recovery proof (`runnerdStateProvesIncompleteBootstrap`, #19665) shows that no turn began and every recorded process is gone. Any other root, and any remote runner, stays `ambiguous_state` as before. - `resetAbandonedBootstrapRecord` (runner ACPX adapter) runs on a fresh open (no runner-recorded identity). If ACPX has a saved record for the session key with no messages, no prompt and no events, it is marked `reset_on_next_ensure`. Claude saves nothing before the first prompt, so reusing such a record fails the first turn with `session_resume_required`. A record that holds any conversation is unchanged. - A chat retry of an exhausted `provider_session_open_timeout` run is accepted like the other pre-provider causes. - `doc/execution-semantics.md` and the Runner README describe the cause and the bound. ## Verification **Measured startup distribution** (ACPX Claude `runner.session.bootstrap` / `runner.session.startup` spans from restart-chaos run events, 1-minute load 8 to 44): | Source | Bound | Successful cold bootstraps (s) | Failures | | --- | --- | --- | --- | | Earlier storm evidence | 30 s | 2.9, 4.7, 7.4, 8.6, 8.7, 26.2, 27.4, 27.6, 29.2 | startups ended at 34.9 to 40.3 s (the 30 s `session.open` bound plus spawn and `run.prepare`) | | This branch, wave s3601 | 60 s | 6.3, 7.8, 8.1, 13.4, 14.4, 22.3, 22.8, 22.9, **37.3** | 0 `session.open` timeouts | | This branch, wave s3602 | 60 s | 13 samples, p50 10.5, p90 17.9, max 25.7 | 0 `session.open` timeouts | | This branch, wave s3610 (final head, load 28 to 42) | 60 s | 10.4, 11.3, 11.7, 12.0, 13.3, 14.2, 29.6, **30.4**, **30.9** | 0 `session.open` timeouts | Under load, successes clustered just under 30 s and failures sat just above it, so the 30 s bound was cutting the distribution off. With 60 s, cold starts of 30.4, 30.9 and 37.3 s succeeded that the old bound would have failed. 60 s is about 1.6 times the largest observed success, and it matches Pi and OpenCode. **Live restart-chaos waves** (local, ACPX Claude only, 4 in flight, `--rc06-masked`, real provider, K1/K2 restarts): - s3601 (8 tasks, load 19 to 32): 7/8 PASS. No `session.open` timeout. The one failure is an R3 Retry signature after K1 (`native_restart_recovery_blocked`), not this cause. - s3602 (8 tasks, load 17 to 39): 8/8 PASS. No `session.open` timeout. Three Retry clicks were for `native_provider_terminal_failed` (`provider_turn_lost_on_restore`), a different restart signature. - Forced timeout, s3609 (4 tasks, B5, K2): a local, uncommitted fault injection gave every other cold startup a 1.5 s `session.open` deadline. Result 4/4 PASS. Run 4fe6537a timed out, retried itself on the same run, and then succeeded through four K2 restarts with no person involved. Run 02799a85 hit the injection three times in a row; the budget stopped it after attempt 3, and one Retry completed it. - Before the two clean-state fixes, the same forced wave (s3607 and s3608) showed the retry reusing leftover state: `turn.start` failed with `session_resume_required`, and then "ACPX session is not open". Those runs are the red evidence for the two clean-state fixes. - Final head, s3610 (8 tasks, K1/K2, no injection, load 28 to 42): **8/8 PASS**, 0 `session.open` timeouts. The one Retry click (t06) was for `provider_turn_lost_on_restore`, a different restart signature. **Tests:** - Red before the fix: `native-startup-retry.integration.test.ts -t timeout` (embedded PostgreSQL, real executor, real resume claim). The 3 retry cases failed with `native_session_interrupted` / `ambiguous_state`. - Green: `native-startup-retry.integration.test.ts` passes 36/36. New cases: - for each startup failure code, a timeout retries on the same run with backoff, keeps the task `in_progress` with an agent-owned `resume_native_run` wake, and blocks after attempt 3 with a board-owned action; - no automatic retry when `session.started` or `turn.started` is durable, when the code is `provider_transport_failed`, or when another command timed out; - a real runner root with an opened session and no turn is quarantined; a root with a turn command stays `ambiguous_state` and intact; a remote runner is never prepared. - `codex-runtime-adapter.test.ts`: 113 pass. New cases: a 45 s delayed handshake is admitted for Claude and rejected for Codex; a never-prompted record is reset only on a fresh open; records with messages, a prompt or events are kept. - `abandoned-bootstrap-record.test.ts` (real `createAcpRuntime` with `createRuntimeStore` and a controlled ACP peer that, like Claude, cannot load a never-prompted session): without the reset, a fresh open reuses the record and the first turn fails with SESSION_RESUME_REQUIRED after `session/load`. With the reset, the provider receives `session/new` and the turn completes. - `vitest run src/drivers/acpx src/cli` in the runner: 107 files, 1,935 tests pass. - `native-session-executor.test.ts -t "session.open|startup|bootstrap"`: 144 pass. `runnerd-codex-transport.test.ts -t "cold process admission"`: 4 pass. `cargo test -p paperclip-runner-core --lib admission_budget` passes. - `npx tsc --noEmit -p server` and the runner `tsc -p tsconfig.test.json` pass. ## Risks - **This changes Runner Rust code. The Daytona runner image must be rebuilt after merge** for remote sandboxes to get the 60-second sidecar bound. Until then, a remote Claude `session.open` still fails at 30 seconds in the sidecar (as a definite failure, which already retries). A remote timeout stays operator-owned, because a remote root cannot be proven clean here. - A cold ACPX Claude start that hangs now fails after about 60 seconds instead of 30 seconds. - A timed-out `session.open` may have created an empty provider session. No prompt was sent. The retry quarantines that runner root and asks ACPX for a new session. - Same-run retries of a fresh-attempt Retry run can still fail with `isolated_task_directory_binding_mismatch`. This is RC-35 (workspace re-preparation), which has its own fix in progress. - No schema or API change. The new error code appears on runs and in run error codes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
b94963f91e |
fix: retry temporary repository catalog failures before dispatch (#19690)
## Thinking Path > - Paperclip manages AI agents and their work. > - A task must check repository access before it starts agent work. > - Repository discovery can fail while the user's existing grant remains valid. > - Discovery previously replaced a known temporary provider refusal with an empty catalog. > - The task then reported lost access and did not schedule recovery. > - This change keeps temporary discovery failures distinct and uses the existing bounded retry owner. > - Each retry must pass a fresh access check before it can use the saved repository. ## Linked Issues or Issue Description **What happened?** A task with a saved catalog repository stops with an access denial when GitHub returns an exhausted-quota response or HTTP 503. The task does not start provider work, but the discovery layer discards the temporary failure and returns an empty catalog. **Expected behavior** Known temporary failures before provider dispatch use the existing bounded infrastructure retry. The successor checks the same responsible user's current grants and reads the saved files only after fresh discovery succeeds. A completed denial remains terminal. **Steps to reproduce** 1. Bind a task to an authorized catalog repository with saved local changes. 2. Make its authorized GitHub discovery return HTTP 403 with zero remaining quota, or HTTP 503. 3. Let discovery recover before the existing retry deadline. 4. Observe one successful successor using the original repository bytes. **Paperclip version or commit** Reproduced on master at `b8d94b3ad1`. A search found no existing PR for this temporary catalog failure. ## What Changed - Keep a safe typed failure for known HTTP availability and quota responses. Honor the later provider deadline and bound it to one day. - Propagate that failure only when fresh discovery cannot find a required repository. Preserve successful partial catalogs and the existing connection setup error contract. - Classify the failure only before provider dispatch. Reuse existing bounded retry, interaction ownership, and bootstrap settlement rules. - Keep current grant checks, saved repository selection, normal Stop, and provider-started failure policy. - Keep provider deadlines when accepted interactions use their own retry owner. Recognize secondary HTTP 403 rate limits with valid retry advice while keeping ordinary permission denials terminal. This matches [GitHub rate-limit guidance](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api). - Keep direct repository request responses outside the automatic bootstrap retry contract. Test successful partial discovery and missing-required-repository recovery with two independent eligible connections. - Document task repository discovery recovery. ## Verification - Behavioral RED: real heartbeat and PostgreSQL fixtures reached actual HTTP 403 and503 discovery responses, started no adapter, and found zero successors instead of one. Both cases failed before the production change. - Behavioral GREEN: both cases waited for the real retry deadline, used the normal queue worker, ran the controlled adapter once after fresh discovery, and read the original saved bytes. Two cases passed in 74.73 seconds. No external model or provider request was made. - Deadline RED/GREEN: a combined Retry-After and quota-reset response initially selected the earlier deadline. The regression now selects the later deadline. - Initial repair at `d8d4714a18` (same production source as `018e8c52d345d79972d7907468080dc9908c299f`): seven affected files passed, 69 cases in 185.16 seconds. That test-only cleanup then passed all five real-database acceptance cases in 134.76 seconds. - Commands: `pnpm exec vitest run` with `heartbeat-repository-catalog-recovery.test.ts`, `github-catalog-timing.test.ts`, `github-grant-metadata.test.ts`, `project-repositories-persistence.test.ts`, `task-workspace-repositories-persistence.test.ts`, `github-repository-catalog-failure.test.ts`, and `services/heartbeat/retries.test.ts` from the server source tree. The acceptance file also ran fresh at `018e8c52d3`. A fresh five-case run after the P1 repairs passed at `f490ca6b78` in 122.06 seconds. The final eight-case file passed fresh at `e5b2efab11919467850f55fd6345317132b9b672` in 124.62 seconds, using real owned embedded PostgreSQL. It covers automatic recovery with saved bytes, genuine denial, provider-started classification, normal Stop, the direct HTTP route, and both partial-catalog outcomes. - Review repairs at `f490ca6b787cf77df96d3d6edfb2ab9115eb19a6`: both reported P1s reproduced RED. The real database interaction scheduler chose a retry after 24 seconds before the provider deadline of one hour. A secondary HTTP 403 with valid Retry-After lost its retry classification. The narrow fixes then passed 44 cases across scheduler, header, and metadata-contract files in 37.66 seconds. - A real direct request through the task workspace router and normal error middleware reproduced a 500 response instead of the existing 403. Required-ID retry mode now applies only to admission and its rechecks. The route and both two-connection partial-catalog cases then passed in 29.09 seconds. Both partial cases use actual owned grants and credential bindings. - All seven server type layers passed again at final head `e5b2efab11919467850f55fd6345317132b9b672`. The affected test type layer passed again after the test-only cleanup. Environment: macOS arm64, Node 26.4.0, pinned pnpm 9.15.4, Vitest 5.0.3, owned disposable PostgreSQL 18 on loopback. The current checkout's shared packages, plugin SDK, and Runner were built first. - One local acceptance run under concurrent owned test/typecheck load hit the unchanged 15-second Stop-case timeout. Its four other cases passed. The unchanged complete file then passed serially. The timeout receipt remains recorded; no assertion or timeout was weakened. - A provider-started failure retains the existing generic adapter retry policy. It does not receive the catalog no-provider certificate or a provider retry deadline. Fixture cleanup cancels its scheduled retry through the normal owner. - Final head `e5b2efab11919467850f55fd6345317132b9b672` has 59 successful checks, 3 allowed skips, no pending or failed check, and Greptile 5/5 with no open findings. [Completed CI workflow](https://github.com/paperclipai/paperclip/actions/runs/42322202977). - Additional local diagnostic for the late recovery-owner advisory: on unchanged `e5b2efab`, the dedicated scheduler returned `retry_exhausted` with attempt 3/max 2. The normal stranded sweeper then escalated once, created no extra run, and started no adapter, even after catalog access recovered. One case passed in 17.32 seconds. The existing failed-predecessor path delegates to that bounded scheduler and returns; it does not enter a new generic retry budget. No production registry change was made. This diagnostic was not a CI case. Its local harness is preserved separately at `74e3854eea3f7c4121fdcd9c8a8ad522cd8c7477` on `codex/catalog-exhaustion-acceptance`; its production source is identical to `e5b2efab`. - Module boundary and whitespace checks passed. ## Risks - A temporary failure from another eligible connection can delay a truly missing repository for the existing limit of two attempts. It cannot grant access. - Unknown network errors and genuine access denials retain their existing behavior. This change handles known HTTP failures only. - Historical provider quota causality is unproven. This repair has a separate local producer regression. It does not rewrite past failures or queued user intents. - Acceptance uses real local database and heartbeat owners with controlled HTTP and adapter fixtures. It does not claim a live native-provider recovery. ## Model Used OpenAI GPT-6.1 Codex (`gpt-6.1-sol`), with high reasoning, repository tools, code execution, and controlled local PostgreSQL tests. The host did not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
254b44f72b |
fix(remote-workspaces): recover retired roots that cannot be renamed (#19674)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspaces must retain saved files across process and sandbox replacement. > - Restore uses identity receipts to distinguish a retired owned root from an unrelated directory. > - Some overlay filesystems reject a directory rename with EXDEV even within one mount. > - That error currently prevents a valid saved workspace from being restored. > - This pull request replaces only a verified retired root in place when that rename fails. > - Runtime state stays in place. Each restore holds a root-specific OS lock. > - A durable workspace epoch fence rejects delayed older commands. A pending marker keeps interrupted cleanup recoverable. ## Linked Issues or Issue Description **What happened?** Workspace restore fails with `remote_workspace_restore_failed: stage=publish code=EXDEV syscall=rename` when the retired directory comes from a lower overlay layer. Retry repeats the same rename. **Expected behavior** Restore should publish the verified saved workspace and preserve runtime state. An interrupted restore should remain safe to retry. **Steps to reproduce** 1. Create an owned retired root with a valid identity receipt, runtime state, and stale files. 2. Make directory rename return EXDEV for the existing lower-layer root. 3. Restore a saved workspace through `restoreRemoteWorkspaceDirectory`. 4. Repeat with an interruption before the new restore receipt is published. **Paperclip version or commit** The regression fails before this change on base `a2aa69bc6230c8b81de3588d7d5571b38db4dd04`. **Deployment mode** Remote sandbox with an overlay filesystem. Local regression: macOS arm64, Node 26.4.0, source checkout. Refs #19231. Refs #19251. Those changes establish the retained-root restore path. This fix handles its EXDEV publication failure. ## What Changed - Keep the EXDEV fallback limited to a verified retired workspace root. Preserve runtime state and the existing ownership checks. - Hold a private SQLite OS lock across receipt checks, file replacement, cleanup, and receipt publication. Keep the lock file outside the replaced root. Process exit releases ownership; a lost reply does not. - Pass the existing workspace authority epoch from the real restore caller. Write and sync a separate epoch fence before root changes. Reject lower epochs, conflicting identities within one epoch, and missing epochs on fenced roots. The fence survives SQLite rollback after process death. - Publish a pending marker without a completed identity until stale preparation-receipt removal and directory-mode cleanup finish. A retry can finish that cleanup. A higher authorized epoch can then restore its selected save. - Preserve later file and permission edits when adopting a completed restore. Refuse unsafe lock files and malformed authority records. Document compatibility and the stop-evidence boundary. ## Verification - RED before the original fix: real restore commands failed with EXDEV. The test preload changes only the kernel rename result. - RED before the review fixes: a live command with an unconfirmed timed-out reply overwrote edits made after retry. Tests also reproduced incomplete cleanup after receipt publication. - RED before the authority fence: a delayed older command replaced a newer completed restore. It also passed after the newer command was killed before root mutation. The final test uses real separate Node processes, OS lock contention, SIGKILL, and a durable on-disk fence. - Final local checks: 142 transport filesystem tests, 16 real PostgreSQL restore-caller cases, and all seven server typecheck layers. The caller cases verify the published receipt uses the current workspace epoch. - Late review RED: an actual separate SQLite owner made the real PostgreSQL restore caller mark an expired lock wait non-retryable. Corrupt fence and receipt bytes also produced a raw syntax error without the required refusal reason. Both regressions now pass. Only the test clock advances the lock deadline; actual OS lock contention and the production caller run unchanged. - Lock contention stays retryable only under the existing same clean writer, epoch, and snapshot proof. Four negative cases change those fields and remain non-retryable. After the owner stops, normal restore returns the useful saved files. Corrupt authority records stay untouched and non-retryable. - Safety cases cover same and different authorized identities, process-death lock release, an unrelated root making progress, pending cleanup interruption, malformed or missing fences, invalid input, and unsafe lock parents/files. - The first final transport attempt passed 139 cases and failed only because the restricted shell could not open a Unix socket (`listen EPERM`). The full suite was then run with the required local socket permission. No production change was made for that setup failure. - Reproduce the filesystem checks from `server/`: `../node_modules/.bin/vitest run --config vitest.config.ts src/services/remote-workspace-transport.test.ts`. Run the caller file with `-t "restores a newer save over this workspace|preserves the published workspace path"` and the repository test PostgreSQL setup. Run `node scripts/typecheck-layers.mjs` from `server/` for all seven layers. - Agent preflight, module boundary audit, and `git diff --check` passed. No added code comments or private issue references occur in the patch. - Local EXDEV proof uses macOS arm64 and Node 26.4.0. It does not claim a new Linux overlay-kernel run or hosted recovery. Final source: [`8092cc6b2ec448ff372337b1e7a43854eae7bb08`](https://github.com/paperclipai/paperclip/commit/8092cc6b2ec448ff372337b1e7a43854eae7bb08). [Exact-head CI](https://github.com/paperclipai/paperclip/actions/runs/42321313715) completed with 59 successful checks, three conditional skips, and one neutral check. Greptile gave this exact head 5/5. The independent review passed. No review threads remain open. ## Risks The EXDEV path changes files in place. It must finish before execution admission. Participating restore commands use a private permanent SQLite lock file and a separately synced epoch fence. The qualified sandbox Node runtime must support `node:sqlite`. Lock contention has a 120-second limit and does not clear live ownership. A legacy command that never acquired this lock remains outside its protection. Hosted recovery must confirm that command stopped; a timeout is insufficient. The new fence does not backfill that evidence. A conflicting identity in the same epoch or a malformed fence stays blocked. A higher epoch can recover an interrupted pending restore only with the existing retired-root replacement authority. No controller database schema, execution permission, or provider-state guard changes. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) authored the recovered patch. OpenAI Codex, GPT-6 family, recovered, reproduced, repaired the review findings, and verified the change with filesystem tests, shell tools, and code execution. The Codex exact serving model ID and context-window size are not exposed in this session. No value is inferred for them. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
06bbcea12f |
fix(budgets): price lost receipts and stop restart-derived accounting from hard-stopping companies (#19673)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget admission and run accounting decide whether any task in a company may start > - #19497 made pending accounting an admission hold and settles a stopped run's lost receipt as an unpriced charge > - Restart-chaos waves still stopped whole companies. A lost Codex receipt with complete token counts settled as unpriced because the rate card only prices complete receipts. Under the default blocking policy that hard-stopped the company at 0.38 of 3000 cents. In another storm, a resumed run's continuation was refused by its own task's recovering-accounting hold. Separately, unpriced charges with zero tokens or a known partial cost hard-stopped a company at 120 of 2000 cents > - This pull request prices a lost receipt's observed usage from the rate card, lets a task's continuation pass its own recovering accounting, and stops restart-derived or evidence-free unpriced charges from tripping the blocking policy > - The benefit is that a restart no longer pauses a company by itself, while genuinely unpriced usage still blocks ## Linked Issues or Issue Description Follow-up to #19497. Refs #19606. **What happened?** - Wave A (K6 crash loop): native Codex run `5037b8d1` was killed mid-turn. Its receipt was lost, but `usage_json` held gpt-6.1-sol with 28690 input and 220 output tokens. The charge was written `cost_status=unpriced`, `pricing_provenance.source=unknown`, because `priceCodexReceipt` returns early when `!receipt.complete`. 0.2 s later the default `unpricedUsagePolicy=block` opened a hard incident at 0.38 of 3000 cents. The company was paused, and 6/6 tasks were blocked with `canRetry=false`. - Wave B (K1 storm): Claude run `2a7a8abf` resumed after the restart. `reserveRunBudget` refused its interaction continuation with "New work must wait while a native run recovers its unfinished accounting", so the task could not settle its own recovery. A 0-token unpriced charge and an unpriced charge with a known 0.81-cent cost then hard-stopped the company at 120 of 2000 cents. **What did you expect?** A restart-lost receipt with known numbers is priced as an estimate, and no restart-derived accounting state pauses a company by itself. A task's own continuation is never held by its own accounting. Real usage above the limit, and genuinely unpriced usage, still stop work. **Steps to reproduce** 1. Give a company a hard-stop budget with the default unpriced-usage policy. 2. Kill a native Codex turn after it has reported tokens, so the final receipt is lost and native recovery ends in terminal failure with a verified stop. 3. Observe an unpriced charge for known tokens, a hard incident, and a paused company. ## What Changed - `run-cost-accounting.ts`: when a lost receipt settles and its usage has known tokens and no known cost, price it with `priceCodexReceipt`/`priceAnthropicReceipt` as a complete per-run checkpoint. The charge is `estimated`, its provenance keeps the rate-card source, and its evidence says "final receipt lost; observed usage is a lower bound". The estimate is also written to the run's usage so a later receipt reconciles against it. Zero observed tokens, and per-model usage, stay unpriced (unknown is never shown as zero, and aggregate totals never take one model's rate). - `budget-exposure.ts`: - The pending/recovering accounting hold excludes the run being admitted and earlier runs of the same task (same `native_issue_id`). A task's continuation and finalization can therefore settle its own accounting. Other tasks stay held, and those runs still count as spend. - The blocking unpriced-usage policy no longer counts these unpriced charges: the settled final charge of a lost receipt (restart-derived, any usage; later growth still counts), a charge with a known partial cost, and metered API or subscription billing with no observed tokens. Unknown billing with no tokens (for example Cursor Cloud) and complete-receipt usage that has no price still block. - `budgets.ts` / `budget-reservations.ts`: admission accepts the admitted `runId` and passes it to the exposure read. `reserveRunBudget`, the final dispatch recheck and the read-only admitted recheck all supply it, so a continuation that reserves can also dispatch. - Daytona wave s12 (run f1d98917, a blocked remote recovery with a 0-cent held reservation and every lease destroyed) is closed by #19606. A Retry for that same task passes the hold through the same-task exclusion here. ## Verification - `closed-native-receipt-accounting.test.ts` (embedded Postgres): - Lost Codex (gpt-6.1-sol) and Claude (claude-sonnet-5) receipts with observed tokens settle as `estimated` rate-card charges with the lower-bound evidence. The company stays active and admission is clear under the blocking policy. - A lost receipt for a model with no rate card stays unpriced but opens no incident and no pause. - A task's continuation reserves budget past its own recovering accounting, while another task's run is still held. - Unpriced charges with no tokens on metered billing, or with a known partial cost, do not hard-stop. - Complete-receipt usage with no price still blocks. - The six new cases fail on master and pass with this change. - Existing budget and accounting suites pass locally: budget-enforcement-scoping, budgets-service, heartbeat-cost-accounting, run-cost-accounting, native-stop-accounting, cost-data-quality, budget-live-attempts, budget-live-usage. Server typecheck passes. ## Risks - A lost receipt's estimate is a lower bound. The provenance says so, and a later complete receipt still reconciles. - Restart-derived unpriced charges no longer trip the blocking policy. Usage lost with a killed provider is not counted against that policy, which is consistent with budget enforcement being best effort. Unknown-billing and complete-receipt unpriced usage still block. - The same-task exemption relies on `native_issue_id`. Legacy runs without it keep the old behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
0b73c805e8 |
feat(environments): fill an unset instance default with a managed plugin environment (#19551)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A managed instance gets its instance-level environments from the `PAPERCLIP_MANAGED_CONFIG` document, which can declare plugin environments next to the managed sandbox > - A platform that runs managed instances wants a new instance to start with one of those plugin environments as its default, without changing the default of any instance that already has one > - #19632 made boot fill the instance default only when it is unset, local or deleted, in one locked transaction, and removed the revert, so every existing managed instance keeps a set default > - This pull request adds an `instanceDefault` flag to a plugin entry. That entry fills an unset default through the same locked fill, before the managed sandbox does > - The fill follows the same readiness rule as an admin's default write, so it never moves ready agents onto a plugin that has not prepared them > - The platform also needs a way to know whether a release understands plugin entries, so the release now ships a small capabilities file > - The benefit: new instances get the plugin environment as their default, and existing instances and admin choices are never changed ## Linked Issues or Issue Description Refs #19632 (the locked fill-if-unset rule this builds on), #19508 (plugin entries in the managed config) **Subsystem affected** Server environments: the managed config parser and the managed environment bootstrap. **Problem or motivation** The managed config can declare plugin environments, but it can't make one the instance default on a new instance. An external platform also has no way to tell whether a given release supports plugin entries before it sends one. **Proposed solution** - Add `instanceDefault: true` to at most one plugin entry. Right after that entry's boot ensure succeeds, it fills the instance default through `fillUnsetInstanceDefault`. The managed sandbox fill runs after it. - On a new instance the plugin environment wins. On an existing instance the default is already set, so neither fill changes it. - The fill declines while an active agent that follows the default hasn't been prepared by that plugin. This is the rule the instance-settings route enforces. When the plugin fill declines, the managed sandbox fills as usual. - Ship `server/src/services/managed-config-capabilities.json`, which lists `pluginEnvironments`. **Alternatives considered** - A recorded one-time decision with a retry worker. It needed pending states and extra checks for many writer and trigger combinations. Filling only an unset default, under the existing lock, needs none of that. - Rejecting the fill whenever agents exist. That is stricter than the admin route and blocks harmless cases, such as agents that are paused or already prepared. **Roadmap alignment** This extends existing managed-environment behaviour. It adds no UI. ## What Changed - `server/src/services/managed-config.ts`: - `ManagedPluginEnvironmentSpec.instanceDefault` (boolean, default `false`). - The parser refuses `instanceDefault` on more than one plugin entry, on the sandbox entry (even `false`), or as a non-boolean. - `MANAGED_CONFIG_CAPABILITIES` and `MANAGED_CONFIG_CAPABILITIES_SCHEMA_VERSION` constants. A plugin entry is refused unless `pluginEnvironments` is listed. - `server/src/services/managed-config-capabilities.json`: `{ "schemaVersion": 1, "capabilities": ["pluginEnvironments"] }`. A test pins it to the constants and to this literal contract. - `server/src/services/managed-environments.ts`: - New `fillPluginInstanceDefault`. It runs right after a successful boot ensure of the flagged entry and logs a fill or a decline. Recovery after boot never fills. - The managed sandbox fill now runs once, after the plugin entries. - `server/src/services/managed-instance-default.ts`: `fillUnsetInstanceDefault` runs `agentsNeedSetupForEnvironmentChange` inside its locked transaction and returns `declined: "agents_need_setup"`. A sandbox target never needs agent setup, so the managed sandbox fill behaves as before. - `server/src/services/plugin-environment-driver.ts`: the readiness query is factored into `agentsNeedSetupForEnvironmentChange`. `assertAgentsReadyForEnvironmentChange` wraps it and still throws the same conflict. - `doc/DEVELOPING.md`: documents `instanceDefault`, the fill order, the readiness decline, and the capabilities file. ## Verification - `cd server && npx vitest run src/__tests__/managed-plugin-environments.test.ts` (embedded Postgres), under `instanceDefault (fill only an unset default)`: - A new instance gets the plugin row as its default. - An existing sandbox default is kept, and so is an admin's choice. - If the plugin was down at the first boot and the sandbox filled, a later recovery and boot keep the sandbox default. - An admin-cleared or local default is declined while a ready or verifying agent follows the default without the plugin's setup. The managed sandbox then fills it. - It fills when the agents that follow the default are paused or already prepared, or are pinned to another environment. - Without the flag, nothing happens. - A new agent that follows the default gets the plugin in `requiredPluginIds`. - `npx vitest run src/__tests__/managed-instance-default.test.ts` (embedded Postgres): the earlier fill, lock and overlap cases pass. A sandbox fill also succeeds while active agents follow the default. - `npx vitest run src/services/managed-environments-plugin-entries.test.ts`: - The fill order is plugin then sandbox. - When the plugin is down there is no fill, and recovery never fills. - A failed ensure gives only the sandbox fill, and a failed fill keeps the ensure counted. - When the plugin fill declines, the sandbox still fills. - `npx vitest run src/__tests__/managed-config.test.ts src/services/managed-config-capabilities.test.ts`: parser refusals and the capabilities contract. - `npx vitest run src/services/managed-environments.test.ts src/__tests__/instance-settings-routes.test.ts src/__tests__/environment-routes.test.ts`: pass. The route still throws the same readiness conflict. - `pnpm --filter @paperclipai/server typecheck` passes. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. ## Risks - An instance whose default is unset at the first boot after this release gets the plugin environment, if its agents are ready for it. That covers a new instance, and an instance whose admin cleared the default or chose the local environment. This is the intended behaviour. - The readiness check doesn't lock agent rows. If an agent becomes ready during the boot fill, it can miss the check, which is the same race the admin route has. Agents created after the fill are prepared for the plugin, because setup reads the current default. - If the plugin is down at a new instance's first boot, the instance keeps the managed sandbox default from then on. - No schema change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (file edits, shell, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
fb3cf1a072 |
feat(ci): retry a failed Vitest case once in PR CI and report each retried pass (#19637)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - PR CI runs Vitest shards with 4 workers on 4-vCPU runners, so timing-sensitive tests sometimes time out under load > - One such timeout fails the whole run, and a re-run costs another 15 minutes or more > - In one 7-hour window on 2026-11-06, about 30 failed jobs were test flakes on commits that passed on a later attempt > - This pull request retries a failed Vitest case once in PR CI only, and reports every pass that needed the retry > - The benefit is fewer whole-run re-runs, while each flaky test stays visible so that we can fix it ## Linked Issues or Issue Description **What existing behavior does this improve?** PR CI fails the whole run when one Vitest case fails once because of runner load. **Subsystem affected** CI: the PR Vitest lanes (general, serialized, workspace and Runner package lanes). **Current behavior** Vitest uses `retry: 0` everywhere. One timeout under load fails the run. A person must notice it and re-run the failed jobs. About 20% of recently merged PRs needed at least one re-run. **Proposed behavior** PR CI retries a failed Vitest case one time. A test that passes only on the retry gets a GitHub warning annotation and a row in the job summary. Local runs, release, cloud readiness and master keep `retry: 0`. **Reason and benefit** A re-run of one test case costs seconds. A re-run of the CI run costs 15 minutes or more. The report keeps flaky tests visible. Separate pull requests fix the known flaky tests. **Breaking changes** None. ## What Changed - Add `scripts/ci/vitest-retry.mjs`. It sets 1 retry when `GITHUB_ACTIONS=true` and `GITHUB_WORKFLOW=PR`, and 0 in other contexts. `PAPERCLIP_VITEST_RETRY` overrides the value. - Add `scripts/ci/retry-reporter.mjs`. It writes a `::warning` annotation and a step-summary table row for each test that passed only on a retry. - `scripts/run-vitest-stable.mjs` passes `--retry` to `vitest run`. The `vitest list` collection step does not retry. - The root `vitest.config.ts` and `packages/paperclip-runner/vitest.config.ts` add the retry reporter beside the existing reporters. The Runner config sets `retry` from the same gate. - `.github/workflows/pr-trusted.yml` reads the `CI_VITEST_RETRY` repository variable. Set it to `0` to turn retries off without a code change. - Add `scripts/ci/retry-reporter.test.mjs` to `test:release-registry`, and document the behavior in `doc/DEVELOPING.md`. ## Verification - `node --test scripts/ci/retry-reporter.test.mjs`: 11/11 pass. The tests cover the gate matrix, the reporter output, a real Vitest run with a test that fails once, the wrapper arguments, and the Runner config. - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: 39/39 pass. - `node --test .github/scripts/tests/*.test.mjs`: 744/744 pass. - Manual check: a temporary test that fails once ran through `run-vitest-stable.mjs` and through the Runner package config. With the PR environment it passed and wrote the annotation and the summary row. With `GITHUB_WORKFLOW=Release` it failed. The temporary test is removed. ## Risks - A retry can hide a flaky test. The annotation and the summary row reduce this risk. - A test that fails for a real reason now runs twice in PR CI, so a failed run takes slightly longer. - `pr.yml` loads `pr-trusted.yml` from master, so the `CI_VITEST_RETRY` switch works only after merge. The retry itself is in the scripts and starts with this pull request. - Low risk otherwise: release and master verification are not changed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
7ac99718a4 |
fix(workspaces): fence legacy sources for online migration (#19642)
## Thinking Path > - Paperclip coordinates AI agents and preserves their saved work. > - Remote workspace storage replaces legacy host directories with durable snapshots. > - Several legacy workspace records can name one physical project directory. > - A task lock alone cannot exclude a new task that selects that directory. > - Terminal recovery records can retain valid sources after physical writers settle. > - This change fences the source through its real preparation and lifecycle owners. > - Idle ordinary sources can then migrate while unrelated remote work continues. ## Linked Issues or Issue Description **What happened?** Online migration rejects ordinary legacy directories when several workspace records name the same path. A source inspection found 45 aliases across three project roots with no root Git marker or previously imported nested child. The instance also had unrelated running remote work. Requiring the whole instance to drain prevented useful migration of these idle sources. Terminal native sources and superseded finalization attempts also remain blocked after their exact physical and durable owners settle. Their files, provider allocations and recovery records must remain available through the change of file authority. **Expected behavior** Migrate an idle ordinary source under a derived physical source fence. Let unrelated object-backed turns continue. Preserve provider and recovery obligations. Keep unknown writers, shared Git stores and uncertain source state blocked. **Steps to reproduce** 1. Create several local workspace records for one managed project directory. 2. Give the aliases different source tasks. Leave their local writers idle. 3. Keep a valid remote workspace turn running on another source. 4. Inspect or apply migration to an alias. The old owner returns `overlapping_workspace_requires_maintenance`. 5. Retain a terminal native prepared source with a null coordinator owner, or an old finalize attempt with an exact later committed receipt. The old online path keeps these sources blocked. **Paperclip version or commit** Current tested head: `7478c817f09f7eed396922c0f7831af07ae6ff14`, based on master `8621bcf53ee2e4bbefb7d193add42dd544ca06e8`. Related recovery work: #19131. **Deployment mode** Hosted native Runner with remote workspace storage. Qualification uses disposable local PostgreSQL databases, actual archive/storage owners and synthetic source trees. The source observations were read-only. ## What Changed - Retain the existing company/task source keys and add one physical managed-project key. Heartbeat source selection, materialization, reopen and lifecycle owners share it. Configured aliases of canonical task directories retain their original task key. - Fence runtime provision and each configured service CWD through both startup owners. Check all unsettled service CWDs for physical source overlap, even when workspace IDs differ. Runs with no services keep their filesystem-free path. - Derive a bounded ordinary alias closure. Lock every alias task and lifecycle key. Reject foreign companies, changed membership, unproven descendants, included Git metadata and other unresolved writer obligations. - Commit remote authority on the connection that holds source exclusion. Refresh a stale selected local workspace after admission if migration published remote authority. Publish activity after commit. - Admit exact terminal native recovery sources under the existing physical finalizer keys. Bind descriptors, allocations and owner checkpoints to journaled fingerprints. Recheck them before capture and publication. - Add a separate company-scoped online entry for exact superseded finalization receipts. Keep the offline entry's deployment prerequisites and keep inventory read-only. - Document the proof and the remaining Git/common-store extension. The original files stay in place. No new scheduler, deployment action or database migration is added. ## Verification - Review found that the real reopen path could hold lifecycle exclusion while waiting for a project source. The new actual nested-reopen regression failed before the repair: lifecycle lock count was 1 while another connection held the project fence. The repair acquires declared and physical project sources before reconstruction and lifecycle. The contention case, ordinary-fetch restore and runtime nested restore passed. The lifecycle-worker restore first completed in 16.64 seconds but exceeded its unchanged 15-second body limit; a bounded warm rerun passed at that same limit. Both wrappers exited and completed cleanup. The source type graph, module-boundary check and diff check pass. Final 7478 CI passed: 58 successful jobs, three skipped, one neutral and a successful Snyk context. Fresh Greptile is 5/5 for exact7478, with no unresolved threads. [Final CI](https://github.com/paperclipai/paperclip/actions/runs/42318150314/job/128240631573). - Full CI on `e4283b3452` passed the other server, build and type checks. One exposure owner case failed: two execution workspaces received one preview pair after source admission serialized their starts. The first local seven-case run reproduced that failure; adding the owner argument alone did not change the hash payload. The completed repair binds run-scoped reuse to its execution-workspace owner. All seven existing real-process/fake-broker cases then passed in 13.81 seconds, with wrapper exit 0 and cleanup complete. Public scope IDs and explicit project/agent sharing are unchanged. The final7478 head passed all required CI and fresh review gates. - The service/source/restore owner canary passed 25 cases across four suites before the final no-service fast-path guard (41.33 seconds). This includes actual project service processes with a different project ID and no execution-workspace ID. Their startup waits for source ownership, migration stays blocked while they run, and the source becomes eligible after their verified stop. The final three-case runtime canary passed at the final source inputs (18.70 seconds). It covers that fast path and both service-CWD variants. Both wrappers exited 0 and completed cleanup. - The boundary-repair canary passed 9 cases across four suites at `7c18f1c170`. It covers the real rehome/conversion gateway, project command provisioning, future task selection, stale imported authority, foreign-company binding, and existing standalone filesystem callers. The shared PostgreSQL wrapper exited 0 and completed cleanup. Duration: 94.15 seconds. - The P1 diagnostic passed 46 cases across five suites at the pre-rebase P1 revision. It covers the actual importer with an export warning created by `recordNativeFinalizationFailure`, strict prepared-source/action bindings, finalizer locks, source/lifecycle races, unrelated remote execution, and bare Git with external alternates. It uses the repository shared PostgreSQL wrapper, `--fsModuleCache`, and a local-only 90-second hook allowance. Duration: 245.44 seconds. No product timeout changed. - The canonical retained-directory compatibility check passed 22 cases across two suites. Missing archived task directories still import from authenticated restore points; they do not need a present-directory source proof. - Both initial P1 regressions were observed before the fix. The real finalizer warning made the importer ineligible. The old archive owner accepted a nested bare repository borrowing external objects. The new cases pass and preserve the original warning, prepared descriptor, allocations and files. - Prior attempts remain non-passes: the first 23-case canary passed on an earlier head; its exact-head rerun hit 30-second database setup/cleanup limits. A read of the owned PostgreSQL server found a template clone waiting on `DataFileExtend` for 58 seconds. A broader owner run was intentionally interrupted during slow existing heartbeat setup. One diagnostic invocation used an invalid CLI option and executed no tests. A later review canary passed 21 cases and failed the new real restore contention case: cwd-only admission missed the declared project of an external worktree. Its fixture also omitted the required file-repository authorization callback and produced an unhandled rejection after the failed assertion. The repaired run passed 22 cases, exited 0 and completed cleanup. These earlier failures are not reported as passes. - The recovery owner passed 37 PostgreSQL cases, including 14 new and 23 existing finalized-recovery cases. The operation reconciliation owner passed 40 cases. The initial archive run passed 103 cases and found a missing closed diagnostic entry; the corrected diagnostic and expanded Git rejection cases pass in the P1 run. - The full server type graph passes with the Runner source exports mapped locally because this checkout has no built Runner declarations. The latest module-boundary check and `git diff --check` pass. Standard package build/typecheck and broad final-head regression coverage remain required in CI. - The initial CI exposed missing admission scopes for standalone filesystem APIs and a changed foreign-company error. The latest repair keeps those APIs independent of a database and holds physical fences at their actual database-backed mutation owners. The branch also includes the existing master fix for unchanged Runner default-model assertions. New review findings and new-head CI are being checked. - The local canary command and bounded source selection are documented in `doc/plans/2026-11-06-online-ordinary-source-fence.md`. Source observations only read data. A fresh bounded metadata probe found 254 pending legacy records and zero unsettled local-process services. No live import, hold, cancellation, provider action or source filesystem write was performed for this change. ## Risks - Run-scoped reuse now includes execution-workspace ownership in its private key. Explicit project/agent sharing keeps its previous identity. - A missing writer owner could make a source proof unsafe. Admission derives physical aliases, includes future project selection and retains all existing source/lifecycle/finalizer locks. Unknown and changed state fail closed. - Cross-company physical alias checks use bounded minimal-path pages. Lexical-prefix filtering cannot safely replace them because a symlink can hide a foreign alias. Unknown paths or more than 10,240 foreign records leave the source explicitly unproven. More than 1,000 same-company local records also fail closed. - The project key can serialize local preparation within one project. A first project waits; additional project acquisition fails on contention to avoid a lock cycle. Remote authority keeps its remote preparation path. - Recovery preservation does not prove provider termination or successful file recovery. Existing provider allocations, checkpoints, task intent and recovery actions remain authoritative. - This slice does not finish shared Git migration. Git administrative/common-object/alternates closure and exact nested imported-child coverage remain the next owned extension. No global writers-stopped shortcut is used online. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7aced7a46c |
docs: restore task composer keyboard guidance (#19680)
## Thinking Path > - Paperclip helps people manage AI agents and tasks. > - Task composers have two keyboard modes. > - The product guide explains these modes. > - The routine feature merge removed that paragraph. > - This pull request restores the paragraph beside the routine guidance. > - Readers can find both behaviors. ## Linked Issues or Issue Description Refs #19128. This change addresses [the late documentation review](https://github.com/paperclipai/paperclip/pull/19128#discussion_r4492144137). **Issue type** Missing documentation. **Where is the issue?** The Task Management section in doc/PRODUCT.md. **What's wrong?** The routine feature merge removed the paragraph about Enter, Shift+Enter, and the optional Command/Ctrl+Enter setting. **Suggested fix** Restore the exact paragraph from before the merge. Keep the routine section. ## What Changed - Restore five lines in doc/PRODUCT.md. - Keep the routine responsibility section. ## Verification - Final qualification at `c423853f51c2bb698a65c746ed5e32d084ef9046`: [required CI passed](https://github.com/paperclipai/paperclip/actions/runs/42317059645/job/128236866986), including typecheck, build, all test lanes, E2E, and verify. The snapshot has 58 successful check runs, three conditional skips, one neutral advisory job, and a successful legacy Snyk status. No pending or failed checks remain. Greptile and Paperclip Review both score this exact head 5/5; all review threads are resolved. - A local content check confirms that the restored paragraph matches the text before #19128. Current keyboard handling and the Experimental setting still match the paragraph. Whitespace checks pass. - Local `pnpm verify` could not start because the restored checkout lacked its `tsx` dependency. The dependency-free selection command completed with exit 2 because PRODUCT.md is a global input. Neither attempt is represented as a test pass; the successful final CI supplies that broader coverage. - This restores documentation only. Runtime code, schema, and behavior do not change. Review verified the source behavior and the removed paragraph; no separate browser acceptance is claimed. ## Risks - Low risk. The change restores existing product guidance. - No runtime behavior or database change is included. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The exact runtime model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (content and whitespace checks pass; runtime tests do not apply to this text restoration) - [x] I have added or updated tests where applicable (no runtime behavior change) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
72f7b42fe3 |
Preserve settled failed controller state during online migration (#19678)
## Thinking Path
> - Paperclip controls agent work across web-server releases.
> - Volume-free servers need retained controller state in shared
storage.
> - Successful settled sources already support online migration.
> - Some retained failed sources have no accepted result and must stay
available for recovery inspection.
> - This change copies those sources under task, coordinator and
publication fences while unrelated work continues.
> - The original files, failed status and recovery warnings remain
unchanged.
## Linked Issues or Issue Description
Refs #19038. Refs #19631.
**What are you trying to improve?**
Preserve retained failed controller files without waiting for global
idle.
**What problem does this solve?**
The online producer skips terminal failed controllers. Their files can
still be needed for recovery inspection even when no accepted provider
artifact proves executable continuation.
**What would the improvement look like?**
A settled exact source can enter the existing encrypted archive store.
An accepted committed failed source can use the existing checkpoint
import. Busy, changed, unowned or foreign sources remain blocked.
## What Changed
- Add a source owner bound to the selected task session, run,
coordinator generation, admitted input and file digests.
- Check the task, physical finalizer lock, released coordinator,
canonical resume source and active related turns before upload and
before publication.
- Preserve failed sources through the existing encrypted archive format.
Do not treat archived bytes as execution authority or old-writer stop
proof.
- Allow the existing accepted committed checkpoint import for a finished
failed run without changing its status.
- Validate admitted input bindings before looking up local controller
files.
- Document the online preservation path and the final static writer
fence.
## Verification
- Real PostgreSQL: 125 cases passed across migration, controller
recovery and durable backup storage. One added assertion inspected an
unadmitted queued fixture; it was corrected while retaining the
queued-turn preservation and stale-publication assertions. The final
migration rerun passes 51/51; the other three affected suites pass
75/75.
- Genuine prepatch RED: online failed-source preservation and accepted
committed failed-source import both fail on the previous producer and
recovery owners.
- New cases cover independent restore after deleting the original
source, an expired but non-null coordinator owner, source mutation after
upload, a new related queued turn during upload, foreign input binding
and an accepted committed failed result.
- The full rebased server source and test-tree typecheck passes after
restoring its missing ignored workspace SDK link and building that
existing dependency.
- Module boundaries and whitespace checks pass.
- Rebased onto current master; both provider-artifact imports and the
new archive owner are preserved. Rebased type and module-boundary checks
pass. All six archive cases pass on the exact rebased head (39.68
seconds). All CI gates pass on exact head
|
||
|
Nicky LeachandClaude Opus 5.5
|
8ed712f00a |
test(server): split workspace-recovery tests into seven feature files (#19664)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - PR CI runs the general server tests in 12 shards, and each shard runs files in parallel on 4 workers > - `server/src/services/workspace-recovery.test.ts` takes about 699 seconds in CI by itself. A file runs whole on one worker, so shard 2/12 takes 11.5 minutes (median) when the other shards take about 7 minutes > - A different shard count cannot fix this, because the time of one file sets the minimum time of its shard > - This pull request splits the file into seven feature files with a shared fixture > - The benefit is that the shards can balance again. This removes about 2 minutes from the median PR CI time and about 5 minutes from the p90 ## Linked Issues or Issue Description **What existing behavior does this improve?** The time of the general server test shards in PR CI. **Subsystem affected** Server tests: workspace recovery (`server/src/services/workspace-recovery*.test.ts`) and the general-server shard weights. **Current behavior** One file holds 460 tests and takes about 699 seconds in CI. Its shard is the slowest in most PR runs. **Proposed behavior** Seven files of about 60 to 140 seconds each. The shard partitioner places them on different shards. **Reason and benefit** The general server shards finish close together. The PR CI critical path becomes shorter. **Breaking changes** None. No product code changes. ## What Changed - Split `server/src/services/workspace-recovery.test.ts` into seven files by feature area: - `retry-copyback` (76 tests); - `preparation` (80); - `preparation-proof` (53); - `preparation-dispatch` (25); - `preparation-gates` (43); - `warm-handoff` (36); - `admission` (147). - Add `server/src/services/workspace-recovery.test-fixture.ts` with the shared hooks, `seed`, helpers and an embedded Postgres per file. - Test bodies, hooks and helpers are moved without change. - Four lines that assigned the shared `db`, `root` or `connectionString` now call `replaceFixtureState(...)`. - `scripts/general-server-shard-durations.json`: replace the old key with the seven new files. The weights divide the current CI cost of the old file (636 seconds, master runs 42296943853 and 42294279748) in proportion to measured local time. A `splitEstimates` entry records the method, as for earlier splits. - Update the file references in `doc/task-recovery.md` and `maps/feature-map/recovery.md`. ## Verification - `vitest list`, before and after: 136 declarations each, with identical sorted names. At run time, 460 tests before and after. - Each new file passes when run alone: 76, 80, 53, 25, 43, 36 and 147 tests. - All seven files run together with `--maxWorkers=1`: 460/460 pass. - A timing check of the original file: - `afterEach` took 4.7 seconds in total and `beforeAll` 4.1 seconds, out of about 1,000 seconds, so the teardown is not the cost; - most of the time is in three `it.each` blocks. - Focused `tsc --noEmit` on the eight new files: clean. - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: 39/39 pass. - `node scripts/check-maps.mjs`: aligned. - With the new weights, the 12-shard partition puts the seven files on seven different shards. ## Risks - The CI time of each new file is an estimate from local runs. If `workspace-recovery-preparation.test.ts` takes more than about 150 seconds in CI, the next split point is the "keeps ancestor Retry held" block. - Each file starts its own embedded Postgres, which adds a few seconds of setup per file. - The optional external-database environment variables keep their old behavior but were not tested. CI does not set them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent made the split, and the parent session reviewed it and corrected the shard weights. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
2a4a001d37 |
Preserve GitHub access repair guidance for MCP callers (#19655)
Return GitHub permission denials as failed MCP tool results with trusted, connection-specific recovery guidance. Explain the personal-account and organization App installation boundary without replaying writes or bypassing provider-content checks. Align the repair steps with the Permissions UI. Cover enabled model checks and malicious provider failures across direct calls, connection tests and stored approvals. Repair the CI-exposed synthetic onboarding environment fixture without weakening cancellation assertions. Verified 311 targeted tests, six onboarding Playwright tests, server typecheck, module boundaries, full CI and Apex 5/5 with all threads resolved. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a3eecf1fc9 |
Preserve admitted runtime skill bundles during server replacement (#19675)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs keep the exact skill files admitted for execution and recovery. > - Older inputs can name immutable local bundles without a skill version ID. > - Installed skill heads alone do not preserve those admitted bundles. > - Server replacement needs a durable source for each retained bundle. > - This pull request copies authenticated bundles through the existing package owner and restores their exact projections. > - The benefit is preservation during online migration without stopping active work or settling recovery warnings. ## Linked Issues or Issue Description Refs #19656. Related: #19660. **What happened?** Older native execution inputs can contain skill bundles with a null version ID. Recovery skips those bundles. Startup installation checks can pass even when those retained inputs still require the old filesystem. **Expected behavior** Preserve each retained admitted bundle in object storage. Restore its original bytes, modes and digests on an empty replacement host. Keep its null pin and all run, controller and recovery state. Block startup and enrollment while a concrete local asset obligation remains. **Steps to reproduce** Retain a native run with an admitted local skill bundle and a null version ID. Convert its installed skill head to durable storage. Replace the server home and attempt recovery. The old bundle path remains in the recovered context. ## What Changed - Add a bounded `runtime-assets` phase to the existing skill package maintenance CLI. It copies exact admitted files under the runtime asset publication lock, verifies the durable package, and appends a run-bound receipt after source revalidation. - Keep original files, null pins, task intent, warnings and controller leases. Support an active run through copy-only preservation. Do not start a turn or claim provider ownership. - Verify and reuse a healthy canonical local bundle before storage reads. Restore missing or damaged bundles through the existing package and native asset materializers. Use the authenticated original manifest to preserve exact read and execute modes. Accept a later contract revision only when the run binding and exact skill selection remain unchanged. - Check retained recovery obligations and current task sessions during both startup and migration eligibility. Ignore ordinary admission that has no completed execution input. - Protect receipt-backed packages from orphan collection. Serialize receipt publication and deletion on the package row, then check references in a fresh statement after the lock. - Add real PostgreSQL and filesystem regressions. Replace a delete-method proxy with an actual upload-renewal row-lock race. Publish mutation activity only after commit. ## Verification - Baseline production startup gate failed the new concrete-asset regression as expected. The fixed gate passed. The actual before-fix preservation owner also failed recovery from a healthy local bundle during a simulated storage outage. A verified local-first path fixes that regression. - Eight preservation tests passed with real PostgreSQL, streaming storage, archive extraction and native materialization. They cover cold recovery, exact modes and binary bytes, active ownership, changed sources and bindings, contract revisions, missing or corrupt receipts, and publication/collection races. - Six affected package collection/deletion controls and three existing startup controls passed. The existing versioned recovery consumer passed a sequential retry. - One fixture attempt failed on its cleanup foreign-key order; the fixture was corrected. One existing suite hit its unchanged 30-second database startup hook before test execution. The unchanged single consumer passed on retry. These failures remain recorded. - Source, changed-test and CLI TypeScript checks passed. Rebase preserved both the local instruction-copy and runtime-asset guards. Current-head changed-test checks passed. Final PostgreSQL rerun passed all eleven preservation/startup cases. Current CLI check passed. Required CI remains pending at the trusted-runner gate. - Current canonical bundled creation uses package preparation plus atomic content/head publication. Native preparation requests immutable version snapshots before delivery. Converted installations and new canonical bundled creation therefore do not recreate the historical null-version asset dependency. - Fresh Greptile review on exact head `54c7dd048c9016c1af9fc68213d5ae053ac16499` passed at 5/5 with zero unresolved threads. - Module boundaries, whitespace, and credential/internal-reference scans passed. No live source writes, provider actions or migration apply were performed. ## Risks Missing, changed or unsafe local files remain blockers. This command does not repair a previously published package whose objects later disappear or become corrupt. It also refuses malformed preservation receipts. It retains the original local files and prior evidence and keeps removal blocked. Automatic repair requires a separate canonical package-owner recovery operation; direct database edits are not an accepted recovery path. Failed publication keeps the original files and any unreferenced package for normal collection. The readiness check is explicit startup or enrollment work, not a background scan. Its cost grows with retained admitted asset obligations. Older readers must be upgraded before the old volume is removed. This change preserves asset bytes; it does not settle business recovery or qualify unrelated volume dependencies. ## Model Used OpenAI GPT-6 (Codex). The runtime did not expose a more specific backend model ID or context-window size. Used reasoning, code editing, and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
6d1ecfc420 |
fix(accounting): close a remote run's lost receipt once its sandboxes are terminated (#19606)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget admission waits while completed runs await accounting; #19497 made that a hold on new work, never a hard stop > - #19497 also closes a lost usage receipt once a local provider is verified stopped, so the hold ends > - A remote (Daytona) run has no local stop event and keeps its recorded remote process identity, so its lost receipt never closed. The company then held all new work until a receipt that can no longer arrive > - This pull request closes a remote run's lost receipt once every sandbox it used carries a provider termination receipt > - The benefit is that a restart that kills a Daytona turn no longer stops a company's new work ## Linked Issues or Issue Description Follow-up to #19497. Refs #19544 (Daytona chaos harness). **What happened?** In a Daytona restart wave, run `a261c30b` failed after native recovery was blocked. Its usage receipt stayed incomplete (`accountingReceiptReady: false`). Both of its sandbox leases were released with a `destroyed` termination receipt, but accounting stayed pending. Admission then refused new company work with "New work must wait while a native run recovers its unfinished accounting". **What did you expect?** Once every sandbox of the run is stopped or destroyed, no runner can report usage for it. The known part of its usage should settle as an unpriced charge, as it does for a verified local stop, and new work should be admitted. **Steps to reproduce** 1. Run a native task on a Daytona environment with a hard-stop budget. 2. Kill the control plane during the provider turn so the turn's receipt is lost and native recovery ends in terminal failure. 3. After the sandbox lease is released with a termination receipt, observe that the run stays `cost_accounting_pending` and new work is held. ## What Changed - `run-cost-accounting.ts`: the closed-receipt predicate gains a remote branch. A native run in terminal failure without a result (failed, cancelled, timed out or interrupted) is closed when it has at least one environment lease and every lease carries a `paperclip.remote-termination.v1` receipt bound to that lease, run, company, provider and provider lease ID, with state `stopped` or `destroyed`, after a successful release. This mirrors `hasRemoteTerminationReceipt`. One SQL predicate still serves the reconciler scan and settlement. - The local branch is unchanged: a failed run with a verified local stop event and a cleared process identity. - A sandbox that may still run (no receipt, lease still active), a receipt bound to another sandbox, or a run with no lease keeps its debt pending. - `budget-enforcement-worker.ts`: the accounting worker also wakes when environment-cleanup or retained-session-cleanup work commits. Those are the signals the lease writers already send when they release or stop a sandbox. - `doc/SPEC-implementation.md`: documents the remote closure rule. ## Verification - `closed-native-receipt-accounting.test.ts` (embedded Postgres): - A failed and a cancelled remote run, with one destroyed and one stopped sandbox, settle as one unpriced charge, settle the reservation, and clear admission. These are red on master. - Debt stays pending for an active sandbox with no receipt, for one of two sandboxes without a receipt, for a receipt bound to another sandbox, and for a run with no lease. - `budget-enforcement-worker.test.ts`: the accounting worker wakes on environment-cleanup and retained-session-cleanup commits, and stops waking after `stop()`. These are red on master. - Existing suites pass locally: native-stop-accounting, budget-enforcement-scoping, heartbeat-cost-accounting, run-cost-accounting, cost-data-quality, budgets-service. Server typecheck passes. - Live Daytona restart wave: results in the comments. ## Risks - A cancelled remote run now closes once its sandboxes are terminated. Stop recovery could otherwise reattach a recorder, but a stopped or destroyed sandbox has no runner to reattach to. Local cancelled runs keep their drain. - Workspace export can restart a stopped sandbox and removes its termination receipt first. A receipt that arrives after settlement still posts through late-receipt reconciliation. - Reusable-lease chains (`resolveRemoteTerminationLease`) and recovered-cleanup stop receipts are not mirrored, so those runs stay pending. That is the conservative side. - Low query cost: the predicate runs only for pending terminal runs. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests, live restart-chaos harness). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6c7cc2c5d6 |
fix(native): do not spend a recovery attempt resuming a cleanly drained run (#19625)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs have an execution recovery budget of three provider attempts, so a run that keeps failing stops retrying > - A graceful shutdown (deploy or SIGTERM) drains native runs: it suspends them for reattachment by the next server > - When the runner did not survive, the next boot resumed the turn and charged one attempt > - Restarts that the run did not cause used up its budget, so recovery stopped and the task needed a manual Retry > - This pull request does not charge an attempt for resuming a run that a graceful drain suspended > - The benefit is that deploys and graceful restarts no longer exhaust a healthy run's recovery budget ## Linked Issues or Issue Description **What happened?** In restart-chaos testing, native runs reached `execution_recovery_budget_exhausted` (3/3) only because of server restarts. In one wave, two restarts crossed each of two runs. In a second schedule (K1, then K2 when ready, then K2 again), each drain correctly suspended the local ACPX Claude run, yet the fourth boot blocked it with `native_restart_recovery_blocked`. Every resume after a drain was classified `resume_dead_runner` and charged an attempt (`nextNativeProviderAttempt` in `native-restart-recovery.ts`, applied by the executor's execution claim). **Expected behavior** A restart that the run did not cause should not consume its recovery budget. A graceful drain records `restart_suspended` and `awaiting_runner_reattach`. The resume that follows should keep the attempt count. **Paperclip version or commit** `master` at `4ef3fa4271`. **Steps to reproduce** Start a native local run. Send the server SIGTERM three times while the run is active, with a full restart each time. The third resume is blocked with `execution_recovery_budget_exhausted`. ## What Changed - `suspendedByCleanDrain()` in `native-restart-recovery.ts` recognizes a run whose most recent recovery claim is older than the drain's `restart_suspended` entry. Ownership waits and deferred blocks between the drain and the next claim do not erase it. The next claim consumes it. - A scan that waits for the old controller to exit (`awaiting_evidence`) records one history entry per reason, so repeated scans cannot push the drain record out of the bounded history. - Restart claims carry `chargesProviderAttempt`. It is `false` for a claim that follows a clean drain. - `nextNativeProviderAttempt()` takes that flag. The executor passes it when it takes the execution lease, so a drained resume keeps the current attempt. - The first execution is always counted: a run drained at attempt 0 (no execution claimed it yet) still charges its first provider start. - A run drained during its third attempt may continue that attempt. The local resume and remote bootstrap gates skip the `>= 3` exhausted check only for a non-charging resume, and the executor still refuses a fourth attempt. - A crash restart (no drain record) still charges one attempt. Reattachment to a live runner still charges none. - `doc/execution-semantics.md` states the rule next to the three-attempt budget. ## Verification - New test in `server/src/__tests__/heartbeat-native-restart-suspension.test.ts` (embedded Postgres, real heartbeat drain and restart recovery): - Red before the fix: the claim after the drain had no `chargesProviderAttempt: false`. - Green after the fix, for drained attempts 0, 1 and 3: the crash claim charges (`true`). After the drain, a controlled backend lets the executor take its execution lease and records the saved `native_run_finalizations.attempt` at provider dispatch: 1, 1 and 3. Attempt 3 is claimed instead of blocked. - Red at the executor: when the executor ignores the flag, the drained attempt-1 resume saves 2. - Ownership-wait case: the drained controller (the test process) is still alive, so two scans wait. The history keeps `restart_suspended` followed by one `awaiting_evidence` entry. After the controller is gone, the claim does not charge, and attempt 3 continues. - Classifier cases (local resume and remote bootstrap): attempt 3 after a crash is still `execution_recovery_budget_exhausted`. Both new cases were red on the previous revision. - `heartbeat-native-restart-suspension.test.ts` 12/12, `native-runner-restart-recovery.integration.test.ts` and `native-runner-ownership.test.ts` 94 passed. - `tsc --noEmit` for `server` passes. ## Risks - A run that itself makes the server shut down gracefully, again and again, would no longer be stopped by this budget. Graceful shutdowns are operator or deploy actions, so this is unlikely. A crash loop (SIGKILL, OOM) still charges an attempt on every restart, and a resumed turn that fails still goes through the existing failure and replacement budget. - This changes only the attempt count. Ownership checks, checkpoint verification and the exhausted gate are unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
56c2b28c04 |
Settle saved monitor results after a newer user request (#19644)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native results must finish their durable finalization before later
work can reuse a saved workspace.
> - An accepted result can wait on a monitor while its workspace files
are recovered.
> - A newer user request can replace that old wait before finalization
resumes.
> - The old finalizer then reports lost monitor authority and blocks the
saved-workspace lineage.
> - This pull request proves the newer user request and settles only the
old result bookkeeping.
> - The benefit is released recovery work without replaying an old turn
or changing the current request.
## Linked Issues or Issue Description
Refs #17641 and #16975.
**What happened?**
An older successful native result has saved files and a server-issued
monitor receipt. Its task is now blocked after a newer user-comment
continuation failed during preparation. The old coordinator stays in
retryable_failure with monitor_wait_authority_lost. The retained
allocation ownership check correctly refuses the unresolved predecessor,
so later file recovery also stays held.
**Expected behavior**
The normal finalizer can settle the old accepted result when exact newer
user-request evidence proves the old wait is superseded. It must
preserve the current task, user input, ownership, schedule, accepted
result, and attempt history. It must not start provider work or publish
the old wait summary.
**Steps to reproduce**
Create an accepted successful monitor result and mark workspace
finalization successful. Leave the task blocked. Record a newer
canonical user comment and its ordinary user wake, either queued or
consumed by a same-task continuation that retains the exact comment
revision and source-run lineage. Call the real native finalizer. Before
the repair, both cases remain retryable_failure instead of settling.
**Paperclip version or commit**
|
||
|
DottaandPaperclip
|
752cb8d3da |
fix(ui): contain patch previews and highlight diff source (#19478)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tasks show documents and file attachments in cards and file views. > - Patch source needs clear added and removed lines in each view. > - Long patch lines can exceed the width of an inline preview. > - This pull request keeps the preview inside the card and adds patch colors. > - Users can read a compact patch preview and open the full source. ## Linked Issues or Issue Description **What happened?** Patch text can exceed the inline document box. Patch files and patch code blocks show plain text in the document view. **Expected behavior** Keep source inside its preview. Show added lines in green and removed lines in red. Keep headers, source markers, copy controls, line numbers, and downloads. **Steps to reproduce** 1. Open a document card with a diff code block that has long unbroken lines. 2. Open a `.patch` or `.diff` attachment in a task tab. 3. Open the same patch in the workspace file view. **Paperclip version or commit** Source checkout based on `e82210386f1e`. **Deployment mode** Built from source. The browser checks use Storybook fixtures. Related work: #19269 adds inline diffs for file edits. This change uses its line classifier for document and file source views. ## What Changed - Share the unified diff line classifier with file and document views. Track hunk lengths so later file headers keep their metadata style. - Add safe patch source rendering for patch files and `diff` or `patch` code blocks. - Clip long lines with an ellipsis in compact cards. Wrap the full source in file views. - Add a Preview patch action for small `.patch` and `.diff` attachments, including generic binary MIME types. Opening a large artifact list does not fetch patch source. Bound the actual response bytes. Reject remote preview URLs and redirects. Keep download and retry fallbacks. - Add patch text colors with sufficient contrast in both themes. Keep source-line selection visible over patch row colors. Copy the original patch text without layout line breaks. - Add five production component stories and browser checks at two widths in both themes. - Document the focused Storybook browser command. ## Verification - Final base qualification at `a120e0ada3f1eee9746f47a10384b5aa95f79824`: normally integrated the already-merged #19610 model-fixture repair (`8621bcf53ee2e4bbefb7d193add42dd544ca06e8`), preserving the reviewed patch preview changes. Official preflight passes (1.8s); all 129 focused tests pass again (23.67s), with UI typecheck and token gates passing. Current revision CI and reviews are pending; prior-head review receipts below are not current-head approval. - Conflict qualification at `befc954a6116f7292506200b96623b673f9385b5`: normally merged current master `c66b4a55efdc5675b5a5146197078b94de6407da`, resolving the append-only `ui/src/index.css` conflict by retaining both the patch rules and all current agent identity rules. No feature behavior was changed during resolution. Official isolated preflight passes. The seven focused component/parser suites pass 129 tests; UI typecheck and token gates pass. Fresh head CI and reviews are pending. - Qualification setup limits: the first focused command used the repository root instead of the UI working directory, so setup failed before any test ran. It was corrected to the UI directory and actual attachment-test path. The first UI typecheck found the unbuilt local plugin SDK; building that checkout’s SDK and repeating typecheck passes. These setup attempts are not test/typecheck passes. - Focused Vitest run: 139 tests passed in eight files. This includes source preservation, HTML escaping, large patches, tabs, Markdown wrapping, line numbers, and transcript diff parsing. A 200-card check confirms no patch fetch occurs before a preview request. CRLF regressions cover no-newline markers and added lines that start with `+++`. - `pnpm --filter @paperclipai/ui typecheck`: passed after the plugin SDK build prerequisite. - `pnpm check:token-gates`: passed. - Storybook catalog check and 30 map/catalog tests: passed. - `STORYBOOK_TEST_BASE_URL=http://localhost:6006 pnpm exec playwright test patch-preview.spec.ts --config=tests/storybook-visual/playwright.config.ts --project=stories --workers=2 --retries=0`: 20 passed on Linux Chromium. The checks include accessibility, theme colors, overflow, opening a document, wrapping source, exact clipboard contents, and the selected source line. Screenshots were inspected and saved to the task. - The first browser runs found contrast failures in diff text and source gutters. Patch text tokens and patch gutter opacity now fix those failures. The final run passes without retries. - `pnpm verify`: exit 2. The new browser spec is a global CI input. Required CI must supply broad coverage. - The local full Storybook build was killed with exit 137. This is not a build pass. External screenshot baselines are not published, so no baseline comparison was run. - Reviews found selected-row background precedence, file headers in unified patches without git separators, eager patch loading across a collection, and CRLF no-newline markers. Each finding has a focused fix and regression checks. The Storybook hunk count now matches its source lines. - Prior feature commit `b3cb9d6d3c0423315540514b1ad68325a64c1cc2`: [CI typecheck, build, tests, runner checks, and E2E passed](https://github.com/paperclipai/paperclip/actions/runs/42247974154). [Security scan passed](https://github.com/paperclipai/paperclip/runs/127995160859). [Greptile is 5/5](https://github.com/paperclipai/paperclip/pull/19478#issuecomment-6688482753), and [Paperclip Review is 5/5](https://github.com/paperclipai/paperclip/runs/127998788846). All six inline review threads are resolved. ## Risks - Compact previews omit the end of long lines. The full document and downloads keep the source. - Patch colors apply to unified diff syntax. Other patch formats keep literal text and do not have full format parsing. - The shared classifier keeps the existing task chat behavior. Regression tests cover that consumer. - No database, API, or access rules change. ## Model Used - OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The runtime did not expose a more specific model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
f4b19e4231 |
fix(runner): finalize a durable proposed result when restore loses its Codex turn (#19658)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs survive control-plane restarts: runnerd restores the provider conversation, and the control plane continues the task from its checkpoint > - A hard kill can land after the agent proposed its result (`run.result.proposed`) but before the provider turn ended > - The restored runner then proves that the turn ended with the accepted result. It records `session.reconciled` and `run.terminal`, but no provider turn terminal > - The Codex recovery read did not see that turn as ended, so recovery failed with "persisted active turn … is missing from thread/read". Retries ran out, and the Retry re-ran finished work > - This pull request lets recovery settle that turn with its durable result > - The benefit is that a task that already finished its work continues to finalization after a restart, instead of failing and redoing the work ## Linked Issues or Issue Description Refs #19525, #19591, #19539, #19501 **What happened?** In a restart storm, a native Codex task (B1) proposed a complete `paperclip.run_result.v1` with passing tests. The control plane accepted the completion tool call. About 9 s later, a K1 (SIGKILL and restart of the control plane) killed the controller, runnerd, and the Codex app-server. After startup, `resume_dead_runner` resumed the same run two times. Each resume spent about 25 s in `runner.session.startup` and failed with `native_session_recovery_failed: HarnessReconciliationError: persisted active turn <id> is missing from thread/read`. Then `native_session_retry_exhausted`, replay was blocked (`preserve_without_replay_v1`), and the run ended `native_session_interrupted`. The task needed a Retry, which re-ran the finished work. **Steps to reproduce** 1. Start a native Codex task through runnerd. 2. Let the agent call `paperclip_finish`, and let Paperclip accept the result. 3. Kill the controller, runnerd, and Codex before the provider turn ends. 4. Let restart recovery resume the run. Durable evidence from the kept instance: - The run events have `run.result.proposed` for the turn, then a `completion.tool_accepted` span, then the kill. - The saved checkpoint has `activeTurnId` set to that turn and `semanticResult: null`. - The Codex rollout on disk does contain the turn. The provider did persist it. - The runnerd journal for the first resume has `session.reconciled {previousProviderTurnId: <turn>, activeProviderTurnId: null}` and then `run.terminal {turnTerminalState: completed, runTerminalState: succeeded, reportedWorkDisposition: done}`. It has no `turn.*` event for that turn. The runnerd Codex provider state shows `completedTurnAuthoritative: true` and `completedProviderTurnId: <turn>`. **Expected behavior** Recovery finalizes the accepted result. It does not re-run the task. **Actual behavior** The runnerd Codex transport builds the recovery `thread/read` reply from runnerd's journal. It looked only for `turn.completed|failed|interrupted|cancelled` events that name the turn. Runnerd's "turn ended with an accepted result" branch (`provider_backend.rs`, `recovered_turn_ended_with_result`) emits only the reconciliation and the run terminal. So the reply omitted the turn, and `CodexHarnessSession.reconcile()` threw at `codex-harness-session.ts:796`. The second resume read the same journal and failed the same way. ## What Changed - `runnerd-codex-transport.ts`: a new `runnerdRecoveredTurnTerminal` helper selects the terminal for a restored turn. A provider `turn.*` terminal that names the turn wins, as before, with its cause. A `run.terminal` that follows a `session.reconciled` that names the turn as the ended previous turn (no active turn, no later `turn.started`) now also settles that turn, with the run terminal's `turnTerminalState`. The `thread/read` wait loop uses this helper. - `codex-harness-session.ts`: `reconcile()` now settles a checkpointed active turn that provider history no longer contains when the session holds the semantic result that this exact turn proposed (`resultTurnId` equals the turn). Recovery restores that result from the durable `run.result.proposed` event. Without such a result, a missing turn still fails closed. - `codex-harness-session.ts`: the same rule applies when the restored runner reports that turn `failed` with `provider_turn_lost_on_restore`. This happens when the runner cannot prove the accepted result itself. Later recoveries keep the settled outcome. Without a held result, a restore-lost turn keeps its existing failure and same-conversation continuation. - `fake-codex-app-server`: a test-only `--hold-after-terminal-result` flag keeps the turn open after an accepted completion report. - `doc/execution-semantics.md`: one additive paragraph on this recovery rule. ## Verification - Red before the fix, real `paperclip-runnerd` and fake Codex: `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-restart-continuation.test.ts -t "durable proposed result"` failed with `native_session_recovery_failed: HarnessReconciliationError: persisted active turn provider-turn-1 is missing from thread/read` at `native-session-runtime.ts:2267`. This is the same stack as the incident. - Green after the fix: the run finalizes on `provider-turn-1` with the proposed result and `runTerminalState: succeeded`. It has no `session.failed` or `turn.failed`, and Codex received only one `turn/start` (no re-run). With the transport fix alone, the test passes in about 5 s. With only the session fix, it also passes (about 30 s, through the 5 s read wait). - Driver red/green: `src/drivers/codex/codex-app-server-driver.recovery.test.ts -t "missing persisted turn"`. The settle case failed before the fix (`recovered: false`) and passes after it. The guard case (the held result belongs to another turn) still fails closed. The existing "fails recovery explicitly when the persisted active turn is missing or replaced" test still passes. - Driver red/green for the restore-lost case: `-t "restore-lost turn"`. The held=true case failed before the fix and passes after it. The held=false case still maps `turn.failed` with the lost cause. - Transport unit test: `runnerd-codex-transport.test.ts -t "reconciled result terminal"` covers the run-terminal rule and its negative cases. - Suites: `src/drivers/codex/`, `src/native-session-runtime.test.ts`, `src/backends/`, `src/live/runnerd-restart-continuation.test.ts`, and `src/live/runnerd-codex-transport.test.ts` pass. Some load-sensitive tests time out at host load 25–30 and pass when rerun alone. The two `proves local suspension after an event backlog … (1024 suffix deltas)` cases also time out on unmodified master. - `pnpm --filter @paperclipai/paperclip-runner typecheck:typescript` passes. - Live chaos replay with real Codex: K1 0–3 s after `run.result.proposed`, 3/3 PASS (R1). Each case resumed the same run, which ended `succeeded`. No Retry was needed and no work was re-run. Details are in the comment below. ## Risks - The transport rule accepts a run terminal only directly after runnerd's own reconciliation that names the ended turn with no active turn. Runnerd emits that pair only after it restored the conversation and proved the turn ended. A provider turn terminal still wins. Other run terminals do not settle a turn. - The session rules use only a result that Paperclip already holds for the exact turn. The control plane accepted that result before the kill. A later provider terminal for that turn cannot change the outcome. The finalizer can already interrupt a provider after an accepted result. - A missing turn without a durable result still fails closed. Its outcome is unknown, and the existing Retry stays available. - No profile, schema, pack contract, or database changes. The Rust change touches only the fake Codex test binary. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
6be71028ad |
Prepare online ephemeral enrollment through signed Cloud control (#19660)
## Thinking Path
> - Paperclip manages agent work through a control-plane web server.
> - A replacement web server needs shared durable data and the original
signing identity.
> - Existing enrollment exports that identity only after a global drain
and offline migration.
> - Online enrollment needs a separate preparation operation while agent
work continues.
> - This change adds a purpose-bound Cloud control operation with
existing data and key owners.
> - The result is an audited preparation receipt, with no permission to
stop agents or detach storage.
## Linked Issues or Issue Description
Refs #19038. Refs #19024.
**What are you trying to improve?**
Allow Cloud to prepare an already migrated tenant for volume removal
without the offline migrator or a global-idle gate.
**What problem does this solve?**
Cloud must preserve the original decision key before starting a
replacement with static ephemeral mode. A tenant session must never gain
key-export authority.
**What would the improvement look like?**
A fresh exact-action Cloud assertion obtains a private, deployment-bound
preparation receipt. Unfinished durable-data migration and invalid
replacement configuration block export.
## What Changed
- Add the signed `ephemeral-enrollment:prepare` control action and a
strict UUID-only POST route.
- Check existing durable-data eligibility and ephemeral configuration
before reading the original key. Never create a missing key.
- Audit only public preparation metadata. Keep HTTP diagnostics
secret-sensitive and the response non-cacheable.
- Bind the receipt to the attempt, stack, build commit, Railway
deployment, time, static mode and key digest.
- Always return `enrollmentReady: false`. Cloud still needs to arm a
replacement, verify old writers have stopped, repeat eligibility and
qualify native handoff before detaching the retained volume.
## Verification
- PASS: 80 focused tests across the preparation owner, signed route,
existing Cloud control, HTTP privacy and OpenAPI route coverage.
- PASS: 2 real PostgreSQL cases. Concurrent copies of one signed request
export once; the durable receipt still rejects reuse after clearing the
process fence. Normal run admission before skill pins is allowed;
missing durable skill installation remains blocked.
- Genuine prepatch RED: the same two PostgreSQL cases exported twice and
rejected ordinary admission. The fixed owner was restored automatically
after the comparison.
- PASS: server TypeScript check with workspace dependencies built.
- Fixed the first CI route-coverage failure by declaring this private
control endpoint in the existing explicit OpenAPI exclusion registry.
- Initial local HTTP execution lacked sandbox socket permission; the
reviewed run passed. Initial PostgreSQL fixtures enabled durable files
too late and reused an issue prefix; those fixture defects were
corrected before the red/green comparison.
- All required CI gates pass on
|
||
|
DottaandPaperclip
|
9197abf673 |
fix(native): preserve controller state when remote archiving fails (#19663)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native continuity replacement must preserve the state needed to recover saved work. > - The current replacement moves local controller files before it knows whether the remote archive succeeded. > - A remote failure then leaves the local controller without its recovery files. > - This pull request confirms the remote archive before it moves those files. > - A user turn that never started can use the existing bounded Retry path after verified preservation and cleanup. > - The task retains its saved work and can continue once under current user authority. ## Linked Issues or Issue Description Refs #19638, #19547, #19399 and #19659. The prerequisite #19638 merged as `0819f4ea78dd37fba87341b314be9dac078c1410`. This PR now targets master. A normal dependency integration at `febed950f4` keeps only the six-file archive repair in the PR diff. Fresh master-based CI and review are required before merge. **What happened?** A native continuity replacement moved its local controller journal and provider checkpoint. The following remote archive failed. A later recovery attempt could no longer read the local recovery files. The first-attempt failure also had no normal Retry receipt for the saved user turn. **Expected behavior** A failed or timed-out remote archive retains the local recovery files and starts no replacement provider. A user turn that has not started can request one normal Retry. The existing user, provider-turn, Stop, preservation, cleanup, budget and approval gates remain in force. **Steps to reproduce** 1. Keep a local controller journal and provider checkpoint for a retained remote conversation. 2. Request continuity replacement and fail or time out the remote archive command. 3. Check the local controller files before any later recovery. 4. Settle the native failure under a current user wake and recover the saved workspace. ## What Changed - Confirm remote continuity archiving before moving the local controller state. - Add the exact remote archive failure to the shared bounded user Retry admission from #19638. - Verify failed, timed-out and successful archive ordering with real local files. - Verify ordinary failure settlement and saved-file recovery produce one fresh successor. Keep Stop, system wake and unsettled-provider holds. - Document the file preservation and Retry contract. ## Verification - RED: failed and timed-out remote archives removed the local controller files. Reading those files failed with ENOENT. The saved-user Retry request returned false instead of true. - Initial archive unit verification: eight cases passed, including the three archive outcomes and five existing Cloud archive safety cases. - Initial stacked revision: `9104ee399b`, based on #19638 at `d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. - Final master-based revision: `febed950f4`. All 13 affected cases passed across three Vitest files after dependency integration. They include the seven-mode real PostgreSQL completion case, saved-file successor and negative-hold cases, and real-file archive outcomes. All seven server typecheck layers passed again. Module boundary, whitespace and added-code-comment checks passed. All required CI passed on this exact head. The final snapshot has 59 successful checks, three intended skips, one neutral result, and no pending or failed checks. Current-head Greptile and the independent review are both 5/5, with no open findings. - Final local revision: `9104ee399b`, macOS ARM64, Node 26.4.0, pnpm 9.15.4 and embedded PostgreSQL. The three real-file archive outcome cases passed. The real PostgreSQL completion test passed all seven internal scenarios, including archive failure, prior provider activity, system wake, Stop before and after terminal settlement, and stale runtime promotion. Own Runner/shared/SDK builds and all seven server typecheck layers passed. Nine saved-work continuation and negative-hold cases passed. They verify exported file contents, one user-attributed fresh successor, idempotent hooks, no second automatic chain, and first-use manual Retry for held cases. - `node scripts/check-module-boundaries.mjs`, whitespace checks and the added-code-comment audit passed. - Focused commands from `server/`: Vitest with `-t 'retains the local controller checkpoint until remote continuity archiving succeeds'`; `-t 'replays saved user intent through normal Retry'`; and `-t 'continues a user turn once|does not continue automatically'`. PostgreSQL cases used a fresh local cluster and a 120-second setup hook budget. Application timeouts were unchanged. - The initial stacked head passed all test, build, typecheck and E2E jobs. Greptile and the independent review both scored it 5/5 with no findings. These results do not replace the fresh master-based qualification now running. ## Risks - A remote command can fail after it changed remote state. This change retains local evidence in that case. It does not claim a distributed atomic transaction. - The exact archive error can request one normal Retry only under the existing saved-user admission. The normal path must prove stop, saved files and cleanup before a successor starts. - The protected launch-argument rejection remains enforced. Its exact hosted argument difference was not available through ordinary access. #19659 separately addresses a saved-thread feature-policy argument difference. This PR preserves local archive evidence and does not claim direct hosted verification of that primary cause. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use and code execution. The model is inherited; no override was used. The exact serving model ID and context-window size are not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0819f4ea78 |
fix(native): verify controller ownership before retrying saved turns (#19638)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native task turns can keep a provider conversation from an earlier run. > - A copied conversation checkpoint names the new run before its controller has attached. > - A failed attachment can make recovery treat that copied identity as current execution authority. > - The current run needs its own durable controller checkpoint, or a valid staged attachment, before exact resume. > - This pull request checks that ownership and uses the normal Retry path to continue a saved user turn after verified cleanup. > - The task can continue once with its saved work and history. ## Linked Issues or Issue Description Refs #19399 and #18860. **What happened?** A new user comment resumed a completed native conversation. The retained runner process was gone before the new controller attached. The new run still had a copied provider session ID. Recovery selected exact resume for that new run and then failed its unchanged owner and lease checks. Ordinary first-attempt failure settlement also omitted the saved-user Retry handoff that restart recovery already uses. **Expected behavior** Exact resume requires current controller evidence. A saved user turn can request one normal Retry after this infrastructure failure. The normal path must verify provider stop, saved files, current user authority and all execution gates before it starts a fresh session. **Steps to reproduce** 1. Finish a native warm run and retain its provider checkpoint. 2. Admit a user comment as a new run and copy the provider checkpoint through the real resume producer. 3. Stop the retained runner before attachment. The real remote process check reports that the process is not running. 4. Observe that the copied provider ID selects exact current-run resume. Observe that ordinary failure settlement saves no Retry request. ## What Changed - Check a bounded encrypted controller header for fenced remote Runner recovery. Keep local Runner recovery on its existing disk-backed path. Match the run, conversation and runner identity. Preserve the existing staged-attachment path. - Share the existing saved-user Retry admission between restart recovery and ordinary terminal native settlement. Keep its exact infrastructure errors, user wake, comment, provider-turn and retry-chain checks. - Add real PostgreSQL and stopped-process regressions. Verify malformed and foreign checkpoint holds, current-owner resume, user authority and Stop. Verify saved-file recovery creates one fresh-session successor. - Document the ownership and Retry contract. ## Verification - RED: the real checkpoint rebind producer and stopped-process check selected exact current-run resume without a current controller checkpoint. - RED: ordinary settlement created zero Retry receipts for the real native failure code; expected one. - Review RED: a real current local controller file existed with no database checkpoint rows. The initial gate returned ambiguous state. The fix takes a required trusted execution-target kind and applies the database requirement only to the remote persistence path. - Initial local revision: `3662055880ccf8e09c3c50a340897e16c78627e7`, macOS ARM64, Node 26.4.0, pnpm 9.15.4, embedded PostgreSQL. - Initial server Vitest: 42 selected cases passed across controller recovery integration, remote restart adoption, ordinary run completion and retained workspace recovery. The cases cover the stopped-process producer, current/foreign/malformed ownership, automatic user continuation, Stop, unsettled turns, system wakes and a manual Retry on first use. - `pnpm --filter @paperclipai/server typecheck`: passed with this worktree's Runner, shared and SDK builds and all seven server layers. - `node scripts/check-module-boundaries.mjs`, whitespace checks and the added-code-comment audit: passed. - Review correction: `8c795e2e0e44b5209da3e228593c5cd35e948cae`. The real local/remote controller case passed against a fresh PostgreSQL cluster. It verifies local disk recovery without database rows, remote current-owner resume, and remote missing/foreign/malformed holds. Fresh `node scripts/typecheck-layers.mjs` passed all seven server layers. The existing offline chat reader now requests metadata only and retains its full independent native-state safety check. - A broader local recheck hit PostgreSQL setup/cleanup hook timeouts. These are retained as failed setup attempts; the fresh-cluster regression above passed. Required CI supplies broad final-head coverage. - Settlement-order correction: `dafb721c3bbb7ae86fbc3c9c6c953ace29ca088b`. The real PostgreSQL regression first proved both failures: Stop after an already-failed row created one Retry instead of zero; a DB-native promotion with an old legacy caller snapshot created zero instead of one. All six settlement scenarios now pass. The hook requires the local failed outcome and the canonical settled native row. Fresh seven-layer server typecheck passed again. - Required shared fixture base integrated at `d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. The six runner-skills fixture cases and all seven server typecheck layers passed. Required CI passed for this head: [workflow](https://github.com/paperclipai/paperclip/actions/runs/42309419592), including all server layers, tests, build, Runner verification and E2E. The final snapshot has 57 successful checks, three intended skips and one neutral result. Current-head Greptile is 5/5; both concrete review threads are resolved. The independent re-review ended without a final response; its prior Stop finding has a real RED/GREEN repair. The focused command was run from `server/`: ```sh ../node_modules/.bin/vitest run --config vitest.config.ts src/services/heartbeat/run-completion.test.ts src/services/native-runtime/native-controller-recovery.integration.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/workspace-recovery.test.ts -t 'replays saved user intent|does not assign current-run resume authority|remote controller restart adoption|continues a user turn once|does not continue automatically' ``` ## Risks - A terminal failure of a fenced Runner adds one exact-run header read. The read selects one encrypted header and limits inspection to 1 MiB. Invalid or missing ownership evidence stays held. - Saved user input can request one automatic Retry. It uses normal preservation, current authorization, budget, approval, Stop and cleanup gates. Unsettled provider turns and system wakes do not start an automatic successor. - This PR changes no provider process, credential, model or lease authority checks. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use and code execution. The model is inherited; no override was used. The exact serving model ID and context-window size are not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bb621880e6 |
fix(storage): preserve agent and skill files during online conversion (#19656)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent files and installed skills need durable storage before a legacy volume can be removed. > - Their maintenance commands currently require every writer to stop. > - Normal file edits already use agent row locks and skill publication locks. > - Shared agent heads also remain durable while their agents run. > - This pull request uses those owners to copy settled files online and keeps unresolved recovery states intact. > - The benefit is useful migration progress while unrelated work continues. ## Linked Issues or Issue Description Related: #19642. **What happened?** Agent-file adoption and skill installation conversion reject apply requests without a global stopped-writer acknowledgement. Agent-file maintenance also reports an active agent as blocked when its shared head is already durable. Live enrollment omits the existing unresolved-local-instruction-copy guard. **Expected behavior** Copy settled agent files and skill installations under their existing writer locks. Keep old skill directories, versions, and inline bytes. Keep unresolved local copies and unknown historical pin obligations visible to their owners. **Steps to reproduce** Run agent-file maintenance with `--apply` on the original host with durable storage enabled. A managed agent without active execution is rejected without `--writers-stopped`. The same gate rejects skill installation conversion. An agent that already has a shared head is also reported as blocked while it runs. ## What Changed - Adopt settled agent files under the existing agent row lock. Recognize an existing shared head before checking active execution, without rewriting its projection. - Reuse the local working-copy preservation predicate for first adoption and live enrollment. Keep its original public export. - Prepare skill uploads outside the shared run-pinning lock. Verify the exact local tree and source identity under the cache, slug, and row publication locks. Publish catalog origin history in that same transaction. - Retain old skill directories, versions, and inline bytes. Keep pin retention and volume-removal guards unchanged. - Add PostgreSQL and filesystem coverage for active durable heads, unresolved copies, concurrent source edits, and linked files. Update the maintenance docs. ## Verification - Agent-file durability and live enrollment suites: 16 tests passed with real PostgreSQL and filesystem fixtures. - Managed skill suite: 22 tests passed. Seven final controls also passed: upload-time file edits, source relocation, unknown historical pins, hard-linked files, pinning during a blocked upload, and both rejected/successful catalog-origin publication. - Server source typecheck and focused changed-test typecheck passed. - Module boundaries, diff whitespace, and added-line credential/internal-reference scans passed. - Baseline production rejected both online paths at their stopped-writer guards. Review regressions also reproduced a busy pinning lock during upload and extra catalog history after rejection; their fixes passed real PostgreSQL controls. All expected red results are preserved. - One final-control attempt failed the existing 30-second fixture startup hook before executing tests. A sequential retry passed all four controls without changing guards or timeouts; the failed attempt remains recorded. - Exact head `001a5beed9d0e55c7c178906fbad1f9a47f87788`: [required CI passed](https://github.com/paperclipai/paperclip/actions/runs/42309446426), including typecheck, build, tests, Runner and end-to-end gates. Greptile completed successfully at 5/5 with zero unresolved review threads; GitHub reports mergeable/clean. ## Risks Changed or unsafe sources remain blocked. Storage failures keep the original files and recovery state. This conversion does not settle retained provider copies or historical execution. Unknown pin obligations still block retention and volume removal; this PR alone does not qualify removal of the original volume. ## Model Used OpenAI GPT-6 (Codex). The runtime did not expose a more specific backend model ID or context-window size. Used reasoning, code editing, and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bee0f0e2fa |
fix(server): recognize recoverable result finalization during upgrades (#19651)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution saves an accepted result before it finishes
workspace export and status projection.
> - The upgrade report currently recognizes only a live execution in the
observed phase.
> - A durable accepted result can therefore block an upgrade after its
execution lease is released.
> - Existing finalization owners already resume that saved work with
task binding, physical worker stop proof, and an atomic owner claim.
> - This pull request lets the report inspect those existing contracts
without taking authority.
> - Qualified remote work can continue through a normal server
replacement.
## Linked Issues or Issue Description
Refs #18037 and #19596.
**What happened?**
An accepted remote native result remains in result acceptance, workspace
finalization, retry, or committed projection. The upgrade report marks
it blocking because it is no longer a result-less observed execution
with a live execution lease.
**Expected behavior**
Report a saved result as recoverable only when the existing replacement
owners can resume it. Preserve all task, result, contract, remote
workspace, provider lease, physical owner, decision, and workspace
barrier checks. Keep unsupported or unknown work blocking.
**Steps to reproduce**
Run the new PostgreSQL post-result readiness fixture. It persists a real
encrypted Runner header and an accepted terminal result with a bound
remote workspace. The unchanged report returns one blocker.
**Paperclip version or commit**
Regression base:
|
||
|
DottaandPaperclip
|
374a918d67 |
fix(runner): retain qualified OpenCode artifacts for cold recovery (#19622)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native recovery must keep admitted provider state and useful workspace files. > - OpenCode binds its saved session to exact executable paths and hashes. > - A replacement image can remove those files even when session data is saved. > - Retaining a manifest alone cannot restore the missing launch files. > - This pull request saves the qualified files before provider work and restores the same owned paths. > - The benefit is cold continuity without relaxing the Runner guard or replaying a prior turn. ## Linked Issues or Issue Description Refs #19558. That fix retains the selected pack for ordinary continuation. This change adds durable file retention for a cold allocation. Related #19487 adjusts the admission budget; it does not retain launch files. A search of open and closed OpenCode cold/durable artifact PRs found no duplicate. **What happened?** A saved OpenCode session could outlive the image that supplied its Node executable, proxy, and native executable. A new image supplied different bytes. The Runner then correctly refused the changed launch profile. The saved harness did not contain the old files. **Expected behavior** Core must save qualified launch files before the first turn. It must restore those exact files under the same owned account and absolute paths. Unknown ownership, missing files, and changed hashes must still stop recovery. **Steps to reproduce** Start an admitted remote OpenCode session. Save its workspace, encrypted checkpoint, and harness data. Destroy the original allocation and controller files. Restore into a new allocation with an image whose qualified proxy has different bytes. Resume the same session. The new opt-in local qualification command automates this path with actual OpenCode and a local synthetic model. ## What Changed - Store company-bound, encrypted launch objects and canonical run/backup references. Generate migration 0438 through the normal database workflow. - Stage qualified Node, proxy, and native OpenCode files under a private content-addressed OS-account home. Record the account UID and exact launch paths. Preserve the Rust profile guard. - Gate first provider work on durable capture under the current native claim. Bound Stop/deadline waits while the actual late transfer keeps its worker and scratch ownership until settlement. Restore cold caches with verified staging and atomic rename. Keep live reattachment and historical image-rooted sessions on their existing paths. - Reuse bounded canonical expiration owners. Atomically fence old checkpoint authority with a strict company/run-scoped retirement marker before external deletion. Keep its indexed cleanup anchor across failed deletes and restart. Preserve earlier resume handles and reject later header or source-consumer writes. Add no idle scan or network call inside a transaction. - Add storage, race, executor, filesystem, and actual Rust resume regressions. Add the opt-in local Linux qualification command and operating limits in `doc/opencode-cold-recovery.md`. ## Verification - RED before repair: real PostgreSQL/AES cold restore lacked the launch files; normal executor preparation chose the new image pack. Separate regressions exposed lost upload receipts, missing cache repair, last-owner retirement, an account-UID mismatch, and a resume-handle race during external deletion. Each expected failure was captured before its repair. - Local qualification before the final master integration: 1,169 executor tests, 233 storage/archive/admission/expiration tests, all seven server typecheck layers, migration safety, and the actual Rust cold-resume test passed. - The maintained Linux command passed with two cached immutable Paperclip images whose proxy hashes differ. It removed the original account/controller trees, restored the production published cwd under a new lease, resumed the same actual OpenCode session, kept a useful file and prior model context, and made one new synthetic model request. No external model call or network was available in either container. - The object-store fixture uses real encrypted StorageService streams with an independent owned local directory. It is not hosted S3 acceptance or a complete task HTTP/actor workflow. Historical tasks still require normal effects reconciliation and recovery after deployment. - Review regressions reproduced partial-cache publication, resume-versus-deletion, fresh-writer and source-consumer resurrection, and unbounded Stop/deadline waits before each repair. They now pass through real filesystem, encrypted storage, PostgreSQL, and SDK boundaries. File validation is split from database ownership (384/831 source lines). - Clean runtime head `aadab7acc69ff959678030db46ffe2dcfbefe28f`: the maintained actual OpenCode cold qualification passed with `dirty:false`. It preserved the provider session and recorded launch paths, restored useful workspace content and prior model context, and made exactly one resumed synthetic model request. The 314,560,000-byte launch capture took 23.118 seconds; the later harness was 833,536 bytes. Subsequent repairs change backup-reference retirement and governed continuity-cache classification; capture, restore, launch, and published-workspace inputs are unchanged, so that runtime receipt remains scoped to those paths. The continuity-break consumer is verified separately. - Retention repair `c14d386cecf7123257918ba6a8d47307df2da397`: 68/68 affected real PostgreSQL/AES storage tests and 15/15 focused executor authority/admission cases pass. All seven server typecheck layers pass (source layers and unchanged test shards, with the final 590-file test layer repeated after fixture edits). All review threads have evidence-backed resolutions. Its subsequent CI caught the separate lifecycle-classification gap described below. The refreshed Paperclip review reported paused work, not a completed new review. - Final head `4d1cd3e3309e328b11abc85adc6d78c6d0c7f55d`: all 19 focused root-entry/executor cases and all three source declaration layers pass. [Greptile reviewed this exact head at 5/5](https://github.com/paperclipai/paperclip/pull/19622#issuecomment-6695984874). Final-head [CI verification](https://github.com/paperclipai/paperclip/actions/runs/42308908167/job/128214100339) passes: 60 successful checks, five conditional skips, and the explicitly neutral paused Paperclip review. No check is pending or failing. - CI also caught the launch cache missing from the session-root lifecycle contract. Both the writer scan and a real governed-replacement consumer failed locally before repair: old launch bytes remained in the active root. The cache now archives with the prior session state; the consumer verifies exact bytes are preserved in the archive and absent from the active root. - The latest review found that backup retirement could delete an independent run pin while a selected session retained the checkpoint. A real PostgreSQL/AES regression failed before the repair. Backup-only ownership now uses a NULL-safe exact backup predicate; the run pin belongs to fenced checkpoint expiration. The regression verifies checkpoint decryption and launch-file restoration after pending-backup cleanup, followed by deletion only after removing the final resume handle. - An executor first-session case checks new-session durable capture through the real capture helper and receipt propagation into the actual harness manifest at the backup boundary. Real storage cases verify the run-profile receipt and canonical backup/run pins. - Sequential closure reuse and simultaneous first admission are different scopes. A deterministic two-uploader test measures both independent uploads, one verified ready winner, loser-only deletion, and usable restoration for both runs. Each capture is bounded to 512 MiB and five minutes; up to three pending receipts per run are retained. There is no claim of global single-flight capture or a company-wide egress bound. - Earlier optional warm-ownership verification encountered missing local binaries. After exact-source setup, all nine ownership assertions passed; two other suites encountered local PostgreSQL setup/teardown contention, so those interrupted attempts are not passes. Required current-head CI supplies the broad final coverage. No deployment or hosted historical recovery is claimed. ## Risks - This adds two tables and a forward migration. The file object is bounded to 512 MiB. Capture adds one initial transfer; later harness archives exclude those large files. Company-scoped ready objects are shared and retained until the last canonical owner retires. Concurrent initial admissions can duplicate bounded transfers; each keeps independent ownership and cancellation rather than waiting on another controller. A continuity break retains the controller cache with its prior state until normal session-root cleanup, adding up to one bounded launch closure per retired state. - Cold continuity requires the same qualified Linux platform, supported runtime dependencies, owned account UID/home, and published absolute cwd. A mismatch fails closed. No alias or Rust guard change is included. - Existing image-rooted profiles have no retained file object. This change cannot backfill missing historical binaries. A fresh governed session must reconcile prior effects first. - Live reattachment does not read artifact objects. Storage or provenance failures before a new turn hold admission. They do not grant new execution, model, tool, or replay authority. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and local browser tools. The exact served model ID and context window are not exposed in this session. No claim is made for unavailable model metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
349b8b3f7b |
Migrate settled controller checkpoints while unrelated work runs (#19631)
## Thinking Path > - Paperclip manages work performed by AI agents. > - Server replacement must preserve retained Runner sessions. > - Older sessions can retain controller checkpoints on the server volume. > - The existing importer requires all execution writers to stop, including unrelated runs. > - This change adds a separate online path for successful, settled sources. > - It locks the source task and finalizer while it reads and publishes that source. > - Unrelated work continues. The offline importer keeps its existing authority. ## Linked Issues or Issue Description **What happened?** A retained controller checkpoint cannot be converted by the existing maintenance importer while another task runs. Requiring a busy server to become globally idle prevents online migration. **Expected behavior** An operator can convert a successful retained source after its finalizer releases ownership, while unrelated tasks continue. An expired owner remains an owner. Related active work and uncertain evidence remain blocked. **Steps to reproduce** 1. Retain a valid legacy controller for a successful run with an accepted result and a released committed finalizer. 2. Start a different task. 3. Run controller preparation. The old importer refuses because execution writers are active. Related preservation contracts: #19038 and #19024. This change does not replace workspace migration or authorize volume removal. ## What Changed - Add `prestageNativeControllers` with task, finalizer, and source-run row locks. Recheck related runs and exact checkpoint identity under those locks. - Allow legacy checkpoint material through normal settled-owner import checks. Keep offline authority separate. - Compare captured local controller and supervisor bytes before commit. A changed source rolls back publication. - Add `--controllers-only` to the existing finite native preparation entry. This avoids re-reading provider archives during controller conversion. - Add PostgreSQL cases for unrelated work, expired owners, related active work, source-lock contention, and source changes. - Document the finite command and its limits. ## Verification - Real PostgreSQL affected suites: 56 passed across controller migration and recovery. A first per-server run was interrupted because it was slow; the completed run used the repository's shared PostgreSQL test wrapper. - Initial online cases: 3 passed, covering unrelated work, an expired owner, and related active work. - Source lock and changed-file cases: 2 passed. Publication rolled back on a source change. - Final source-evidence regression: 2 passed. A first unproven source deferred and a later valid source imported; changed-file rollback remained intact. - Server TypeScript check passed with the SDK and Runner declarations built. - Server package build passed, including the Runner vendor checks and compiled command. - Runtime bundling suite: 10 passed. - Full repository CI and exact-head automated review remain required before merge. This draft is not staging acceptance. - PASS: 7 CLI cases after adding the controller-only dependency fixture and explicit-mode coverage. The first CI run exposed missing mock/export coverage; the fixture repair passed after rebase. - The unrelated model-default failures were fixed on master; this branch now includes that fix. ## Risks - A source task can wait briefly on its own migration lock. A one-second lock acquisition budget defers a contended source. No global admission hold is acquired. - Only successful sources with a released committed finalizer are eligible. Failed or unsettled sources stay deferred and require their preservation owners. - The operator must run the command on the host that owns the original files. The command does not certify enrollment readiness or detach any volume. - The existing offline path remains unchanged and still requires stopped writers. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
15ee6a25a6 |
fix(native): re-bootstrap a claimed Daytona run killed before runner launch (#19616)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. Restart recovery decides what to do with each live run after
the control plane restarts.
> - #19583 lets recovery start the same-run bootstrap again when the
Postgres controller checkpoint proves that the runner received only
bootstrap commands.
> - A live replay found an earlier window. The control plane died after
the executor claimed the run and staged the sandbox workspace, but
before it asked for a runner process. No runner checkpoint existed, so
#19583 did not apply.
> - Recovery classified that run as `ambiguous_remote_recovery_evidence`
and blocked the task. The task also showed "Workspace repair required",
which did not describe the real cause.
> - This pull request proves the "nothing reached a runner" case from
durable evidence and starts the bootstrap again. It also names the
restart in the settled hold note.
> - The benefit is that a restart at any point of Daytona startup before
runner launch continues automatically.
## Linked Issues or Issue Description
**What happened?**
A chaos replay killed the control plane about 2 s after a claimed ACPX
Claude run started on Daytona. The kill came after `stage.workspace` and
environment acquisition, before `native.process_start_requested`. The
restarted server took over because the previous controller process was
dead. It then returned `ambiguous_remote_recovery_evidence` with
disposition `blocked`. The task rested blocked as "Workspace repair
required".
Evidence in the saved database at that point: coordinator `attempt: 1`,
`controllerGeneration: 1`, previous controller identity recorded and
dead. Native workspace sync was `prepared`. There were no runner
checkpoint rows, no session checkpoint, no process PID or start time, no
provider session, and no execution-start or runner-sourced event. The
local session root was an empty directory.
**Expected behavior**
When durable evidence proves that no runner process was requested and no
controller checkpoint exists, recovery continues the run with a fresh
bootstrap. Any evidence that a runner may have started keeps the hold.
**Steps to reproduce**
1. Start a native run on a Daytona environment.
2. Kill the control plane after workspace staging and before the
executor requests the runner process.
3. Restart the control plane.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.
Root cause:
`server/src/services/native-runtime/native-restart-recovery.ts`,
function `classifyNativeRunnerRecoveryEvidence`, remote branch. The only
no-checkpoint remote rule required an unclaimed coordinator (`attempt:
0`, no controller identity). A claimed run with no launch evidence fell
through to "ambiguous".
Refs #19583, #16349
## What Changed
- `native-restart-recovery.ts`: add the `remote_runner_never_launched`
proof, which leads to `bootstrap_incomplete`. It applies only to a
classic sandbox run with a prepared workspace and the preparation
fencing version. The run must have a real executor claim (controller
identity and lease owner) whose controller is gone (takeover allowed).
It needs no durable checkpoint, no session checkpoint, no process
fields, no provider session, and no provider evidence. The run must also
have no `native.process_start_requested`, process identity,
`adapter.invoke`, `tool.execution.started`, or runner-sourced event. The
same three-attempt budget applies. The shared event query moved into
`hasNativeExecutionStartEvidence`.
- `native-session-executor.ts`:
`restoreRemoteBootstrapCheckpointForRecovery` removes an empty session
root that preparation created. Without that step, the fresh bootstrap
read it as `durable_identity_unreadable`.
- `execution-recovery-resolution.ts`: the new
`executionRecoverySettlementNote` names a settled
`native_restart_recovery_blocked` hold as a stop after a server restart.
It no longer shows "Workspace repair required" because a later workspace
export was retried. Other causes keep their notes.
- `doc/run-log-events.md`: document the new reason.
## Verification
- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "unlaunched|never requested|Postgres checkpoint"`. Before the fix,
the proven case (lease live with a dead owner, and lease expired) and
the budget case returned `blocked` /
`ambiguous_remote_recovery_evidence`. The empty-root executor case
failed.
- Fail-closed cases: these keep the hold:
`native.process_start_requested`, a process identity event,
`adapter.invoke`, a runner-sourced event, a recorded PID, a recorded
provider session, and a spent budget. A live claiming controller returns
`awaiting_evidence` / `controller_still_alive`. The existing
"contradictory evidence" cases (`claimed` with no controller identity,
`launch`, `process`, and others) stay blocked.
- Wording: `server/src/services/execution-recovery-resolution.test.ts`.
- Regression set (10 files, including `workspace-recovery`,
`native-safe-replacement`, `heartbeat/recovery`,
`native-session-executor`, and `native-restart-recovery`): 2035 passed,
14 skipped.
- Live Daytona replay of the failing chaos task, 3 runs, sequential,
ACPX Claude, double SIGKILL, current master plus this change. Image:
content-matched `paperclip-daytona-runner@sha256:5a29a797…` built from
|
||
|
DottaandPaperclip
|
c9391720a1 |
fix(heartbeat): send saved follow-ups after task completion (#19600)
## Thinking Path > - Paperclip manages agents and their work. > - Task messages remain in a durable queue while an earlier run saves its result. > - That result can mark the task done before the user sends the saved messages. > - Send records current user intent, but its wake guard allowed only active task states. > - The old queue therefore remained pending with an obsolete finalization message. > - This change admits that exact Send receipt and reopens the task through the issue service in the successor transaction. > - The agent can receive the saved input once while the completed result remains unchanged. ## Linked Issues or Issue Description **What happened?** Send on a saved message can leave the message queued after its native predecessor succeeds and commits. The old wait message remains visible. Allowing admission alone also fails: continuation construction rejects a task that remains done. **Expected behavior** Explicit Send must deliver the saved input once through normal admission. It must retain the previous run and its result. **Steps to reproduce** 1. Save a follow-up while a native run finishes its result. 2. Let that run succeed, commit its accepted result, and mark the task done. 3. Send the saved follow-up. Before this change, the receipt stays deferred. Related public work: Refs #19542. That change selects recovered workspace files. This change handles the later saved-message admission. Refs #19602 for the independent workflow fixture repair required by CI. ## What Changed - Allow the exact explicit Send wake guard to accept a done task. Keep automatic recovery and cancelled task guards unchanged. - Reopen the task through the company-scoped issue service after admission gates pass. Commit reopening, queue consumption and the successor together. Publish the audit after commit. - Add a native succeeded/committed predecessor regression with unchanged saved work and one successor after two resume calls. - Verify real heartbeat startup under the clicking operator. Keep automatic input, cancelled tasks and paused agents held. - Check workspace preflight for the prospective reopened task. Use the normal company-scoped transition if preflight blocks a done task, so its completion metadata is cleared. - Check a successful native predecessor for its committed accepted result, current ownership and pending cleanup. Keep approvals and questions effective even without a recovery action. - Document the saved Send contract and update the queue test dependency. ## Verification - Final merge qualification for `675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`: all 56 applicable status checks pass, with no pending or failed checks. [Final CI](https://github.com/paperclipai/paperclip/actions/runs/42307623559) includes typecheck, build, all applicable test and E2E shards, and aggregate verification. Current-head Greptile is 5/5; all five review threads are resolved. Any paused or queued application review is not claimed as a new completed review. - Final base qualification at `675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`: official preflight passes (1.5s); all six integrated model-normalization cases pass (19.07s); both affected test typecheck layers pass. The integration changes no production server/shared/Runner/adapter inputs, so the preceding 12-case Send proof, two ordinary status guards, and unchanged production typecheck layers remain applicable. Fresh final-head CI supplies broad coverage. - New review regression: both failed and cancelled native predecessors on a task manually marked Done fail before the repair with `continuation_task_ownership_changed` through real `resumeQueuedCommentInterrupt` admission. At `bfb0eff7941090ac3ac43ce4366ba2e3d46612e9`, all 12 affected Send/source/cleanup/workspace/decision cases pass (246.80s), ordinary Done/cancelled rejection cases pass (2; 100.31s), and all seven server typecheck layers pass. History validation uses the prospective reopened status only for an already-verified explicit native Send. No status is changed during a dry run. - Prior `e64e4bb1` CI passed the affected serialized server suites, including the complete saved-message route suite, typecheck and build. General server shard 4 still failed two unrelated stale default-model expectations (`gpt-5.6-sol` versus the new `gpt-6.1-sol` default). The equivalent repair has already merged in #19610 as `8621bcf53ee2e4bbefb7d193add42dd544ca06e8`; the separately prepared #19645 is closed as superseded. This source fix does not weaken or edit those assertions. - Original RED: the committed-source and completed-task startup cases left the saved receipt deferred. - Review RED: Done plus an unrunnable isolated worktree created one successor instead of zero. A successful committed predecessor with pending lease cleanup also created one instead of zero. The new Done-to-blocked path retained stale completion time. Pending approvals and questions each allowed a successor. All failures were captured before their respective repairs. - Integrated production revision `99018264dd1ef002ff3500378184bcd167d9206b`: 417 affected cases pass across the complete native continuation, responsible-user startup, and workspace preflight suites (386 + 18 + 13). These include the original successful startup case and six cleanup/finalization faults that hold the exact saved receipt until settlement, then deliver it once. - `pnpm --filter @paperclipai/server typecheck` passes all seven layers on that integrated head, including the local Runner/SDK dependency build. Module-boundary, whitespace, and added-code-comment checks pass. - Normal master integrations preserve branch history. No force push or assertion weakening was used. Fresh CI and automated review are running on `675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`, which normally integrates the already-merged #19610 fixture repair after the reproduced inherited CI failure. - The earlier CI attempt failed in five unchanged workflow fixture cases because IMAGE_ONLY was unset. The merged #19602 repairs that fixture. A later inherited duplicate default in that independent fixture has a separate repair; this PR does not edit it. - Follow-up route-fixture RED: three existing native queue delivery cases failed because their fixture marked a run successful without an accepted result or committed finalization. The same three cases pass after the fixture records the complete native success contract (42.08s); delivery assertions and production gates are unchanged. The affected test-tree TypeScript layer also passes. The full route run passed all 119 assertions but failed its 30-second database teardown hook, so it is not recorded as a suite pass. That result is retained. One unchanged bounded rerun was stopped after more than 15 minutes without a final verdict; it is inconclusive, not a pass. - Final integrated head `e64e4bb1`: all 13 focused route and completed-task admission cases pass across two real-PostgreSQL suites (188.95s), including the three repaired route fixtures and every new cleanup/finalization/decision hold. Fresh preflight passes; all seven server typecheck layers pass. Required final CI supplies the broader route and server coverage. - These tests use real PostgreSQL and normal admission, cleanup, queue consumption, and startup services. The startup suite controls the provider boundary. Hosted provider acceptance remains pending normal deployment. ## Risks - Explicit Send can reopen a done task. This is limited to its validated saved receipt and current assignee. A cancelled task remains held. - Reopening uses the existing company-scoped issue transition and shares the successor transaction. A failed admission does not reopen the task. Provider Stop, workspace, approval, pause and budget checks retain their existing owners. - Cleanup and pending decisions retain the done task and saved receipt. An unrunnable workspace uses the existing blocked-task diagnostic. No schema or runtime package changes. ## Model Used - OpenAI Codex, GPT-6 family; exact serving snapshot/context window not exposed; repository analysis, code editing, tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
795c3c4c1d |
fix(recovery): continue saved input after historical retry pause (#19576)
## Thinking Path
> - Paperclip manages work performed by AI agents.
> - Its scheduler parks work when another run holds a shared workspace.
> - A pause can refuse the scheduled retry before provider dispatch.
> - Older schedulers recorded that refusal in the run log without a JSON
marker.
> - After the pause was released, a saved user message could remain held
for execution recovery.
> - This change verifies the original refusal and parent workspace-wait
proof before admitting one continuation.
> - Existing execution, Stop, budget, approval, and workspace guards
remain in force.
## Linked Issues or Issue Description
Refs #19466, #19552, #19580, #19602.
**What happened?**
A saved user message remained deferred after a pause was released. Its
source was a never-claimed workspace retry. The historical source had an
`issue_paused` error and no JSON cancellation marker. The continuation
proof accepted only a different system-cancellation shape.
**Expected behavior**
The normal saved-message worker admits the existing input once when
trusted scheduler evidence proves no provider dispatch occurred and
current admission gates permit work.
**Steps to reproduce**
1. Record a host-certified pre-provider workspace wait and its scheduled
retry.
2. Let the real scheduler refuse promotion under an active task pause,
including after the user presses Retry Now.
3. Keep the historical run-log-only cancellation shape, then release the
pause.
4. Run concurrent normal saved-message recovery scans.
**Paperclip version or commit**
The regression fails on master `f0006c7783`. It leaves zero queued
successors. The branch now normally integrates the fixture repairs in
#19602 and #19628.
## What Changed
- Accept the historical cancellation shape only when a trusted system
lifecycle warning records the exact scheduler pause refusal. A prior
Retry Now event does not invalidate that proof.
- Bind the event to a released same-company pause and its task
membership.
- Reuse the existing parent workspace-wait certificate,
no-provider-output checks, and retained-execution rejection.
- Preserve historical cancellation records and consume the existing
saved input through normal admission.
- Document the contract and add real PostgreSQL coverage using actual
scheduler suppression and pause release.
## Verification
- RED: the real scheduler suppressed the retry under a pause. After
release, two normal recovery scans left zero successors.
- GREEN: the same path creates one successor and coalesces the original
saved request.
- All 389 tests in the complete native continuation suite pass before
the final base update. The only changed continuation source from master
is the separately merged one-line repair in #19552.
- Fourteen focused cases passed at
`2aaeb50f78ba0a22809ab96a00076ce50fcea559`, including a parent pause
covering a child task. Thirteen cases also passed before the base
update. They cover the normal path and missing event, wrong task,
missing membership, active or mistimed pause, provider activity,
retained lease, operator Stop, and invalid ancestry.
- The earlier seven-layer server typecheck passed after building this
worktree's Runner and SDK dependencies. The current review fix passes
the exact CI test-layer typecheck. Fresh required CI will supply
current-head source coverage.
- The first sandboxed database run could not bind loopback. A later
fresh dependency install omitted native PostgreSQL library symlinks and
failed before executing tests. The package's own postinstall hydration
repaired that setup. Previous-head verification passed all 14 focused
cases at `2aaeb50f78ba0a22809ab96a00076ce50fcea559`.
- Module-boundary and whitespace checks pass.
- Commands: `cd server && pnpm exec vitest run
src/services/explicit-native-continuation.test.ts`; `pnpm --filter
@paperclipai/server typecheck`.
- Environment: macOS arm64, Node 26.4.0, pnpm 9.15.4, real local
PostgreSQL.
- Previous head `2aaeb50f78ba0a22809ab96a00076ce50fcea559` had Greptile
5/5 and 56 passing checks, with three skipped. A later advisory review
found the Retry Now event-order case. Those checks do not qualify the
new head. Hosted recovery remains unverified.
- Review RED: the real `retryScheduledRetryNow` producer wrote
lifecycle/info at sequence 1 and the exact pause warning at sequence 2.
After real hold release and two concurrent normal queued-message scans,
both task-rooted and parent-rooted cases produced zero successors
instead of one. Both failed before the production change (17.24s).
- Review GREEN: all 17 focused cases pass at
`f38e9934b8ce3f7ddfb23eaa0f979c3f44e7a990` (4.74s). The previous 14
cases remain covered. Two new cases use the real Retry Now producer, and
one new case rejects contradictory provider execution evidence. No
pause, membership, timing, ancestry, output, retained-execution, or Stop
check was removed.
- Current focused command: `pnpm exec vitest run
server/src/services/explicit-native-continuation.test.ts -t 'historical
pause|Retry Now pause history' --fsModuleCache`. Current test-layer
typecheck: `cd server && node ../node_modules/typescript/bin/tsc
--noEmit --project
node_modules/.cache/paperclip-typecheck/tsconfig.tests.json --pretty
false`. Both pass.
- Final-head verification: all 17 focused cases passed again at
`c88687d69f2c87fe1b81b5fffd670e3cf26a884d` (8.00s). The prior exact CI
test-layer typecheck qualifies the unchanged typed logic; fresh CI
supplies broader final-head coverage.
- The follow-up commit removes only the new code comments to follow
CONTRIBUTING.md. Recovery and test behavior are unchanged. That revision
had Greptile 5/5 with its actionable review thread resolved. It is
retained as historical verification; current-head qualification is
below.
- The first final-head CI attempt failed in the unchanged CLI company
import/export fixture before assertions: its child server did not reach
the runtime-info health check within 30 seconds. Other suites passed.
The exact fixture passes locally with its original timeout. The single
failed-shard retry passed at the unchanged head, together with its
dependent CI checks. [Original failed
job](https://github.com/paperclipai/paperclip/actions/runs/42279773036/job/128106195975).
No assertion or timeout was changed.
- The independent required Runner support failure reproduced five
`IMAGE_ONLY: unbound variable` errors. #19602 repaired its fixture
inputs and merged as `4d554f08218b04df49ec0f2a0d5c5dfdfaf65660`. This
branch normally merges that master with no conflicts or force push. The
current host-preparation log verifier from #19580 remains intact.
- Final integrated head `ffc9e8d9516343a9823b4b80b5fcfccc548844ba`:
17/17 focused recovery cases pass (12.71s), all nine workflow-target
cases pass through the official Runner Vitest config (1.10s), and all
seven current server typecheck layers pass. Worktree preflight rebuilt
current Runner dependencies and reused the Cargo seed; it completed in
61.3s with Rust debug binaries skipped.
- Prior integrated head `ffc9e8d9516343a9823b4b80b5fcfccc548844ba`
finished with 54 successful checks. Its only code-check failure was a
global unhandled readiness rejection after 2,285 assertions passed in
the server8 shard, plus dependent verify. The real-child race is fixed
separately in #19624; it was not hidden by a rerun here.
- Final normal integration head
`550d1732b6960d6b349d03903f448fa1d71c17f6`: exact 17 paused-retry
lifecycle regressions pass (71.06s), all nine official workflow target
cases pass (2.43s), all seven server typecheck layers pass, and
module/whitespace checks pass. Required preflight completed in 45.641s.
Current preparation-log guards from #19580 remain intact. The diff
against master contains only the original three owner/test/doc files.
- Prior head 550d had two inherited model-default assertion failures
plus a retained-cleanup backlog 15s timeout. The exact unchanged cleanup
case passes locally (41.91s total including DB/hooks); no timer or
assertion was weakened and no old CI job was rerun. The model fixture
repair was independently merged in #19610 at
|
||
|
Nicky LeachandClaude Opus 5.5
|
a2aa69bc62 |
perf(ci): split the Runner package Vitest lane into three duration-balanced lanes (#19646)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - The Runner package Vitest lane is the PR CI critical path. It finishes a median 15.3 minutes after the push > - That lane runs all 311 package test files in one job with 2 workers: 1,381 seconds of serial work and 727 seconds of wall time > - Ten files account for 61% of the time. Vitest's `--shard` splits by file count, not by duration, so it cannot balance this work > - This pull request splits the package files into three lanes balanced by duration, and runs the native server group in its own lane > - The benefit is about 232 seconds of tests per package lane instead of 727 seconds ## Linked Issues or Issue Description **What existing behavior does this improve?** The time that PR CI needs to run the Runner package Vitest tests. **Subsystem affected** CI: the `Verify Paperclip Runner` Vitest lanes in `pr-trusted.yml`, and `packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs`. **Current behavior** The `vitest package` lane takes a median of 13.6 minutes. It is the last job to finish in most PR runs. The lane keeps a 2-worker limit so that Runner and sidecar processes can start within their deadlines. Average CPU use is 29%. **Proposed behavior** Three package lanes each run about one third of the files, balanced by measured duration. Each lane keeps the 2-worker limit. The native server group runs in its own lane, as before. **Reason and benefit** The Runner lane stops being the critical path. More workers per lane would add CPU contention and timeouts, so this change adds lanes instead. The cost is about 3 runner-minutes per run for the extra setup. **Breaking changes** None. The `master` ruleset requires only `ci / verify` and `ci / e2e`, so the new job names do not change required checks. ## What Changed - Add `packages/paperclip-runner/scripts/vitest-package-lanes.mjs`. - It gets the file inventory from `vitest list --filesOnly --json`, so it follows `vitest.config.ts`. - It partitions the files with the `partitionGeneralServerSuites` function from `scripts/general-server-shard.mjs`. - Add `packages/paperclip-runner/scripts/vitest-lane-durations.json`. - It uses the same schema as `scripts/general-server-shard-durations.json`. - It is seeded from CI run 42290920882. It measures 310 of the 311 files. - `run-pr-vitest-lane.mjs` accepts two new arguments: - `--package-lane=K/N` runs one partition. - `--native-server` runs only the native server group. - The `--shard=1/2` and `--shard=2/2` arguments keep their exact behavior for the matrix that master still sends. - `.github/ci-task-matrix.json` and the fallback matrix in `pr-trusted.yml` have three `vitest package (K/3)` lanes and the `vitest native server` lane. The labels keep the `vitest` prefix because `scripts/ci-profile/trace.mjs` matches `Verify Paperclip Runner (vitest`. - Tests: - the lanes cover the inventory exactly once, checked with real `vitest list`; - the old `--shard` commands plan the same commands as before; - the manifest schema is correct; - bad lane arguments are refused. - Update the Runner Vitest lane paragraph in `doc/DEVELOPING.md`. ## Verification - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: 44/44 pass. - `node --test scripts/__tests__/runner-release-crate-cache.test.mjs .github/scripts/tests/ci-plan.test.mjs`, together with the file above: 71/71 pass. - Related workflow, cache, routing and profile tests: 446/446, 153/153 and 14/14 pass. - Local end-to-end run of `GITHUB_WORKFLOW=PR node scripts/run-pr-vitest-lane.mjs --package-lane=3/3` on macOS: - the prep steps ran, and the lane ran 104 of 311 files in 123 seconds; - 2 tests failed: a macOS `kill EPERM` and a 5-second timeout under full local load without the CI worker limit; - both files pass when run alone (35/35 tests). - This PR's own CI runs the matrix from master, so it uses the old `--shard` commands. The new lanes run in CI only after merge. Reviewers can compare the job durations of the first PR runs after the merge. | Lane | Files | Serial estimate | Wall estimate with 2 workers | | --- | --- | --- | --- | | 1/3 | 103 | 464.9 s | ~232 s | | 2/3 | 104 | 465.0 s | ~232 s | | 3/3 | 104 | 465.0 s | ~232 s | ## Risks - The weights come from one CI log and do not include collection or setup time. The balance can be uneven until a refresh measures real lane runs. - Each lane passes its file paths to Vitest as filters, and Vitest matches a filter as a substring of the path. A path that is a substring of another path would run a file in two lanes. No current path does this. If one does, the script prints a warning, and no file is dropped. - Each package lane runs `ensure:eval-build-deps` and `build:rust` again, which adds about 3 runner-minutes per run. - A comment in `scripts/run-vitest-stable.mjs` (about line 110) still says that the last Runner shard runs the native group. A follow-up will correct it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it and ran the local lane check. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
2e340d6235 |
fix(ui): show unread message counts in agent chat pickers (#19627)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - People switch between agent chats to read replies. > - The agent picker does not show how many messages are unread. > - People need that count before they select a chat. > - The chat list already has the viewer's read state. > - This pull request adds numeric badges from that shared list. > - People can find unread replies without extra requests for each agent. ## Linked Issues or Issue Description **What existing behavior does this improve?** The chat header agent picker and the secondary sidebar's Add chat picker. **Subsystem affected** Agent Chat UI, shared task types, and the chat read-state service. **Current behavior** The pickers do not show the unread message count for each agent. **Proposed behavior** Show the unread count beside each agent. For example, show 3 for three unread agent messages. Hide the badge when the count is zero. Keep switching available while chat data loads. **Reason and benefit** People can see which chats need attention before they switch. The count uses the existing shared request and cache. **Breaking changes** The chat response gains an optional `unreadMessageCount` field. Existing fields and read receipts keep their behavior. There is no database migration. **Additional context** Related: #19425 changes chat structure. This change adds counts to the current agent pickers. The duplicate search found no PR for these count badges. ## What Changed - Count unread agent comments in the existing grouped read-state query. Keep company and viewer scope. - Exclude human messages, deleted comments, system notices, and fast-response receipts. - Pass memoized count maps to both agent pickers. Use a direct lookup for each row. - Show numeric badges and accessible singular or plural labels. - Test count changes, account isolation, cached opens, search, and large agent lists. - Update the chat stories, fixture data, and development guide. - Make the Slack owner retry-deadline test deterministic after reproducing its CI failure. ## Verification - On current head `05f5fa6201`, the [main CI workflow](https://github.com/paperclipai/paperclip/actions/runs/42305845930) passed, including build, typecheck, all applicable test shards, browser shards, and final verification. The prior server shard 2, 6, and 7 failures do not recur. - [Greptile's current-head review](https://github.com/paperclipai/paperclip/pull/19627#issuecomment-6696161631) is 5/5, its check succeeded, and both previous review threads are resolved. Current head check inventory is complete: 55 successful checks, three intentional skips, and no pending or failed checks. The Paperclip application review summary reports paused; no new completed Paperclip review is claimed. Its existing findings are resolved, and the independent current-head Greptile review is 5/5. - Current head `05f5fa6201` merges master `8621bcf53e` to resolve an import conflict. Master now includes the same model-default test repair. This branch keeps its upstream import, so that file no longer differs from master. - After this merge: all 21 server tests in the Slack owner and model-default suites pass, all 22 tests in the three picker/sidebar suites pass, and token gates pass. The standard preflight still cannot build Rust because `cargo` is absent. Fresh CI must supply the broad checks. - Head `7d125fb8c7` replaces the Slack owner test's real timer wait with the worker's existing clock input. It checks that the intent stays pending one millisecond before the retry deadline and clears at the deadline. All 15 Slack owner integration tests pass locally. Runtime retry behavior is unchanged. - On `4cfb1e9d55`, both review checks gave 5/5. CI [server shard 2](https://github.com/paperclipai/paperclip/actions/runs/42302458830/job/128183671349) failed the Slack credential-lease retry test. The same assertion failed in an unchanged local run. A diagnostic run with added logging passed; the exact timer deviation was not captured. The new test removes the real-time wait assumption and checks the clock boundary directly. The earlier model-default and cleanup failures did not recur in that CI run. - On `a3378dfc40`, both Greptile and Paperclip Review gave 5/5 with no open findings. Typecheck, build, all eight browser shards, and most test shards passed. - That CI attempt failed two server shards. The model-default test still expected `gpt-5.6-sol`, while the current default is `gpt-6.1-sol`. I reproduced both failures locally. Commit `4cfb1e9d55` uses the exported default-model constant; all six tests in that suite now pass. Production model selection does not change. - The retained-cleanup backlog test exceeded its 15-second limit in [server shard 6](https://github.com/paperclipai/paperclip/actions/runs/42299858777/job/128175193320). The unchanged test passes locally with the same limit. Its CI timeout cause is not established. No assertion or timeout was changed. The fresh CI run must recheck it. - Review repair: commit `0db631b215` keeps the existing-chat and unread-agent arrays stable when chat data and the active agent do not change. A new 500-agent test fails before the repair and passes after it. The test also checks count refreshes. - The first CI typecheck failed on duplicate `IMAGE_ONLY` and `TESTOPS_REQUESTED` keys in `tests/runner-e2e/workflow-target.test.ts`. This was present in upstream code. Master commit `e482f0f632` removes the duplicate keys. This branch now includes that commit. All nine workflow-target tests pass locally. - After the latest master merge: all 22 tests in the three affected picker/sidebar suites pass. Before that merge, all 38 focused UI tests passed. - The current head is `05f5fa620134ad199a3d85b66fb2eb93d3bb6077`. All current status checks are complete, the aggregate verify gate succeeds, mergeability is CLEAN, and no unresolved review threads remain. The maintainer has instructed normal merge after checks pass. - After merging current master: 37 focused UI tests pass across AgentChatPicker, CoordinatorAgentSwitcher, AgentConversationsSidebar, useAgentChatRead, and SidebarAgentChats. - After merging current master: all 38 proactive-chat integration tests pass with a real disposable PostgreSQL database. They check counts of 2, 1, and 0 at exact read cutoffs. - The prior revision passed UI typecheck and build, shared build, isolated low-memory server typecheck, token gates, and 30 map/catalog tests. - The 500-agent test bounds index work and count lookups. Search does not rebuild the existing-chat index. - Browser checks used production components with simulated API data. Dark, light, mobile, and Add chat cases passed. Counts of 3 and 12 stayed visible through search. Read agents had no badge. Cached opens made no chat API requests. - Twenty local cached opens took 34.5–57.6 ms, with a 38.4 ms median. This is an unthrottled fixture measurement. - Earlier full local checks could not complete because Rust cargo is absent and the workspace has limited memory. The standard worktree preflight also stops at missing cargo. Required CI must supply broad checks for this PR head. - The first server typecheck ran out of memory while Storybook was active. Its isolated low-memory retry passed. No visual baseline comparison or live provider delivery is claimed. ## Risks - Low risk. The count replaces a boolean aggregate in the same grouped query. Very long chat histories still need that existing scan. - A count can be stale until the existing chat-list cache refreshes. Read receipts still control the unread cutoff. - The change adds no polling, per-agent history request, database migration, or permission change. ## Model Used OpenAI GPT-6 through Codex, with code editing, terminal tools, and browser verification. The runtime did not expose a more specific served model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
f2299cb00f |
feat(ci): re-run failed PR jobs once when every failure was a lost runner (#19640)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - PR CI runs on self-hosted AWS runners, and these runners are sometimes lost mid-job > - On 2026-11-06, 21 of 51 failed jobs on commits that later passed were runner loss. The losses came in fleet-wide bursts, for example 10 jobs on 10 instances in 51 seconds > - After each loss, a person must notice the failure and click "Re-run failed jobs" > - This pull request adds a workflow that re-runs the failed jobs once, only when every failure was a lost runner > - The benefit is fewer manual re-runs and a shorter wait for CI to pass ## Linked Issues or Issue Description **What existing behavior does this improve?** The recovery of PR CI runs that failed only because a self-hosted runner was lost. **Subsystem affected** CI: the trusted PR workflow (`pr.yml` → `pr-trusted.yml`). **Current behavior** When an AWS runner is lost, the job fails with "The self-hosted runner lost communication with the server" or "The runner has received a shutdown signal". The PR stays red until a person re-runs the failed jobs. **Proposed behavior** When the `PR` workflow completes with a failure on attempt 1, a new workflow examines the failed jobs. If every failed job lost its runner, it re-runs the failed jobs one time. If any job failed for another reason, it does nothing. **Reason and benefit** A runner loss is not a code problem. The re-run starts without waiting for a person to notice the failure. **Breaking changes** None. ## What Changed - Add `.github/workflows/ci-runner-loss-rerun.yml`. It triggers on `workflow_run` for `PR`, and it acts only on attempt 1 failures of `pull_request` runs in this repository. It runs on `ubuntu-latest`, so it does not depend on the AWS fleet. - Add `.github/scripts/runner-loss-rerun.mjs`. The script: - lists the attempt-1 jobs and ignores the aggregate jobs `ci / verify` and `ci / e2e`; - finds runner loss in the check-run annotations. If the only annotation is "The operation was canceled.", it reads the `##[error]` lines in the job log; - calls `rerun-failed-jobs` only if every remaining failed job lost its runner; - does not re-run when the gate job failed, the PR head moved, the PR is closed, or a newer run exists for the same commit; - writes the decision and a job table to the step summary. A `--dry-run` mode never posts. - Add 16 tests, with trimmed fixtures from four real runs. - Add a paragraph about the workflow to the "Trusted PR Workflow" section of `doc/DEVELOPING.md`. ## Verification - `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 16/16 pass. The tests cover: - re-run when all jobs were lost, including the case where the signal is only in the log; - no re-run for mixed failures, real failures, only aggregate failures, attempt 2, a failed gate, or a stale run; - pagination, incomplete listings, a cross-host link that is refused, and the workflow guards. - `node --test '.github/scripts/tests/*.test.mjs'`: 760/760 pass. - A read-only dry run against the real GitHub API for six recent runs classified all six correctly. - The workflow can run on GitHub only after it is on master. Reviewers can make sure that the first runner-loss failure after the merge triggers one re-run. ## Risks - **Trust:** a re-run of failed jobs uses the outputs of the gate job that succeeded on attempt 1, including its AWS runner admission. A person who clicks "Re-run failed jobs" gets the same result. The only difference is that `github-actions[bot]` triggers attempt 2. The workflow never checks out or runs PR code. - **Failed gate:** if the gate job itself failed, the script does nothing. A re-run by the bot would put all jobs on GitHub-hosted runners. - **Profiling:** `scripts/ci-profile/export.mjs` exports only attempts triggered by authorized users. The bot-triggered attempt 2 is not exported to Honeycomb. Attempt 1 is still exported. - **Events:** GitHub may not create `workflow_run` events for attempts that `GITHUB_TOKEN` started. Then other `workflow_run` consumers do not react to attempt 2. Required checks still update. - **Permissions:** `actions: write`, `checks: read`, `contents: read` and `pull-requests: read`. The last one lets the script confirm that the run still belongs to the open PR's current head. - **Cancelled jobs:** a runner loss where the lost jobs end as `cancelled` and only the aggregate jobs show `failure` is not re-run. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
913102870d |
fix(environments): fill the managed instance default once, under a lock (#19632)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On a managed instance under `enableManagedSandboxOnly`, boot points the instance default at the managed sandbox when nothing deliberate is set > - That write read the default and wrote it later without a lock. Two overlapping boots, or a boot and an admin write, could overwrite a value the other had just committed > - A revert path also reset that stamped default whenever the mode was turned off, so a platform-filled default could change again later > - A planned plugin-environment default should fill only an unset default and never change a set one, which needs every instance's default to stay set > - This pull request makes the fill one locked transaction and removes the revert, so a filled default is an ordinary set value > - The benefit is a single, race-free rule: boot fills an unset default once, and nothing changes a set default afterwards ## Linked Issues or Issue Description Refs #19551 (the plugin-environment default that builds on this rule) **Subsystem affected** Server environments: the managed environment bootstrap and the instance settings route. **Problem or motivation** `applyManagedEnvironments` read the instance default, decided, and wrote it in separate steps with no lock, so a concurrent boot or admin write could be overwritten. When `enableManagedSandboxOnly` was not declared, a revert reset a default the platform had filled. That needed a `managedDefaultStamped` marker, plus a marker clear on every admin default write. A later boot could therefore change a default that had already been set. **Proposed solution** Fill the default in one transaction that locks the instance default row and rechecks it. Fill only an unset, local or deleted default. Remove the revert and its marker, so a filled default behaves like any other set value. **Alternatives considered** Keeping the revert and adding locks to it. A revert on a mode change would still move an existing instance's default later, which a fill-only plugin default cannot tolerate. **Roadmap alignment** This narrows existing managed-environment behaviour. It adds no product surface. ## What Changed - New `server/src/services/managed-instance-default.ts`: `fillUnsetInstanceDefault(db, environmentId)`. In one transaction it creates the settings row if needed, locks the instance default row (`FOR UPDATE`), and rechecks the current value. It writes only when the default is unset, names the local environment, or names a deleted environment. Any other set default is kept. - `server/src/services/managed-environments.ts`: - Under `enableManagedSandboxOnly`, the managed sandbox fill now uses that helper, through a new `fillInstanceDefault` test seam. - The mode-off revert and the `managedDefaultStamped` marker are removed. - The unused `instanceSettings` seam option is removed, along with the `getById`, `update` and `findManagedSandboxEnvironment` methods of the environments seam. - `server/src/routes/instance-settings.ts`: an explicit default write is a plain `update` again. The marker clear and its transaction are gone. - `doc/DEVELOPING.md`: the "Managed environments" section states the fill rule. Existing rows may still carry `managedDefaultStamped` in their metadata. Nothing reads it any more, and it is not part of the managed stock hash. ## Verification - `cd server && npx vitest run src/__tests__/managed-instance-default.test.ts` (embedded Postgres): - The fill writes an unset default, including on a fresh instance with no settings row. It also writes a default that names the local environment, or one whose environment was deleted. - It never changes a default set to an ssh, plugin or other sandbox environment, and leaves a default that already names the target alone. - Controlled overlap: an uncommitted admin default write makes the fill wait, and the fill then keeps the admin's value. This test fails without the row lock. - Two overlapping fills: exactly one writes. - Real boot pass: the default is filled under managed-sandbox-only and stays the same after a boot with the mode off. A set default stays on every boot. - `npx vitest run src/services/managed-environments.test.ts src/services/managed-environments-plugin-entries.test.ts`: the fill runs only under managed-sandbox-only, with the managed sandbox row. It never runs for a plugin environment. Nothing is written when the mode is off. A failed fill does not fail the ensure. - `npx vitest run src/__tests__/instance-settings-routes.test.ts`: an explicit default write is a plain update that touches no environment row. - `npx vitest run src/__tests__/managed-plugin-environments.test.ts src/__tests__/environment-routes.test.ts src/__tests__/server-startup-feedback-export.test.ts src/__tests__/instance-settings-service.test.ts`: pass. - `pnpm --filter @paperclipai/server typecheck` passes. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. ## Risks - Behaviour change: when `enableManagedSandboxOnly` is turned off for an instance, a default the platform filled with the managed sandbox now stays. Local execution is still allowed then, but it is no longer the default unless an admin picks it. - No schema change. Stale `managedDefaultStamped` keys stay in some row metadata and are ignored. - The fill holds the instance default row lock for one short transaction at boot. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (file edits, shell, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
8621bcf53e |
Cancel subscription sign-in requests when leaving agent setup (#19610)
Back during pending subscription sign-in could accept a late response and
reopen agent setup. Abort pending reads, release abandoned sessions, and
ignore stale responses. Wait for cleanup before starting a replacement,
and preserve sessions belonging to another account or environment.
Cover deferred resume/start/prompt responses for both providers, local
login cancellation, quick reselection, and browser request abort. Stabilize
Runner fixtures and normalization assertions exposed by integration CI.
Verified: 483 focused UI tests, six onboarding browser cases, UI typecheck
and token gates, Runner fixture checks, and the complete CI run on
|
||
|
DottaandPaperclip
|
fdda02a26e |
fix(ui): remove stale plan approvals from the composer (#19528)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - A task can ask a person to approve a saved plan. > - The server refuses approval after a newer plan replaces that plan. > - The document and request queries can finish at different times. > - The composer can then keep controls for a plan that is already out of date. > - This change removes those controls when a newer saved revision is known and refreshes the request state. > - People can review the current plan without an obsolete approval above the composer. ## Linked Issues or Issue Description **What happened?** An old plan approval can stay above the task composer after the plan changes. Selecting Approve returns a superseded error. The failed action does not refresh the request list. **Expected behavior** Remove the old approval and its pending count as soon as the UI knows that the plan has changed. Keep a newer approval visible if its document query still has an older revision. Preserve accepted decisions and unrelated requests. **Steps to reproduce** 1. Create a confirmation for a saved plan revision. 2. Save a newer plan revision while the old request remains in the browser cache. 3. Read the composer before the request query finishes. 4. Try an old approval after the server has expired it. **Paperclip version or commit** Source checkout based on `5d25117ffa`. **Deployment mode** Source checkout. The regressions use the production React components and query cache in jsdom. Related work: Refs #14907 (replaced request receipts) and Refs #15025 (server replacement for requests from board users). This change fixes the interval between document and request reads, plus the refresh after a failed resolution. It does not change those server or history rules. ## What Changed - Remove pending approvals for older saved plan revisions from the task display and composer. - Compare revision numbers, or document and request times for targets that have only a revision ID. Keep newer requests when the document cache is behind. - Refresh request queries after document activity and settled approval errors. - Use the same plan freshness check for recovery retry. An old hidden approval no longer blocks Retry agent. - Test the recovery controls and the guards for missing plans, accepted decisions, questions, other tasks, other companies, and older document reads. - Add regressions for both workspace presentations, task and conversation modes, revision formats, unrelated requests, and accepted history. - Document the display behavior in the development guide. ## Verification - TDD: five assertions failed before the main fix. Old controls stayed in the composer, document activity did not refresh requests, and superseded accept or reject responses did not read the new state. - The previous session recorded 815 passing focused UI tests after the recovery retry fix at `d95cf3235ed83dc1f6c7f51372dff408814eb15c`. UI typecheck, the production UI build, and token gates also passed. - The main fix at `5147fd29a03413596d5b2445ae3b480352cbff1a` passed all CI gates. This includes typecheck, build, general tests, and eight E2E shards. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42266101662). - The prior affected local run passed 7,656 UI tests, with one skipped, and 38 shared tests. Its CLI phase had two file-mode failures under the agent's `0077` umask and a child-server timeout during first-use builds. All three CLI files passed after a new check with `umask 022` and warm build dependencies: 25 tests passed. No CLI source changed. The initial full command did not pass or store a reusable pass. - On this continuation, the local test command could not start because the restored workspace has no installed Vitest dependency. No test verdict came from this attempt. - Final [CI](https://github.com/paperclipai/paperclip/actions/runs/42302544154) passes all applicable required gates for `d95cf3235ed83dc1f6c7f51372dff408814eb15c`, including typecheck, build, general tests, and eight E2E shards. Greptile and Paperclip reviews both report 5/5 at that exact revision, with no unresolved findings. The separate advisory review workflow remains queued. - No live browser acceptance or deployment is claimed. ## Risks - The UI must distinguish an older approval from an older document cache. Tests cover both query orders and preserve newer approvals. - This is a display and cache refresh change. Stored decisions, server authorization, approval gates, and database schemas are unchanged. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. The exact deployment model ID and context window size are not exposed in this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with available version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My remote branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass for the final revision (scope and initial environment failures are stated above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c66b4a55ef |
fix(ui): explain monitor waits during active runs (#19519)
## Thinking Path > - Paperclip manages work performed by AI agents. > - Task monitors schedule later checks through execution admission. > - A native run must save its work before a monitor starts another run. > - The monitor banner can show an overdue warning during this valid wait. > - It also promises the next tick and offers a check that cannot start. > - This change explains the active run hold on both monitor surfaces. > - Users can see why the check waits without changing execution guards. ## Linked Issues or Issue Description Refs #16975 and #17863. The branch also preserves the quiet workspace-wait behavior from #19507. These changes added the native settlement guard and monitor deadline worker. A search found no open PR for this display case. **What happened?** A monitor became due while its native run was still saving work. The banner showed an overdue warning and “Fires on next tick.” The check later ran. The warning did not explain the execution hold. **Expected behavior** Show the active run wait. Keep the saved check time. Restore the normal display when the run clears. **Steps to reproduce** 1. Keep a task run active and set a monitor on that task. 2. Let the saved check time pass. 3. Inspect the monitor banner and composer strip. **Paperclip version or commit** Reproduced in mounted production components at |
||
|
|
ee6c101854 |
Reject failover backups behind the accepted event cursor (#19613)
A verified failover backup can still trail events already accepted by the controller. Compare the backup produced cursor with the registered controller state, or the supported local journal before registration, and reject stale files before remote sync. Keep the permanent replay-conflict classification ahead of generic Runner errors.
Validation: 1,192 native session and execution-driver tests and 10 workflow tests pass locally, together with the maintained low-memory Runner test project typecheck and fresh server declarations. All applicable checks on
|
||
|
DottaandPaperclip
|
44c60c7221 |
Verify default skill sync through two real cold sleep cycles (#19191)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Automatic skill sources check upstream every five minutes. > - The domain deadline contract can hand future work to a durable wake alarm. > - A saved deadline or a worker unit test does not prove that a stopped tenant resumes the real producer. > - This pull request checks the default cadence through the actual app and a matching service checkout. > - It checks two cold starts and three distinct persisted skill revisions with real clocks. > - The benefit is repeatable local evidence without changing production timing or authority. ## Linked Issues or Issue Description Refs: #19001, #17008, #17923. This is a test and documentation follow-up to #19001. #19001 is now merged; this PR targets master. Feature code remains in that merged foundation. #17008 covers the earlier source worker change. #17923 covers separate skill-cache retention; this PR does not change that behavior. A scoped search of public PRs and issues found no duplicate default-cadence cold-cycle acceptance test. No matching public issue was found. The enhancement fields follow. **Pre-submission checklist** This improves checks for an existing feature. It adds no product endpoint or execution authority. The public related-work search is complete. **What existing behavior does this improve?** Verification of automatic skill-source updates across a durable idle handoff and cold restart. **Subsystem affected** `server/` test fixtures and development documentation. **Current behavior** Domain tests cover claims, access and future deadlines. They do not prove the full default-300-second path from ordinary source attachment through signed tenant reports, durable alarms, stopped compute and a changed persisted skill. **Proposed behavior** An opt-in launcher checks bootstrap first, then two full real-clock cycles. Ordinary CI can run the transport controls without the host fixture. The launcher-specific worker assertion runs only when prewire is requested, and fails closed in that lane. **Reason and benefit** The test can detect an early refresh, a lost deadline, a stale proof, an ACK after physical stop, a failed cold restart or a producer that does not publish changed contents. **Breaking changes** None to product APIs, schema, cadence, authorization, claims or retry policy. Existing maintenance-helper defaults stay unchanged. **Additional context** The opt-in host flow needs an authorized matching service checkout, its normal compiled acceptance bridge, Node 22 and two owned PostgreSQL test databases. The standalone controls do not need that service or PostgreSQL. Roadmap review places this under existing Skills Manager and Cloud deployment verification, not a new core feature. ## What Changed - Add an opt-in default-cadence acceptance test and checked-in launcher. - Reuse real server startup, managed tenant authority and the ordinary project attachment route. Control only the remote Git snapshot boundary; keep actual authorization, scanning, claims and publication. - Use actual signed tenant routes and the matching service's gate, sleep controller and PostgreSQL wake-alarm store. Stop and restart real app children through a controlled local provider. - Preload the Node transport guard before coordinator, worker and child module work. Check worker-local and child PID, thread and source-hash receipts. Keep caught unexpected transport attempts fatal. - Require both persisted refresh times and actual upstream handler timestamps to be at or after each original deadline. - Join physical callbacks during bridge close, stop a late wake-created child, and preserve the original cleanup budget. Add a real IPC/HTTP-child counterfactual. - Accept valid owned HTTP options paths without allowing Unix sockets or unknown origins. Remove the unused App helper export. - Add finite transport and launcher controls. Defer only the launcher-specific parent assertion in vanilla CI. Keep all subprocess controls default-runnable. - Add optional child entry, preload and bounded pool settings to the existing test helper. Preserve its defaults and the real observer test body. - Remove added explanatory comments under the contribution policy. Preserve executable syntax, directives and literals. - Document pinned setup, fresh owned databases, bootstrap and full two-cycle commands in `doc/DEVELOPING.md`. The diff against the intended base contains nine test/helper/launcher files and this one document. It adds no production feature code or Cloud source to the App repository. ## Verification The latest two-file CI repair preserves both original failed jobs. A newer master workflow introduced two required step inputs absent from its extracted-step test environment; the faithful false/false inputs now pass all five existing target-authorization cases, after an actual missing-input counterfactual failed. A controlled socket case also reproduces legal reuse of the released runner port; the fixture now holds that reservation until the preview binds, releases it in finally, and asserts distinct ports. All original retry/result/accounting assertions and timeouts stay unchanged. The original Linux listener owner remains unknown; the control proves this fixture collision mechanism, not that original attribution. Fresh current-head full CI and complete reviews are pending. Current merge-integration candidate: `f6f686103b7bfd0a4ab83085634c622af3896ecb`, a normal merge after #19001 landed, including actual master `1c9767d323cf872a266962b9e6b4bd5cbcb57f1c`. All nine acceptance code files are byte-identical to148a82e5; both parents' documentation additions survive. Every other path equals actual master. The only conflict was the old foundation's startup block; current master was retained in full. Fresh exact-head CI and complete reviews are pending. The local measurements below belong to their explicitly named revisions; no fresh full614-second run is claimed. Current candidate: `148a82e54c502759785407357001c6b50c1439dd`, normally carrying the current base `8c912e1ba4fc66397a7341ade59373af8075068f`. All nine acceptance code blobs remain identical to qualified repair `e092746f4da181c8414c63be71474619a84b97e7`; documentation retains both parents. The matching built service for the fresh bootstrap is `389878364b42dd129835d263f91882c5774d5123`, using Node 22.23.4. The App uses Node 24.22.2 and pnpm 9.15.4. Use the exact controlled setup and commands in [doc/DEVELOPING.md](doc/DEVELOPING.md#controlled-default-cadence-sleep-acceptance). Fresh owned databases and the matching compiled service are prerequisites; the launcher does not create a cluster or build dependencies. - **Current affected semantic check passed**, using actual native TypeScript 7 and the current generated CI options/references (0.640 seconds). Normal declaration references passed in 11.726 seconds, with source-matched SDK/shared/kernel/Runner prerequisites and the frozen current lockfile. - **Current guarded controls: 44 passed**, plus the same single explicit real-PG observer deferral, in 4.790 seconds. The executing worker checked its own global PID/thread/preload source hash and matching receipt. Guard hash: `7daf9ae7d2f26e8fcd389b1d560fac1d4e218a8304a2c58e49f646af108165bc`. No unexpected attempts or PG sockets occurred. Vanilla controls on the identical repair files passed 22 cases with one explicit launcher-parent assertion deferral (2.044 seconds). - **Real cleanup counterfactual:** unchanged old helper failed the actual endpoint assertion in 1.568 seconds: its late HTTP child remained reachable after close returned. The same fixed-helper case passed in 1.485 seconds: close waited, the child exited, the port was unreachable and no IPC error occurred. The isolated driver records only the known legacy closed-IPC error long enough to inspect cleanup; current helper requires zero such errors. All recorded processes and ports were independently checked empty. - **Fresh current bootstrap passed once**, 16.159 seconds wall / 14.12 seconds Vitest / 12.237 seconds body. The ordinary enabled producer persisted useful revision 1, cleared its lease and retained exactly 300,000 ms between success and original next deadline. A signed current-generation report produced the matching generation/operation-lock SQL ACK before physical child stop (ACK 06:54:02.020 UTC; stop signal 06:54:02.025 UTC). The alarm retained the original D and D-120 eligibility. Both fixture databases ended with zero sessions; the coordinator independently audited and normally removed them. This bootstrap does not execute a due wake or the new v2/v3 timing assertions. - **Historical full two-cycle result remains 1/1 passed in 614.213 seconds wall**, at measured App `74f7c012ca17ad0d7302d76403f5033affb1430b` and service `2cb54f904031dcf69a32902cd853b43f74d6f301`. Two new processes published distinct immutable v2/v3 at original D+115 ms and D+98 ms. Exact signed SQL ACKs preceded both physical stops; real SQL D-120 claims caused both cold starts. Features, claims and original business deadlines stayed unchanged. - Both historical 10.002-second quiet windows kept app SQL, upstream and outgoing HTTP counts unchanged. Two awake PG sockets remained. Quiet counts do not mean all connections closed. - The stronger timing checks are additional assertions, not retroactively executed results. Offline inspection of the original receipts confirms persisted successes at D+115/+98 ms and earliest upstream handler entries at D+44/+30 ms; all five requests in each refresh group started after D. The exact new pre-sleep request counter and dispatch-before-handler time were not independently recorded historically. - Source review found 19 critical automatic Git producer/runtime files identical to the measured App. The current active cache helper change does not run on automatic Git publication and adds no pool. The matching service carries unchanged measured helper/sleep/controller/alarm seams through its normal master merge. The current bootstrap and focused controls qualify the changed cleanup/guard and parent integration. No fresh current-head 614-second pass or whole-tree identity is claimed. - Runtime is serialized: worker 1, per-pool limits at most 4, configured App potential 10 + shared service pool 2 + one observer against server max 40. Current bootstrap point samples observed at most 3 owned clients plus the observer; those samples are not a continuous aggregate peak. App/service guard peaks were 2 each, with zero remaining sockets at cleanup. Failures remain accounted for. The first historical bootstrap cleanup masked an unknown primary error. Later attempts exposed missing fixture-owned domain metadata and lost origin snapshots; test-only ownership/cleanup corrections preserved the original assertions and timeouts. The new causal driver first lacked a final observation, then encountered sandbox bind EPERM, then exposed the old closed-IPC error; only the final real endpoint failure above is labeled the cleanup counterfactual. An offline dependency miss and a generator-cleared ignored compiler config were retained and corrected without weakening compiler options. The former inherited policy failure (397 findings) is resolved by the genuine current-master base/scanner, with zero findings and seven scanner tests; no unrelated UI site was edited and the gate was not relaxed. Historical real TCP observer coverage passed 1/1 in 3.688 seconds. Comment-only and opt-in-selection carries remain documented in prior evidence; self-hash changes are metadata exceptions with fresh executing-worker checks. The full workspace typecheck, test, build and browser commands were not run locally for this test-only slice. The current base passes its full exact-head CI and Greptile 5/5; this leaf now has all 56 exact-head check runs terminal (54 success, one neutral, one skip), the additional legacy security status passing, complete current Greptile 5/5 and Ponytail 5/5 reviews, and all four review threads resolved across all pages. No live provider or deployed host/database suspension was exercised. ## Risks - The host test requires a matching service checkout and compiled bridge. An unrelated service revision is not a valid substitute. - Controlled localhost Git, S3 and physical process operations do not qualify deployed Railway or Neon suspension, external permissions or provider uncertainty. - The Node guard does not prove native subprocess transport isolation. Only finite local Git metadata and Node version commands were needed. - Two short SQL/HTTP quiet windows do not establish strict 90-second host acceptance. Separate plugin-host LISTEN, MCP wake and other domain gaps remain open. - Explicit test URLs are destructive fixture inputs. Use fresh, owned `_test` databases. The launcher does not start a cluster or choose a database implicitly. - Passing CI and review do not qualify deployed suspension; the current proof boundaries above still apply. > For core feature work, check `ROADMAP.md` first and discuss it in `#dev` before opening the PR. This PR adds verification to the existing feature in its base. ## Model Used OpenAI `gpt-6.1-sol` assisted with implementation, repository review and local code execution. OpenAI Codex, based on GPT-6, performed managing review and integration. The context-window sizes and exact served backend snapshots were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green — fresh merge-head gates pending - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — fresh merge-head review pending - [x] I will address all Greptile and reviewer comments before requesting merge ### Current master reconciliation (2026-11-06) Merged actual master `e28158b353fd274f0322be76d62cd223e0df599f` normally. All nine qualified acceptance code files and the held-port repair remain byte-identical. Upstream independently landed the workflow input fix in #19602; the fixture now uses that exact upstream version once, avoiding duplicate object keys in a squash merge. All other master changes are preserved. The earlier two-cycle local evidence remains source-specific; no fresh 614-second run or deployed suspension is claimed. This new head requires fresh CI and bot reviews. ### Actual combined-tree typecheck failure and repair CI job `128156887548` failed with TS1117 at workflow-target.test.ts:22: two independently landed master fixes declared the same `IMAGE_ONLY` and `TESTOPS_REQUESTED` keys. The complete original log is retained. A further normal master merge now removes the duplicate declarations, retaining exactly one false/false pair and every original assertion. All nine qualified acceptance code paths and the held-port repair remain byte-identical; unrelated master changes are preserved. This is a concrete compiler failure and source repair, with fresh exact-head CI/reviews required. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c09fb60ca6 |
fix(recovery): retain worker-stop audit in preparation retries (#19612)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery must preserve saved files before it admits another execution. > - A failed preparation can need a verified stop of its previous workspace worker. > - Recovery keeps that verified stop receipt in the run profile for audit. > - The preparation retry guard treats the retained receipt as unknown execution evidence. > - This pull request recognizes a valid same-run audit receipt while it retains all current execution and file checks. > - A saved Retry can then continue once after recovery completes. ## Linked Issues or Issue Description **What happened?** A native preparation failed before provider dispatch. The recovery worker verified that the previous workspace worker had stopped and saved its files. The same HTTP Retry stayed deferred because the run profile retained `nativeWorkspaceOwnerStopProof`. **Expected behavior** The saved Retry should continue once after all preparation, stop, file, ownership, and authorization checks pass. The old run and its audit receipt should remain unchanged. **Steps to reproduce** 1. Fail native preparation with a retained remote workspace and no provider activity. 2. Let the recovery worker acquire the old workspace owner through verified platform stop evidence. 3. Submit one HTTP Retry and let preservation complete. 4. Observe that the old guard retains the retry hold even after saved-file verification passes. **Paperclip version or commit** Reproduced against `d728be8fbf`. Related work: #18908, #19021, #19026, and #19466. This change covers the retained workspace-worker stop audit left by a later repair. ## What Changed - Add a typed reader for a host-persisted same-run worker-stop audit receipt. Keep signature and expiry validation at ownership acquisition. - Recognize only that validated audit field in failed-preparation settlement. Keep unknown profiles and active owners blocked. - Exercise real preservation, HTTP Retry delivery, native dispatch, saved files, accepted results, and task completion. Add malformed, foreign-run, unstopped-worker, and active-owner cases. - Document the distinction between retained audit evidence and current execution authority. ## Verification - Failing before the fix: the current-worker preservation case passed; the stopped-worker case produced zero successor runs after the same Retry, where one was expected. - Passing after the fix: 47 recovery cases, 10 native dispatch/preservation cases, and 6 platform-proof tests. The two preservation success cases overlap between the first two groups. - Server typecheck passed all seven layers after the final test changes. The Runner and SDK dependencies were built in this isolated worktree. - Final comment-free commit `44f8f33f8d917dddb51852283a479786102eab4d`: a fresh combined run passed all 61 affected cases in 67.92s. Fresh server typecheck passed all seven layers. - The earlier `95d0376c` combined run passed 60 cases and hit one `socket hang up` in the existing `catalog_provider_warning` HTTP test. That case passed in a bounded isolated run (12.48s), and the final 61-case combined run passed. The socket failure remains recorded as an intermittent HTTP harness limitation. No retry or weaker assertion was added. - Final commit `44f8f33f8d917dddb51852283a479786102eab4d` completed all required CI gates: 57 checks succeeded, 3 intentional skips, and the advisory Paperclip review is neutral with a substantive 5/5 verdict. Greptile is 5/5 and there are no unresolved review threads. [Final CI run](https://github.com/paperclipai/paperclip/actions/runs/42293625696). The follow-up commit only removes six new code-comment lines to follow CONTRIBUTING. - The native provider and signed platform API are controlled test boundaries. The tests use real PostgreSQL, recovery, ownership, file persistence, retry admission, dispatch, and finalization services. Deployed recovery remains unverified until normal rollout. ## Risks - The main risk is mistaking an audit receipt for current stop authority. The reader checks its exact shape and run binding. The receipt cannot replace stopped leases, saved-file receipts, settled preparation, current authorization, or the absence of an active owner or provider evidence. - The negative matrix runs with the audit field present. Provider activity, unknown usage, bad file receipts, Stop, actor changes, and unsettled leases still block continuation. - No schema or migration change is required. ## Model Used OpenAI GPT-6 in Codex. The exact deployment model ID and context-window size are not exposed in this session. Used reasoning, code editing, shell execution, and test tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
02a2299173 |
feat(codex): default Codex to gpt-6.1-sol, half the gpt-5.6-sol price (#19623)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through Paperclip Runner or the legacy
`codex_local` harness, and the runner E2E suites run paid Codex cells
> - The shared Codex default and many selections preferred `gpt-5.6-sol`
> - The reviewed OpenAI rate card in
`server/src/services/codex-pricing.ts` (`openai-standard-2026-09-30`)
prices `gpt-6.1-sol` at $2 input / $0.10 cached / $10 output per 1M
tokens, and `gpt-5.6-sol` at $4 / $0.40 / $20
> - So every default run and every paid Codex eval cell cost twice as
much as necessary
> - This pull request moves every place that prefers or selects
`gpt-5.6-sol` to `gpt-6.1-sol`, and keeps historical facts, rate cards
and compatibility fallbacks on `gpt-5.6-sol`
> - The benefit is half the token price for new default Codex agents and
for the Codex E2E and TestOps cells
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The default Codex model for new agents and the Codex model that the
runner E2E suites and TestOps plans select.
**Subsystem affected**
Adapters (`adapter-utils`, `codex-local`), server native runtime
fallback, UI AI connection pools, Paperclip Runner devtools and live
evals, runner E2E catalog and TestOps mappings.
**Current behavior**
New Codex agents without an explicit model store `gpt-5.6-sol`. The
runner E2E `runner-codex` profile, the Reliability `queue-steer` plan
cell, the Product regression TestOps mapping and the ACPX Codex
qualification selection all use `gpt-5.6-sol`.
**Proposed behavior**
They use `gpt-6.1-sol`. Existing agents that store a model keep it.
`gpt-5.6-sol` stays selectable.
**Reason and benefit**
`gpt-6.1-sol` costs half as much per token on the in-repo reviewed rate
card. The current OpenAI pricing page (retrieved 2026-11-06) shows the
same short-context rates and also lists long-context rates of $4 / $0.20
/ $15 for `gpt-6.1-sol` against $8 / $0.80 / $30 for `gpt-5.6-sol`.
**Breaking changes**
None for stored configurations. See Risks.
## What Changed
Changed defaults and selections (all now `gpt-6.1-sol`):
- `PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` in
`packages/adapter-utils/src/paperclip-runner-permissions.ts`. This also
changes `DEFAULT_CODEX_LOCAL_MODEL`,
`resolvePaperclipRunnerModel("codex", …)`, runner config normalization,
`codex_local` → Paperclip Runner conversion, the UI new-agent and
agent-config defaults, the composer default model and the public MCP
effort lookup.
- `codex-local` `models`: the 5.6 entry is now the literal
`gpt-5.6-sol`, because it used to be `DEFAULT_CODEX_LOCAL_MODEL`. The
list order and contents do not change.
- AI connection pool: a new OpenAI member profile uses
`PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` instead of a hard-coded
`gpt-5.6-sol`.
- Paperclip Runner devtools ACPX Codex default and preset, the
`runner-codex-sol` local provider smoke, and the `acpx-codex-sol` live
workflow candidate.
- `tests/acpx-qualification-models.ts` Codex selection (runner E2E and
runner acceptance ACPX Codex profiles).
- Runner E2E: the `runner-codex` profile follows the default, so all
native Codex cells move. This includes the `runner-codex` cells among
the 52 standard `everyday-workflows` cells and the seven Codex
reliability cells. `runner-codex-mini` stays on `gpt-5.4-mini`. The
Reliability plan `queue-steer` `steeringProfile` is now
`['runner-codex', 'gpt-6.1-sol']`. Its executionId
(`everyday-workflows.runner-codex.daytona.queue-steer`), its fullName
and its TestOps case identity do not change. No separate profile is
necessary.
- `product-regression-testops.mjs`: the declared model for every
`runner-codex` case is now `gpt-6.1-sol` (it was already that for
`configured-repository`).
- `tests/runner-e2e/installed-cli.ts`: the credential-free installed
startup proof opens a session on the default model.
- `tests/runner-e2e/scheduled-prices.json`: add a `gpt-6.1-sol` price
(`openai-standard-2026-11-06`, including 30m/12h cache writes and
long-context rates from the OpenAI pricing page). The weekly scheduled
Codex cells need an exact price for their model, or admission fails
closed.
- Storybook: the composer model picker fixtures, the connection pool
stories and host, and the agent settings preview show `gpt-6.1-sol` as
the default.
- Docs: `doc/DEVELOPING.md` (the Codex conversion default) and
`tests/runner-e2e/EVERYDAY-WORKFLOWS.md` (local matrix, reliability
extras, `queue-steer` row).
- Tests that assert the default now expect `gpt-6.1-sol`. New
assertions: the default has the `0.159.0` CLI floor, the compatibility
fallback is floor-free, the `queue-steer` plan cell runs `runner-codex`
on `gpt-6.1-sol`, and all native Codex everyday cells (standard and
reliability) use `gpt-6.1-sol`.
Deliberately kept on `gpt-5.6-sol`:
- Rate cards: `server/src/services/codex-pricing.ts` and
`packages/paperclip-runner/src/evals/model-pricing.ts`. Old receipts
need them. (The runner eval rate card has no `gpt-6.1-sol` entry. An
unpriced candidate is reported as unpriced, not as zero. I did not add a
rate there because that card has its own version and source.)
- The remote Codex CLI compatibility fallback. A new
`CODEX_LOCAL_COMPATIBILITY_FALLBACK_MODEL = "gpt-5.6-sol"` replaces the
default in `codexModelFallbackCandidates`. `gpt-6.1-sol` needs Codex
0.159.0 or newer, so it cannot be the floor-free last resort. Fallback
behavior is unchanged.
- `tests/runner-e2e/provider-usage.ts`: the implicit request
cache-policy attribution stays qualified only for `gpt-5.6-sol`. I did
not verify that policy for `gpt-6.1-sol`. A `gpt-6.1-sol` response that
omits the write-TTL split keeps its cache-write cost unknown, which is
the existing behavior for unqualified models.
- `tests/runner-e2e/scheduled-measurements.json`: historical
`gpt-5.6-sol` measurements. They no longer match the current suite
definitions, so the forecast reports them as historical only. The
forecast tests now build their synthetic cohorts on the current
definition hash.
- Model catalogs that list `gpt-5.6-sol` as available (`codex-local`,
`cursor-local`, `opencode-local`, Speko voice diagnostics, Pi picker
fixture).
- The `gpt-5.6` → `gpt-5.6-sol` alias for old stored configs.
- Dated or evidence docs: `doc/plans/*`, `doc/adapter-model-audit-*`,
`QUALIFICATION-*`, `doc/connections/MEMORY.md`,
`doc/connections/TASK-REVIEWS-VERIFICATION.md`,
`tests/runner-e2e/CHAT-FEATURES.md`, the Slack eval evidence, and the
S49 soak signature text (the soak failure happened on `gpt-5.6-sol`).
- Test fixtures that parse, price, migrate or round-trip an existing
`gpt-5.6-sol` value (Rust runner tests, driver tests, pricing tests,
stored-config tests, stories that show an existing agent).
## Verification
- `npx vitest run` on every server, UI, adapter-utils and codex-local
test file that mentions `gpt-5.6-sol` or the default, plus all of
`packages/adapter-utils` and `packages/adapters/codex-local`: 5,483
passed. Three timeouts under load (`sandbox-managed-runtime`,
`acp.quota`) passed when rerun.
- `packages/paperclip-runner`: `vitest run` for `src/eval`, `src/evals`,
`src/drivers/acpx`, `src/drivers/codex`, `src/cli`, `src/contracts`,
`src/backends`: 2,641 passed.
- `node --test tests/runner-e2e/reliability-testops.test.mjs
tests/runner-e2e/product-regression-testops.test.mjs`: 28 passed.
- `node tests/runner-e2e/run-unit.mjs`: all model, catalog, reliability,
scheduled and TestOps tests pass. Failures remain only in
`copilot-protection-flow`, `denied-target-isolation-flow`,
`pi-controls-flow`, `workflow-target` and `unit-launcher`. These five
also fail on unmodified master in this local environment (filesystem
watch and git environment), and they do not touch models.
- `vitest run --config tests/runner-acceptance/vitest.config.ts`: 19
passed.
- `pnpm test:e2e:runner -- --list --suite everyday-workflows --group
reliability`: the seven `everyday-workflows.runner-codex.daytona.<case>`
cells show `gpt-6.1-sol` with `OPENAI_API_KEY,DAYTONA_API_KEY`.
`--matrix-json` gives 16 reliability cells with the same credentials as
before. `--list --all` shows no `gpt-5.6-sol` cell.
- Typecheck: `adapter-utils`, `adapter-codex-local`, `server`, `ui`,
`paperclip-runner` (TypeScript), and `pnpm test:e2e:runner:typecheck`.
- `pnpm check:token-gates`: clean.
- Not run: paid cells, Rust tests (no Rust change), the browser suites,
and the full `pnpm -r typecheck` / `pnpm build` (CI runs them).
## Risks
- Agents without an explicit model change model on their next save or
normalization, and Paperclip Runner rows with a blank model use
`gpt-6.1-sol` at execution. Stored models, including `gpt-5.6-sol`, are
never rewritten. No migration is necessary.
- `gpt-6.1-sol` needs Codex CLI 0.159.0 or newer with ChatGPT sign-in.
The runner pins `@openai/codex@0.160.0`. On a remote target with an
older image CLI, the existing compatibility check falls back to
`gpt-6-sol` and then `gpt-5.6-sol`, and that probe now runs for
default-model agents too. A legacy `codex_local` agent created with the
new default on a host with an older Codex CLI gets the existing
actionable "upgrade Codex" Test result.
- Qualification: the native Codex profile uses `adapter_constant` model
qualification, so it follows the default. No qualification gate is
dropped. The ACPX Codex qualification selection moves to `gpt-6.1-sol`.
Its next qualification run uses the new model. The provider-usage
cache-policy attribution stays limited to `gpt-5.6-sol`.
- Runner E2E history: the Codex cells change their suite definition
hashes, so earlier `gpt-5.6-sol` results are a separate cohort and do
not count as current.
- The `gpt-6.1-sol` scheduled price uses the file's existing `expiresAt`
(2026-11-19). It needs the same review as the other entries before then.
- No `.github/` change. The Reliability queue-steer cell keeps the
`runner-codex` profile id, so the existing workflow credential and setup
steps still match.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, running in
Claude Code with tool use (shell, file edits, web fetch of the OpenAI
pricing page).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #19604 (commits the Reliability TestOps identities; this PR
changes only `steeringProfile` in the same file).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
6148fbd31f |
feat(ui): distinguish GitHub resources in inline Markdown links (#19476)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Task conversations and documents use the shared Markdown renderer. > - GitHub repository, pull request, and issue links currently use the same icon. > - Long pasted URLs make these references hard to scan in a conversation. > - This change identifies each resource from its URL and shows a compact label with a distinct icon. > - The exact destination and custom link text stay intact. Connected status data keeps its existing behavior. ## Linked Issues or Issue Description **What existing behavior does this improve?** Inline GitHub links in rendered Markdown. **Current behavior** Repositories, pull requests, and issues all use the GitHub mark. Pasted resource links show the full URL. **Proposed behavior** Pull requests use the pull request icon. Issues use the issue icon. Repositories and other GitHub pages keep the brand mark. Pasted resource URLs show `owner/repository#number` or `owner/repository`. Custom labels keep their text and formatting. **Reason and benefit** Readers can identify GitHub resource types and scan references in a narrow conversation. Full URLs, query strings, and comment anchors stay available through the link destination. Link rendering makes no remote metadata requests. **Breaking changes** The visible text of a bare GitHub resource URL becomes compact. The destination does not change. No API, database, or connection changes are required. Searches found no duplicate work for GitHub inline resource presentation. Related work: [same-origin Markdown routing](https://github.com/paperclipai/paperclip/pull/17864). This change preserves that route handling. ## What Changed - Add a pure GitHub URL classifier with exact host checks and resource labels. - Move ordinary URL presentation into `MarkdownUrlLink`. Keep internal references, workspace links, custom resolvers, and connected metadata in their existing order. - Add separate icons and accessible resource names. Preserve authored Markdown titles and formatted labels. - Add Playground, Narrow, and Streaming Storybook examples that mount `MarkdownBody`. - Add parser and renderer tests, eight Chromium browser cases, and the browser command in the development guide. - Consider Open Graph loading. Use the existing connected metadata path for remote status and titles. Add no arbitrary page fetches. ## Verification - Focused Vitest coverage: 129 passing tests across nine files for GitHub parsing, Markdown rendering, connected metadata, document rendering, and workspace-file rendering. - `pnpm check:token-gates`. - `pnpm --filter @paperclipai/ui typecheck`. - `pnpm --filter @paperclipai/ui build`. - `pnpm build-storybook`. - Storybook catalog generation/check and 30 catalog/map tests. - Site-route regression: Stars, Education, and enterprise pages keep general GitHub branding and their full URL text. Three parser cases failed before the fix; parser and renderer cases now pass. - Streaming regression: three resource cases failed before the fix and pass after it. The fix preserves the generated word span and custom formatted labels. - Chromium: `STORYBOOK_TEST_BASE_URL=http://localhost:6106 pnpm exec playwright test inline-links.spec.ts --config=tests/storybook-visual/playwright.config.ts --project=stories`. - Browser scope: eight passing cases for ordinary/streaming modes, dark/light at desktop and 390px mobile, completed play assertions, exact comment destinations, safe external attributes, keyboard focus, wrapping, and no GitHub requests. Screenshots are saved as task artifacts; no review images are committed. - The first browser attempt in the resumed environment could not launch Chromium because its executable was missing. The browser was installed, and all eight cases then passed. - The browser cases use synthetic text and one connected metadata fixture. They do not prove provider authentication or server persistence. No visual baseline comparison is claimed; the repository baseline manifest is unpublished. - Initial typecheck found two unsupported story assertion options and an unbuilt SDK dependency in the fresh checkout. The options were removed and the SDK was built before the successful retry. - `pnpm verify` returned 2 because the new browser spec requires broad CI coverage. Focused tests and the browser suite supply local evidence. Required CI supplies broad coverage. ## Risks - Compact labels hide URL detail from the visible line. The exact href and default tooltip preserve that detail. - GitHub site routes can change. Unknown paths retain the general GitHub treatment. Lookalike hosts and credential-bearing URLs do not get GitHub resource treatment. - Connected external-object status and title rendering keeps priority over URL-derived presentation. ## Model Used OpenAI Codex, GPT-6 (`gpt-6` family), with reasoning, tool use, browser checks, and code execution. The exact deployment ID and context window are not exposed in this run. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b6e9615aed |
feat(ci): request advisory merge review after PR verification (#19480)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Pull requests already run CI and automated code review. > - A completed check does not start a final advisory merge review. > - A private routine webhook can connect these two steps. > - The webhook credential must stay outside PR code and public logs. > - This pull request adds an optional notification from trusted branch code. > - The routine records a recommendation. A person keeps merge control. ## Linked Issues or Issue Description **What existing behavior does this improve?** It connects successful PR verification to an advisory Paperclip routine. **Subsystem affected** GitHub Actions and the existing routine webhook API. **Current behavior** A PR can finish verification without asking a routine for a final merge recommendation. **Proposed behavior** A trusted workflow reads the current PR and its complete check list. It sends the PR identity to a private webhook when required checks pass and other reported checks finish successfully, as skipped, or as neutral. The receiving routine runs the configured merge review skill and records its decision on its Paperclip task. **Reason and benefit** This gives maintainers an advisory decision after verification. It does not grant merge control to the routine. **Breaking changes** None. Missing secrets or delivery errors leave verification results unchanged. Searched open and closed PRs and issues for related work. No duplicate notification workflow was found. Related API work: #18058 and #18581. This change uses the routine API and does not alter chat credential setup. ## What Changed - Add a default-branch workflow for PR workflow completions, external checks, and commit statuses. - Use read permissions and a GitHub environment restricted to master for the private URL and bearer token. - Require successful `ci / verify`, `ci / e2e`, and additional branch-rule checks. Read every status page and refresh the PR before delivery. - Use a stable idempotency key for each PR, head, and base. Reject stale events and changed candidates. - Verify the two core check runs belong to the same real GitHub Actions suite for the unchanged PR caller and master-owned reusable CI. Reject forged names, renamed callers, and incomplete file listings. - Keep HTTP redirects disabled, bound request time, and print fixed delivery states only. - Add regression checks and document private setup, pause, rotation, and acceptance steps. - Grant commit-status read access. Add a separate PR API probe with the same read permissions and the real Actions token. The probe has no private environment or webhook secrets and sends no notification. ## Verification - Pass: `node --test .github/scripts/tests/notify-pr-merge-review.test.mjs` (21 tests after the security and token-permission fixes). - Pass: actionlint v1.7.12 on `.github/workflows/pr-merge-review.yml`. - Pass: live GitHub query of a current PR and its complete status list. This found a GraphQL field mismatch during development. The corrected query passed with 62 checks. - Pass: the configured bearer webhook returned `test_received` with `routineStarted=false` in setup mode. Readback confirmed an active routine and environment secrets restricted to the master branch. - Not run: `pnpm verify --explain` could not start because this checkout has no installed workspace dependencies. Required CI will supply broad coverage. - The first quality-gate run rejected a CI-only title because this change adds a script. The feature title passed the later gate. - The security review found that PR-controlled check names could forge eligibility. A focused regression failed before the fix and passed after suite, run, caller, and reusable-workflow checks were added. The corrected provenance query also passed against a completed historical CI run without sending a webhook. - Earlier CI at |
||
|
|
2e35730637 |
feat(chat): receive sealed Slack events through the current SDK owner (#19019)
## Thinking Path > - Paperclip manages AI agents and their work. > - Slack connectors turn authenticated messages into company work. > - Direct callbacks need an awake tenant. > - Cloud can retain authenticated asynchronous events until the tenant wakes. > - Replay must preserve the original callback proof and current endpoint authority. > - Configuration changes must retain registration responsibility through failures. > - This PR enrolls verified Slack identities and drains sealed events through the normal SDK owner. ## Linked Issues or Issue Description **Subsystem affected** Server chat connectors and signed Cloud connector delivery. **Problem or motivation** Direct Slack Events API delivery depends on an awake tenant. Broker outages, lost responses and configuration changes must preserve delivery and registration responsibility. Enrollment failures must also leave other connector events able to drain. **Proposed solution** Stage nonsecret enrollment or retirement intent inside the local configuration transaction. Let the existing connection-event owner perform broker requests and retries. Replay sealed events through the current Slack SDK with their original authentication evidence. **Alternatives considered** An early acknowledgment for every callback changes command and modal semantics. A second delivery ledger duplicates existing ownership. Reusing a new stack identity to retire old registrations crosses an authorization boundary. A companion Cloud broker change is required. Related setup fixes: Refs #18837, Refs #18878. Kickoff searches found no competing async enrollment consumer. The ROADMAP's connector setup and access controls stay in place. ## What Changed - Persist enrollment and credential-free retirement intent before local configuration commits. Broker failure does not make a completed pause or removal look unsuccessful. - Bind company, endpoint, connection, public ID, generation and authorized actor. Preserve lost-response responsibility and recover only the supported same-instance configuration outage. - Permit an explicit move to a new authorized stack only after confirmed old retirement. Keep the old identity until the new broker receipt matches the current intent and generation. - Show permanent signing-credential failures as attention. Preserve transient retries and reject uncertain inbound authentication before SDK admission. - Drain events and acknowledgments before enrollment reconciliation. Persist a bounded retry on the exact busy intent so other due endpoints can enroll or retire. Preserve current custody under concurrent changes. - Stage retirement in the Apps removal transaction too. Archive the endpoint and connection, retain the original subject and instance, then revoke credentials. Retirement does not need the deleted Slack credential. - Acknowledge a malformed sealed Slack receipt as invalid once. Keep actual broker outages on the retry path. - Run the real configuration, SDK ingress, generation fences and public Apps removal paths in the default SQL suite. Keep Vitest transport helpers outside production services. - Preserve original callback bytes, signature, time and public URL evidence. Use the existing transactional chat-delivery ledger for redelivery. - Provide a checked-in prewire guard and launcher using the current Node executable. Add broker, worker, SDK, SQL and controlled-process acceptance. - Stabilize AI account summaries by ID and grant ID. This fixes a CI migration-repeat equality failure without changing SQL, authorization or reconnect behavior. - Open the four configured database connections before the enrollment test freezes its clock. Keep all 35 original assertions and all timeouts. - Measure status-card dispatch entry before its mock does database work. Keep the original strict 500 ms bound and check any still-armed earlier deadline. Bound that fixture's pool to four connections. ## Verification - Candidate: `54fa701d41b4ced62f4ff338d39b946015670ae3`. Normal merges preserve master `47027e7` and all owned changes. A read-only merge assessment against fetched master `cb5d7e44` is conflict-free. The only overlapping file has separate AI test additions. The branch has not merged that later master; fresh remote checks must validate the current PR test tree. - At this head, normal server production typechecking and declaration emission passed in 35.24s. The affected actual CI test shard passed in 6.27s. Other unchanged CI layers, 92 affected pure controls and 12 guard controls retain their source-matched results. - All **15 default real-PostgreSQL cases passed in 34.66s**, with no skips or filtering. They cover normal configure and SDK ingress, rollback, stale generations, malformed receipts, busy-lease fairness, concurrent intent custody, unknown errors, and public Apps removal. Removal tests check actual credential revocation, retained retirement through outage and repeat removal, one ordinary retry and confirmed retirement, transaction rollback and foreign-company refusal. - The public removal counterfactual failed before the fix: archival and credential revocation passed, but the inactive intent was absent. The same test source passed after the staging call. Separate counterfactuals also reproduced busy-endpoint starvation and malformed-receipt retries. Original failures remain retained. - The paired broker/SDK/process journey passed in 31.98s at ancestor `0324c011`. It preserves callback proof, lost-ACK redelivery, exactly one comment, one delivery and one successful controlled process. Its test, producer, SDK, guard and runtime bytes are unchanged. The new public Apps archive branch is not called by that journey; its evidence carries on that checked basis. - The new SQL cases need no companion Cloud checkout. Normal CI runs them with the repository fixture. For an existing dedicated PostgreSQL fixture, set `PAPERCLIP_SLACK_OWNER_TEST_DATABASE_URL` and `PAPERCLIP_TEST_POSTGRES_ADMIN_URL`, then run `node server/scripts/test-slack-cloud-owner.mjs` with Node24. The optional paired launcher still requires the normally built companion Cloud checkout. - Every pool was bounded to four or fewer connections. The final 15-case run sampled three runtime clients plus one observer. The independent final audit found zero clients and all five recorded processes absent. Earlier closing-socket observations are kept separate from final cleanup. - Earlier unrelated CI failures are documented: [enrollment timeout](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483754) and [status-card observation bound](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483843). The enrollment test retains all 35 assertions and its original timeouts. A controlled connection probe established the frozen-clock mechanism; the original remote access history remains unknown. The status-card fixture measures dispatch entry with the original strict 500ms bound. A real delayed-dispatch control fails that bound. - Full local workspace typecheck, build and test inventory was not run because its embedded-cluster and wider resource paths exceed this local lease. Exact-head remote CI now passes: 54 successful checks, three skips and Ponytail's neutral check, with aggregate verification green and the legacy security context independently confirmed successful. Both complete bot reviews are 5/5, with all 13 paginated review threads resolved. Seven execution jobs originally reported runner shutdown; one diagnostic rerun recovered them and the dependent aggregate check. Original logs remain retained, and the underlying shutdown cause is unknown. This is recovery evidence, not a product-fix claim. - Provider responses are controlled. This proves local SQL, broker, SDK and process behavior. It does not prove live Slack, agent tools, synchronous callbacks or deployed suspension. ## Risks The companion broker is required. Outage or conflicting authority can leave explicit pending or attention responsibility. A move between stacks requires confirmation from the old authorized owner first. Synchronous commands, actions, modal validation, options and challenges retain their deadlines and endpoint blocker. This slice grants no full Slack endpoint sleep exemption. No deployed suspension or savings claim is made. ## Model Used OpenAI `gpt-6.1-sol` workers through Codex, with reasoning, repository tools and code execution. The served backend revision and context limit are not exposed. A GPT-6-family coordinator performed technical review and integration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandClaude Opus 5.5
|
876a2e4857 |
fix(recovery): recover a prior boot's unstarted run at lease expiry (#19578)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a server restart, a task must continue automatically and promptly when that is safe > - A user clicks Retry, and the server is killed before the new run starts a process > - Recovery leaves that run `running` until the orphan reap's five-minute stale-age threshold passes > - The user sees nothing happen for about five minutes > - This pull request recovers that run when its controller lease from the earlier boot expires > - The benefit is that the automatic retry starts about one lease period after the kill, not five minutes later ## Linked Issues or Issue Description **What happened?** A Retry admitted a legacy run. The server was killed before the run recorded a process. The run kept a controller lease from the killed boot and `executionStage: preparing`. After the restart, live recovery skipped the run until `updatedAt` was five minutes old (`server/src/services/heartbeat/recovery.ts`, `staleThresholdMs ?? 5 * 60_000`). Restart-chaos measured 5 m 01 s from run creation to `process_lost` and the automatic retry, twice (4 m 20 s after the server was ready). **Expected behavior** Recovery should claim the run as soon as the earlier boot's lease expires. The lease is the only liveness proof for a legacy controller, and no process was ever recorded. **Paperclip version or commit** `master` at `71313207eb`. **Steps to reproduce** Click Retry on an interrupted task, then SIGKILL the server within about one second. Observe the retried run stay `running` for about five minutes. ## What Changed - `recoverActiveExecutions` skips the stale-age gate for a legacy `running` run when all of these are true: its controller lease is expired (checked by the existing live-lease gate), the lease belongs to an earlier boot (`controllerBootId !== legacyControllerBootId`), the run is still `preparing` (no adapter dispatch), and the run has no recorded `processPid` or `processGroupId`. - The stage and process fields committed with the revocation are checked again. If the old controller dispatched before its lease expired, the run keeps the age guard. - The existing revocation (`revokeExpiredLegacyControllerWithState`) still requires the expired boot ID, so a concurrent renewal wins. The existing `restartInterruptedBeforeDispatch` rule still decides the single process-loss retry. - `doc/execution-semantics.md` describes the rule in section 10. - Two regressions in `server/src/__tests__/live-recovery-work.test.ts` (embedded Postgres, real residual-recovery workers). ## Verification - Red before the fix: `recovers a fresh run from a prior boot at lease expiry ...` failed with `expected 'running' to be 'failed'`. - Green after the fix: `pnpm exec vitest run src/__tests__/live-recovery-work.test.ts -t "stale-age|fresh foreign controller|suppressed live runs"` passes 4/4. The run fails `process_lost` with `executionRecovery.providerWorkStarted: false` within seconds of lease expiry, and exactly one retry runs to `succeeded`. - Race case: a DB trigger makes the row committed with the revocation say `dispatching` while the scan saw `preparing`. The run stays `running`, and no retry is created. - Guard cases: a run with an expired lease of the current boot, a run from an earlier boot with a recorded process, and a run from an earlier boot that was `dispatching` (for example a remote HTTP adapter) all stay `running` after four minutes. - Related suites: `heartbeat-process-recovery`, `heartbeat-observed-workspace-preservation`, `native-startup-cleanup-recovery`, `services/heartbeat/recovery.test.ts`, full `live-recovery-work`. - Live K8 probe (restart-chaos supervisor, local instance, ACPX Claude agent): the first run fails with a provider error, the probe clicks Retry, and the server is SIGKILLed 100 ms later, before the Retry run records a process. The probe then measures how long the Retry run stays `running`. | Build | Retry run created to `process_lost` | Server ready to `process_lost` | Automatic retry run | | --- | --- | --- | --- | | Base (`71313207eb`) | 300.1 s | 293.5 s | started at 5 m 00 s | | This PR | 60.1 s | 54.5 s | started 0.1 s after `process_lost` | The claim now happens when the killed boot's 60-second controller lease expires. In this probe the restart took about 6 s, so the claim came about 54 s after ready. In the restart-chaos waves, kill to ready took about 41 s, so the claim would come about 20 s after ready. - A chaos wave with the `retry` trigger (`w-20261106T195715-s2727`, branch #19513 merged locally) missed the trigger: no Retry was offered because recovery completed the task automatically (claude t01 PASS). The probe above makes the Retry-then-kill timing deterministic. ## Risks - Low risk. The change only removes an age wait. Ownership is still proved by the expired lease of another boot. A rolling deploy or hot restart whose earlier instance is still alive keeps renewing its lease, so the run is not touched. - Runs with a recorded process, and dispatched runs, keep the five-minute guard. A process on another host or a remote request is not inferred dead. - Native runs are not changed. Their ownership is in `native_run_finalizations` and startup native recovery. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
da9a4cb96c |
fix(native): retire a finished local Runner that stopped before suspending (#19597)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner path runs each per-turn agent turn through a
runnerd process, and the next turn must prove that the previous Runner
is finished before it takes over the saved session root
> - Under load, a local turn can have its result accepted while its
`runner.suspend` command does not finish, so the Runner's own receipt
stays unsuspended
> - The retirement proof that handles the same case for sandbox Runners
refused every local Runner with `target_not_sandbox_runner`
> - So the next turn failed with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate` and its session root was
quarantined
> - This pull request extends the retirement to a local Runner when the
prior Runner process is proven stopped by exact PID and start time
> - The benefit is that a follow-up turn (for example, after a user
answers a question) continues instead of failing
## Linked Issues or Issue Description
Refs #19525 (keeps the accepted result in the same incident; this is the
next run).
**What happened?**
In the runner E2E cell
`local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`,
the first turn asked a question and its result was accepted. The Runner
then logged `runner did not prove durable suspension before checkpoint`
(`runner.suspend` pending, Runner lifecycle `ready`). After a server
restart and the answer, the next run failed with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=target_not_sandbox_runner`, and its root was
quarantined.
**Expected behavior**
When the prior local Runner is provably stopped and every other
retirement proof holds, the next run retires that owner as a continuity
break and continues. If the stop cannot be proven, the run keeps the
named rejection.
**Steps to reproduce**
1. Run a local per-turn native turn whose `runner.suspend` does not
finish after its result is accepted (host load average ~40).
2. Start the next turn for the same session.
3. See `retirement_refused=target_not_sandbox_runner`.
## What Changed
- `server/src/services/native-runtime/native-restart-recovery.ts`: add
`proveNativeLocalRunnerStopped`. It uses the existing restart-recovery
checks: the recorded Runner PID is dead or now belongs to a process with
a different start time; its recorded process group is not alive; each
recorded provider process is dead or recycled. A missing PID, start time
or process group, a live recorded process, or a start time that cannot
be compared is not proof. When settlement already cleared the run's
process fields, the saved `native.local_process_stopped` receipt is the
Runner proof. Extract `nativeProviderProcessIdentities` (checkpoint plus
run events) so restart recovery and this proof read provider PIDs the
same way.
- `server/src/services/native-runtime/native-session-executor.ts`
(`retireFinishedPerTurnRunnerdOwner`, `readSettledRunnerdProjection`): a
local target is now eligible. For a local Runner, the projection accepts
the Runner's own unsuspended receipt for the same identity and
unfinished Runner lifecycle commands (`runner.suspend`, `runner.drain`,
`runner.shutdown`) only. All other proofs are unchanged: committed
result, released lease, no recovery state, stopped controller, delivered
terminal, other commands complete, tool inputs answered, quarantine
copies. The process proof runs before retirement and again just before
the archive.
- `doc/execution-semantics.md`: describe the local case.
- Tests: seven integration cases in
`native-recovered-owner-followup.test.ts` and ten unit cases in
`native-restart-recovery.test.ts`.
## Verification
- Red before the fix: the four new integration cases failed with
`retirement_refused=target_not_sandbox_runner`, which is the incident
error.
- Green after the fix: `pnpm exec vitest run
src/services/native-runtime/native-recovered-owner-followup.test.ts`
35/35. The new cases prove that a stopped Runner is retired (root
archived under `continuity-breaks`, no quarantine), and that a live
recorded Runner (`runner_process_alive`), a missing identity
(`runner_process_identity_missing`), and a pending non-lifecycle command
(`journal_commands_unsettled(turn.start.pending)`) keep the rejection.
All existing sandbox cases still pass, including "keeps an unsuspended
Runner receipt decisive" for sandbox targets.
- `native-restart-recovery.test.ts` unit cases: recycled PID, live
process, unknown start time, live group, live provider, missing PID or
start time.
- `native-session-executor.test.ts` + `native-restart-recovery.test.ts`:
1190 pass. `native-runner-restart-recovery.integration.test.ts` +
`native-restart-recovery-isolation.test.ts` (shared provider-PID
extraction): 81 pass. `tsc --noEmit -p server/tsconfig.json`: pass.
- Live: `pnpm test:e2e:runner -- --max-parallel 1
--max-automatic-retries 0 --id
local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`,
3 runs at
|
||
|
DottaandClaude Opus 5.5
|
586caa5f49 |
fix(native): hand a shutdown-interrupted startup to the next controller (#19596)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native Runner path runs an agent turn through a runnerd process that the control plane owns > - A restart of the control plane must keep the task going without a click, if that is safe > - A graceful shutdown closes the HTTP listener first and then drains the heartbeat runs > - A native run that was between provider dispatch and runnerd backend setup in that gap failed with `local_runtime_api_listener_unavailable`, so the drain did not see it as running and did not suspend it for the next boot > - This pull request makes that failure a restart detach: the run stays running with its claim, the drain suspends it, and the next controller adopts it > - The benefit is that a SIGTERM in this window no longer spends a failure and a same-run retry, and the task continues by itself ## Linked Issues or Issue Description Refs #19530 (the same restart window, before provider dispatch). **What happened?** In a restart chaos replay (ACPX Claude, local runner), a restart-recovered run got SIGTERM while `createRunnerdBackendWithinSessionClaim` built the runnerd backend. `resolveLocalRuntimeApiUrl()` threw `local_runtime_api_listener_unavailable` because shutdown had already closed the listener. The run settled `failed`. The drain then logged `restartSuspendedRunIds: []`. The next boot used the one bootstrap retry. A second SIGTERM hit the same window, and the task stopped at a manual Retry. **Expected behavior** A run that this process cannot continue because it is shutting down is handed to the next controller. It is not settled as failed. **Steps to reproduce** 1. Start a native Runner task and restart the server (K1) so that startup recovery resumes the run. 2. Send SIGTERM while the resumed run sets up its runnerd backend. 3. See the run fail with `local_runtime_api_listener_unavailable` and the drain report no suspended runs. ## What Changed - `server/src/runtime-api.ts`: `resolveLocalRuntimeApiUrl()` throws the new `LocalRuntimeApiListenerClosedError` when this process had a listener and closed it (the `listening` flag is false or the `close` event fired). The error message stays `local_runtime_api_listener_unavailable`, so the existing classifiers do not change. A listener that never started still throws the plain error. - `server/src/services/native-runtime/native-session-executor.ts`: for a local Runner, the startup failure handler treats `LocalRuntimeApiListenerClosedError` like a restart detach. It stops lease renewal, keeps the claim and throws `NativeControllerDetachedForRestartError`. Heartbeat already keeps such a run `running`. A remote Runner keeps the same-run bootstrap retry from #19462, because startup recovery has no process evidence to adopt a claimed remote startup (`native-runner-restart-preparation.integration.test.ts` still covers that path). - `doc/execution-semantics.md`: describe the new rule. - Tests: a heartbeat regression that closes the real runtime API listener before backend setup, then drains and restarts; a unit test for the closed versus never-started listener; two `vi.mock` factories keep the real module exports. ## Verification - Red before the fix: `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts -t listener` failed with `restartSuspendedRunIds: []` and the run `failed` (`local_runtime_api_listener_unavailable`, `bootstrap_retry`). - Green after the fix: the same test passes. The drain suspends the run, the run stays `running` with `awaiting_runner_reattach`, and the next controller claims it and reaches the backend once. - `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts src/http/server.test.ts src/__tests__/server-startup-feedback-export.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-recovered-owner-followup.test.ts`: 1333 tests pass. One timing test in `native-session-executor.test.ts` failed once under host load average ~30 and passed alone. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: pass. - Live, restart chaos harness, local ACPX Claude (`claude-sonnet-5`), `--rc06-masked`, this branch at |
||
|
DottaandClaude Opus 5.5
|
d728be8fbf |
fix(native): re-bootstrap a Daytona runner restarted before session.open (#19583)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a `paperclip-runnerd` process in a sandbox. The local control plane keeps the runner's controller state in an encrypted Postgres checkpoint. > - When the control plane restarts, restart recovery classifies each live native run and decides to reattach, resume, bootstrap again, or hold the task. > - A restart chaos wave killed the control plane twice while a Daytona runner was connecting. The run was claimed and the sandbox runner was launched, but `session.open` never completed. No provider evidence existed. > - Recovery waited correctly for the old 60 s controller lease to expire. Then the classifier had no rule for a claimed remote bootstrap. It returned `ambiguous_remote_recovery_evidence` and blocked the task as "Workspace repair required". > - The controller writes each command to the Postgres checkpoint before it sends it. After the lease expires, that checkpoint is the complete record of what reached the sandbox runner. > - This pull request lets recovery prove from that checkpoint that no provider work exists, and then start the same-run bootstrap again. > - The benefit is that a restart during Daytona runner startup continues automatically instead of stopping the task. ## Linked Issues or Issue Description **What happened?** A restart during Daytona remote runner startup (native ACPX Claude, two SIGKILLs about 20 s apart) left the task blocked. The run history showed `remote_controller_lease_live` twice, then `ambiguous_remote_recovery_evidence` with disposition `blocked`. The task showed "Workspace repair required" (`native_restart_recovery_blocked`) and did not continue. **Expected behavior** When the durable controller record proves that the runner received only bootstrap commands, recovery continues the run with a fresh bootstrap. Any evidence of provider work keeps the hold. **Steps to reproduce** 1. Start a native run on a Daytona environment. 2. Kill the control plane after the sandbox runner connects and before `session.open` completes. 3. Restart the control plane and wait for the old controller lease to expire. 4. See that restart recovery blocks the task with `ambiguous_remote_recovery_evidence`. Root cause: `server/src/services/native-runtime/native-restart-recovery.ts` `classifyNativeRunnerRecoveryEvidence`, remote branch. It admitted only an unclaimed bootstrap (`remote_execution_never_claimed`, #16349) or a run with a provider checkpoint. A claimed bootstrap with no provider evidence fell through to `ambiguous_remote_recovery_evidence`. The decrypted checkpoint of the failed run held `connectionCount: 1`, `run.prepare` delivered once and pending, `session.open` pending and never delivered, no committed events, and `ackedSourceSeq: 0`. Refs #16349, #19524 ## What Changed - `native-bootstrap-proof.ts`: move the incomplete-bootstrap predicates from the executor into a small pure module. Restart recovery and the executor now use the same proof. No behavior change. - `native-runner-checkpoint.ts`: add `nativeRunnerCheckpointProvesIncompleteBootstrap`. It reads only a checkpoint with the bootstrap parts (header, cursor, `command:1`, `command:2`), checks the run, runner instance, and session identity, and applies the shared proof. A remote runner journal is not on this host, so the proof uses the controller record alone. - `native-restart-recovery.ts`: for a classic sandbox run whose owner lease has expired, with a durable checkpoint, no session checkpoint, no provider evidence, and no staged attachment, claim `bootstrap_incomplete` with the reason `remote_bootstrap_proven_incomplete` when the checkpoint proof holds. At three provider attempts, block with `execution_recovery_budget_exhausted`. - `native-session-executor.ts`: add `restoreRemoteBootstrapCheckpointForRecovery`. For a `bootstrap_incomplete` recovery on a sandbox target, it restores the Postgres checkpoint over the local external-state marker. The existing incomplete-bootstrap proof then checks the restored state before it quarantines the root and bootstraps again. - `doc/run-log-events.md`: document the new disposition reason. ## Verification - Red, then green: `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "remote bootstrap"`. Before the fix, the proven case and the budget case returned `blocked` / `ambiguous_remote_recovery_evidence`. - New cases: the proven claimed bootstrap gets `bootstrap_incomplete` (attempt kept, generation +1, run still `running`, no recovery action). The hold stays for a queued `turn.start`, an acknowledged event, adopted state, no checkpoint, and a spent budget. The executor helper restores the checkpoint only for a bootstrap recovery on a classic sandbox and only over a marker for the same run. - Regression set: `native-runner-restart-recovery.integration`, `native-session-executor`, `native-restart-recovery`, `native-bootstrap-retry.integration`, `heartbeat/recovery`: 1383 passed, 14 skipped. - Live Daytona replay of the failing chaos task (double SIGKILL during remote runner startup), 3 runs: REPLAY_RESULTS ## Risks - A fresh bootstrap acquires a new sandbox. The old sandbox keeps its lease until run settlement releases all run leases. The old runner gets no turn command, because only the controller issues one and the new controller never connects to it. - If `session.open` reached the old runner, the old provider session can exist without a turn. No provider work or tool call can happen in it. - Any checkpoint shape other than the bootstrap-only shape keeps the previous fail-closed hold. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4938a930d6 |
fix(budgets): stop pending or restart-lost accounting from hard-stopping companies (#19497)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget admission and run accounting decide whether any task in a company may start > - A control-plane restart that kills a provider turn mid-flight can lose that turn's final usage receipt. ACPX Claude reports usage only at turn end, so the receipt never becomes complete > - Accounting kept such a run "awaiting receipt" forever. Every later admission in the company then refused with "completed runs await accounting", and the budget evaluator opened a hard-stop incident that paused the company. No product action could clear it: raising the budget refuses while runs await accounting, accounting retry finds no new data, and allowing unpriced usage does not apply to pending runs > - Restart chaos waves showed this as a dead task Retry, a new task blocked as "over budget" at 22 of 500 cents, and a hard stop that cancelled queued Retries and an unrelated running task > - This pull request closes the debt from evidence: once native recovery reaches terminal failure and the provider stop is verified, the known part settles as an unpriced charge. Until then the run holds new work as recovering accounting, without a hard stop > - The benefit is that a restart no longer bricks a company with a hard budget. The operator's unpriced-usage policy decides admission, and unknown spend still never shows as zero cost ## Linked Issues or Issue Description Refs #19399, #19426, #19431 (earlier restart Retry fixes; this is the budget layer under the same journey). **What happened?** After a control-plane SIGKILL during an ACPX Claude turn, the killed run ended `native_provider_terminal_failed` with `usage_json.accountingReceiptReady = false` and `cost_accounting_pending = true`. It stayed that way for good. The task Retry was accepted but parked with `budget.blocked` ("Company cannot start work while completed runs await accounting"), which the UI showed as "Retry is waiting for execution checks". A new Codex task was escalated to `blocked` with "the original recovery target is over budget" while the company had spent 22 of 500 cents. After the next restart the budget evaluator opened a hard-stop incident (`amount_observed` 22.08 < 500), paused the company and cancelled both queued Retries. **What did you expect?** A restart should not stop the company. Unknown usage from a stopped provider should become an honest, operator-governed state, and Retry should start a run. **Steps to reproduce** 1. Give a company a hard-stop budget. 2. Start an ACPX Claude task and SIGKILL the control plane mid-turn, so the resumed turn fails. 3. Observe the run stay `cost_accounting_pending` with an incomplete receipt, and every new admission refuse. ## What Changed - `budgets.ts`: completed runs that await accounting now hold new work only. They no longer open a hard-stop incident, pause the company, place an agent budget hold, or deliver Stop to running work. Only spend at the limit, or unpriced usage under a blocking policy, does that. Chaos wave A and a pre-restart lifecycle cell showed the wedge: a run that had just succeeded stayed terminal for about 100 ms before its charge committed, the evaluator paused the company in that gap, and the Stop it delivered left a cancelled run whose receipt never closed. - `run-cost-accounting.ts`: a failed native run is a closed receipt source only with positive stop evidence: its finalization is `terminal_failure` with no result (not a finalization-deadline failure), it no longer records a process identity, and its latest server-authored process event is `native.local_process_stopped`. One SQL predicate serves the reconciler scan and settlement. Accounting now settles its known part as an `unpriced` charge instead of waiting forever. The background reconciler selects these runs. The reservation settles. Other incomplete receipts stay pending as before. A cancelled run keeps its existing accounting drain, so Stop recovery can still deliver its final receipt. - `native-local-process-stop.ts` and its transactional callers: releasing a verified local or remote process stop wakes the accounting worker, so the closed debt settles promptly. - `usage-receipts.ts`: a charge settled without its receipt is marked `accountingReceiptUnavailable`. A later complete receipt that carries its own price (including a rate-card price) corrects it. A receipt with no own price and unchanged usage does not reopen the hold. - `budget-exposure.ts`: the blocking unpriced-usage policy no longer counts a charge settled from a lost receipt with no observed usage. That is unknown, not "usage with no reliable price". Complete receipts that report no tokens (for example Cursor Cloud) still count. A live wave under the default policy hard-stopped its company at 0 of 500 cents on one such charge. - `budget-exposure.ts`: a terminal-failure native run with an incomplete, not capture-failed receipt counts as recovering accounting. Admission holds new work ("New work must wait while a native run recovers its unfinished accounting") without a hard-stop incident, a company pause or cancellation of running work. - `recovery/service.ts`: the stranded-work comment names the actual budget gate instead of always saying "over budget". - `task-interruption-presentation.ts`: a Retry parked by `budget.blocked` says it waits for the budget, not for "execution checks". - `doc/SPEC-implementation.md`: documents the closed-receipt contract next to the existing incomplete-receipt rule. ## Verification - New embedded-Postgres suite `server/src/__tests__/closed-native-receipt-accounting.test.ts`, built from the persisted shape of the killed run: - Red before the fix: the run stayed pending, `reconcileRunCosts` accounted 0, and admission refused with "completed runs await accounting". - Debt stays pending with no verified stop event, with a finalization-deadline failure, with a recorded process identity, and while the coordinator can still produce a result. - Under a blocking policy, a lost receipt with no usage does not hard-stop; one with tokens still blocks. A run with a still-recorded process identity paused the company with a hard-stop incident. - Green after: the stopped run settles as one `unpriced` charge with the reservation settled; admission is clear under an `allow` policy and names "no reliable price" under `block`; a run with a recorded process identity or a non-terminal coordinator stays pending; the closing window holds admission without an incident or pause; a final receipt that arrives later still enters late-receipt reconciliation. - Pending accounting from a run that just finished holds admission with no incident, no pause, no enforcement version bump, and an unrelated running run stays running. Red before the `budgets.ts` change: the company was paused. - A later priced receipt corrects the unavailable charge to its price; an unpriced one leaves it settled. - `task-interruption-presentation.test.ts`: the `budget.blocked` Retry wait message. - `budget-enforcement-scoping.test.ts`: the accounting-hold release cases now assert that pending accounting places no budget hold and that admission clears after settlement. - Existing suites pass locally: all budget suites (control publication, enforcement scoping, enforcement worker, evaluation work, live attempts, history, service, live usage, budget cancellation), native-stop-accounting, explicit-native-continuation, native-case-handoff, cost-data-quality, heartbeat-cost-accounting, run-cost-accounting, native-usage-delivery. Server typecheck passes. - Runner e2e cell `lifecycle-baseline.runner-codex.local.lifecycle-dependency-restart-neutral`, which failed on master with a false agent budget hold before any restart, passes on this branch. - Live restart-chaos wave (local runner, K1 SIGKILL, codex + 2× ACPX Claude + OpenCode, 4 restarts, company policy `unpricedUsagePolicy: allow`) with the accounting fix: every killed run settled (no `cost_accounting_pending` left), the company stayed active, the OpenCode Retry completed the task on the first click and the Codex Retry finished its task. That wave also exposed a 1-second false hard stop in the closing window (observed 0 of 500 cents) that cancelled the running OpenCode turn; the `budget-exposure.ts` change fixes it and has its own red/green test. - Confirming live wave with both commits, same plan (2× ACPX Claude, OpenCode, Codex; 4 K1 restarts): both Claude tasks passed R3 (one Retry click started a run that completed the task), OpenCode passed R1 (continued by itself), and the Codex Retry completed its task. Zero budget incidents, zero cancelled runs, zero runs left pending accounting, company active; three killed runs recorded as `unpriced` charges. The only open signature is an unrelated workspace-durability check (`R2:pre_fault_files_lost` on the Codex long-bench task). ## Risks - Behavior change: a stopped native run's lost receipt now becomes an `unpriced` ledger charge with its known usage. If that usage has tokens, the default `block` policy still stops new work with "recorded usage has no reliable price". If it has none, the charge is visible in Costs but does not stop the company. Other unpriced charges keep their existing effect. Unknown spend is never recorded as a priced receipt. Budget enforcement stays best effort, as the spec says; killed OpenCode and Codex requests already lose their in-flight usage. - Remote runs and runs without a verified local stop keep their debt pending (an admission hold only, no hard stop). - Contract change: pending accounting no longer pauses a scope or stops running work. New work still waits with "completed runs await accounting". A scope's real hard stop (spend at the limit, or blocked unpriced usage) is unchanged. - If a provider stop can never be verified, the run stays "recovering accounting": new work waits without a hard-stop incident. Before, the same state paused the company. The hold is visible in admission errors and Costs. - A receipt that arrives after settlement posts a late correction. The original unpriced charge stays in the ledger. - No schema or migration change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests, live restart-chaos harness). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |