mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
2fb4837cca929954d026d4663e367a669b8ad591
6954
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
2fb4837cca |
fix(chat): tell chat how to link execution runs and name their agents (#19889)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat lets a person ask an agent about work, such as the runs behind
a task
> - The chat guidance gives a link template for tasks only. It does not
give one for execution runs
> - The issue runs API returns `agentId` and the run `id`, but no agent
name and no link
> - So when a person asks "Show me the runs behind this website task",
the agent lists raw UUIDs. It does not link each run. It does not name
the agent that did the work
> - This pull request adds the registered run route to the chat link
guidance. It also tells chat to read the agent and name it, not show a
raw ID
> - The benefit is that a person can open each run (including the
cancelled attempt) and sees who ran it
## Linked Issues or Issue Description
**What happened?**
A chat agent was asked to show the runs behind a task that had a
cancelled attempt and a successful retry. It read `GET
/api/issues/{id}/runs` correctly, but its reply showed both runs as raw
UUIDs in code spans and named the agent as "`fe03dd15…`, the Website
Brief worker" instead of its name. It gave no run links.
**What did you expect?**
Each run links to its run detail page
(`/<prefix>/agents/<agentId>/runs/<runId>`), and the reply names the
agent that executed it.
**Steps to reproduce**
1. Create a task assigned to an agent, cancel its first run, and retry
it so a second run succeeds.
2. In chat, ask "Show me the runs behind this website task."
3. The reply lists run UUIDs without links and identifies the agent by
ID.
Refs #19614 (the chat catalog eval case `activity.find-runs` shows this
failure).
## What Changed
- `server/src/services/agent-conversations.ts`: the chat link guidance
now gives the run route `/<prefix>/agents/<agentId>/runs/<runId>` and
tells chat to read and name the agent instead of showing a raw ID.
- `server/src/__tests__/agent-conversations.test.ts`: a regression test
checks that the run template matches the registered UI route and reaches
the chat task prompt.
## Verification
- `cd server && npx vitest run src/__tests__/agent-conversations.test.ts
-t "chat prompt policy"`: 7 passed. The new test fails without the
guidance change.
- `src/services/heartbeat/task-markdown.test.ts` passes.
- The embedded-Postgres suite in `agent-conversations.test.ts` did not
start in my local sandbox (Postgres init error). It does not depend on
this prompt text. CI runs it.
- End-to-end: the chat catalog case `activity.find-runs` is rerun with
this commit on its eval branch.
## Risks
- Low risk. This is prompt guidance text only. No API, schema, or UI
change.
- It conflicts textually with #17985, which edits the same guidance
line. The merge is simple: keep both sentences.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits). Default reasoning mode.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
7fd98682ca |
fix(workspaces): give every subtask its own execution workspace (#19860)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents split work into child tasks, and each task runs in an execution workspace (a checkout, branch, or sandbox folder) > - Since early 2026, a new child task used its parent's realized workspace by default: the same checkout, branch and sandbox > - A parent and its child often run at the same time. Two writers in one folder lose each other's files, the waking parent is cancelled with `workspace_busy`, and sandbox lease ownership crosses from the parent into the child, so retained-file recovery refuses the workspace as ambiguous > - The product rule is simple: a subtask gets its own workspace, independent of the parent's saved files. Paperclip never needs to carry a parent's unmerged changes into a child > - This pull request makes that the only behaviour on every creation path, and moves existing shared children to their own workspace on their next run, with no user or admin action > - The benefit is that parents and children never block or overwrite each other, and recovery ownership stays with one task ## Linked Issues or Issue Description **What happened?** A child task inherited its parent's realized execution workspace by default (`createChild` defaulted to `executionWorkspaceInheritanceMode: "linkage"`, and `issueService.create` fell back from `inheritExecutionWorkspaceFromIssueId` to `parentId`). When the parent woke while the child wrote to the shared workspace, the parent run was cancelled with `workspace_busy`. In sandbox environments, the lease ownership chain crossed from the parent into the child, and retained-file recovery refused the workspace with `allocation_ownership_ambiguous`. **Expected behavior** Every new child task gets its own execution workspace, realized from the project base exactly as a top-level task in that project. It never uses the parent's checkout, branch, saved snapshot or unmerged changes. A parent and its descendant never share a workspace by any route. **Steps to reproduce** 1. Run a parent task in a project with isolated (git worktree) or sandbox workspaces, so the parent has a realized execution workspace. 2. From the parent run, create a child task with only `parentId` (or through the runner `create_task` tool, chat suggestions, decision effects, plugins, or the New sub-task dialog). 3. See that the child's `executionWorkspaceId` is the parent's, with `executionWorkspacePreference: "reuse_existing"`. 4. Wake the parent while the child runs in a sandbox environment. The parent run is cancelled with `workspace_busy`. **Additional context** Refs #9233 (the existing `strategy_only` opt-out). The opt-out `executionWorkspaceInheritanceMode: "strategy_only"` already existed for accepted-plan decomposition. This change makes that behaviour (tightened to drop a pinned existing branch) the only behaviour. ## What Changed - **One rule for every creation path (`server/src/services/issues.ts`).** `issueService.create` still uses the parent (or `inheritExecutionWorkspaceFromIssueId`) as a source for the project, project workspace and workspace *strategy*, but it never copies the source's `executionWorkspaceId`, `reuse_existing` preference, or pinned existing branch. All server paths that create a task go through `create()`/`createChild()`: issue routes (`POST /issues`, `POST /issues/:id/children`, the run-bound default parent and run workspace source), runner `create_task`, chat suggestion acceptance, native status decisions, accepted-plan decomposition, decision effects, plugins (`ctx.issues.create`), external agents, watchdog follow-ups, monitor recovery issues, case handoffs and email channels. - **Explicit ancestor workspaces are dropped.** If a create names an `executionWorkspaceId` (or an `existing` workspace selection) that an ancestor of the new task created or is bound to, `create()` drops it and keeps only the strategy. Older clients and skills that copied the parent's id keep working and get an independent workspace. - **`createChild` no longer has a "linkage" mode.** `executionWorkspaceInheritanceMode` is still accepted from internal callers and is ignored. - **`inheritExecutionWorkspaceFromIssueId` decision.** I kept the field for API compatibility, but it now copies only the source issue's project and workspace strategy. I removed its workspace-sharing behaviour for every pair, not only parent/child: the source task and its follow-up can run at the same time (the runner, watchdog and status-decision callers all create the follow-up while the source run is live), so sharing would again let two tasks write one folder. Explicit sharing between unrelated tasks stays available through an explicit `existing` workspace selection. - **New `server/src/services/task-workspace-lineage.ts`.** `buildIndependentTaskWorkspaceSettings` (strategy only: mode, environment, runtime, network egress, shared-workspace concurrency, base ref, branch template, commands; no `reuse_existing`, no `operator_branch`, no `existingBranch`), `loadTaskLineageIssueIds` (the parent chain plus, for a task that depth placement put beside its requester, the requester's chain via `privacyParentIssueId`) and `isAncestorExecutionWorkspace` (the workspace's `sourceIssueId` is in that lineage, or a lineage issue is bound to it). A child in another project does not take the parent's strategy. - **Upgrade-safe rebind at admission.** `executionWorkspaceService.releaseAncestorWorkspaceBindingAtAdmission` runs at the same admission point as the pending workspace selection (`heartbeat.ts`). For a child (or a flattened task beside its requester) that an older release bound to an ancestor's workspace, it clears the child's binding (keeps the project workspace and strategy, bumps the binding revision) so the run realizes the child's own workspace. It does nothing while another run of the child is running, while the child's earlier run still owns workspace finalization or recovery, for native restart/lease-owner recovery, for a process-loss retry, or for a Retry that keeps the current workspace. The parent and its workspace are never changed. - **Projects without a project workspace (per-task repository acquisition).** A parent with no project workspace worked in its own task files, so its settings (for example `isolated_workspace`) do not describe the project base. A child of such a parent, and an older child released from one, takes the project's defaults exactly like a top-level task, instead of an isolated mode that would resolve to a git worktree with no git base ("needs a project / project workspace"). When the child's new workspace is created at admission, `executionWorkspaceRepositoryService.prepareForAdmission` repeats the parent's (or requester's) repository requests as the child's own pending requests (identity, URL, catalog id and requested ref only; no pinned commit, branch or files). The normal acquisition path then clones a fresh copy into the child's task files with the child's own access checks. This runs only for a workspace created by that admission, so existing tasks never gain repositories. - **UI.** The New sub-task dialog no longer pre-selects the parent's workspace. `buildSubIssueDefaults` drops `executionWorkspaceId` and `reuse_existing`, and derives the parent's independent mode (`shared_workspace`, `isolated_workspace` or `agent_default`) from its preference, saved settings or current workspace. The "This sub-task will no longer use the parent task workspace" warning is removed. - **Contracts and docs.** Shared validator and plugin SDK comments describe the new meaning of `inheritExecutionWorkspaceFromIssueId`. Updated `doc/execution-semantics.md` (new rule in §6 and a wording fix in the retained-sandbox paragraph), `doc/SPEC-implementation.md` (child workspace rule, requester keeps *strategy* inheritance, watchdog follow-ups), `doc/SPEC.md` (task workspace ownership), `docs/specs/external-task-protocol.md`, `skills/paperclip/SKILL.md` (step 9 and the workspace bullet), `skills/paperclip/references/issue-workspaces.md`, the CEO onboarding `HEARTBEAT.md`, and the plugin SDK README. ### Previous-release story (`rules/upgrade-safe-run-state.mdc`) - **Migrated on next admission.** A child that the previous release bound to its parent's (or any ancestor's) execution workspace keeps that binding until its next normal admission. Then the server clears the binding and the run realizes the child's own workspace from the project base. No schema migration is needed. No user or admin action is needed. - **In-flight writers finish first.** A running child is not interrupted. Its current run, its native restart recovery (admitted native input or lease owner), a process-loss retry, and a Retry that keeps the current workspace all finish in the shared workspace. While a child's earlier run is still running or its workspace finalization or recovery is unsettled, the binding stays and a later admission releases it. - **No files are dropped.** The release changes only the child's binding row. The parent keeps its binding, and the shared workspace row, folder, branch and saved snapshots are not touched. Anything the child left there stays in the parent's workspace. - **No new refusal codes.** Saved run identity (native execution input, leases, checkpoints) is not changed or refused; the rebind only affects new admissions. ## Verification - New `server/src/__tests__/subtask-workspace-independence.test.ts` (embedded Postgres): `create` with `parentId`, `createChild` with no mode / `"linkage"` / `"strategy_only"`, `inheritExecutionWorkspaceFromIssueId` pointing at the parent, explicit parent and grandparent `executionWorkspaceId`, a non-child follow-up, and unchanged top-level defaults and explicit top-level reuse. 7 creation cases fail on `master` and pass with this change. Admission cases: an old-release child is rebound while its parent is running and the parent keeps its workspace; a running child is not rebound (deploy lands mid-run); an unrelated explicit share is kept; the parent is never rebound. - New heartbeat case in `server/src/__tests__/heartbeat-workspace-busy.test.ts`: an old-release child bound to its running parent's workspace in a sandbox environment runs to `succeeded` with no `workspace_busy`, gets a new workspace, and the parent run and binding stay unchanged. It fails on `master` (the child waits with `workspace_busy`). - Updated tests that asserted the old inheritance in `issues-service.test.ts`, `task-workspace-binding.test.ts`, `chat-project-tools.test.ts`, `task-workspace-child-trust.test.ts`, `heartbeat-git-restore-point-resume.test.ts` (a child bound to a closed parent's expired workspace now starts from the project base and leaves the parent's restore point alone), and `ui/src/lib/subIssueDefaults.test.ts`. Remote-workspace reservation, continuation and writer-wait suites now seed the older-release shared shape directly (or an unrelated explicit share), so those safety paths stay covered. - `npx vitest run` on the changed suites and about 45 related server suites (heartbeat workspace, recovery admission, issue routes, routines, plugins), `ui` `subIssueDefaults` and `NewIssueDialog` tests, `pnpm check:token-gates`, `pnpm -r typecheck`. - New `server/src/__tests__/heartbeat-subtask-acquired-repository.test.ts`: a project with no project workspace, a parent whose task files hold an acquired repository with an unmerged change, and a child (new, and the older-release shape bound to the parent's workspace) that runs to `succeeded` in its own task files with a fresh clone at the project base and its own ready repository receipt. The parent's checkout keeps its change. Both cases fail on the previous head with "needs a project / project workspace". ## Risks - **Project-level configuration still applies to children exactly as to top-level tasks.** In a project configured for `shared_workspace`, every task (top-level or child) uses the project's one folder under the existing shared-workspace concurrency policy. A branch that the project itself pins applies to every task in the project. This PR removes only the parent-to-child carry-over of the realized workspace; it does not change those project defaults. - **Standalone follow-ups that an older release bound through `inheritExecutionWorkspaceFromIssueId`** are migrated only when the source is in their recorded lineage (parent or privacy parent). Others keep their explicit share and the existing writer-wait serialization. - **Behaviour change for agents that relied on a shared checkout.** A child or follow-up no longer sees the parent's unpushed or uncommitted changes. Agents must push or merge code they want a subtask to build on. The skill and onboarding text now say this. - **Explicit ancestor workspace ids are silently dropped** instead of rejected, so older clients do not break. The task gets an independent workspace. - **First run of a migrated child starts from the project base.** If an older child had uncommitted work in the parent's workspace, that work stays in the parent's workspace and is not copied into the child's new workspace. - **Admission adds one indexed read** for tasks with a parent and a bound workspace; the locking transaction and ancestor query run only when the bound workspace was created by another issue. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (file edits, shell, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5f1b1bb81d |
fix(recovery): continue from the last save when a retained source cannot be verified (#19867)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a native run cannot save its files, a retained-file recovery hold blocks the task until the worker saves them from the retained sandbox. > - The worker verifies the recorded source strictly. This includes the reusable-lease chain of the sandbox. > - A reusable sandbox chain can cross from a parent task into a child task. The check then refuses the source with `allocation_ownership_ambiguous` on every attempt. > - Retry repeats the same refusal. The task stays blocked by three gates: the saved-work wait, the repair flag and the remote writer hold. The only text left tells a person to inspect the run and its environment. > - The source check must stay strict. Recovery must still work without a person. > - This pull request adds an automatic fallback: the task continues from its last committed workspace save, the unverified files are attached to the task but never applied, and the unverified workspace epoch is abandoned so its root is never reused. > - The benefit is that tasks with an unverifiable retained source continue by themselves, with no lost saved work and no admin action. ## Linked Issues or Issue Description Refs #19840 (removes the "inspect the run and its environment" text and keeps Retry; this change removes the need for that Retry). **What happened?** A child task ran in a reusable sandbox that it inherited from its parent task. The run saved a checkpoint, wrote more files, and then failed to save them. Retained-file recovery refused the source with `allocation_ownership_ambiguous`, because the lease history of the sandbox crosses into the parent task. The hold was exhausted after one attempt. Retry made a new generation that failed the same way. The task stayed blocked with `saved_work_recovery_failed`, `workspaceRepairRequired: true`, and the failed run still the dirty writer of the workspace. A second task had a stale exhausted hold for a run whose files a sibling recovery action had already saved. That hold also kept the task from continuing. **Expected behavior** Recovery works without a person. When the source cannot be verified, the task continues from its last saved version. The unverified files are not applied, but they stay available. A stale hold whose files were already saved settles as recovered. **Steps to reproduce** 1. Use a native run with a remote workspace whose reusable sandbox lease history includes a lease of another task. 2. Let the run save a checkpoint, write more files, and fail its final save. 3. Let retained-file recovery run, or press Retry. 4. Before this change, the hold is exhausted with `allocation_ownership_ambiguous` and the task stays blocked. **Paperclip version or commit** Observed on deployed builds that include `630411af53`. Holds saved by the Nov 3–6 releases have this shape. ## What Changed - New `server/src/services/workspace-recovery-fallback.ts`: - `continueFromLastSavedWorkspace` runs when a hold is exhausted by a permanent source refusal: `allocation_ownership_ambiguous`, `source_binding_invalid`, `source_unavailable` or `historical_source_missing` with retry class `service_required` or `use_alternative_strategy`. It applies only to remote workspace sources in the `prepared` state. - Quarantine (secrets omitted under the workspace file policy: `.env`, keys, repository metadata, dependency directories): if the retained root is readable, the worker copies it with the existing read-only capture (no write to the sandbox), converts it to a `.tar`, stores it, and attaches it to the task as `unverified-files-run-<runId>.tar`. A `workspace_recovery_quarantine` operation records it as `verified: false, applied: false`. If the root is not readable, the worker records a closed reason label. - Stop proof: every allocation of the source run must carry a provider stop receipt. A missing receipt is obtained through the same stop-only cleanup as normal recovery (`prepareSandboxStopAndRetain` / `prepareHistoricalSandboxStopVerification`, `retryPendingSandboxTeardown`, `settleStopOnlyCleanup`). A sandbox that another lease holds (`active` / `pending_cleanup`) is never read or stopped from here; the hold waits. - One transaction (lock order: task, coordinator, run, recovery action, workspace) re-checks all facts including the stop receipts, abandons the epoch, resolves the hold with the named outcome `retained_source_unverifiable_continued_from_last_saved`, records an `executionReconciliation` decision with `retainedSourceFallback`, and denies native replacement. A Blocked task belongs to the hold only while the hold's own `workspaceFinalizationBlock` receipt still owns that status version, or, for receipt-less holds saved by older releases, while the block was set by the run's terminal failure (within its finalization window) and no status decision followed. Only then is the task restored to `todo` and a continuation scheduled; a newer block is kept and no continuation starts. The decision carries machine `workspaceRepairEvidence`, which the reconciled admission requires for runs with a restore failure. - Continuation: with no waiting user input, the hold uses `continuationDelivery: "pending"`, so the existing reconciled delivery starts one continuation. Waiting user input continues the task itself (`not_requested`), so there is never a second continuation. - Agent note: the decision's `outcomeEvidence` (part of the continuation's recovery outcomes) says that the run's unsaved changes could not be verified and were not applied, where the attached copy is, that the agent should check what to redo, and that completed external actions must not be repeated. - `settleSiblingRecoveredHold`: an exhausted hold whose run has a verified repair receipt from a different recovery action settles as recovered. It runs for every exhausted hold, including adoptable native sources. A sibling marked recovered whose receipt no longer verifies records a permanent deferral, so discovery does not reselect it. - Deferral: when the fallback cannot settle yet, the hold records `retainedSourceFallbackCheck` (reason, generation, count). Transient reasons (live writer, running operation, missing stop, sandbox in use, pending status effect, lock busy, storage failure) are checked again after 5 minutes, at most 12 times. Permanent reasons (uncertain external action, non-remote source) wait for new input, Retry or a new generation. Storage or database failures after a successful read are retryable and never recorded as "unreadable". - New `server/src/services/workspace-recovery-fallback-discovery.ts`: SQL for pending and due fallbacks. `retainedWorkspaceCandidates` (the production worker's obligations sweep) and `nextWorkspaceRecoveryAt` (its scheduler) include them, so the worker reaches old holds without saved input. - New `server/src/services/remote-workspace-epoch-abandonment.ts` (kept out of `remote-workspaces.ts`, which is near 1,000 lines): authority transition `abandonUnverifiedRemoteWorkspaceEpoch`. It clears the writer and the dirty flag, keeps `snapshotId`, drops `location`, advances the epoch, clears `providerRef` and the migration root, and records the abandoned epoch. The next preparation restores the last save into a new `<workspace>-<epoch>-<lease>` root. - `server/src/services/workspace-recovery.ts`: hooks. An exhausted hold first tries the sibling settlement, then the fallback (under the existing workspace ownership lock). A fresh permanent refusal in an attempt runs the fallback in the same pass, under the same ownership. - `remote-workspace-transport.ts`: export the existing read-only capture. - `packages/shared`: optional `retainedSourceFallback` on `ExecutionReconciliation`. - `doc/execution-semantics.md`: document the fallback. - No new polling or worker: the fallback runs inside the existing retained-workspace recovery pass (`rules/sleep-aware-background-work.mdc`). Safety: - The cross-task lease check (`retiredReusableLeasePredecessors`) is unchanged and stays strict. Unverified files are never applied. - No second executor: the source run, its controller and its finalizer must be stopped. A live writer of the task or workspace, or a running workspace operation, keeps the hold. - No repeated external side effects: pending status effects, non-completed or unknown API receipts, and uncertain tool invocations keep the existing reconciliation path. The continuation keeps the existing API receipt fences. - Never overwrite another run's files: only the exact epoch that the failed run wrote is abandoned, and the old root is never reused. - Bounded and idempotent: a settled hold is never revisited. The quarantine record is reused. A changed generation or revision leaves the hold unchanged. Previous-release story (`rules/upgrade-safe-run-state.mdc`): holds saved by earlier releases carry forward. A hold saved by the Nov 3–6 releases (`systemRecovery` exhausted with `allocation_ownership_ambiguous` / `service_required`, board owner, old admin text) takes the fallback on the next worker pass after deploy (restart sweep, Retry, or a new message). No migration is needed. Refusals that do not qualify keep today's behavior. The only new persisted shapes are additive: `retainedSourceFallback` and `siblingWorkspaceRecovery` on the hold, `remoteWorkspaceAbandonedEpochs` on the workspace, and a `workspace_recovery_quarantine` operation. Earlier releases ignore them. ## Verification - `cd server && npx vitest run src/services/workspace-recovery-fallback.test.ts` — 15 tests: - cross-task hold, saved by the previous release and refused fresh: continues from the checkpoint snapshot, the quarantine `.tar` holds the unverified file, the epoch is abandoned, `getExecutionBlocker` is `null`, the task is `todo`, explicit continuation is not blocked by `saved_work_recovery_failed` or `workspace_repair_required`, one reconciled continuation is delivered, and the next writer restores into a new root while the old root keeps its files; - unreadable root: continues and records the reason; - idempotency: a second pass changes nothing and delivers no second continuation; - waiting user input: no second continuation; - negatives (live writer, pending side effect, uncertain external action, sandbox held by another lease): the hold stays, the authority is unchanged, nothing is attached or delivered, the deferral reason and recheck time are recorded; - an allocation without a stop receipt is stopped (stop-only) before the hold settles; - the production worker (`reconcileRetainedWorkspaces` with `obligationsOnly`) and `nextWorkspaceRecoveryAt` discover and settle an old hold with no saved input; a transient deferral is not revisited until due, then settles; - an old-release hold without a Blocked receipt keeps its status; a current hold with the receipt is restored to `todo`; - stale exhausted hold with a sibling-saved receipt settles as recovered; one whose sibling receipt no longer verifies is recorded and no longer discovered; - a newer Blocked decision is kept and no continuation is delivered, while the file hold settles; - the attached copy omits `.env` and key files. - RED: with `workspace-recovery.ts` and `system-recovery.ts` reverted to master, all tests fail. - Regression suites pass: `workspace-recovery-warm-handoff`, `workspace-recovery-retry-copyback`, `workspace-recovery-preparation-gates`, `workspace-recovery-source`, `task-retry`, `execution-blocker-read` (288 tests), and `workspace-recovery-admission`, `workspace-recovery-preparation*`, `remote-workspaces.integration`, `native-failed-workspace-recovery`, `remote-workspace-transport`, `workspace-restore-repair-route` (683 tests). - `server` typecheck passes. ## Risks - A task now continues without the unverified changes of its failed run. The agent receives a note and the attached copy, so it can redo or re-apply the work. This replaces an indefinite block. - Abandoning an epoch on a workspace shared with a parent task also moves the parent to the last save. This is correct: the unverified changes are not part of the saved lineage. - Clearing the migration root means sessions bound to the old root start fresh. The fallback continuation already forces a fresh session. - The quarantine capture reads a sandbox that can be shared with another task. It writes nothing there, and it verifies the copy against inventories taken before and after. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edits, read-only browser for API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
feb8dacb86 |
fix(skills): teach the board skill how people find things in the web UI (#19891)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - People use chat with a board agent. The board agent reads the
`paperclip-board` skill.
> - People also ask the agent how to use the web UI. An example is "How
can I quickly find Launch tasks without leaving what I am doing?"
> - The skill does not describe the web UI search. The agent searched
for docs and found none. Then it said that it did not know the UI and
told the person to ask in chat.
> - The correct answer is quick search: Cmd+K or Ctrl+K opens the
command palette over the current page.
> - This pull request adds a short "Finding Things in the Web UI"
section to the board skill.
> - The benefit is that the agent gives the real entry point and
shortcut, and does not claim that it can operate the person's browser.
## Linked Issues or Issue Description
No issue exists. This is the problem description (bug template labels):
**What happened?**
A person asked the board chat agent how to quickly find Launch tasks
without leaving their page. The agent did not mention quick search or
Cmd/Ctrl+K. Its reasoning said that it did not know the exact UI
capabilities. It told the person to ask in chat.
**Steps to reproduce**
1. Open board chat in a company that has a "Launch" project.
2. Ask "How can I quickly find Launch tasks without leaving what I am
doing?"
3. Read the reply. It does not name the quick-search shortcut.
**Expected behavior**
The agent names quick search (Cmd+K on macOS, Ctrl+K on Windows/Linux,
or the sidebar search icon), operators such as `status:backlog`, and
"Search all" for full search. It tells the person to do these steps. It
does not claim to press keys for them.
Found by the chat-feature eval `navigation-preferences.quick-search`
(runner-opencode cell).
## What Changed
- `skills/paperclip-board/SKILL.md`: add a "Finding Things in the Web
UI" section. It describes the Cmd/Ctrl+K command palette, the sidebar
search icon, search operators, the "Search all" and Cmd/Ctrl+Enter
handoff to `/{prefix}/search`, and permission-scoped results. It tells
the agent that it cannot press keys or focus fields in the person's
browser.
- `skills/paperclip-board/SKILL.md`: add "help finding things in the
Paperclip web UI" to the skill description, so agents select the skill
for UI how-to questions.
- `server/src/__tests__/board-skill-ui-guidance.test.ts`: new test. It
checks that the documented shortcuts match the key bindings in
`ui/src/components/CommandPalette.tsx`, that the full-search route
exists, and that the description makes the section discoverable.
## Verification
- `pnpm exec vitest run
server/src/__tests__/board-skill-ui-guidance.test.ts
server/src/__tests__/cli-invocation-safety.test.ts
server/src/__tests__/agent-skills-routes.test.ts
server/src/__tests__/onboarding-seed-route.test.ts
server/src/__tests__/board-claim.test.ts
cli/src/__tests__/managed-agent.test.ts` passes.
- The new test fails on `master`, because the section does not exist
there.
- End-to-end: the change is cherry-picked onto the eval branch
`chat-eval/case-quick-search`, and the
`navigation-preferences.quick-search` chat eval runs on it.
## Risks
- Low risk. This is a documentation change to one skill and one new
test.
- If the palette shortcut or route changes, the new test fails. Then
update the skill and the test together.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, GitHub CLI).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Nicky Leach
|
35cbd15175 | fix(native): recover a stopped Runner whose journal stayed ready at a settled cut (#19907) canary/v2026.1108.0-canary.0 | ||
|
Nicky Leach
|
94718ce168 | fix(native): continue on a replacement when a retained Runner retires before claim (#19897) canary/v2026.1107.0-canary.24 | ||
|
Nicky Leach
|
91b044858c | fix(runner): send the prepared task scope with retained release and resume (#19896) canary/v2026.1107.0-canary.23 | ||
|
Nicky Leach
|
5020f38f90 | fix(runner): give a later attempt a new task when its stopped task kept no session (#19899) | ||
|
Nicky Leach
|
0363ebec70 | fix(connectors): stop GitHub access refreshes from advancing agent runtimes (#19902) | ||
|
Dotta
|
eb7c7b083a |
Recover Muse verification after contact, restart and access restoration (#19909)
## Thinking Path > - Paperclip manages AI agents and their work under company access rules. > - Muse readiness uses a saved connection, receiver contact and an authenticated challenge reply. > - A failed challenge write could leave a saved contact with no later recovery trigger. > - A server restart or restored access must recover that contact under current authority. > - This pull request uses the existing delivery coordinator and saved Muse state for recovery. > - It keeps the normal receiver schedule, authorization checks and real deadlines. > - It lets eligible verification recover and lets idle verification stop work. ## Linked Issues or Issue Description **What happened?** The first Muse receiver contact could persist while readiness challenge creation failed. The contact then had no durable retry or startup recovery path. Later membership, lifecycle or feature restoration could also leave pending verification without a notice. **Expected behavior** Retry a temporary failure through the existing worker. Recover pending contact after restart. Recheck the current company, agent, operator, credential and binding generation before issuing a challenge. Stop verification work while it is ineligible or complete. Do not revive revoked credentials. **Steps to reproduce** 1. Use the normal invitation and OAuth registration, PKCE and token flow. 2. Send one receiver contact and let its normal persistence schedule run. 3. Fail the first readiness challenge operation before it commits. 4. Observe whether the existing worker retries without another contact or a manual Verify action. 5. Restart the App with a new database owner. Check that it creates one usable challenge from the saved contact. **Paperclip version or commit** The causal baseline test used the unchanged production implementation at `90148317b2a447794ff29a4acae5648ad11a9770` plus its normal fixture. The accepted positive test used `9d383041028776ad8469ac4af3fe14ba6c369425`. The initial published head was `fe518c9b72504daf6374225987f24d45bc7bcf75`. The lease/auth correction was locally qualified at `cfd515a207416097fe8b586277d70c4fd4a679fa`. The current candidate is `6efe722e7eb1e22acd466125c7b4a627611d0273`; its added Cloud actor fixture correction passed the full normal owning program and all 35 tests. Its current normal build, lifecycle owner controls, healthy-wait and restoration measurements are described below. Earlier results below are bound to their stated revisions. Related public changes: Refs #19819 for original-principal authentication. Refs #19750 for the transaction-owned external feature read. These are related foundations, not duplicate fixes. The current Muse title and head-branch searches found no duplicate contact-verification recovery PR. Related maintenance work is #17000 and terminal assignment settlement is #17793. ## What Changed - Add a typed, exact-binding verification notice through the existing Muse delivery queue. - Register the first-contact notice before its durable write in the same transaction. - Recover pending contact from saved state at startup. Retry temporary failures through the existing scheduler. - Recheck current feature, lifecycle, credential and access state after commit. Park unavailable authority without recurring verification scans. - Add narrowly scoped notices to the existing access, lifecycle, configuration, company, budget and settings write owners. - Save a healthy lifecycle verification wait in the existing operation JSONB. Keep generic unresolved-work and error retries unchanged. - Commit authenticated confirmation and its lifecycle recheck in one transaction. Retain a raced recheck under the active lease, and recover saved waits once at startup. - Retain a committed lifecycle recheck while an expired attempt still owns the row, so its late wait cannot park accepted confirmation. Replacement claims keep their existing owner fence and fresh fact read. - Keep the receiver gap, persistence and per-segment contact limits unchanged. - Add real App OAuth, failure, retry, restart, challenge and quiet evidence. Add two normal-route restoration tests. - Document recovery, reconnect requirements and the existing connected-receiver sleep restriction. ## Verification - Causal baseline: the real invitation and OAuth flow saved one contact. The first challenge operation failed. The normal test failed because it observed one verification attempt instead of a retry. No setup or cleanup failure was counted as this regression. - At `9d383041`: normal own Shared/SDK, DB with migration 0441 assets, Runner TypeScript core and surfaces, module boundaries, and all seven normal App programs passed. One new redundant role comparison first failed typecheck; that diagnostic remains recorded and the corrected full App gate passed. - At `9d383041`: all 17 maintenance component tests passed with zero skips. All original deadline, rollback, drain, terminal settlement and startup controls remain. - At `9d383041`: the one normal App case passed. Real OAuth authorized one receiver signal. A retry occurred about 5.01 seconds after the first failure. A new Db/App owner created one challenge, advanced revision 2 to 3, and accepted authenticated confirmation and mailbox ACK. Query instrumentation had an active positive control. All measured query, callback, session and verifier counters stayed equal across the unchanged scheduled 15-second quiet window. Token revocation and old-token/signal rejection passed. - Known App, worker, HTTP and database owners closed without cleanup errors. The tests used bounded local PostgreSQL and HTTP fixtures. They did not call a live Muse model. - Current integrated semantics: the full normal App driver completed all programs. Only a new test callback had two undefined-value diagnostics. An explicit defined-value guard corrected that test. The full 416-root owning program then passed with empty diagnostics. A separate collector incorrectly required an explicit child `noEmit` field; the normal inherited setting was verified without repeating the compiler. The original failures remain recorded. - The first restoration run failed both cases. Permission parking recorded 120 main-pool queries without more verifier calls. The lifecycle counter recorded two denials without their exact scope. These observations remain unexplained at the individual-query/call level. Source separately shows a lifecycle readiness retry every two seconds while the real external reply is absent. - At `fe518c9b`: normal Shared/SDK and DB producers, module policy, all seven full App programs, and all 33 pure diagnostic controls passed. The lifecycle owner file passed all 19 cases, including both confirmation/park orders, rollback, duplicate notices, startup recovery and authority controls. - At `fe518c9b`: the normal OAuth failure/retry/new-Db reconstruction case passed. Its new healthy-wait window measured 15,002 ms with unchanged total activity before genuine confirmation; the existing ready quiet and revocation checks also passed. - At `fe518c9b`: both normal restoration tests passed in Vitest. Ordinary viewer-to-operator restoration and Pause/Resume recovered the same binding, grant and single contact; each committed one challenge and advanced revision 2 to 3. All six parked, restored and revoked/disabled quiet windows measured at least 15 seconds with active positive read controls and unchanged total queries, callbacks, sessions and verifier calls. Both cleanups passed. - The restoration wrapper still recorded failure: it required lifetime lifecycle denial count zero, whereas the measured count was one before the parked quiet window. The reviewed fixture explicitly permits a current-authority denial from an earlier observation and requires no committed challenge before restoration. Precise denial timing relative to Pause was not serialized. The original wrapper failure is retained; no rerun or retrospective explanation of the earlier failures is claimed. - Review exposed an expired-lease race. The added real-store baseline test failed after the late defer restored `waitingForSignal`; no setup or timeout was counted as the regression. On `cfd515a`, the whole lifecycle owner file passed all 20 tests, including that case. The full normal production declaration program (1067 roots), verification program (662), and auth owning program (405) passed; all 1942 declarations remained byte-identical. - The original remote auth-session shard failed three tests because their partial query adapters lacked the new binding query's join/order/limit chain. A fixture-only correction preserved the real notifier and existing authorization assertions. All 12 auth-session tests passed locally. - Complete remote inspection also found 33 failures in the 35-test Cloud actor fixture with the same missing `innerJoin` method and downstream incomplete-auth assertions. Its original full log and exact synthetic Source comparison are retained. At `6efe722e`, the three-line fixture adapter repair passed the complete normal 662-root test program and the whole 35-test file with zero failures or skips. All existing lock, authorization, settings, debounce and unfinished-auth assertions remain. - The earlier healthy/restoration quiet measurements remain bound to `fe518c9b`. Those steady 15-second windows did not exercise a still-owned attempt passing its 120-second lease. The targeted expired-owner regression qualifies that new boundary; it is not a fresh measurement of the earlier windows. - The initial `fe518c9b` remote gates failed the auth fixtures and reported a Greptile 4/5 P1. The aggregate failure reported the underlying shard. Those actual causes were fixed and qualified above. - Current `6efe722e` remote gates completed with **57 successful checks and three explicit skips**. The full latest edited Greptile summary is **5/5**, explicitly reviews this exact head, and has no new finding. The original P1 thread is resolved. All general, inline and nested review pages were read. - One current-head CI attempt timed out after 30 seconds in the Runner evidence/privacy file, with 6,864 passes and one failure. That file and its transitive inputs are byte-identical to the CI base. A separate local diagnostic preserved every assertion and the original timeout, ran the full normal Runner test project and the entire 41-case file, and passed all 41 with zero skips. Its private bounded timing records measured the target body at 74 ms. The diagnostic was then removed, restoring the exact original test. This is local recovery evidence; the original CI stalled await was not recorded and its cause remains **unproven**. - One retry of only the failed jobs on the unchanged head passed the Runner support suite and dependent aggregate verification. The complete second attempt and original failure logs are preserved. No test, timeout, privacy guard or product code was weakened. This result does not turn the original timeout into an explained failure or a diagnosed flake. - Full recursive typecheck, full `test:run` and a full release build were not repeated locally. The normal full App programs, focused producers and tests above passed, and the required current-head remote gates passed. Final review is against the exact unchanged head; local quiet remains separate from deployed suspension. Reviewer commands use the normal project entry points: `node server/scripts/typecheck-layers.mjs`, the whole `server/src/services/muse-maintenance.test.ts` file, and both normal App PostgreSQL files. PostgreSQL fixtures require explicit owned fixture/admin URLs and normal migrations before imports. Run the files serially with one worker. The private controller retains exact Source, output, worker and owner-closure receipts. ## Risks - Missing or early producer notices could lose a restoration trigger. Notices stay in the transaction that commits the write. The worker checks current authority after commit. - A stale notice must not grant access. Binding generation, current membership, company, agent and credential checks remain authoritative. Revoked credentials require reconnecting. - The change adds no table, migration, alternate task queue or periodic verification scan. It adds two optional fields to existing lifecycle operation JSONB. Generic pending/error deadlines, setup holds and manual repair remain obligations. - Confirmation can race with an active verification attempt. The park compare-and-set must retain the durable recheck. Startup recovery uses a fixed cutoff and keyset pages; it must not become another recurring scan. Wrong company, stale version, pause and revoked credentials must stay fenced. - The connected receiver still has its documented tenant-sleep restriction. Worker quiet is not proof of deployed suspension or external wake. - The normal challenge reply is sent by an authenticated protocol fixture. Live Muse background replies, cost/usage, full Muse lifecycle and full hosted Dot behavior remain unqualified. Partial Dot profile ACK and unsigned-device cutover stay withheld. - Restoration coverage for the additional company, budget, configuration and feature producers remains a separate qualification obligation. This pull request does not mark the whole hosted area complete. ## Model Used OpenAI `gpt-6.1-sol` authored the implementation and tests with local code and tool use. A GPT-6-based Codex coordinator reviewed the changes and owns integration. Its exact underlying model ID is not exposed. Context window sizes are unavailable. No context size is inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
adb7633d7b |
fix(status-cards): give agents the person's restore route when archiving a card (#19903)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Status cards are saved summaries of company work. People and agents
can archive and restore them.
> - A person asked a chat agent to archive a card "to reuse it next
quarter". The agent archived it correctly.
> - The archive response had no board route, and the PATCH operation had
no description. The completion guidance tells agents not to guess
routes.
> - So the agent told the person to restore the card "by clearing
`archived`". That is an API instruction, not a route a person can use.
> - This pull request gives single-card responses an `href` to the
Status page and documents archive and restore on the PATCH operation.
> - The benefit is that an agent can give the person a usable link and a
true account of restore, which starts a fresh summary.
## Linked Issues or Issue Description
**What happened**
A chat agent archived a status card. Its reply told the person to
restore the card by clearing `archived` on the card. It gave no link to
the place where a person restores it.
**Expected behavior**
The reply links the card on the Status page, where a person can show
archived cards and click Restore. It also says that restore starts a
fresh summary.
**Steps to reproduce**
1. Enable status cards and create a card as an agent.
2. Ask the CEO agent in chat to archive the card for later reuse.
3. Read the reply. It has no usable restore route, because `PATCH
/api/status-cards/{id}` returns no `href` and has no description.
## What Changed
- `POST /api/companies/{companyId}/status-cards`, `GET
/api/status-cards/{id}` and `PATCH /api/status-cards/{id}` now return
`href`, the company-scoped Status page route for the card.
- The shared `statusCardSchema` has an optional `href` field.
- The `PATCH /api/status-cards/{id}` OpenAPI operation has a
description. It says that archive keeps the card and its history, that a
person restores it on the Status page with the returned `href`, and that
restore starts a fresh full summary.
- The create route reads the company issue prefix before it creates the
card. A failed link lookup can not delete a card whose compile task is
queued.
- Regression tests cover the `href` on the archive and get responses,
the new PATCH description, and a failed prefix read on create (no card
and no task are left).
## Verification
- `pnpm exec vitest run server/src/__tests__/status-cards.test.ts
server/src/__tests__/openapi-routes.test.ts` (71 tests pass). The new
create-route test fails on the first commit and passes on the fix.
- End to end: the chat eval `status-cards.archive-card` runs with this
change on an eval branch. The result is in the eval report for that
case.
## Risks
- Low risk. The change adds one optional response field and one
description. List responses do not change. The UI ignores the new field.
- Each single-card response does one more small company read for the
issue prefix. On create, this read occurs before the card exists, so its
failure leaves nothing behind.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
2d89486c57 |
fix(native): record OpenCode's rejected model as a model rejection (#19865)
## Thinking Path - Paperclip runs agents through native providers; when a provider rejects the selected model, the person needs to see "Model unavailable" and choose another model. - #19548 made the OpenCode driver report OpenRouter's invalid model ID as a closed `model_unavailable` provider failure. The task banner now shows it. - The server's terminal classifier, however, recognized a model rejection only from Codex's exact ChatGPT error envelope. OpenCode's structured failure was stored for the banner but the run was recorded as `adapter_failed`. - So chat showed a generic failure, and the task was not blocked for a model choice. The `runs-adapters.inspect-failed-run` chat eval caught this. - This change maps an action-required `model_unavailable` failure to `native_provider_model_rejected` for any provider. ## Linked Issues or Issue Description Refs #19548. An OpenCode run whose model OpenRouter rejects (`... is not a valid model ID`, HTTP 400) finishes with `errorCode: adapter_failed`. Expected: `native_provider_model_rejected`, as for Codex. ## What Changed - `native-provider-failure.ts`: `forTerminal` returns `native_provider_model_rejected` when the failed turn's provider failure has category `model_unavailable` and retryability `requires_action`. Transient or unknown retryability still returns null. The diagnostic is the closed turn diagnostic; no provider text or model name is copied. - `NATIVE_MODEL_REJECTION_MESSAGE` no longer names a ChatGPT connection, since it is now the blocker text for any provider. - Tests for both providers, plus negative cases for transient/unknown retryability. - `wake-queue/adapters/postgres.ts`: a native run that failed for a rejected model or incomplete configuration, on a task the arbiter already blocked, is released without opening a `native_continuation_requires_reconciliation` incident. That incident put the run in `recovery_needed`, which hid the run page's Retry, though the provider refused the request and nothing ran to reconcile. Queued messages stay deferred. ## Verification - `pnpm exec vitest run src/services/native-runtime/native-provider-failure.test.ts src/services/native-runtime/status-arbiter.test.ts src/services/native-runtime/native-provider-failure.integration.test.ts` (server): 98 passed. - `pnpm exec vitest run src/modules/wake-queue` (server): passes, including a new test that fails on the old code (it opened the incident). - Server `tsc --noEmit`: clean. - End to end: the `runs-adapters.inspect-failed-run` chat eval will be rerun on the chat-eval candidate with this change. ## Risks - Codex model-rejection blocker wording changes from "current ChatGPT connection" to "current AI connection". - Any provider driver that reports `model_unavailable` with `requires_action` now blocks the task for a model choice instead of the generic failure path, which disables automatic adapter-failure retries for that run. That is the intended behavior for a rejected model. - Configuration-failure native runs on a blocked task no longer get a board reconciliation incident. If such a run had already produced side effects in earlier turns, they are not flagged for reconciliation; a provider model rejection happens before the turn does work. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ba3fe09954 |
fix(evals): name why a private evidence transfer refused, without its text (#19908)
## Thinking Path
- Paid chat evals retain their evidence through a trusted transfer
(`tests/runner-e2e/manual-private-transfer.ts`) that runs from master,
separate from the target checkout.
- When the transfer refuses an attempt, it prints one fixed line,
"Restricted manual evidence operation failed; no public fallback", and
keeps nothing. The closed diagnostic on the target side cannot see this
step either.
- So a refused attempt leaves no evidence and no reason.
`runs-adapters.inspect-failed-run` lost three attempts in a row this
way, and `goals.create-goal-tree` lost one; nobody could tell which
check refused.
- This change keeps the privacy rule (no message text in public logs)
but names the refusal with a closed code.
## Linked Issues or Issue Description
**What happened**
A paid chat-eval attempt whose private evidence the trusted transfer
refused printed only "Restricted manual evidence operation failed; no
public fallback". The receipt showed `attemptedSourceBytes: 0` and
`transfer.status: failed_or_unknown`, and no evidence was kept, so the
reason was unknowable. `runs-adapters.inspect-failed-run` lost three
consecutive attempts this way.
**Expected behavior**
A refused transfer still keeps message text private, but names which
check refused it with a closed code, so the attempt can be diagnosed.
**Steps to reproduce**
1. Dispatch `runner-full-stack-e2e.yml` manually for a chat cell whose
private evidence fails transfer validation.
2. Open the "Retain complete approved manual originals" step log: it
shows the single fixed failure line with no reason.
## What Changed
- New `tests/runner-e2e/private-refusal-code.ts`:
`privateRefusalCode(error)` returns a slug of the message only when it
is a fixed literal `new Error("...")` in the trusted evidence modules
(transfer, handoff, retention, report assets, history, framework report,
history budget, evidence, redaction). Otherwise it returns `fs-<code>`
for Node error codes, `error-type-<Name>` for typed errors,
`non-error-thrown`, or `dynamic`. Message text is never returned.
- `manual-private-transfer.ts`: the failure line now includes `(refusal:
<code>)`.
- Tests: fixed literals are named; dynamic, filesystem, typed and
non-error throws never leak their text.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/private-refusal-code.test.ts
tests/runner-e2e/manual-private-transfer.test.ts`: 15 passed.
- `npx tsc --noEmit -p tests/runner-e2e`: clean.
- The three files the transfer requires to match between trusted and
target checkouts (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
unchanged.
## Risks
- The public log line gains a code. Codes are either fixed literal
messages (already public in this repository's source), Node error codes,
or error type names. Dynamic messages, which can contain paths, are
reported only as `dynamic`.
- Reading the module sources once at failure time adds a small file read
on the error path only.
## Model Used
Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
0f19eb39d3 |
fix(chat): link the profile of an agent chat acted on (#19893)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat lets a person ask the CEO agent to act on other agents, for example to resume a paused agent > - The chat guidance tells the agent how to link tasks, but it does not tell the agent how to link an agent profile > - In a chat eval, the CEO resumed Morgan and reported the result, but the reply had no link to Morgan > - The person then has no direct route to check the saved availability of the agent > - This pull request adds one guidance sentence for agent profile links > - The benefit is that a reply about an agent gives the person a link to that agent ## Linked Issues or Issue Description **What happened?** A person asked the chat agent: "Resume Morgan so they can pick up their work." The agent resumed Morgan through the API and reported the new status. The reply did not link Morgan's profile. The chat guidance gives only the task-link template (`/<prefix>/issues/<identifier>`), so the agent had no route for an agent. **Expected behavior** When a reply is about a specific agent, the reply links that agent's profile at `/<prefix>/agents/<urlKey or id>`. The link is a plain profile link. It is not an `agent://` mention, because a mention notifies the agent. **Steps to reproduce** 1. Pause an agent named Morgan. 2. In Agent Chat with the CEO, send "Resume Morgan so they can pick up their work." 3. Read the reply. It reports the resume but has no link to Morgan. Refs #19889 (adds the run-link template in the same guidance; this change merges cleanly with it). ## What Changed - `server/src/services/agent-conversations.ts`: add one sentence to the chat directive. It tells chat to link an affected agent's profile at `/<prefix>/agents/<urlKey or id>`, and to use `agent://` only to notify an agent. - `server/src/__tests__/agent-conversations.test.ts`: add a regression test that checks the directive and the built task prompt contain the profile-link guidance. ## Verification - `cd server && pnpm exec vitest run src/__tests__/agent-conversations.test.ts -t "chat prompt policy"` passes (7 tests). - The new test fails on `origin/master`, because the directive does not contain the sentence. - The route exists in `ui/src/App.tsx` (`agents/:agentId`), and the UI accepts a url key or an id. - End to end: the `agents.resume-agent` chat eval runs with this change on its case branch. ## Risks - Low risk. The change adds prompt text only. No API, schema, or UI behavior changes. - The prompt becomes one sentence longer. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edit), extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
29b6411660 |
fix(chat): give chat replies the approval route (#19892)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat lets a person ask their agent about company state, such as "What decisions are waiting on me?" > - The chat directive tells the agent how to link a task: `/<prefix>/issues/<identifier>` > - The directive gives no route for an approval, and the approval tools return no href > - The completion feedback tells the agent not to guess routes, so the agent names a pending approval but cannot link it > - The person then cannot open the approval from the chat to decide it > - This pull request adds the approval route that the board UI already serves to the chat directive > - The benefit is that a chat reply can link each pending approval to its exact source ## Linked Issues or Issue Description **What happened?** In the Agent Chat eval `inbox-search.show-decisions`, the person asked "What decisions are waiting on me?". The agent found the pending hire approval and listed it. It did not link it. Its recorded reasoning said that it had no known href for approvals and that it must not invent URLs. Task sources in the same reply had working links. **Expected behavior** The reply links the pending approval, so the person can open it and decide it. **Steps to reproduce** 1. Create a pending `hire_agent` approval in a company. 2. In Agent Chat, ask "What decisions are waiting on me?". 3. See that the reply names the approval but gives no link to it. **Paperclip version or commit** `master` at the base of this branch. **Agent adapter(s) involved** Not adapter-specific (core bug). The chat directive is shared by all runners. ## What Changed - `AGENT_CHAT_DIRECTIVE` now says to link approvals as `/<prefix>/approvals/<approvalId>`, with the same company prefix as task links. The board UI serves this route (`approvals/:approvalId` in `ui/src/App.tsx`). - Added a regression test that the directive gives routes for both tasks and approvals and keeps the "never invent a hostname" rule. ## Verification - `pnpm exec vitest run server/src/__tests__/agent-conversations.test.ts` passes (41 tests). The new test fails without the directive change. - End to end: the change is cherry-picked onto the eval branch `chat-eval/case-show-decisions` and run in the Runner Full-Stack E2E workflow for `chat-features.runner-opencode.local.inbox-search.show-decisions`. ## Risks - Low risk. The change adds one sentence to the chat prompt. It does not change routes, permissions or tool results. - The prompt becomes slightly longer for every chat turn. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
cbf811cd3b |
fix(recovery): prove a replaced deployment's copyback owner gone automatically (#19913)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save their workspace after the result is accepted, under
a copyback owner receipt
> - A deploy can restart the server while that save is in progress
> - The new server runs in a new container, so it cannot see the old
process and fails closed
> - The run then waits on a board action that asks a person to verify
the previous controller
> - Recoveries must continue without a human step, and copy must not ask
anyone to inspect a server
> - This pull request adds a deployment-level rule to the shared
owner-liveness primitive and uses it in workspace finalization
> - The benefit is that a save interrupted by a deploy resumes on its
own, with no second writer
## Linked Issues or Issue Description
**What happened?**
A deploy restarted the server while three runs were saving their
accepted results. Each run kept a copyback owner receipt that names the
old Railway deployment on another container. The restarted server
reported `native_workspace_finalization_owner_unverified`, owned by the
board, with the next action "Verify the previous controller and its
workspace-copyback processes have stopped...". The cloud harness stop
receipt did not arrive, so the runs stayed in "Recovery needed" for
about an hour, until a person confirmed the stop by hand.
**Expected behavior**
The restarted server proves that the previous controller is gone,
resumes the save, and settles the run with no human step. No user-facing
text asks a person to verify a server or controller.
**Steps to reproduce**
1. Start a native run on a Railway service with a volume.
2. Let the result be accepted and the workspace save start.
3. Deploy, so that the server restarts during the save.
4. See that the run waits for a board confirmation.
**Paperclip version or commit**
Observed on
|
||
|
Devin FoleyandPaperclip
|
ee556a266d |
perf(storage): upload workspace snapshots in parallel verified parts (#19917)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - With remote (sandbox) execution, every agent turn ends by saving the
workspace: capture an archive, encrypt it, upload it to object storage,
and verify it.
> - On a busy S3-backed instance, the upload step is the slowest part of
a save. A 1.5 GB workspace can hold a run open for minutes after the
agent finished.
> - The upload step reads the archive several times, uploads through one
stream with 4 parts in flight, and then downloads the whole object again
to verify it.
> - Parts that use the stream path share one process-wide 32 MiB buffer
window, so saves that run at the same time slow each other down.
> - This pull request uploads parts in parallel straight from the
encrypted file, lets the backend verify every part with Content-MD5, and
replaces the full read-back with a size check for those verified
uploads.
> - The benefit is faster saves, a smaller run-finalization window, and
no interference between concurrent saves.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Remote workspace snapshot upload (`uploadWorkspaceSnapshot`) to
S3-compatible storage.
**Subsystem affected**
Server storage (`server/src/storage/*`) and remote workspace snapshot
storage.
**Current behavior**
For one save, the upload step:
1. hashes the archive;
2. reads the archive again to encrypt it into a temporary file;
3. hashes the encrypted file;
4. uploads it as one stream with 4 parts in flight;
5. downloads the whole object to verify it.
On one hosted instance over 7 days (724 saves, median 731 MB), the
upload step ran at a median of 6.5 MB/s. With no other save running it
reached 11–14 MB/s. With six or more saves at the same time it fell to
4.4 MB/s. From the same region and with the same part settings, a plain
client reached 31 MB/s to the same kind of bucket.
**Proposed behavior**
Upload parts in parallel from byte ranges of the encrypted file, with
Content-MD5 on each part. Confirm only the stored size after a verified
upload. Digest the archive in the same pass that encrypts it.
**Reason and benefit**
Saves finish faster and hold their run open for less time. Concurrent
saves no longer serialize on the shared buffer window. A transient part
failure is retried from disk instead of failing the whole save.
**Breaking changes**
None. `StorageProvider.putObject` may now return a receipt. Providers
that return nothing keep the current behavior, including the full
read-back.
Related: none found.
## What Changed
- `storage/types.ts`, `storage/service.ts`: an upload can name its
`sourceFile`, a private file whose bytes are exactly the body.
`putObject` may return `{ integrity: "server_verified_md5" }`, and
`putFile` passes it through as `PutFileResult.integrity`.
- `storage/s3-multipart.ts`: new `putS3MultipartFile`.
- Each part reads its own byte range of the file. Up to 8 parts upload
at once, with no shared buffer window.
- Each part carries the Content-MD5 of its range.
- A failed part is resent from disk, up to 3 attempts.
- The first part that runs out of attempts stops its siblings. The
multipart upload is aborted only after every part settles.
- A source file whose size differs from the declared length is refused
before any request.
- `storage/s3-provider.ts`:
- A `sourceFile` larger than 16 MiB uses `putS3MultipartFile`. A smaller
one uses one PUT with Content-MD5.
- These requests use a second client with `requestChecksumCalculation:
"WHEN_REQUIRED"`, so bodies are plain and the backend verifies
Content-MD5 directly. The default client wraps bodies in aws-chunked
framing with a CRC32 trailer.
- Other uploads are unchanged.
- `services/remote-workspace-snapshot-storage.ts`:
- Encryption digests the plaintext in the same read. An unchanged file
identity (dev, inode, size, mtime, ctime) across that read replaces the
separate digest pass. A change still fails with `archive_changed` before
any upload.
- The encrypted file is still digested from disk, so the recorded hash
covers the exact bytes that the parts read.
- The upload passes `sourceFile`. When the provider reports
`server_verified_md5`, a HEAD confirms the stored size instead of
downloading the object.
- The full read-back stays for:
- other providers (local disk);
- recovery of earlier attempts;
- the caller that keeps a verified archive file.
## Verification
- `pnpm --filter @paperclipai/server typecheck`: exit 0.
- `vitest run src/__tests__/storage-s3-multipart.test.ts
src/services/remote-workspace-snapshot-storage.test.ts`: 59 passed. New
tests:
- every part carries the Content-MD5 of its range, and 8 parts are in
flight;
- a failed part is resent and the upload completes;
- a part that keeps failing aborts the upload only after in-flight
siblings settle;
- a size mismatch is refused before any request;
- a small file goes as one PUT with Content-MD5, through a real HTTP
server;
- the read-back is skipped only for a server-verified upload, and the
snapshot still restores;
- the read-back stays when the caller keeps the read-back file;
- a stored size mismatch is rejected;
- an archive that changes during encryption is rejected before upload.
- With the source change reverted, the three new snapshot tests fail.
- Real S3-compatible bucket (a Railway bucket, run from a workstation):
- A 3 MiB single PUT stored exactly: HEAD size, ETag and read-back
SHA-256 all matched.
- A 96 MiB, 13-part upload stored exactly, with the same checks.
- A part with a wrong Content-MD5 was rejected with `BadDigest`.
- The same backend stored a wrong `x-amz-checksum-sha256` without
checking it. That is why this change uses Content-MD5.
- Measured from a container in the app's region, 1 GiB with the app's
settings (8 MiB parts, 4 in flight), to the same kind of bucket: 31
MB/s. With 16 in flight: 69 MB/s.
- 77 storage-related suites: the only failures are the same 19 that fail
on unmodified `master` in this checkout. They are in 4 files: missing
bundled connector skills and an unbuilt local runner daemon.
## Risks
- Integrity now depends on the backend enforcing Content-MD5 for file
uploads. AWS S3 and the tested S3-compatible backend enforce it. A
backend that ignores Content-MD5 would accept corrupted parts without a
read-back. A later restore still fails closed: AES-GCM authentication
and the plaintext SHA-256 are checked on load.
- Up to 8 parallel part requests per save, instead of 4 shared ones.
Memory per part is a 64 KiB stream buffer.
- Archive-change detection now uses file identity instead of a second
digest. The archive lives in a private temporary directory that the
capture step creates.
- Medium risk overall: the change is limited to file-sourced S3 uploads
and the snapshot verify step.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs, S3 API probes). AI-assisted
investigation, change and tests, with human direction.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
docs describe this path)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
DottaandClaude Opus 5.5
|
99529f9d71 |
feat(observability): local speed budgets and a run-log budget check (#19863)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When someone messages an agent, the reply must start as fast as
possible, and restart-safety work must not slow it
> - The durable ownership plan makes this invariant 8 ("fast turns") and
asks Phase 1 to turn the Phase 0 baseline into p50/p95 budgets per
adapter
> - Without budgets, a regression in time to first token, agent boot or
restart recovery shows up only when someone reruns the baseline by hand
> - This pull request records the budgets and adds a per-run check that
writes a run-log row when a run exceeds its p95 budget
> - The benefit is that every later phase can show "no regression
against the budgets" from the run log, without new waits on the turn
path
## Linked Issues or Issue Description
Refs #19831 (plan, "Performance constraint", rule 4, and invariant 8).
Refs #19846 (the Phase 0 baseline these budgets come from). Stacked on
#19861.
**Problem**
The plan requires p50/p95 budgets per adapter for TTFT, agent boot and
restart recovery latency, and a check against them. Neither exists.
**Expected behavior**
Budgets are recorded in the plan and in code. A run that exceeds a p95
budget gets one run-log row that names the metric, the measurement and
the budget.
**Impact**
Phase 2–5 exit criteria can cite budget breaches from chaos waves and
staging. The check adds no wait before `turn.start`.
## What Changed
- New `server/src/services/native-runtime/native-speed-budgets.ts`: the
local budget table (codex, claude, opencode) for TTFT and agent boot
(cold and warm) and restart recovery latency. Each budget is 1.2× the
baseline, rounded up to 0.5 s. `evaluateSpeedBudget` judges one run
against p95. An unknown start kind uses the looser budget. Remote runs
and adapters without a baseline are not judged.
- `native-run-trace.ts`: the trace takes an optional `budget` subject.
It checks `task.time_to_first_token`, agent boot (end of
`heartbeat.queue` to the end of the first `runner.turn.submit` or
`runner.session.resume`) and restart recovery once each. It writes
`run.performance.budget_exceeded` through `defer`, so the write is
detached from the event path. `finish` now drains diagnostics that defer
more work.
- `native-session-executor.ts`: passes the adapter and environment to
the trace, labels cold or warm at the existing `provider_session` reuse
receipts, and checks restart recovery at a recovered run's first
provider event, measured from the startup recovery `ready` time.
- Docs: budget tables in
`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` ("Speed
budgets (Phase 1)"), and the new event in `doc/run-log-events.md`.
## Verification
- `npx vitest run
src/services/native-runtime/native-speed-budgets.test.ts
src/services/native-runtime/native-run-trace.test.ts` (in `server/`): 34
tests pass. They cover:
- every budget has p50 ≤ p95;
- the provider-to-adapter mapping;
- p95 judgement, and the looser budget when the start kind is unknown;
- one breach for a slow boot, and none for a second submit or a fast
TTFT;
- a deferred TTFT breach is written before the trace closes;
- restart recovery is checked once;
- remote runs and runs with no budget subject write nothing.
- `npx tsc --noEmit -p server`: no errors.
## Risks
- Low risk. This is a run-log change only (AGENTS.md §5.7, no extra
review). The OTLP and Telemetry paths are unchanged.
- The baseline samples are small (2–12 per cell) on a shared host.
Budgets carry 20% headroom, and the codex restart budget (8.0 s p95) is
tight. Expect some breach rows on a loaded machine. They are diagnostics
and change no behavior.
- Restart-recovery budgets use run 4 of the baseline, as merged in
#19846.
- Restart recovery is measured from the startup recovery `ready` time to
the first provider event, which matches the baseline definition.
- Previous-release story: no persisted run state changes. The new event
is additive.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1107.0-canary.22
|
||
|
DottaandClaude Opus 5.5
|
d5dd09c966 |
docs(plans): design turn-scoped ACPX MCP authority; correct the Claude baseline (#19912)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Follow-up messages must start fast. #19888 made Codex and OpenCode follow-ups 2.6–5× faster by keeping their runners warm. > - Claude (ACPX) did not gain, because ACPX relaunches the Claude process and reloads its session on every follow-up (3.2–4.2 s). > - That relaunch is a safety boundary: the ACPX MCP endpoint lives as long as the process and cannot attribute a tool call to a turn, so reaping the process is the proof against late calls. > - Removing it needs a new authority model, which needs a reviewed design before any code. > - This pull request adds that design and corrects the Claude baseline. ## Linked Issues or Issue Description Refs #19888 (warm default), #19846 (TTFT baseline, finding 4). **Docs issue** The TTFT docs did not explain why Claude follow-ups relaunch, what removing that would require, or the corrected Claude numbers. **What is wrong or missing** `doc/plans/2026-11-07-warm-follow-up-ttft.md` reported a Claude follow-up at 10.5 s, measured under heavy host load. A quieter re-measurement gives 6.3 s per-turn against 7.1 s warm. There was no design for removing the per-turn relaunch. **Proposed fix** A design document for turn-scoped ACPX MCP authority, and a corrected-baseline section in the warm follow-up doc. ## What Changed - New `doc/plans/2026-11-07-turn-scoped-acpx-mcp-authority.md`, which covers: - the measured cost breakdown; - how the current boundary works (tool bridge bearer, call-ID tombstones, `restart_idle_provider`, `run.attach`); - a threat model (late tool calls from a settled turn, replay, credential replay, cross-turn and cross-run confusion, restart replay); - three candidate designs with their safety arguments: turn-scoped credentials in a live session, a proof that no calls are in flight, and an in-process session reload with per-generation credentials, which is recommended for a spike; - restart and chaos implications, the expected TTFT gain, and next steps. - `doc/plans/2026-11-07-warm-follow-up-ttft.md`: corrected Claude baseline. Per_turn stays the default; the earlier 10.5 s was host noise. Finding 4 now points to the design and drops the idle-retirement idea, which would only bring warm level with per-turn. - Docs only. No code changes until Dotta has reviewed the design and a security review is done. ## Verification - The Claude numbers come from `ttft-baseline.ts --adapters claude --samples 6 --phases coldwarm --lifecycle <per_turn|warm>` on master `fc6e1b0346` (load average about 9). - The step breakdown comes from temporary timing in runnerd's ACPX backend. It was not committed. - `node scripts/check-maps.mjs` passes. ## Risks - Docs only. The design proposes no behavior change until it has been reviewed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
177f642da7 |
fix(recovery): continue native runs that failed before their first claim (#19911)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs go through a coordinator claim before any provider work
starts
> - A run can fail before that claim, for example while it restores the
previous harness backup
> - The failure path then records a no-replay hold, as if the run had
acted
> - Workspace recovery saves the files, but the task stays blocked and
asks a person to send a message
> - Recoveries must continue without a human step when no side effect is
possible
> - This pull request lets the existing safe-replacement worker continue
such runs once, after their files are saved
> - The benefit is that these tasks resume automatically, and runs that
may have acted keep today's reconciliation
## Linked Issues or Issue Description
**What happened?**
Native runs that failed before their first coordinator claim were never
continued automatically. They had no claim, no provider process and no
provider or tool events. Their result recorded `executionRecovery: {
kind: "bootstrap", providerWorkStarted: false }`. The failure path still
moved the coordinator to `terminal_failure` with
`native_continuation_requires_reconciliation` and recorded a resolved
`preserve_without_replay_v1` hold (replay blocked, board owner).
Workspace recovery then saved the files, but the next action stayed
"Saved work recovered. Send a message to continue."
`nativeReplacementPending` accepted only a few failure codes, and the
interrupted-preparation path covers only legacy runs.
**Expected behavior**
When a native run provably failed before its first claim, Paperclip
continues it once, automatically, after its files are saved (or when it
had no workspace to save). Runs with a claim, a process, a provider or
tool event, or recorded provider work keep the existing reconciliation.
**Steps to reproduce**
1. Start a native run with a remote workspace.
2. Make it fail before `native.coordinator.claim` (for example, abort
the harness-backup restore).
3. Let workspace recovery save the files.
4. See that the task stays blocked and asks for a message.
**Paperclip version or commit**
Observed on
|
||
|
DottaandClaude Opus 5.5
|
fc6e1b0346 |
perf(native): keep local Codex and OpenCode runners warm by default (#19888)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When someone messages an agent, the turn must start fast. Time to first token (TTFT) is a first-class metric. > - The Phase 0 baseline (#19846) found that every local turn launches a new runner and loads the provider session again, because `per_turn` is the default lifecycle. That made a follow-up message slower than a new task. > - Warm runners keep the runner and provider session alive between runs. #19874 fixed the defect that broke warm OpenCode follow-ups, and #19879 bounds an idle warm runner whose controller dies. > - This pull request makes warm the default for local Codex and OpenCode agents, and migrates existing agents that only carried the old creation default. > - The benefit: follow-up TTFT p50 drops from 12.0 s to 4.7 s (Codex) and from 21.7 s to 4.2 s (OpenCode), with no new restart-chaos failures. ## Linked Issues or Issue Description Refs #19846 (TTFT baseline), #19852 (startup sub-spans), #19874 (warm completion contract), #19879 (warm reconnect grace). **What existing behavior does this improve?** The default runner lifecycle for native agents, and how a follow-up message starts. **Current behavior** Agent creation writes `lifecycleMode: "per_turn"` into every native agent. Every follow-up launches a new runner and restores the provider session: 4.2 s (Codex) and 16.3 s (OpenCode) at p50 before the turn starts. **Proposed behavior** New native agents store no lifecycle. On a local target, an unset lifecycle resolves to `warm` with the default 5-minute idle window for Codex and OpenCode. Other harnesses, remote targets and managed providers without warm support stay `per_turn`. An explicit choice always wins. Migration 0443 clears the stored creation default from existing agents, except where an edit set "Turn by turn" explicitly. **Reason and benefit** Measured locally (`doc/plans/2026-11-07-warm-follow-up-ttft.md`). Warm follow-ups are 2.6–5× faster for Codex and OpenCode, and restart recovery latency is equal or lower. ## What Changed - `native-lifecycle.ts`: an unset `agentMode` resolves to warm (5-minute idle window) for local `codex_app_server` and `opencode_server` agents. `runtime-selection.ts` passes `localTarget`. Explicit `per_turn` or `warm` wins. An admitted session keeps its persisted policy. ACPX Claude stays `per_turn`: it relaunches its provider process on every follow-up by design, so warm did not shorten its follow-up (10.7 s warm vs 10.5 s per_turn). - Creation and configuration writes no longer store `lifecycleMode: "per_turn"` for Codex, Claude or OpenCode. This covers `normalizePaperclipRunnerAdapterConfig` (runs on every create and update), `paperclipRunnerTransitionConfig` (switching to the native runner), `agent-runner-selection.ts`, `codex-local` `build-config.ts`, and the simplified new-agent setup (`paperclipRunnerCreationLifecycle` in `@paperclipai/shared`). Grok, Pi and Cursor keep their explicit value. - Agent form: the Runner lifecycle field shows the unset state as "Automatic". "Turn by turn" and "Warm session" remain. - Migration `0443_native_warm_follow_up_default.sql`: clears `lifecycleMode = 'per_turn'` from native Codex, OpenCode and Claude agents unless `agent_config_revisions` shows an edit that set it to `per_turn`. Each migrated agent gets an `agent.updated` activity entry with the old value (`actor_id = migration:native-warm-follow-up-default`). Creation records no revision, so a "Turn by turn" pick made at creation cannot be told apart from the old default. The product owner approved migrating those agents. Claude agents follow the default, which is `per_turn` today. - Restart chaos harness and TTFT driver: `--lifecycle per_turn|warm|default` (saved in the wave plan). `default` removes the catalog's pinned lifecycle, so a wave exercises the real default path. - Docs: `doc/DEVELOPING.md` "Runner lifecycle default", and the results in `doc/plans/2026-11-07-warm-follow-up-ttft.md`, which also documents baseline finding 4 (Claude's resumed turn-start gap is ACPX's per-turn process generation, not the provider). ## Verification - `server`: `native-lifecycle.test.ts` (new default cases: local Codex and OpenCode warm, Claude, remote, unknown target, unmeasured harness, managed Claude, admitted per-turn session, explicit choices, non-native adapter), `agent-runner-selection.test.ts`, `runtime-selection.test.ts` (335 pass). - `packages/db`: `native-warm-follow-up-default-migration.test.ts` on embedded Postgres. Default agents are cleared, the explicit edit is kept, other harnesses and legacy adapters are untouched, audit rows are written once, and a rerun is a no-op. Without the explicit-edit filter the test fails. `check:migrations` passes. - `codex-local` `build-config.test.ts` (65), `ui` `config-fields.test.tsx` (40), restart-chaos unit tests (134), `tsc --noEmit` for `server`. - `adapter-utils` `paperclip-runner-lifecycle-default.test.ts`: configuration writes and adapter conversion store no lifecycle for Codex, OpenCode and Claude, keep `per_turn` for the others, and preserve explicit choices. It fails without the fix. - End-to-end default check on this branch (`ttft-baseline.ts --lifecycle default`, codex and opencode, 2 samples): the agents read back with no stored lifecycle, and every follow-up reused the runner and provider session (no `runner.process.launch`). Follow-up TTFT was 3.6–4.8 s. This check found the per-write `per_turn` injection fixed here. - Restart chaos (codex and opencode, A1/B1/B5, K1/K2/K6, seed 4242), 8/8 PASS each with no open signatures: forced warm, forced per_turn, and `--lifecycle default` on this branch (agents created without a lifecycle). - `pnpm verify` selects broader CI (harness config is a global input), so required CI supplies the broad coverage. ## Risks - Behavior change: local Codex and OpenCode agents keep a runner and provider process for up to 5 minutes after a run. Instance sleep waits for that idle window. The process is reaped by the idle timer, by graceful shutdown, or by the #19879 reconnect grace if its controller dies. - Migration: agents whose "Turn by turn" was chosen at creation become Automatic. The activity entry records the old value; choosing "Turn by turn" restores it. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): a task session keeps the lifecycle policy it was admitted with, so sessions saved before this change stay `per_turn`. New sessions get the default. No run, checkpoint or launch state changes shape. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1107.0-canary.21 |
||
|
DottaandClaude Opus 5.5
|
76994c5027 |
fix(native): bound failover backup restores by inactivity, not total time (#19906)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native per-turn session that continues an earlier conversation first restores its failover backup from durable storage. This happens before the coordinator claim, so no provider has started. > - That restore ran under a fixed 300 s total deadline (`AbortSignal.timeout(300_000)`). The deadline covers the download and the local decrypt, digest, extraction and provider-artifact steps. > - On staging, restore time for resumed sessions grew during the day. It reached 300 s, and then every resumed run of one review agent failed with the generic "The operation was aborted". Fresh sessions were not affected. > - A failure at this point also enters the saved-work reconciliation path, so the user request stopped. > - This pull request bounds each durable read by inactivity instead of total time. A slow transfer continues while bytes arrive. A read that stops sending bytes fails with a named code. > - The benefit is that resumed sessions no longer fail only because their backup is large or the store is slow. ## Linked Issues or Issue Description Refs #19894 (adds the same idle-deadline rule for workspace snapshot reads; this PR covers the separate native failover-backup reads). **What happened?** On staging, a PR-review agent had 17 runs fail in about 80 minutes. Each failure had the message "The operation was aborted" and the code `native_continuation_requires_reconciliation`. Every failing run resumed an existing session (`sessionIdBefore` set, per-turn lifecycle). Every fresh-session run in the same window succeeded. In each failing run, the error came 304–309 s after the `environment.startup` span, and no `native.coordinator.claim` span was recorded. The run result had `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. Earlier resumed runs of the same agent show the time from `environment.startup` to `native.coordinator.claim` growing from about 30 s to about 200 s before the failures started. The only fixed 300 s deadline on that path is the failover-backup restore in `restoreDurableHarnessForExecution`. **Expected behavior** A resumed session restores its backup when the store keeps sending bytes, even when the total time is more than five minutes. A read that stalls fails with a named, diagnosable code. A failed restore publishes no partial backup and keeps the saved backup. **Steps to reproduce** 1. Persist a native failover backup with provider artifacts. 2. Restore it through a storage reader that sends bytes slowly, so the full restore takes longer than the deadline. 3. Before this change, the restore fails with "The operation was aborted". A reader that stops sending bytes, with no caller deadline, waits indefinitely. ## What Changed - `native-harness-backup-storage.ts`: add `downloadIdleTimeoutMs` to `restoreDurableNativeHarnessBackup` and `verifyDurableNativeHarnessBackup`. When set, each durable `getObject` (the backup object and the provider artifacts) runs under an idle timer. Each received chunk renews the timer. The timer is active only while a read is in flight, so local decrypt, digest, extraction and copy are never timed. On expiry the whole restore aborts and throws `native_harness_storage_download_stalled`. Caller cancellation keeps its own error. - `verifyRecoverableNativeHarnessBackup` and the executor's `restoreDurableHarnessForExecution` use the five-minute idle limit (`NATIVE_HARNESS_DOWNLOAD_IDLE_TIMEOUT_MS`) instead of `AbortSignal.timeout(300_000)`. - The whole restore or verification keeps a 30-minute outer limit. It fails with `native_harness_storage_restore_deadline_exceeded`, so a hung local step stays bounded and named. - The executor passes the startup Stop signal (`providerPackSourceAbort`) into the restore, so Stop (also a Stop that came before the restore began) ends it. Only a startup that already completed does not reuse its signal. The restore checks the signal after the local copy and before it publishes `current`. - The wrapped storage is a `Proxy` that replaces only `getObject` and forwards every other member to the real service. - `doc/DEVELOPING.md`: document the idle limit and the named code. Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted shape changes. Backups, artifact rows, receipts and checkpoints saved by the previous release restore unchanged. A deploy that lands during a restore stops the process. The new process starts the restore again from the saved durable backup under the new limit. The existing hash, size, encryption and atomic publication checks are unchanged. ## Verification Run from `server/`: - `pnpm exec vitest run src/services/native-runtime/native-harness-backup-download-deadline.integration.test.ts` — 4 tests passed (embedded PostgreSQL, local-disk storage, an OpenCode backup with provider artifacts). The tests are in a focused suite, so the existing 990-line storage suite does not grow. - `pnpm exec vitest run src/services/native-runtime/native-harness-backup-storage.integration.test.ts` — 68 existing tests passed. - New: "keeps restoring a backup whose download outlasts the idle window while bytes keep arriving". With the former total-deadline policy, the same slow transfer fails with "The operation was aborted" and publishes nothing (the staging message). With the idle limit, the restore completes after more than twice the window, restores the session and the OpenCode launch artifacts, and verification also completes. - New: "names a download that stops sending bytes and publishes no partial backup". It checks restore and verify, checks that no `current` backup is published, and then restores the same saved backup successfully. - New: "stops at the outer bound during local copy and publishes nothing". - New: "lets Stop end a progressing download with its own reason, including a Stop before it starts". - RED: with the production files reverted to `origin/master`, the stall test does not finish and times out after 10 s, because no stall is detected. The other two tests pass on the old code, because the total deadline was the executor's caller policy. The first test reproduces that policy directly. - `pnpm exec vitest run` on `native-harness-online-preservation`, `native-harness-filesystem-independence` and `native-controller-migration` integration suites passed. `native-warm-checkpoint-ownership.integration.test.ts` fails locally on master as well, because the local Rust `paperclip-runnerd` binary is not built. - `node scripts/typecheck-layers.mjs` passed for all layers. Not verified: a live resumed run on staging with this build. The staging evidence shows where the time goes (before the coordinator claim, on resumed sessions only) but does not show which restore step was slowest. The fix removes the total deadline for every step on that path. ## Risks - A restore that progresses very slowly can now take longer than five minutes. Before, it failed. The run still stops on caller cancellation. - A local step that hangs (for example a stalled disk) is now cut off by the 30-minute outer limit instead of 5 minutes. A stalled local writer also pauses the read through backpressure, so it can be reported as `native_harness_storage_download_stalled`. - A failure before the coordinator claim still enters the existing saved-work reconciliation path. This PR does not change that classification. It removes the cause of these failures. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edits, read-only browser for API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
57ae2fc309 |
perf(workspaces): reduce snapshot stream scheduling overhead (#19900)
Use bounded 1 MiB filesystem buffers during workspace archive hashing, encryption, and restore. Preserve integrity, ownership, cancellation, and snapshot format guarantees. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d7f02c51b4 |
fix(skills): scope file protection to admitted selections (#19904)
Allow unused skill deletion when unstarted admission entries have no retained execution. Limit unknown legacy selections to skills that could have existed when the validated immutable run finished. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
c9959814ba |
fix(workspaces): cancel stalled snapshot reads and preserve restore refusals (#19894)
Cancel encrypted snapshot reads after five minutes without incoming bytes. Renew the deadline on progress, preserve caller cancellation, settle the physical request and private scratch, and retain saved snapshots for normal retry admission. Keep Node warnings out of server-owned helper refusal labels. Verified both complete owner suites (145 tests), the prior unchanged storage and transport coverage, server typechecks, final-head CI and Greptile 5/5. The earlier CI cancellation-signal assertion failure was reproduced locally and corrected with checks for abort state, reason, stream closure and cleanup. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
988a4b6749 |
feat(recovery): ownable-kind registry and report-only ownership reconciler (#19861)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart leaves in-progress records owned by dead processes, and about 31 per-subsystem proofs decide when to release them > - The durable ownership plan (section C) asks for one registry of ownable record kinds and one reconciler > - Phase 1 must measure first: how many in-progress records exist, who owns them, and which invariants they break, with no behavior change > - This pull request registers 46 kinds from the Phase 0 inventory's section C and adds a report-only pass to the existing native finalization reconciler > - The benefit is per-kind invariant-violation counts from chaos waves and staging. That data unlocks Phase 2 (moving kinds onto one settle path) and the removal of the company-wide admission gate ## Linked Issues or Issue Description Refs #19831 (plan, section C and Phase 1). Refs #19846 (Phase 0 inventory, section C). Stacked on #19853 (the prover) and #19851 (one process identity). **Problem** No single place lists the control plane's in-progress records or checks the plan's invariants. Each subsystem releases its own records by its own rules, so nobody can count how many records break "owned", "settled" or "visible next action". **Expected behavior** One report-only pass lists every registered kind, classifies each owner as gone, alive or unprovable, and records each invariant violation once. It changes no record and stays quiet when nothing is in progress. **Impact** Phase 2 moves kinds onto the shared settle path in the order these counts show. Phase 3 removes the admission gate once the I3 data exists. ## What Changed - New `server/src/services/recovery/ownership-registry.ts`: 46 `OwnableKind` entries (`id`, `obligation`, `storage`, `listInProgress`, `settle: null`, `invariantChecks`). Each record names its owner as a process stamp, a fence token, a run, an operator, or none. The skill-cache lock is registered with a named coverage gap because its lock files live in per-skill roots. - New `server/src/services/recovery/ownership-invariant-report.ts`: one pass classifies every owner with `proveOwnerGone` (or the owning run's status), then runs: - I1 owned: an in-progress record with no owner; - I2 settled: an owner proven gone on two consecutive passes while the record is still in progress; - I3 admission independence: terminal runs whose accounting outlives the 10 min closure, which hold company admission; - I4 exactly-once intents: wake requests ended `skipped`/`failed` on a temporary refusal (heuristic until section E); - I7 visible next action: an agent-assigned `in_progress` issue with no live run, no pending wake and no open recovery action for 5 min. - I5, I6 and I8 are reported as named coverage gaps; I8 comes in the speed-budget PR. - Violations are written once per process to the activity log as `recovery.invariant_violation` (`actorId: ownership-reconciler`, `details.mode: report_only`), insert-only with no live-event or plugin fan-out. Nothing goes to Telemetry (AGENTS.md §5.7). - `native-finalization-reconciler.ts`: `registerOwnershipReconciliationWork` runs the pass in the shared delivery-work scheduler (new queue `ownership-invariant-report`). There is no second loop. - `index.ts` registers it after residual recovery and does not await its first pass, so readiness is not delayed. ### Sleep-aware trigger (rules/sleep-aware-background-work.mdc) - **Trigger:** one pass at registration (startup reconstruction from durable truth). After that, a settled `active-execution-recovery` signal, which run lifecycle, controller renewal and process-evidence writes already emit. Signals are coalesced to at most one pass per 30 s. No producer gained a write. - **Pending predicate and next deadline:** a 30 s follow-up only while the last pass saw a changing in-progress set, or an owner proven gone that needs the second I2 pass. - **Quiet:** an empty set or the same stuck set twice sets no deadline. The test asserts `nextWakeAt() === null` after an idle pass. - **Turn path:** the pass never runs before `turn.start`. Stamps read only cached identity. ## Verification - `npx vitest run src/__tests__/ownership-reconciliation-report.test.ts` (embedded Postgres, in `server/`): 4 tests pass: - every kind is registered once with no settle path; - verdicts per class: alive (this process's claim), gone (lease of a failed run), unprovable (no owner; another PID space); - each rule fires on its fixture, and I2 fires only on the second pass; - every touched table is byte-identical before and after (report-only); - activity rows are written once per violation, and none are written with `record: false`; - the worker is quiet when idle and schedules a follow-up only while records change. - `npx vitest run src/__tests__/delivery-work-coordinator.test.ts src/services/owner-proof.test.ts src/__tests__/server-startup-feedback-export.test.ts`: pass. - `npx tsc --noEmit -p server`: no errors. ## Risks - No record changes, so no settle risk. The risk is load: one pass runs about 46 small indexed or `limit 201` queries, at most once per 30 s while runs are active, and none when idle. - I4 matches refusal text with a heuristic. It can over- or under-count until section E gives wake requests closed terminal reasons. It is labelled as such in the code. - Violations dedupe per process, so a restart writes a still-open violation again (once per boot). Chaos reports count distinct `(rule, entityId)`. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted run state changes. The pass reads rows of any age, and a missing field classifies as unprovable. - `heartbeat_runs` and `native_run_finalizations` record no host or PID space, so foreign-boot owners there classify as unprovable (`missing_identity`). This is the inventory's finding 2, and the report makes it countable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
12681d1611 |
fix(sentry): stop resending retained task interruption failures (#19041)
Keep task-state Sentry reports within the recent saved-change window while retaining reports for newly failed recovery and delayed Retry preparation. Include the safe run error code without changing event grouping. Verified 402 relevant local tests, every server typecheck layer, all final-head CI checks, and Greptile 5/5 with no unresolved review threads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1107.0-canary.20 |
||
|
Devin FoleyandPaperclip
|
25273aa69b |
fix(onboarding): start the agent when the opening question is answered (#19886)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A new company starts with an onboarding first task. The task opens
with a greeting and an opening question card that the assignee agent
posts.
> - Onboarding does not wake the agent when it creates the task. The
user's answer must start the agent.
> - When the user clicks an answer, run dispatch cancels the agent's run
0.4 seconds later with `issue_not_in_progress`. Answer continuations
only start on `in_progress` or `in_review` tasks, and the onboarding
task is still `todo`.
> - The user sees nothing until an unrelated bounded retry starts the
agent about two minutes later. This is the first thing every new user
does.
> - This pull request lets an answer start a `todo` task when the task
has never started and no run asked the question.
> - The benefit is that the agent starts immediately after the first
answer. The protection for parked tasks does not change.
## Linked Issues or Issue Description
**What happened?**
I created a new company and clicked an option on the onboarding opening
question ("What would you like to do?"). The agent's run was cancelled
immediately with `issue_not_in_progress`: "Cancelled because
resolved-interaction continuation issue is no longer in_progress
(current status: todo) before the queued run could start". The task
stayed in `todo`. About two minutes later, a `transient_failure` bounded
retry of the cancelled run started the agent through the normal checkout
path.
**Expected behavior**
The agent starts as soon as the user answers the opening question.
**Steps to reproduce**
1. Create a new company and hire the first agent through onboarding.
2. Open the onboarding first task.
3. Click an option on the opening question card and submit.
4. Look at the agent's runs. The first run is `cancelled` with
`issue_not_in_progress`.
**Paperclip version or commit**
`master` at `86bc3db6938`. The same code is on `00ad9503fe3`.
**Relevant logs or output**
```
heartbeat_runs: status=cancelled error_code=issue_not_in_progress
"Cancelled because resolved-interaction continuation issue is no longer in_progress (current status: todo) before the queued run could start"
two minutes later: "Scheduled bounded retry 1/2" (retryReason transient_failure)
```
Related: #13068 added the onboarding opening question. #13318 made
answer continuations require `in_progress` or `in_review`.
## What Changed
- New `server/src/services/unstarted-task-question-answer.ts`.
`isUnstartedTaskQuestionAnswer` returns true only when all of these are
true:
- the interaction is an answered question (`ask_user_questions`, status
`answered`) on this task, and it has no `sourceRunId` (no run asked it);
- the task is `todo`;
- the task's `startedAt` is null.
It reads only server-owned rows. `startedAt` is never cleared, so a task
that started and was later moved back to `todo` does not match.
- `run-dispatch/adapters/postgres.ts`: the staleness facts include
`isUnstartedTaskQuestionAnswer`. The adapter reads it under the issue
lock, and only for a resolved-interaction continuation on a `todo` task.
- `run-dispatch/domain/policy.ts`: `decideQueuedRunStaleness` lets such
a continuation start from `todo`. The other answer continuations still
require `in_progress` or `in_review`.
- `services/heartbeat.ts`: the final continuation checkout adds `todo`
to the expected statuses only for the same case. The task then moves to
`in_progress` in the same guarded update.
## Verification
- `heartbeat-stale-queue-invalidation.test.ts` (embedded Postgres):
- New test: an answer to a question that no run asked, on a
never-started `todo` task, starts the assignee. The task moves to
`in_progress` under the run's execution lock. With the fix reverted,
this test fails with the reported error (`cancelled`,
`issue_not_in_progress`).
- New test: three regression cases stay cancelled with
`issue_not_in_progress`:
- an answer on a task that started and was moved back to `todo`;
- an answer to a question that an agent run asked;
- a rejected confirmation that no run asked.
Without the kind and status check, the confirmation case fails.
- `policy.test.ts`: the new fact allows `todo`. The same fact on a
`backlog` task is still cancelled.
- `pnpm --filter @paperclipai/server typecheck`: exit 0.
- `npx vitest run` on these suites: 8 files, 497 tests passed.
- stale-queue invalidation
- run-dispatch policy
- onboarding first-task routes
- onboarding first-task admission
- question-response delivery
- issue-thread interaction routes
- heartbeat retry scheduling
- legacy continuation authority
## Risks
- Low risk. The change only affects answer continuations on `todo` tasks
that have never started, where the question has no source run. All other
continuations behave as before.
- The exception is not limited to the onboarding card. Any answered
question that no run asked, on a never-started `todo` task, can now
start the assignee. This matches what a plain comment wake does on a
`todo` task. Confirmations and other interaction kinds keep the existing
rule. For example, rejecting a confirmation that no run asked still does
not start the task.
- The final checkout does one extra indexed read. It only runs for
resolved-interaction continuations on `todo` tasks.
- I did not find which component scheduled the two-minute
`transient_failure` retry. After this change, onboarding does not depend
on that retry.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs). Investigation, fix, and tests were
AI-assisted, with human direction.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
docs describe this behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
DottaandClaude Opus 5.5
|
f439150cb3 |
fix(recovery): limit the null-host owner fallback to single-host servers and record each use (#19883)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #19868 and #19871 made restart recovery judge a recorded pid only in the PID space that recorded it > - Rows saved before those host columns existed have no recorded host. They kept the previous local dead-pid proof, so in-flight native runs would not be stranded at upgrade > - On an ephemeral (multi-host) server that fallback is unsafe: the row may have been written on another host > - This pull request limits the fallback to single-host servers, records every use, and gives ephemeral servers a working board action > - The benefit is that single-host upgrades still recover on their own, multi-host servers never accept a cross-host proof, and the next release can remove the fallback with evidence ## Linked Issues or Issue Description **Bug description** After #19868 and #19871, a `native_run_finalizations` row with `controller_host_hash = null`, or a run with `process_host_hash = null`, still accepted "the pid is dead here" as proof on every server. That includes an ephemeral server, where the row may come from another host. Nothing recorded how often the fallback was used, so it had no safe removal path. **Expected behavior** - Single-host server: the fallback applies for one release, and each use writes a `legacy_host_unknown_dead_pid_proof` activity-log entry. - Ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`): the row is unprovable, and the board has an action to confirm the previous server stopped. - A mismatched host is always unprovable. Refs #19868, #19871, #19846 ## What Changed - `native-restart-recovery.ts` `evaluateNativeControllerTakeover`: - A coordinator row with `controllerHostHash: null` returns `controller_host_unrecorded` when `ephemeralServerEnabled()` is true. - Otherwise a pid proof from that row carries `legacyHostFallback: true`. - `operator_attested` is accepted as proof (`operator_attested_controller_stopped`). - Receipt-built owners, which have no host key, are unchanged. - New `recordLegacyHostFallbackProof` writes the activity-log entry. It is not a run-log event: run events are progress evidence for restart recovery's crash-loop credit, and the record must not change any recovery decision. It is called from restart recovery, the dead-controller result-lease release, native retirement, and remote warm-session recovery. - New `native-controller-host-recovery.ts`: - `requestNativeControllerStopConfirmation` (board-owned `native_controller_host_unverified` recovery action, idempotent, never displaces another active action). - `confirmNativeControllerStopped`. It verifies the exact recorded controller and stamps `operator_attested` with compare-and-set. It then resolves the action, wakes live-run recovery, and logs `issue.native_controller_stop_confirmed`. - Route `POST /issues/:id/recovery-actions/confirm-controller-stopped`. It is board-only and requires `runtime:confirm_owner_stop`. Generic `restored`/`false_positive` resolution of this cause is refused. A shared `confirmNativeControllerStoppedSchema` validates the body. - UI: `WorkspaceExportRecovery` reuses the "Confirm the previous server stopped" control for this cause, and the generic recovery card hides it. Adds the `issuesApi.confirmControllerStopped` client call. - `recovery/service.ts` orphan reaper: a null `process_host_hash` is judged only when not ephemeral. A successful use records the entry best-effort, after the terminal write. On an ephemeral server the run is left to its existing Stop control. - `ephemeralServerEnabled` moves to a leaf `ephemeral-server-mode.ts` (re-exported by `ephemeral-server-contract.ts`). Importing the contract module from recovery code created an import cycle in the runner child process. - The new route is registered in OpenAPI. - `doc/execution-semantics.md`: documents the host rule and the one-release sunset. ## Verification - `native-restart-recovery.test.ts`: null host on a single-host server is released with the flag. Null host on an ephemeral server is unprovable. A mismatched host is unprovable in both modes. An attested controller is accepted. Receipt owners are unchanged. - New `native-controller-host-recovery.test.ts` (embedded Postgres): - The fallback writes the activity-log entry and no run event. - Ephemeral mode creates exactly one board action. - A wrong run id is refused. - Confirmation stamps the attestation, after which takeover is allowed. The action resolves and the activity is logged. A second confirmation is refused. - `recovery-stale-issue-lock-sweep.test.ts`: on a single-host server a null host is terminalized and records one event. On an ephemeral server it is left alone with no event. The existing audit-failure case still clears the lock. - `WorkspaceExportRecovery.control.test.tsx`: the controller confirmation posts to `confirmControllerStopped` and never to the copyback endpoint. - Locally: these suites plus `remote-warm-session-recovery`, `native-controller-recovery.integration` and `native-startup-cleanup-recovery` (136 passed), and after the CI fixes `native-runner-restart-preparation` and `native-runner-restart-recovery` integration, `openapi-routes` and the suites above (256 passed); the UI recovery suites (52 passed); `node scripts/typecheck-layers.mjs` with no errors in the touched files; `pnpm check:token-gates` clean. ## Risks - Previous-release story: no schema change. Null-host rows saved by earlier releases recover as before on single-host servers, now with an event, and wait for the board on ephemeral servers. The next release removes the fallback; the event count shows whether any rows remain. - On an ephemeral server, a null-host orphaned run process is not reaped automatically. The operator uses Stop. That is the intended "unprovable" outcome. - A board confirmation is an attestation. It applies only to the exact recorded controller (boot id + pid) and only while the host is still unrecorded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
5a70999306 |
Add a versioned online workspace migration command (#19882)
## Thinking Path > - Paperclip preserves the files and recovery state produced by its agents. > - Canonical online migration can preserve selected local workspaces while unrelated work continues. > - Operators sometimes need a newer migration owner before the serving web process can take that release. > - The existing packaged maintenance entry requires stopped writers. > - This pull request adds a finite online entry from one coherent reviewed release tree. > - The entry delegates capture and publication to the existing owners and keeps tool and serving identities separate. ## Linked Issues or Issue Description Refs #19854, #19855 and #19872. The duplicate search found related owner fixes and no separate versioned online entry. **What happened?** The finite online workspace importer was available as a library in the serving image. The packaged all-phase migration entry required maintenance mode. **Expected behavior** Run the reviewed online owner from a separate complete release tree without replacing the web process, stopping unrelated work or changing its release policy. **Deployment mode** A managed instance with its original local volume, database, existing encryption key and shared S3 storage. ## What Changed - Add strict inspect, single, cohort and exact single-journal resume requests with explicit company, workspace, deadline and concurrency bounds. - Verify the private tool root and its own build stamp separately from the serving source commit and deployment. - Reuse the nonmutating full migration-manifest inspector and require the additive controller host-identity column before publication. - Hydrate existing configuration and key bindings, then use the original ESM database pool and canonical storage, source locks and migration owners. - Keep multi-member inspection with the optional canonical cohort preflight in a read-only transaction with denied storage. Releases without that owner refuse cohort operations; single import and resume remain available. - Register the optional runtime bundle entry and document the supported full-tree source mode, resource admission and reconciliation. - Add hash-bound old-writer/reader compatibility and encrypted restore tests. ## Verification Final local revision: `c9345b72f27aee1158047dccd0b1060f9a0be6df`, on master `fbe689938c3bf8c5849d2cd56b9da90029db9389`, macOS arm64 with Node 24.22.2. - Entry and runtime packaging: **34/34 PASS**. - Real PostgreSQL/filesystem compatibility: **6/6 PASS**, disposable database wrapper exit 0 and cleanup complete. The frozen old source lock owner contends with the new entry; unrelated work proceeds. The old authority and snapshot readers decrypt and restore newly published bytes. - Pending and uploaded journal cases use the unchanged old persisted format and preserve operation/snapshot IDs through resume. - Full layered server source and test typecheck: **PASS**. - Module boundaries and `git diff --check`: **PASS**. - Actual source-mode module import under Node 24 succeeds without starting the application. - A separate temporary full-old-producer fixture did not get past module setup: 90 seconds under Vitest and a 15-second direct Node control. Neither is counted as an executed old-producer test or a product failure. The smaller frozen old-writer/reader proof and old-format journal cases are the local evidence. - Hosted resume of the actual saved operation and Linux artifact qualification remain operational acceptance gates. This PR does not claim those ran. - Review regression: a real successful import returned a null operation ID (genuine RED). The repaired entry binds the exact returned snapshot to its succeeded migration journal after commit (GREEN). Committed or superseded snapshots are accepted; a missing readback requires reconciliation rather than a blind retry. - Final affected check: **40/40 PASS** across the entry, packaging and actual PostgreSQL compatibility suites; wrapper exit 0 and cleanup complete. Full layered typecheck and boundaries pass after the user-requested rebase and current Runner rebuild. - A second real postcommit database-read fault also has genuine RED/GREEN proof. The succeeded journal, saved authority and original bytes remain intact, and every readback error now preserves the reconcile-required warning. - Final-head [CI run 42404116001](https://github.com/paperclipai/paperclip/actions/runs/42404116001): **PASS**, with all 59 check contexts terminal (55 success and 4 intentional skips). Fresh Greptile review is **5/5** on `c9345b72f27aee1158047dccd0b1060f9a0be6df`, with no unresolved threads; check and thread pagination are complete. ## Risks - The tool must come from one authenticated complete release tree with its own dependencies. A copied bundle chunk can retain old absolute paths and is not a portable release. - Concurrency is not resource admission. The operator must check memory, CPU and scratch headroom and start with one import or two cohort members. - Schema must be prepared by the existing schema owner. This entry never applies DDL. - Cancellation awaits the canonical owner. An SSH interruption does not prove the process stopped; durable journals must be reconciled before retry. - The entry preserves remote obligations and source files. It does not prove enrollment readiness or zero-downtime operation. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact serving snapshot and context-window size are not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Héctor Villarreal Ortega
|
576d6f6430 |
fix(ssh-restore): reset the local index to HEAD after importing remote git history (#16644)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents can run on a remote SSH environment. After the run, Paperclip
restores the workspace and imports the remote git history.
> - The import moves the checked-out branch with `git update-ref`.
`update-ref` does not change the index.
> - The index then still matches the old tip. `git status` in the
workspace shows the new files of the run as staged deletions.
> - An operator who commits in that workspace can remove the work of the
agent by mistake.
> - The sandbox restore already resets the index after the import
(#8595). The SSH restore does not.
> - This pull request calls the same `resetLocalGitIndexToHead` after
the SSH restore.
> - The benefit is a clean `git status` after each SSH run, the same as
after a sandbox run.
## Linked Issues or Issue Description
Refs #8595 (the same fix for the sandbox restore). Refs #15662, Refs
#16440 (other open changes to the same SSH restore code).
**What happened?**
After a remote SSH run imported a commit, `git status` in the local
workspace showed every new file of that commit as a staged deletion, and
every changed file as a staged modification. The files on disk were
correct.
**Expected behavior**
`git status` is clean after the import, because the working tree and
HEAD are the commit of the run.
**Steps to reproduce**
1. Use an opencode_local agent on an SSH environment with a git_worktree
project workspace.
2. Let the agent add a file and commit on the remote copy.
3. After the run, run `git status` in the local worktree. The new file
shows as `D` in the index.
**Paperclip version or commit**
paperclipai 2026.916.1 (also present on master
|
||
|
DottaandClaude Opus 5.5
|
0189fedffc |
fix(native): bound an idle warm local runner whose controller died (#19879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A warm runner (`lifecycleMode: warm`) keeps runnerd alive between runs so a follow-up turn starts fast. The follow-up default flip (next PR) makes warm the default for local Codex and OpenCode agents. > - The controller owns a warm runner's idle window. Runnerd ignores the idle timeout itself. > - A graceful controller shutdown suspends idle warm sessions. A controller that dies (SIGKILL, crash) does not. Local runners launch with no process reconnect grace, so the idle runner kept running until its 1-hour connection lease ran out, owned by nobody. > - This pull request gives warm local runners the 7-minute reconnect grace that remote runners already use. > - The benefit: idle runners are reaped by proof within a bounded time, never left orphaned, before warm becomes the default. ## Linked Issues or Issue Description Refs #19874 (warm completion contract), #19846 (TTFT baseline), #19852. **What existing behavior does this improve?** Ownership of an idle warm local runner after its controller dies. **Current behavior** `runnerProcessReconnectGraceMs` is `undefined` for local targets, so runnerd launches without `--reconnect-grace-ms`. An idle warm runner whose controller was killed stays alive until its connection lease (one hour) expires. The restart-chaos harness cannot show this: its K1 kills the controller's whole process tree, runner included. **Proposed behavior** `localRunnerProcessReconnectGraceMs(policy)` returns the remote grace (7 minutes) for a warm policy and `undefined` for per-turn. After the grace, runnerd records `transport_reconnect_grace_exceeded`, stops its provider, keeps durable state and exits. The next run resumes cold. **Reason and benefit** The restart plan requires idle runners to be reaped by proof, never orphaned. Seven minutes is far longer than a controller restart (13–20 s locally) and the long-outage fault holds (3–5 minutes), so a live turn still reattaches. ## What Changed - `native-runner-recovery-contract.ts`: `localRunnerProcessReconnectGraceMs`. - `native-session-executor.ts`: the local runnerd transport passes it as `runnerProcessReconnectGraceMs`. Remote targets keep `null` (lease-owned). - `runnerd-warm-idle-orphan.test.ts` (new): real runnerd and fake Codex. An idle warm runner, after its turn completed, whose controller detaches. - `doc/DEVELOPING.md`: the grace and its bound. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-warm-idle-orphan.test.ts`: with a process grace, the idle runner exits with `transport_reconnect_grace_exceeded` and the next transport resumes the same provider session cold. With no grace (`null`, today's local launch), the runner is still alive after the controller left. - `cd server && npx vitest run src/services/native-runtime/native-runner-recovery-contract.test.ts`: warm gets the remote grace; per-turn keeps none; the grace outlasts a controller restart and a long outage. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - Local restart measurement with warm runners (K1/K2, codex, claude, opencode, n = 3 each): every interrupted run recovered, with recovery latency equal or lower than per-turn. The chaos waves with this change are in the default-flip PR. ## Risks - A warm local runner whose controller stays down for more than 7 minutes exits. A turn in progress then resumes through `resume_dead_runner`, the same path as a controller crash that kills the runner. Remote runners already behave this way. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted state changes. A warm runner launched by the previous release keeps the launch arguments it started with (no grace) until it exits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Harjoth KharaandClaude Sonnet 5.5
|
d53fa727e1 |
docs: use a supported agent role in role examples (#16449)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every agent has a role, and the API only accepts the values in `AGENT_ROLES` (`z.enum(AGENT_ROLES)` in `packages/shared/src/validators/agent.ts`) > - The board-operator guide, the API docs and the agent-facing API reference are where operators and agents learn which role values to send > - They name `manager` as a role, but `manager` is not in `AGENT_ROLES`, so creating an agent with it is rejected with `invalid_value` > - This pull request replaces the unsupported value with supported ones in the guide and in the two places that show the same invalid value in an example > - The benefit is that the documented role values match what the API accepts ## Linked Issues or Issue Description - Fixes: #15687 ## What Changed - `docs/guides/board-operator/managing-agents.md`: the Role list now reads `ceo`, `cto`, `pm`, `engineer`, `researcher`, etc. (`pm` replaces the unsupported `manager`). The next bullet already says Reports to is the agent's manager in the org tree, so the hierarchy is still described and no extra wording was added. - `docs/api/agents.md` and `skills/paperclip/references/api-reference.md` (two examples): the same invalid value appeared as `"role": "manager"` in response examples for an "EngineeringLead" / "VP Engineering" agent. Changed to `cto`, which is a supported role and fits those examples. These are single-word changes made so the documented examples do not contradict the guide. - Left alone on purpose: ordinary uses of the word "manager" in prose (reports-to wording, AWS Secrets Manager), the "Worked Example: Manager Heartbeat" heading, the planning docs under `doc/plans/`, and a code test fixture (`ui/src/components/RoutineOverview.test.tsx`). - No code, validation or enum change; no new role. ## Verification - Before: the guide lists `manager`, and `AGENT_ROLES` (`packages/shared/src/constants.ts`) is `ceo, cto, cmo, cfo, security, engineer, designer, pm, qa, devops, researcher, general`, with no `manager`; the create validator uses `z.enum(AGENT_ROLES)`. - After: every role value named in the changed lines (`ceo`, `cto`, `pm`, `engineer`, `researcher`) is in `AGENT_ROLES`. `git diff --check` is clean. `pnpm-lock.yaml` is unchanged. - This is a docs-only change (3 files, 4 lines), so I did not run the test suite and did not add a test; there is no docs lint script that runs without installing dependencies. - To check by hand: send `POST /api/companies/{companyId}/agents` with `"role": "manager"` (rejected with `invalid_value`) and with `"role": "pm"` (passes role validation). ## Risks - Low. Documentation only, no runtime behavior change. - `cto` in the two example responses is a judgment call for the "EngineeringLead" / "VP Engineering" sample; any supported role would also be valid there. The two extra files are example-only and can be dropped if a maintainer prefers this PR to touch only the guide. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Implementation: Anthropic Claude Sonnet 5.5 (`claude-sonnet-5-5`), medium reasoning effort, with tool use and code execution through Claude Code. Context window not reported by the runtime. - Coordination and this description: Anthropic Claude Sonnet 5.5 (`claude-sonnet-5-5`). - Independent review of the exact head: OpenAI Codex `gpt-5.6-sol`, medium reasoning effort, read-only, no findings. - The issue was picked by an automated screening agent; a human contributor directed the work and owns the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ff53eb84db |
feat(recovery): add OwnerStamp and proveOwnerGone (gone | alive | unprovable) (#19853)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Restart recovery must decide when a dead process's in-progress record may be released > - About 31 sites decide "owner gone" today, with different rules for EPERM, PID reuse and other hosts (Phase 0 inventory, finding 1) > - The durable ownership plan (section A) asks for one owner stamp and one prover that returns `gone | alive | unprovable` > - This pull request adds that stamp and prover. Nothing calls it yet; the next PR uses it in a report-only reconciler > - The benefit is one tested definition of "owner gone" that later phases move each record kind onto ## Linked Issues or Issue Description Refs #19831 (plan, section A and Phase 1). Refs #19846 (Phase 0 inventory). Refs #19848 (Track 1: 1 s start-time gap and EPERM as alive). Stacked on #19851. **Problem** Each subsystem proves "owner gone" with its own rules. Some accept a dead PID on another host as proof. None returns "unprovable". **Expected behavior** One prover with one set of rules. A different host and an expired fence alone are never proof. "Unprovable" never releases a record. **Impact** This is the primitive that Phase 1's report-only reconciler and Phase 2's migrations use. ## What Changed - New `server/src/services/owner-proof.ts`: - `OwnerStamp`: a local stamp (`host`, `processSpace`, `bootId`, `pid`, `processStartedAt`, `processGroupId`, `generation`, `scope`) or a remote stamp (`fenceToken`, `generation`, `scope`, `provider`). - `proveOwnerGone(stamp, …)` returns `{ verdict: "gone" | "alive" | "unprovable", reason }`. - Local rules: this boot is alive; the stamp must name this host and PID space (host, kernel boot, PID namespace); the PID must be dead or reused (start-time gap of at least 1 s, via Track 1's `processStartsDefinitelyDiffer`); EPERM is alive; the process group must be empty or led by a newer process. - Remote rules: a superseded fence token or generation is gone; a provider stop proof is gone; a held fence is alive; a matching fence that only expired is unprovable. Process liveness is never consulted for a remote owner. - `currentLocalOwnerStamp()` reads only cached identity, so writing a stamp adds no OS call to the turn path. - `controller-process-identity.ts` exports `observedProcessStartedAt` and `UNKNOWN_PROCESS_SPACE` for the prover. - New `owner-proof.test.ts`: the proof matrix (EPERM, ESRCH, PID reuse inside and at 1 s, unreadable start time, a different host, a different or unknown PID space, process groups, a held, superseded, expired or provider-proved remote fence). ## Verification - `npx vitest run src/services/owner-proof.test.ts` (in `server/`): 20 tests pass. - `npx tsc --noEmit -p server`: no errors. ## Risks - Low risk. No caller uses the prover yet, so no behavior changes. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted shape changes. Stamps from older rows that lack fields return `unprovable` (`missing_identity`), which never releases a record. - Track 1 is still fixing proofs in place. The prover reuses Track 1's helpers (`processExists`, `processStartsDefinitelyDiffer`), so a later Track 1 rule change applies here too. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
965ca91dd0 |
Capture legacy workspace cohorts under one source fence (#19873)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Workspace migration preserves local files as authenticated portable
snapshots.
> - Workspaces that share Git storage require one complete source fence.
> - Independent imports contend on that fence and cannot safely run as a
batch.
> - This change adds an opt-in cohort owner that captures members
concurrently and publishes them atomically.
> - Existing source, recovery, snapshot and publication checks remain
authoritative.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The administrative remote-workspace migration owner for workspaces in
one shared physical source component. Related changes: #19688, #19854
and #19855.
**Current behavior**
Each independent import acquires the complete source fence. Parallel
independent calls therefore contend instead of overlapping archive work.
A caller cannot use a surrounding transaction and assume that a
successful savepoint confirms the outer commit.
**Proposed behavior**
An explicit `migrateRemoteWorkspaceCohort` call selects one to eight
leaves from one proven component. It owns the full
source/lifecycle/copyback fence, uses the original pool for bounded
archive work, validates every member, publishes serially, and returns
success only after the outer commit.
**Reason and benefit**
This supplies a correctness-gated path for a controlled two-member
hosted qualification. It makes no wall-clock speedup claim. No running
app call site or scheduler selects cohorts automatically.
**Breaking changes**
The existing single-import API retains its guards and rejects a journal
owned by a cohort. New cohort journals require the same complete member
set on resume.
## What Changed
- Extract the existing reservation, capture, validation and publication
stages for reuse by both owners.
- Add an opt-in cohort API with default concurrency two and a maximum of
eight. Every member must share the same complete physical/Git/alias
component; selected roots must be leaves relative to each other.
- Persist exact cohort and operation membership in the existing journal.
Compare all selected fingerprints with the aggregate proof before the
first reservation under the complete fence, and recheck each member when
reserving its journal. Refuse foreign journals, incomplete or mixed
replay sets, lost fences and changed sources. Keep all authorities and
activity events unpublished if the outer commit fails.
- Share the merged nested-child verifier inside one complete fence.
Queue its prepare calls, retain its two-child bound and authenticated
immutable proof, and recheck local bytes and database bindings on every
pass.
- Document the API, exact-membership resume protocol, resource bounds
and required hosted qualification.
## Verification
- Integration head `fce1896010`: 32/32 PostgreSQL/filesystem cases
across the cohort owner, the complete merged nested-verifier suite, and
five existing parent-import cases.
- Final source-proof repair: 19/19 cohort-owner cases pass, including
real first- and last-member metadata changes before reservation,
changed-source partial-resume refusal, mixed local/imported refusal with
storage forbidden, cancelled/failed/finished admission, a late finished
timestamp, complete fences, queued child-proof reuse, cold restore,
cancellation, commit failure and exact replay. All local database checks
use the canonical owned wrapper, one Vitest worker and Node 24.22.2 on
macOS arm64/PostgreSQL 18.1.
- Final rebased head is based on
`fbe689938c3bf8c5849d2cd56b9da90029db9389`, including current
source-owner, database-schema and Runner contract changes. The 19-case
owner suite, full server layered source/tests/test-tree typechecks,
module boundaries and diff checks are rerun on that base; required CI
supplies the native Runner qualification.
- Selection-only `node scripts/local-verify.mjs --base
|
||
|
Can H. Tartanoglu
|
410aa6d1b4 |
test(runner): resolve provider fixture signals through PATH (#14476)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its native runner tests verify exact provider shutdown and recovery. > - Four fixture call sites invoke `/bin/kill` directly. > - NixOS supplies `kill` through PATH, so those tests fail before checking process state. > - This pull request resolves `kill` through PATH, as the neighboring runner tests already do. > - The same stop, probe, and resume assertions can run on non-FHS hosts. ## Linked Issues or Issue Description **What happened?** Four Codex provider tests fail with `No such file or directory` on NixOS. Each failure comes from a test-owned `/bin/kill` invocation. The executable is installed on PATH. **Expected behavior** Tests should probe and signal their disposable provider processes with the installed `kill` command. **Steps to reproduce** On a NixOS host with `kill` on PATH and no `/bin/kill`, run the `codex_provider` Rust integration target. Three `durable_stop_*` cases and `structured_question_round_trips_through_the_normalized_backend` fail. All four failures were reproduced separately on unmodified `e9debd3eac328e6abe87ab89e9ff40ee2bb56b73`. Related: these failures also blocked the broader local qualification of #14474. This fixture fix has no dependency on that runtime change. ## What Changed - Replace four test-only `/bin/kill` executable paths with `kill`. - Retain every signal, PID/process-group operand, assertion, and unwind cleanup. - Match existing PATH lookup in this file and `tests/process_supervisor.rs`. ## Verification Rebased onto master `820c458465` (single commit `ad1fb1043`). - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --test codex_provider --locked -j4 -- --test-threads=2`: 94 passed, 0 failed, 2 ignored (subprocess helper entries that their parent tests run). The four tests that used `/bin/kill` pass on a host without `/bin/kill`. - No production code or documentation changes. - Hosted CI: `General tests (workspaces-b)` timed out in `packages/db/src/backup-lib.test.ts` and `e2e shard (1/8)` hit `ERR_STREAM_UNABLE_TO_PIPE`. Neither area is touched by this test-only diff. Master pushes currently run the "CI profile skipped" workflow, so there is no master run of those jobs to compare against. ## Risks Low risk. Tests require a working `kill` on their test PATH. Fixture PIDs and arguments stay unchanged. ## Model Used - Anthropic Claude Sonnet 5.5 (`claude-sonnet-5-5`) through Claude Code with tool use and code execution, for the rebase and verification. Original change: OpenAI `gpt-6-astra` through OpenCode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Devin Foley
|
fbe689938c |
docs(release): canonicalize stable notes for v2026.1106.0 (#19770)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The release process promotes a soaked beta to stable. The stable run publishes the curated notes from `releases/beta/v<beta-version>.md` on `master` as the GitHub Release body > - After the publish, the notes must move to `releases/v<stable-version>.md` on `master`. That path is the canonical location that the website changelog and the next release notes use > - Stable `v2026.1106.0` was published on 2026-11-06 from beta `2026.1023.0-beta.0`. The `canonicalize_stable_notes` job in run 42295041150 was skipped, so the workflow did not push this branch > - This pull request does by hand what that job does: it renames the beta notes file to the stable path. It changes no content > - The benefit is that `releases/v2026.1106.0.md` exists on `master` and is byte-identical to the published GitHub Release body ## Linked Issues or Issue Description Refs #17400 Refs #19607 **Issue type** Missing content **Where is the issue?** `releases/` on `master`. The stable notes for `v2026.1106.0` are only at the beta path `releases/beta/v2026.1023.0-beta.0.md`. The canonical path `releases/v2026.1106.0.md` does not exist. **What's wrong?** The GitHub Release https://github.com/paperclipai/paperclip/releases/tag/v2026.1106.0 is published, but the `canonicalize_stable_notes` job of the release run was skipped. The website changelog sync reads `releases/v2026.1106.0.md` on `master`, so the changelog page for this release cannot appear until this rename is merged. **Suggested fix** Rename the beta notes file to the stable path with no content change. This is the same step the workflow job performs. ## What Changed - `releases/beta/v2026.1023.0-beta.0.md` is renamed to `releases/v2026.1106.0.md`. Git reports a 100% rename - No line of the file changes. The blob is `16f05f22d4d71de3a568b493de2ddcb8eadb7859`, the same blob the release run pinned at `master` commit `da9a4cb9` when it created the GitHub Release - The branch has two commits (add the stable path, then remove the beta path) because it was written through the GitHub API. Squashed, it is one pure rename - This pull request does not change the release workflow ## Verification - `git diff --stat -M100% master...release-notes/v2026.1106.0-canonicalize` shows one renamed file with 0 insertions and 0 deletions - `git rev-parse release-notes/v2026.1106.0-canonicalize:releases/v2026.1106.0.md` prints `16f05f22d4d71de3a568b493de2ddcb8eadb7859` - `git rev-parse da9a4cb96c46945d693f0ddffdcb0e0743649a01:releases/beta/v2026.1023.0-beta.0.md` prints the same blob - `gh api repos/paperclipai/paperclip/releases/tags/v2026.1106.0 --jq .body` piped through `cmp` against the file (without the trailing newline that `--jq` adds) reports no difference. The release body and the file have the same bytes - After merge, https://github.com/paperclipai/paperclip/blob/master/releases/v2026.1106.0.md returns 200 and the old beta path returns 404 ## Risks - Low risk: one markdown file moves and no content changes - The release workflow's `canonicalize_stable_notes` job was skipped again, as on the `v2026.1023.0` release (#17400). `publish_stable` succeeded and the notes mode was `master_beta`. The job's `if` has no status function, so GitHub applies an implicit `success()`. That check is false when an upstream job in the chain was skipped (jobs inside `release_runner_binaries` were skipped in this run). A `!cancelled() &&` guard, as `publish_stable` already uses, fixes it. That fix is not in this pull request so this stays a pure rename ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5` (Claude Opus 5.5), tool use via Claude Code ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code)canary/v2026.1107.0-canary.19 |
||
|
DottaandClaude Opus 5.5
|
dcaada5f59 |
fix(native): carry each warm run's own completion contract (#19874)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A warm runner (`lifecycleMode: warm`) keeps runnerd and the provider
process alive between runs, so a follow-up message can start fast.
> - Each run has its own completion contract. The revision changes with
each follow-up, and the agent must name that revision when it finishes.
> - On a warm attachment the transport rebuilt the `run.attach` payload
from its constructor options. Those options belong to the first run.
Every later run kept the first run's contract.
> - The OpenCode proxy checks the finish claim locally against that
stale contract. Every warm OpenCode follow-up was refused ("Semantic
result completion contract revision does not match"), and the agent
looped until the run timed out.
> - This pull request carries each run's own contract into the warm
attachment, and lets runnerd adopt it without a cold provider resume.
> - The benefit: warm follow-ups finish correctly, and they keep the
live provider process, which is the reason to be warm.
## Linked Issues or Issue Description
Refs #19852 (startup sub-spans), #19846 (TTFT baseline).
**What existing behavior does this improve?**
Warm cross-run attachment for native Codex and OpenCode agents.
**Current behavior**
In a local measurement wave with `lifecycleMode: warm`, 4 of 4 OpenCode
follow-ups looped for 8+ minutes. Each finish call was refused with
"Semantic result completion contract revision does not match". The saved
`runAttachTemplate` after the attachment still had revision `1`, while
the follow-up turn named revision `2`. Codex hid the defect because its
finish claim goes through the controller's tool path, which uses the
current input.
**Proposed behavior**
The attachment carries the attaching run's contract. Runnerd updates the
contract in place for Codex and OpenCode and keeps the provider process.
The OpenCode proxy receives the current contract with the run identity
on `turn/start`.
**Reason and benefit**
Warm follow-ups are the largest TTFT lever in the baseline. A
correctness-only fix (cold-resume the provider when the contract
changes) would make every warm follow-up relaunch Codex or OpenCode,
because the revision changes on every follow-up.
## What Changed
- `attachRun` carries `completionContract` through `NativeSession`, the
harness backend, the Codex harness session and the runnerd transport.
`executeNativeSession` passes the current input's contract. The
transport uses it instead of its first-run `resumeCompletionContract`.
- Runnerd (`codex_provider.rs`, `attach_run_in_place`): a changed
contract no longer forces a cold resume for `codex` and `opencode`.
Codex app-server never receives the contract, and the OpenCode proxy now
receives it per turn. Other providers keep the cold-resume boundary. A
changed tool set still forces it.
- Runnerd sends `paperclipCompletionContract` with each OpenCode
`turn/start`, next to the existing `paperclipRunId`.
- The OpenCode proxy passes that contract to `attachRun` when the run
changes. `OpenCodeHarnessSession.attachRun` replaces its envelope's
contract and validates it.
## Verification
- `src/live/runnerd-codex-transport-warm-attachment.test.ts` (new case):
real runnerd and fake Codex. After a warm attachment with a new
contract, the saved `runAttachTemplate` has the new revision and the
Codex pid is unchanged.
- `src/cli/opencode-proxy-input.test.ts`: the real proxy bundle serves a
warm second run with revision `proxy-v2`. The second
`paperclip/runResult` is produced and names `proxy-v2`. Without the fix
the finish call is refused.
- Each test fails without its part of the fix: the proxy test without
the proxy change; the transport test without the transport change (stale
saved revision); and, with `PAPERCLIP_ATTACH_TRANSITION_RUNNER` pointed
at a runnerd built without the `codex_provider.rs` change, the Codex pid
changes.
- `cargo test -p paperclip-runner-core --lib warm` (11 tests),
`native-session-runtime.test.ts` (232), the OpenCode driver, backend and
warm-attachment suites, and `tsc --noEmit` for the runner package pass.
## Risks
- Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted shape changes. Saved `runAttachTemplate` values carry forward,
and the next attachment writes the current contract. A new runnerd with
an old OpenCode proxy behaves as today: the proxy ignores the new
parameter and keeps the stale contract. An old runnerd sends no
parameter, so a new proxy keeps its launch contract.
- In-place adoption widens the warm-retention path for Codex and
OpenCode to changed contracts. The idle, accounting and pending-work
checks still apply first.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
5a20fca9f7 |
fix(recovery): judge an orphaned run's pid only in the PID space that recorded it (#19871)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The orphan reaper (`terminalizeOrphanedRunningRun`) ends a `running` run whose recorded process is gone > - It judged "gone" with `kill(pid, 0)` on the local host, wherever the process actually ran: another host, another PID namespace, or a sandbox > - So a run whose process is alive elsewhere could be terminalized while it still works > - #19868 fixed the same gap for native coordinator takeover and the runtime-control sweep. This pull request finishes finding 2 for the reaper > - The benefit is that a dead local pid never proves an owner on another host is gone ## Linked Issues or Issue Description **Bug description** The Phase 0 ownership inventory (finding 2) names the orphan reaper `terminalizeOrphanedRunningRun` (`recovery/service.ts`) as a proof that accepts "the pid is dead here" for a process anywhere. `heartbeat_runs` recorded `process_pid` with no PID space, and a sandbox pid could collide with a local one. **Steps to reproduce** 1. A run records `process_pid` on host A, or in a sandbox. 2. On host B, that pid number is not alive. 3. B's stale-lock sweep terminalizes the run as `orphaned_running_run`. **Expected behavior** The process-death authority applies only in the PID space that recorded the pid. Issue-terminal authority is unchanged. Refs #19846, follows #19868 ## What Changed - Migration `0442`: add nullable `heartbeat_runs.process_host_hash`. - `heartbeat/run-lifecycle.ts` `persistHeartbeatRunProcessMetadata`, the one writer of `process_pid`, stores `currentProcessSpaceHash()` for a local pid and `"remote"` for a sandbox pid. - `recovery/service.ts`: the process-death authority runs only when `process_host_hash` is null (rows saved before this column, for one release) or `sameProcessSpace` matches. - Tests: the reaper leaves a dead pid alone for another host, a sandbox, and an unreadable space, and terminalizes it in this space. The writer stamps local and remote pids. ## Verification - `npx vitest run src/__tests__/recovery-stale-issue-lock-sweep.test.ts`: 46 passed. - Red before the fix: with the old `recovery/service.ts`, the three cross-space cases fail (the run is terminalized). Green after. - `pnpm db:generate` produced `0442_low_salo.sql` (one `ADD COLUMN`), and the snapshot set was pruned to five. `pnpm --filter @paperclipai/db check:migrations` passed. Server `tsc --noEmit` shows no errors in the changed files. ## Risks - Previous-release story: runs saved with `process_host_hash = null` keep the previous local proof for this release. That matches #19868, because stranding in-flight runs at upgrade would be a restart failure. New rows are strict. - A run whose process really lived on another host now stays `running` until issue-terminal authority, its controller lease, or another owner settles it. - No new I/O before `turn.start`. The hash is computed at module load, and the stamp rides the existing process-metadata update. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
00e627a23e |
test(logs): give the 4 MiB archive segment test a 60 s budget (#19877)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native run logs migrate into a bounded object-storage archive (#19719) > - One integration test migrates a 4 MiB log and reads it back across a segment boundary against embedded PostgreSQL > - On loaded CI runners it exceeds Vitest's 15 s default, so `General tests (server (9/12))` fails on unrelated PRs > - This pull request gives that one test an explicit 60 s budget > - The benefit is that unrelated PRs stop failing on a timing-only timeout, and the test keeps its exact-bytes assertions ## Linked Issues or Issue Description Refs #19719 (the test's origin). **Problem** `native-log-archive.integration.test.ts > preserves exact bytes across the bounded archive segment boundary` times out at 15 s on CI. On 2026-11-07 it failed on at least eight unrelated branches, among them `fix/skill-cache-lock-owner-proof`, `chat-eval/master-candidate`, `fix/subtasks-own-workspace` and `feat/owner-gone-prover` (three attempts). **Expected behavior** The test passes when the code is correct, regardless of runner load. **Impact** It blocks merges on unrelated PRs and wastes reruns. ## What Changed - Give that single test an explicit 60 s timeout, with a comment that names the reason. No assertion changes. ## Verification - `npx vitest run src/services/native-log-archive.integration.test.ts` (in `server/`): 15 tests pass locally. - The CI shard `General tests (server (9/12))` should pass on this PR. ## Risks - Low risk. This is a test-only timeout change. A real hang still fails, after 60 s. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
00ad9503fe |
feat(commentary): agents read only their own commentary unless granted commentary:view (#19842)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents can submit complaints and suggestions as commentary, and
people with commentary access read them on the Commentary page
> - When a person asks an agent in chat what agents have been
complaining about, the agent cannot read commentary and gets "Board
access required"
> - That response does not say where a person can see it, so the agent
can name the feed but cannot link it
> - Product rule: an agent cannot read other agents' commentary unless
it holds `commentary:view`, which is off by default; it may read its own
> - This pull request implements that rule and adds a remediation to the
denial naming the human page for the agent's own company
> - The benefit is a clear permission boundary and a working link for
the person instead of a dead end
## Linked Issues or Issue Description
**What happened?**
In the `activity.agent-feedback` chat eval, the agent correctly said it
could not read agent commentary and that only the person could open it,
but it gave no link, because the denial named no page.
**Expected behavior**
An agent denied a commentary read learns which page a person with
commentary access can open.
**Steps to reproduce**
1. As an agent, call `GET /api/companies/{companyId}/agent-commentary`.
2. Before this change, the 403 body is only `{ "error": "Board access
required" }`.
## What Changed
- `server/src/routes/agent-commentary.ts`:
- An agent with the `commentary:view` grant reads all commentary its
task access allows. The grant is not among the default agent grants.
- Without the grant, an agent may read its own entries: `?agentId=<its
own id>` on the list, or its own entry by ID.
- Without the grant, listing all commentary, filtering by another agent,
or reading another agent's entry returns 403 "Board access required".
The response's `remediation` names `/<issuePrefix>/activity/commentary`
for the agent's own company.
- A request for another company still fails on company access first,
with no page path.
- Board reads are unchanged: `commentary:view` for the user, plus task
read conditions.
- `agent-commentary.integration.test.ts`: one test covers the full-list
and other-agent denials with the page path, own-list and own-entry
reads, the cross-company denial, the grant allowing a full list, and
that `commentary:view` is not a default agent grant.
## Verification
- `npx vitest run src/__tests__/agent-commentary.integration.test.ts` in
`server`: 21 tests passed.
- The new test fails against the previous route.
- Negative control: with the previous route, the new test fails.
- Related commentary tests: `agent-commentary-reader`,
`agent-commentary-helper`, `commentary-permission-migration` passed.
`agent-commentary-skills` fails the same way with and without this
change (a missing skill file in a temporary directory locally).
- The chat eval rerun with this change is recorded in #19614.
## Risks
- Agents can now read their own commentary when they ask for it
explicitly; before, every agent read was denied.
- The denial's status code and message are unchanged; a `remediation`
string is added for same-company agents.
- There is no UI yet for granting `commentary:view` to an agent.
- An agent learns the company's issue prefix, which it already sees in
task identifiers.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1107.0-canary.18
|
||
|
DottaandClaude Opus 5.5
|
05e3309740 |
fix(accounting): settle a failed usage capture as unpriced at the closure bound (#19857)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every finished run settles its usage into the cost ledger. Until it settles, budget admission makes new company work wait > - When usage capture fails (`accountingCaptureFailed=true`), accounting skips the run forever. It is also not "recovering", so it counts in `waitingRunCount` > - So one failed capture holds new company work until the UTC budget month rolls over > - This pull request bounds that wait with the existing accounting closure timeout. It then settles the run's known usage as unpriced, with provenance that names the capture failure > - The benefit is that admission recovers by itself, while unknown spend stays visible and still counts under a blocking unpriced-usage policy ## Linked Issues or Issue Description **Bug description** The Phase 0 ownership inventory (finding 3) found that runs with `accountingCaptureFailed=true` are excluded from accounting (`run-cost-accounting.ts` `actionableRunAccounting` and `accountRunCost`). They are also excluded from "recovering" in `budget-exposure.ts`, so they count as waiting until the month window rolls over. Nothing else releases them. **Steps to reproduce** 1. Finish a run whose usage capture failed (for example, a fence write failure or usage after a source handoff). 2. Wait any length of time and run `reconcileRunCosts`. 3. The run stays `costAccountingPending`, and `getInvocationBlock` keeps holding new company work. **Expected behavior** After a bounded wait, the run settles with its known usage marked unpriced, and held work resumes. A blocking unpriced-usage policy still treats the run as unknown spend. Refs #19846 ## What Changed - `run-cost-accounting.ts`: new `captureFailureBoundPassed`. It applies to a terminal capture-failed run whose native coordinator (if any) settled, after `ACCOUNTING_CLOSURE_TIMEOUT_MS`. This is the same bound used for missing closure evidence. - `actionableRunAccounting` selects these runs, and `readCostAccountingWork` schedules a wake at their bound. A past bound never schedules a wake, as before. - `accountRunCost`: a capture-failed run is never treated as pre-provider work. Before the bound it stays pending. After the bound it settles through the existing lost-receipt path as `receiptUnavailable` + `closureTimedOut`: known cost becomes a lower-bound estimate, otherwise the charge is unpriced, and the run gets `accountingClosureTimedOut: true`. - Provenance evidence and the `cost.accounting_closure_timed_out` activity say that the usage capture failed (`details.captureFailed: true`). - Tests: three new cases in `closed-native-receipt-accounting.test.ts`. `cost-data-quality.test.ts` now pins both sides of the bound for a board retry. ## Verification - `npx vitest run src/__tests__/closed-native-receipt-accounting.test.ts`: 54 passed. - `npx vitest run src/services/run-cost-accounting.test.ts src/__tests__/idle-accounting-work.test.ts src/__tests__/cost-data-quality.test.ts`: 40 passed. - Red before the fix: with the old `run-cost-accounting.ts`, all 3 new capture-failure cases fail. The run never settles, nothing schedules a wake, and work stays held. Green after. - Hard stop: "keeps a settled capture failure under the blocking unpriced-usage policy" shows that a blocking policy still blocks after settlement. Real observed spend (`budgetExposure`) counts toward the hard stop before and after. - Server `tsc --noEmit` shows no errors in the changed files. ## Risks - Behavior change: capture-failed runs no longer stay pending forever. The 2026-10-28 Cursor runner plan recorded "capture failures remain pending" as an interim rule. This PR keeps them pending through the closure bound and then settles them as unknown spend, never as free. A complete receipt that arrives later still reconciles through late-receipt handling. - A board retry of a capture-failed run past the bound now settles it. Before the bound it is still refused. - Previous-release story: no persisted shape changes. Existing capture-failed rows past the bound settle on the next reconcile pass. That can release work that was held, and under a blocking policy it records unknown spend that blocks. - No new waits or I/O before `turn.start`. This is background accounting only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
4d5265fe7e |
Preserve fully covered historical workspace sources (#19872)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace migration must preserve local files while retaining unfinished recovery obligations. > - A shared Git source needs proof for its companion workspaces before import can publish. > - Recovery backfill can add a historical-source marker to an exhausted prepared-export hold. > - The hold owner rejected that marker even when every recorded allocation already belonged to its proven prepared source. > - This pull request accepts only that fully validated form through the existing source owners and publication fences. > - Unrelated work can continue while import saves the local files and keeps the remote recovery hold intact. ## Linked Issues or Issue Description Refs #19830. No separate public issue was found. A search for the marker found no open duplicate PR. **What happened?** A companion workspace with an exhausted prepared-export hold blocked the whole shared Git cohort after canonical recovery backfill added its historical-source marker. The prepared and historical source readers both accepted the same allocation. **Expected behavior** Local file preservation can proceed when all historical references are already covered by the proven prepared source. The recovery hold and remote obligations remain unchanged. Unknown or uncovered sources stay blocked. **Steps to reproduce** 1. Create a shared Git repository with a selected leaf and a companion workspace. 2. Give the companion a settled failed export, accepted result, exhausted recovery and blocked task. 3. Run canonical historical recovery backfill for its retained allocation. 4. Import the selected leaf through the normal online migration owner. **Paperclip version or commit** Reproduced on `4185e308659abf0887930e20049ca6e7cbfc0e81`. **Deployment mode** A source installation with legacy local Git workspaces and object storage. ## What Changed - Validate the exact historical marker schema, run and unique lease references against the already proven prepared source and declared allocations. - Reuse the canonical historical-source inspector and compare the same provider allocation and remote working directory. - Reject unknown fields, malformed markers and foreign, duplicate or uncovered references. - Keep the full marker in the existing hold fingerprint and retain the existing source, task, lifecycle, finalizer and publication checks. - Add real PostgreSQL/Git tests through shared-cohort inspection and the encrypted import/restore owners. ## Verification - The unchanged owner failed the actual selected-leaf importer regression with `online_workspace_source_scope_unproven`. The canonical prepared/historical reader controls passed before that refusal. - Production head `3e678e10fa` passed 41 focused PostgreSQL/Git cases across three files in 18.93 seconds after rebasing onto the new finalizer schema migration. This includes 14 new cases; 80 unrelated cases were filtered out. The final test-only review update passed both additional multi-allocation controls plus the original selected-leaf cold restore (3/3, 11.29 seconds); TypeScript passed again. - The positive uses the actual historical backfill producer. It cold-restores an encrypted selected-leaf snapshot with the original Git tree unavailable and keeps the peer's task, finalizer, result, allocation and recovery action unchanged. - The tests retain an unrelated active object-backed run, prove the peer finalizer lock is held, refuse bad bindings and reject late marker changes without discarding the uploaded snapshot. - Server TypeScript, module-boundary checks and `git diff --check` passed. The canonical PostgreSQL wrapper exited and cleaned up. - Final-head CI passed: 56 successful checks and four intentional skips, with no failures or pending jobs ([run](https://github.com/paperclipai/paperclip/actions/runs/42397090541)). Greptile reviewed `0cbf0a0255` at 5/5 with no open findings; GitHub reports CLEAN/MERGEABLE. ## Risks - The allowance applies only to the existing exhausted prepared-export hold. Other recovery forms and unknown or uncovered references remain blockers. - A marked hold adds database reads proportional to its already declared allocations during maintenance inspection. It adds no provider calls, payload download or recurring background work. - Import saves the current local files. It does not claim that unsaved remote files were recovered, that a provider stopped or that execution may resume. Existing holds and warnings remain authoritative. - Local tests use the real encryption and restore owners with a local object backend. Hosted backfill and normal-delivery qualification remain separate. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools and code execution. The exact serving snapshot and context-window limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
51c352ec2e |
fix(skills): release a runtime skill-cache lock whose holder provably exited (#19849)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a run starts, the server builds a read-only cache of each company skill under a cross-process file lock > - The lock records only a pid and a hostname, so a crash leaves a lock that nothing can release > - If the holder pid is dead on the same host, the next run fails and asks for manual cleanup. If the host name differs, the next run waits 60 s and then fails > - This is on the turn path, so one crash during a skill build breaks later runs that use that skill > - This pull request records the shared controller process identity in the lock and releases the lock at once when `controllerProcessExited` proves the holder is gone > - The benefit is that a crash during a skill build no longer blocks later runs, and the release never relies on lock age or another host ## Linked Issues or Issue Description **Bug description** The Phase 0 ownership inventory (finding 4) lists the runtime skill-cache lock (`runtime-skill-cache.ts`) as a record that nothing releases automatically. When a publisher crashes while it holds `<skillId>.lock`, every later preparation of that skill fails. **Steps to reproduce** 1. Start a skill cache build, then kill the server process while it holds the lock. 2. Restart and start a run that uses the same skill. 3. Preparation fails with "Runtime skill cache publisher exited; remove its stale publication lock before retrying". For a holder whose host name differs, it fails after 60 s with "publisher is busy". **Expected behavior** A lock whose holder has provably exited is released, and the build continues without waiting. A holder that cannot be proven gone is still waited for. Refs #19846 ## What Changed - `controller-process-identity.ts`: add `currentControllerProcessStamp()`. It returns this process's identity with no I/O, so the turn path gets no new wait. - `runtime-skill-cache.ts`: the lock owner file now also stores `controller` (boot id, pid, PID-space hash, start time if already known). `pid` and `host` stay so older readers still work. - New `runtimeSkillCacheLockHolderExited`: true only when `controllerProcessExited` proves the holder gone. Another host or PID namespace, an unknown space, a legacy lock without `controller`, a live pid, and this process all return false. - On proof, waiters race for one exclusive claim named by the stale holder's unique token (`<lock>.released-<token>`). Only the winner unlinks the lock, and only while it still has the stale inode. The claim is kept, so a late waiter does nothing. The waiter then retries the link right away instead of after the 20 ms poll. - A live holder is re-proved at most once a second. `controllerProcessExited` skips the start-time probe (`ps` on macOS) when no start time was recorded. - The busy error now names the lock path. The same-host "publisher exited" hard failure is removed. ## Verification - `npx vitest run src/__tests__/runtime-skill-cache.test.ts src/services/controller-process-identity.test.ts`: 28 passed, rebased on master after #19848. - New case "takes over a release claim whose claimer exited before removing the stale lock" covers the second Greptile finding. - New case "never removes a newer lock for a waiter that read a stale one late" covers the Greptile P1. - New case "releases a publication lock at once when its holder provably exited". On the old code it fails with "Runtime skill cache publisher exited; remove its stale publication lock before retrying". On the new code it builds in well under 5 s and leaves no lock. - New case "never treats age, another host, or a legacy holder as proof": a different host hash, a legacy `{pid, host}` lock, a live pid, this process, and malformed content all stay held. - Server `tsc --noEmit` shows no errors in the changed files. ## Risks - Previous-release story: a lock written by the previous release has no `controller` field. It is unprovable. The waiter waits 60 s and fails as before, and the error now names the path to remove. A new lock still carries `pid` and `host`, so a previous-release reader behaves as before. - Lock release race: while a stale lock exists, only the single claim winner can remove it, and the inode check cannot match a newer lock because the stale holder's owner file stays and pins that inode. A claim whose claimer crashed is itself released by the same proof (depth bounded at 2). - Each released lock leaves one small claim file and the dead holder's owner file next to it. - No new I/O before `turn.start`. The stamp is synchronous. The proof runs only when the lock is contended. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
738058553f |
fix(recovery): resume a continuation that never launched its runner in its own sandbox (#19850)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A chat follow-up on a remote native task runs as a continuation: it starts from its predecessor's saved provider session and, for a warm session, copies the predecessor's runner process identity onto the new run > - When the continuation fails before it asks for a runner of its own, its due same-run retry sees a recorded process identity and reattaches to "this run's" runner. That runner never existed, so reattachment always fails with `native_remote_recovery_lease_mismatch` and the task is blocked > - On staging, 34 OpenCode runs on 31 tasks failed this way in one evening. Each first attempt failed while staging its provider pack, recorded no runner start, and kept its own sandbox lease active > - This pull request proves that shape from durable state, retires the inherited identity, and resumes the run the way its failed attempt ran, in its own retained sandbox > - The benefit is that these retries recover automatically, without a second sandbox and without the unsaved-work hold, while every run that may hold its own provider work keeps verified reattachment ## Linked Issues or Issue Description Refs #19839, Refs #19399, Refs #19755, Refs #19779, Refs #19827 **What happened?** A remote OpenCode chat continuation failed during provider-pack staging. The run had no runner start event, no process identity event, no controller checkpoint, and no provider event of its own. Its saved session (no active turn, no pending request) and its process identity were inherited from the previous run. Its failure detail recorded `providerEventsExist: false`, and its own sandbox lease stayed active. The due same-run retry chose `reattach_remote_runner` because a process identity was recorded. Reattachment failed with `native_remote_recovery_lease_mismatch`, and the run ended in `native_continuation_requires_reconciliation`. **Steps to reproduce** 1. Continue a warm remote native session so the new run inherits the predecessor's runner process identity and saved session. 2. Fail the new run with a retryable error before it requests a runner (for example during provider-pack staging). 3. Let the same-run retry become due. Recovery requests `reattach_remote_runner`, and reattachment fails. **Expected behavior** The retry resumes the run the way its failed attempt ran, in the same sandbox, and continues the inherited provider session. A run with any provider work, runner start, or controller checkpoint of its own keeps verified reattachment. ## What Changed - New `server/src/services/native-runtime/native-inherited-runner-retry.ts`: - `retireInheritedRemoteRunnerIdentity` locks the issue, the coordinator, then the run (the same order as same-run resumption). It retires the recorded process identity only when all of these hold: the coordinator is a result-less, unowned `exact_checkpoint_resume` retry with `providerEventsExist: false`; the run has no runner start, process identity, adapter invocation, tool start, runner event, controller checkpoint, provider event or provider session of its own; the saved session has no active turn and no pending runtime request; and a settled predecessor of the same native session recorded the exact same process identity (PID, group, start time). - It records `nativeInheritedRunnerRetirement` on the run profile, with the run's own sandbox location when that lease is still active, and an `inherited_runner_never_launched` lifecycle event. - `readInheritedRunnerRetainedSandbox` returns that sandbox only while it still matches the run's saved remote location. - `heartbeat/recovery.ts`: a due remote retry calls the retirement first. It returns `retired`, `not_applicable` or `transient`. A retired run (including one a concurrent sweep already retired) goes to the ordinary same-run resume (`claimNativeSessionResumptions`), with every existing admission gate. A lock timeout or database error is `transient`: the sweep defers the run and checks again, and never falls back to reattachment. Any other run keeps verified reattachment. - `heartbeat/workspace-preparation.ts`: an ordinary same-run resume with a retired inherited runner acquires its retained sandbox through the existing exact-lease path (`reattachRemoteLease`), which validates and resumes that lease and never acquires a replacement. It acquires as an ordinary resume does only when the exact-lease failure proves the sandbox is unusable (`stage=environment_acquire|environment_resume`, `field=lease|lease_status|lease_expired|provider_lease_id`). Any other failure, such as a scope mismatch on a live lease, is kept and acquires nothing. - New `server/src/services/retained-sandbox-absent-error.ts`: a sandbox provider's resume reply with no provider lease and `expired: true` (the reply Daytona and other providers send when the sandbox is gone) now throws `RetainedSandboxAbsentError`. Every other mismatched reply keeps the existing error. The exact-lease reattachment in `environment-run-orchestrator.ts` maps it to `native_remote_recovery_lease_mismatch: stage=environment_resume field=lease_expired`, so the same-run resume above acquires as an ordinary resume when the provider has removed the retained sandbox. A transient resume failure is not treated as absence and never acquires another sandbox. - `native-restart-recovery.ts` exports the existing `hasNativeExecutionStartEvidence` so both paths use one definition of "execution reached a runner". - `doc/execution-semantics.md` documents the rule. ## Previous-release state - No schema change. The new profile field is additive. - A run saved by an older release without `providerEventsExist`, without a native session ID, or without a matching predecessor keeps verified reattachment. That path is unchanged. When it loses its runner, it still requests the user's Retry once (#19399). - A run that an older release left with its own runner start, controller checkpoint or provider event keeps verified reattachment, because its outcome may be ambiguous. - A profile without the new marker never changes acquisition. ## Verification - `npx vitest run server/src/services/heartbeat/recovery.test.ts -t "never launched its own runner|keeps verified reattachment"` (embedded PostgreSQL): - Staging shape (inherited session and process identity, no runner start of its own, saved remote workspace, active own lease, recorded PID). Before the fix: fails, because recovery requested reattachment. After the fix: passes. The PID fields are cleared, the marker names the predecessor and the retained lease, `executeRun` is called once with a plain `nativeLeaseOwner` claim, the run still has exactly one (active) lease, and no process is signalled. - Negatives, passing before and after: a provider turn of its own, a runner start of its own, a controller checkpoint of its own, no matching predecessor, a pending runtime request in the saved session, and a failure detail without `providerEventsExist`. Each keeps verified reattachment and gets no marker. - `npx vitest run server/src/services/heartbeat/workspace-preparation.test.ts -t "retained sandbox|retired inherited"`: - A retired inherited runner acquires its retained sandbox with `reattachRemoteLease`. Before: fails. After: passes. - When the retained sandbox is gone, it acquires as an ordinary resume. Before: fails. After: passes. - Without the marker, acquisition is unchanged (passes before and after). - A `nativeWorkspaceSync` reference also resumes its retained sandbox. A live lease's `lease_scope` failure is kept and acquires nothing. - `recovery.test.ts` also covers a second retirement call (returns `retired`, one event) and a held issue lock (defers; no reattachment, no dispatch, PID kept). - `environment-runtime.test.ts`: on exact resume, only a provider reply with no lease and `expired: true` (with or without a workspace sentinel mismatch) is `RetainedSandboxAbsentError`; a different lease or a reply without `expired` keeps the existing error. `environment-run-orchestrator.test.ts`: confirmed absence becomes `stage=environment_resume field=lease_expired` and acquires nothing. - Affected suites pass locally: `recovery.test.ts`, `workspace-preparation.test.ts`, `heartbeat-native-remote-failure-retention.test.ts`, `workspace-recovery-retry-copyback.test.ts`, `native-runner-restart-recovery.integration.test.ts`. `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - **No second executor.** The rule applies only when this run never requested a runner, so no runner of this run exists. The predecessor must be settled, and its identity is matched exactly. The resume then takes the normal same-run claim, which holds the issue lock and admission gates. - **No repeated side effects.** The run has no provider event, turn or tool start of its own. A saved session with a pending runtime request or an active turn keeps verified reattachment. The resumed provider session goes through the existing provider recovery, which refuses a pending side-effecting call with a permanent, named code. - **Bounded.** The retirement takes effect once per run, because afterwards the run has no recorded identity. A contended retirement waits for the next sweep. Further failures use the existing three-attempt provider budget. - I did not run a live Daytona sandbox, or a full preparation through a real sandbox plugin, through this path. The orchestrator's exact-lease checks are covered by its own unit tests. The evidence is recovery and preparation tests on a real database with the staging run shape. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file editing, read-only browser access to staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
de82f93443 |
fix(storage): retry S3 reads that the provider times out with HTTP 408 (#19039)
Mark HTTP 408 on GET/HEAD as retryable inside the SDK retry middleware so the standard backoff, retry quota and three-attempt limit apply. Writes are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
68db7ed9e3 |
fix(recovery): never accept a dead local pid as proof for an owner on another host (#19868)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, recovery decides that an in-progress record's owner is gone and takes it over > - Two of those proofs use `kill(pid, 0)` on the local host. A pid that is dead here says nothing about a process on another host or in another PID namespace > - So a controller or runtime-control owner that is still alive elsewhere can be taken over while it still runs > - This pull request records the owner's PID space (the existing `controller-process-identity` host hash) and judges a pid only in that same space > - The benefit is that a different host is never proof. Those records follow their existing lease or TTL instead ## Linked Issues or Issue Description **Bug description** The Phase 0 ownership inventory (finding 2) found three proofs that accept "the pid is dead here" as proof for an owner on another host: `evaluateNativeControllerTakeover` (`native_run_finalizations` has no host column), the runtime-control sweep (`workspace-operations.ts`), and the orphan reaper `terminalizeOrphanedRunningRun`. Only `controllerProcessExited` checked the host. **Steps to reproduce** 1. Run two control-plane processes on different hosts, or in different PID namespaces, against one database. 2. Let host A own a native finalization lease or a runtime-control operation. 3. On host B, run restart recovery. If A's pid number is not alive on B, B takes over the live owner. **Expected behavior** An owner whose PID space differs from this process's, or cannot be read, is unprovable. Its record waits for its lease, fence, or TTL. Refs #19846 ## What Changed - Migration `0441`: add nullable `native_run_finalizations.controller_host_hash`. - `controller-process-identity.ts`: export `currentProcessSpaceHash()` and `sameProcessSpace()`. An unreadable space is stored as `"unknown"`, which never matches. - `native-restart-recovery.ts`: `currentNativeControllerIdentity` carries `hostHash`, which both coordinator claim writes (`native-restart-recovery.ts`, `native-session-executor.ts`) store. `evaluateNativeControllerTakeover` returns `controller_host_unprovable` before any pid probe when the recorded space is not this one. `native-dead-controller-result-lease.ts` selects the column. - `workspace-operations.ts`: runtime-control owner stamps record `hostHash`. `ownerProcessProvenGone` requires the same PID space (EPERM = alive), otherwise the existing staleness TTL applies. - Not in this PR: the orphan reaper `terminalizeOrphanedRunningRun`. `heartbeat_runs` has no host for `process_pid`, and its writers are spread across legacy and native paths. It follows as a separate PR with its own column. ## Verification - `npx vitest run src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/workspace-runtime-control-recovery.test.ts`: 55 passed. New cases cover another host and an unreadable space (dead and recycled pid) for takeover, plus another host, unreadable space, and legacy stamp for the sweep. The sweep cases also confirm that the TTL still reclaims after the heartbeat lapses. - Red before the fix: the 5 new cases fail on the old code. Green after. - `native-controller-recovery.integration.test.ts`, `native-failed-run-owner-release.test.ts`, `native-startup-cleanup-recovery.test.ts`: 71 passed. - `pnpm db:generate` produced `0441_lucky_chameleon.sql` (one `ADD COLUMN`) and pruned to five snapshots. `pnpm --filter @paperclipai/db check:migrations` passed. Server `tsc --noEmit` shows no errors in the changed files. ## Risks - Previous-release story: `native_run_finalizations` rows saved before this column have `controller_host_hash = null`. They keep the previous local dead-pid proof for this release. Making them unprovable would leave every native run in flight at the upgrade in `awaiting_evidence` with no TTL, because local takeovers have no lease fallback. Rows written by this release onward are strict. Runtime-control stamps without `hostHash` are strict now, because that path has a staleness TTL. - Owners built from other receipts (workspace finalization owner, workspace stop proof) pass no host and keep the previous behavior. Track 2's shared `OwnerStamp` should absorb them. - A new row on a host that really moved (redeploy onto a new machine) waits for its lease or an operator instead of being taken over by pid. That is the intended "different host is never proof" rule. - No new I/O before `turn.start`. The hash is computed once at module load. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
4185e30865 |
Recover archived commits from original bare sources (#19855)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace migration must preserve saved files before the web server releases its local volume. > - An archived Git checkout can retain a trusted branch-deletion receipt after its private clone and generated project cache are removed. > - The original repository can still exist at its recorded file URL, but the archive owner did not check that source. > - This pull request permits a narrow, verified original bare-repository source for the exact recorded commit. > - The existing encrypted snapshot and publication owners then preserve the workspace through their normal migration path. ## Linked Issues or Issue Description No public issue exists. Searches for the archived-source error and original bare Git source found no duplicate change. **What happened?** Migration reported `archived_git_source_unavailable` for an archived checkout even though its original bare repository still contained the exact saved commit. The private clone and generated cache were absent. **Expected behavior** The canonical owner can use the recorded original local repository when its full commit identity and source ownership are proven. Uncertain or changed sources remain blocked. **Steps to reproduce** 1. Create an isolated checkout from a local bare repository. 2. Close it through the normal cleanup owner and retain the successful branch-deletion receipt. 3. Remove the private clone and generated cache, while retaining the recorded original repository and full base revision. 4. Inspect the archived workspace for migration. **Paperclip version or commit** Reproduced on `84908d176b2800f4c792c4249c610ec0baebdd28`. **Deployment mode** A source installation with existing local Git workspaces and shared object storage. ## What Changed - Consider the exact configured file-URL repository only after the canonical primary source is definitively absent. - Require a matching source record and successful cleanup receipt, the exact full base/deleted commit, a contained bare SHA-1 repository, bounded standard configuration and alias-free metadata. - Restrict this source path to local Git transport and a minimal environment. Reconstruct the tree only in private migration scratch. - Retain existing source fingerprints, journal identity, encrypted upload/readback and final publication checks. Lock and recheck the configured source and cleanup record before publication. - Add real PostgreSQL/Git coverage for encrypted cold restore, lost-reply retry and refused source or publication changes. ## Verification - The unchanged owner failed the real four-object original-source regression with an ineligible inventory. - The review regressions failed before each repair: four returned-primary retry/publication cases and two actual production metadata/reaper cases. - Final head `8925519310` passed all 40 focused PostgreSQL/Git tests in 56.38 seconds. Four positive formats use canonical/legacy metadata and ordinary/verified-tip cleanup. Each imports an encrypted snapshot, removes the original file URI from reach, then authenticates and cold-restores exact files, executable mode and Git history. - The same file covers source/receipt/writer guards, changed sources before publication, authenticated lost-reply reuse and 18 returned-primary directory/alias/unreadable cases. Refused publication retains returned edits and uploaded ciphertext. - Server TypeScript, module-boundary checks and `git diff --check` passed on this revision. Canonical test wrappers exited and cleaned up. - An earlier focused run also passed 22 existing archived-Git/restore-point controls. `pnpm verify --explain` completed without executing extra suites. Required CI supplies final broad coverage. - Final head passed all 60 CI/review contexts: 56 successful checks and four expected skips. Greptile reviewed this exact head at 5/5, with no open review threads. - One unchanged archive segment-boundary test timed out on both framework attempts in the initial server shard. The one job-only retry passed with unchanged code and timeout. The test and its resolved local import graph match a green control revision. The original failures remain retained; the cause is unresolved. ## Risks - The fallback is intentionally narrow. Original repositories with a different full tip, unsupported configuration, aliases (including an aliased instance root), external object storage or more than 10,000 metadata entries remain blocked. - The first source check and final publication check add bounded local filesystem and Git work to this maintenance path. They add no recurring worker or network Git access. - Original repository files remain intact. Importing a retained workspace does not make its configured local file URL portable for future fresh project work. That separate configuration dependency still requires preservation and qualification before volume removal. - The local regression uses the real encryption and restore owners with a local object-storage backend. It does not claim a hosted backfill or a deployed migration. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools and code execution. The exact serving snapshot and context-window limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1107.0-canary.17 |
||
|
DottaandClaude Opus 5.5
|
a9ac2e2039 |
feat(native): split local runner startup into stage, launch and session-load spans (#19852)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a person messages an agent, the agent must start its turn fast. Time to first token (TTFT) and agent boot are first-class metrics. > - The Phase 0 baseline (#19846) found that local runner startup costs 1.6–17 s at p50 and up to 50 s at p95. It is the largest part of TTFT. > - The run log records that startup as one span (`runner.session.bootstrap` or `runner.session.resume`). We cannot see if the time goes to staging, to the `paperclip-runnerd` process launch, or to the provider session load. > - This pull request splits local runner startup into three sub-spans in the run log. > - The benefit: the follow-up work to cut follow-up-turn TTFT can measure each step before and after. ## Linked Issues or Issue Description Refs #19846 (TTFT baseline, finding 1 and gap "No sub-spans exist inside local runner startup"). Refs #19838. **What existing behavior does this improve?** The run-log performance spans for local native runner startup. **Current behavior** The local path records one `runner.session.bootstrap` or `runner.session.resume` span that covers runtime staging, the runner process launch and the provider handshake together. The remote path already records `runner.runtime.stage` and `runner.process.launch`; the local path does not. **Proposed behavior** The local path records three sequential child spans of `runner.session.startup`: `runner.runtime.stage`, `runner.process.launch` and `provider.session.load`. **Reason and benefit** The TTFT work needs to know which step dominates before it changes the runner lifecycle. Without these spans, a change to warm runner reuse cannot show which cost it removes. ## What Changed - `RunnerdCodexTransport` has a new optional `onStartupStep` callback. `#start` and `#resume` time three milestones and call it synchronously. A failing callback never changes startup. A nested startup (exact adoption from `thread/start`) keeps the outer clock. - `runner.runtime.stage`: launch preparation before the runner process exists. - `runner.process.launch`: spawn until runnerd accepts the run (`run.prepare`, or the recovery attachment on resume). An adopted live runner records no launch. This keeps the existing meaning in `summarizeStartupReuse`: a launch span means a fresh runner. - `provider.session.load`: `session.open` (or the recovery probe) until the provider identity is confirmed. - `recordNativeRunnerStartupStep` (`native-run-trace.ts`) records each step as a `run.performance.span` row with `mode` and `target: "local"`. It uses `trace.defer`, so the run-log write never delays `turn.start`. - The native executor passes the callback only for a local target. Remote launchers keep their own spans, so there are no duplicates. - `redaction.ts`: allowlist `runner.process.launch` and `provider.session.load` as native span names. - `doc/run-log-events.md`: new section "Local runner startup sub-steps". Data path (AGENTS.md §5.7): run log only, plus Observability spans when an operator sets an OTLP endpoint. The attributes (`mode`, `target`, `outcome`) are already in the closed span-attribute allowlist. No Telemetry change. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-startup-steps.test.ts`: a real runnerd and the fake Codex fixture. Bootstrap and a cold resume each report stage, launch and session load in order, with `ok` outcomes and no overlap. A throwing sink does not change startup. - `cd server && npx vitest run src/services/native-runtime/native-run-trace.test.ts src/__tests__/redaction.test.ts`: the rows have the correct parent, offsets, durations and failed level. A stalled run-log sink does not block the caller. The rows settle before `task.provider_session.measured`. The names survive redaction. - The existing `runnerd-codex-transport*` suites pass (412 tests). Under host load (load average 16–27), the first full run had 7 timing failures in suites that do not set the callback. Each failing file passed when I ran it again alone. `runnerd-codex-transport-settlement` passed 3/3 for the failing case alone. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - Low risk. Without `onStartupStep`, the transport behaves as before: the clock is not created and each milestone call is an optional-chain no-op. - More run-log rows: three per local runner startup. - The step boundaries are controller-side observations. `runner.process.launch` includes the runnerd authentication and the `run.prepare` acknowledgement, not only the OS spawn. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e8a418cc4d |
fix(workspaces): settle an uncertain remote provisioning row once its sandbox is stopped (#19862)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote task workspace runs the project's provisioning commands in a sandbox. Each command has a `remote_workspace_provision` journal row, because an arbitrary command cannot be replayed after an ambiguous reply > - A row left `running` with `uncertain: true` blocks every later reservation of that workspace, and nothing ever settles it > - A crash or lost reply during provisioning therefore leaves the task's workspace unusable, with no next action > - This pull request settles such a row only when the earlier run's sandbox is provably stopped. Otherwise the 409 names the row, its run, and the operator's action > - The benefit is that a crashed provisioning attempt no longer locks the workspace forever, and the effect is never assumed to have succeeded ## Linked Issues or Issue Description **Bug description** The Phase 0 ownership inventory (finding 4) lists `remote_workspace_provision` rows left `running`/`uncertain` (`remote-workspaces.ts`) as a record with no automatic release. `reserveRemoteWorkspace` rejects every other run with `remote_workspace_provision_unverified`. A command that simply exited non-zero also stayed `uncertain`. **Steps to reproduce** 1. Start remote provisioning and lose the runner's reply (or have the command exit non-zero). 2. End the run and stop its sandbox. 3. Start another run on the task. Reservation fails with `remote_workspace_provision_unverified`, and the next attempt fails the same way. **Expected behavior** After the earlier sandbox is provably stopped, the row settles and the workspace can be reserved again. While that cannot be proven, the error says what to do. Refs #19846 ## What Changed - `remote-workspaces.ts` `reserveRemoteWorkspace`: for each uncertain provision row from another run, `provisionOwnerProvenStopped` requires two things. The row's run is terminal, and every environment lease of that run carries a provider stop or destroy receipt (`hasRemoteTerminationReceipt` or `hasRecoveredCleanupStopReceipt`). A proven row is settled as `failed` (`uncertain: false`, `settledBy: "owner_sandbox_stopped"`) and logged as `workspace.remote_provision_settled`. It is never marked succeeded. - If a later reservation check (for example a writer hold) rejects the attempt, the proven rows are re-proved and settled in their own transaction, so they do not block again. - An unprovable row still blocks. The 409 now carries `operationId` and `runId` and names the action: stop the earlier sandbox, or set the task's Workspace to "New isolated workspace". That picker option already exists in `IssueProperties`. - `provisionRemoteWorkspace`: a reported non-zero exit (not a timeout) is recorded as `failed` with its `exitCode` instead of staying uncertain. ## Verification - `npx vitest run src/services/remote-workspaces.integration.test.ts src/services/remote-workspace-save-concurrency.test.ts`: 126 passed. - New case "settles an uncertain provisioning command once its run's sandbox is provably stopped". A failed run alone still gets the 409 with `operationId`/`runId`. After a stop receipt, the row settles, and the activity is logged once. - New case "records a provisioning command that reported a failing exit as finished, not uncertain". - Red before the fix: with the old `remote-workspaces.ts`, both new cases fail. Green after. - Server `tsc --noEmit` shows no errors in `remote-workspaces.ts`. - Not run: a live browser check of the "New isolated workspace" path for this error. I confirmed in code that it reserves a different workspace, so it bypasses this check. ## Risks - A settled row means "effect unknown". The next writer provisions again in its own epoch, as every new writer already does. A partially applied, non-idempotent command could see its partial effect. That is the same exposure as any failed provisioning, and it happens only after the earlier sandbox has stopped, so the two commands never run at the same time. - Previous-release story: no persisted shape changes. Uncertain rows saved by the previous release are evaluated by the same proof on the next reservation. - No new waits before `turn.start`. The proof runs only when an uncertain row exists, inside the existing reservation transaction. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |