mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 14:19:00 +01:00
0189fedffc569614299c20dd82693be4079c5bb9
628
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
0189fedffc |
fix(native): bound an idle warm local runner whose controller died (#19879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A warm runner (`lifecycleMode: warm`) keeps runnerd alive between runs so a follow-up turn starts fast. The follow-up default flip (next PR) makes warm the default for local Codex and OpenCode agents. > - The controller owns a warm runner's idle window. Runnerd ignores the idle timeout itself. > - A graceful controller shutdown suspends idle warm sessions. A controller that dies (SIGKILL, crash) does not. Local runners launch with no process reconnect grace, so the idle runner kept running until its 1-hour connection lease ran out, owned by nobody. > - This pull request gives warm local runners the 7-minute reconnect grace that remote runners already use. > - The benefit: idle runners are reaped by proof within a bounded time, never left orphaned, before warm becomes the default. ## Linked Issues or Issue Description Refs #19874 (warm completion contract), #19846 (TTFT baseline), #19852. **What existing behavior does this improve?** Ownership of an idle warm local runner after its controller dies. **Current behavior** `runnerProcessReconnectGraceMs` is `undefined` for local targets, so runnerd launches without `--reconnect-grace-ms`. An idle warm runner whose controller was killed stays alive until its connection lease (one hour) expires. The restart-chaos harness cannot show this: its K1 kills the controller's whole process tree, runner included. **Proposed behavior** `localRunnerProcessReconnectGraceMs(policy)` returns the remote grace (7 minutes) for a warm policy and `undefined` for per-turn. After the grace, runnerd records `transport_reconnect_grace_exceeded`, stops its provider, keeps durable state and exits. The next run resumes cold. **Reason and benefit** The restart plan requires idle runners to be reaped by proof, never orphaned. Seven minutes is far longer than a controller restart (13–20 s locally) and the long-outage fault holds (3–5 minutes), so a live turn still reattaches. ## What Changed - `native-runner-recovery-contract.ts`: `localRunnerProcessReconnectGraceMs`. - `native-session-executor.ts`: the local runnerd transport passes it as `runnerProcessReconnectGraceMs`. Remote targets keep `null` (lease-owned). - `runnerd-warm-idle-orphan.test.ts` (new): real runnerd and fake Codex. An idle warm runner, after its turn completed, whose controller detaches. - `doc/DEVELOPING.md`: the grace and its bound. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-warm-idle-orphan.test.ts`: with a process grace, the idle runner exits with `transport_reconnect_grace_exceeded` and the next transport resumes the same provider session cold. With no grace (`null`, today's local launch), the runner is still alive after the controller left. - `cd server && npx vitest run src/services/native-runtime/native-runner-recovery-contract.test.ts`: warm gets the remote grace; per-turn keeps none; the grace outlasts a controller restart and a long outage. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - Local restart measurement with warm runners (K1/K2, codex, claude, opencode, n = 3 each): every interrupted run recovered, with recovery latency equal or lower than per-turn. The chaos waves with this change are in the default-flip PR. ## Risks - A warm local runner whose controller stays down for more than 7 minutes exits. A turn in progress then resumes through `resume_dead_runner`, the same path as a controller crash that kills the runner. Remote runners already behave this way. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted state changes. A warm runner launched by the previous release keeps the launch arguments it started with (no grace) until it exits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
00ad9503fe |
feat(commentary): agents read only their own commentary unless granted commentary:view (#19842)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents can submit complaints and suggestions as commentary, and
people with commentary access read them on the Commentary page
> - When a person asks an agent in chat what agents have been
complaining about, the agent cannot read commentary and gets "Board
access required"
> - That response does not say where a person can see it, so the agent
can name the feed but cannot link it
> - Product rule: an agent cannot read other agents' commentary unless
it holds `commentary:view`, which is off by default; it may read its own
> - This pull request implements that rule and adds a remediation to the
denial naming the human page for the agent's own company
> - The benefit is a clear permission boundary and a working link for
the person instead of a dead end
## Linked Issues or Issue Description
**What happened?**
In the `activity.agent-feedback` chat eval, the agent correctly said it
could not read agent commentary and that only the person could open it,
but it gave no link, because the denial named no page.
**Expected behavior**
An agent denied a commentary read learns which page a person with
commentary access can open.
**Steps to reproduce**
1. As an agent, call `GET /api/companies/{companyId}/agent-commentary`.
2. Before this change, the 403 body is only `{ "error": "Board access
required" }`.
## What Changed
- `server/src/routes/agent-commentary.ts`:
- An agent with the `commentary:view` grant reads all commentary its
task access allows. The grant is not among the default agent grants.
- Without the grant, an agent may read its own entries: `?agentId=<its
own id>` on the list, or its own entry by ID.
- Without the grant, listing all commentary, filtering by another agent,
or reading another agent's entry returns 403 "Board access required".
The response's `remediation` names `/<issuePrefix>/activity/commentary`
for the agent's own company.
- A request for another company still fails on company access first,
with no page path.
- Board reads are unchanged: `commentary:view` for the user, plus task
read conditions.
- `agent-commentary.integration.test.ts`: one test covers the full-list
and other-agent denials with the page path, own-list and own-entry
reads, the cross-company denial, the grant allowing a full list, and
that `commentary:view` is not a default agent grant.
## Verification
- `npx vitest run src/__tests__/agent-commentary.integration.test.ts` in
`server`: 21 tests passed.
- The new test fails against the previous route.
- Negative control: with the previous route, the new test fails.
- Related commentary tests: `agent-commentary-reader`,
`agent-commentary-helper`, `commentary-permission-migration` passed.
`agent-commentary-skills` fails the same way with and without this
change (a missing skill file in a temporary directory locally).
- The chat eval rerun with this change is recorded in #19614.
## Risks
- Agents can now read their own commentary when they ask for it
explicitly; before, every agent read was denied.
- The denial's status code and message are unchanged; a `remediation`
string is added for same-company agents.
- There is no UI yet for granting `commentary:view` to an agent.
- An agent learns the company's issue prefix, which it already sees in
task identifiers.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
738058553f |
fix(recovery): resume a continuation that never launched its runner in its own sandbox (#19850)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A chat follow-up on a remote native task runs as a continuation: it starts from its predecessor's saved provider session and, for a warm session, copies the predecessor's runner process identity onto the new run > - When the continuation fails before it asks for a runner of its own, its due same-run retry sees a recorded process identity and reattaches to "this run's" runner. That runner never existed, so reattachment always fails with `native_remote_recovery_lease_mismatch` and the task is blocked > - On staging, 34 OpenCode runs on 31 tasks failed this way in one evening. Each first attempt failed while staging its provider pack, recorded no runner start, and kept its own sandbox lease active > - This pull request proves that shape from durable state, retires the inherited identity, and resumes the run the way its failed attempt ran, in its own retained sandbox > - The benefit is that these retries recover automatically, without a second sandbox and without the unsaved-work hold, while every run that may hold its own provider work keeps verified reattachment ## Linked Issues or Issue Description Refs #19839, Refs #19399, Refs #19755, Refs #19779, Refs #19827 **What happened?** A remote OpenCode chat continuation failed during provider-pack staging. The run had no runner start event, no process identity event, no controller checkpoint, and no provider event of its own. Its saved session (no active turn, no pending request) and its process identity were inherited from the previous run. Its failure detail recorded `providerEventsExist: false`, and its own sandbox lease stayed active. The due same-run retry chose `reattach_remote_runner` because a process identity was recorded. Reattachment failed with `native_remote_recovery_lease_mismatch`, and the run ended in `native_continuation_requires_reconciliation`. **Steps to reproduce** 1. Continue a warm remote native session so the new run inherits the predecessor's runner process identity and saved session. 2. Fail the new run with a retryable error before it requests a runner (for example during provider-pack staging). 3. Let the same-run retry become due. Recovery requests `reattach_remote_runner`, and reattachment fails. **Expected behavior** The retry resumes the run the way its failed attempt ran, in the same sandbox, and continues the inherited provider session. A run with any provider work, runner start, or controller checkpoint of its own keeps verified reattachment. ## What Changed - New `server/src/services/native-runtime/native-inherited-runner-retry.ts`: - `retireInheritedRemoteRunnerIdentity` locks the issue, the coordinator, then the run (the same order as same-run resumption). It retires the recorded process identity only when all of these hold: the coordinator is a result-less, unowned `exact_checkpoint_resume` retry with `providerEventsExist: false`; the run has no runner start, process identity, adapter invocation, tool start, runner event, controller checkpoint, provider event or provider session of its own; the saved session has no active turn and no pending runtime request; and a settled predecessor of the same native session recorded the exact same process identity (PID, group, start time). - It records `nativeInheritedRunnerRetirement` on the run profile, with the run's own sandbox location when that lease is still active, and an `inherited_runner_never_launched` lifecycle event. - `readInheritedRunnerRetainedSandbox` returns that sandbox only while it still matches the run's saved remote location. - `heartbeat/recovery.ts`: a due remote retry calls the retirement first. It returns `retired`, `not_applicable` or `transient`. A retired run (including one a concurrent sweep already retired) goes to the ordinary same-run resume (`claimNativeSessionResumptions`), with every existing admission gate. A lock timeout or database error is `transient`: the sweep defers the run and checks again, and never falls back to reattachment. Any other run keeps verified reattachment. - `heartbeat/workspace-preparation.ts`: an ordinary same-run resume with a retired inherited runner acquires its retained sandbox through the existing exact-lease path (`reattachRemoteLease`), which validates and resumes that lease and never acquires a replacement. It acquires as an ordinary resume does only when the exact-lease failure proves the sandbox is unusable (`stage=environment_acquire|environment_resume`, `field=lease|lease_status|lease_expired|provider_lease_id`). Any other failure, such as a scope mismatch on a live lease, is kept and acquires nothing. - New `server/src/services/retained-sandbox-absent-error.ts`: a sandbox provider's resume reply with no provider lease and `expired: true` (the reply Daytona and other providers send when the sandbox is gone) now throws `RetainedSandboxAbsentError`. Every other mismatched reply keeps the existing error. The exact-lease reattachment in `environment-run-orchestrator.ts` maps it to `native_remote_recovery_lease_mismatch: stage=environment_resume field=lease_expired`, so the same-run resume above acquires as an ordinary resume when the provider has removed the retained sandbox. A transient resume failure is not treated as absence and never acquires another sandbox. - `native-restart-recovery.ts` exports the existing `hasNativeExecutionStartEvidence` so both paths use one definition of "execution reached a runner". - `doc/execution-semantics.md` documents the rule. ## Previous-release state - No schema change. The new profile field is additive. - A run saved by an older release without `providerEventsExist`, without a native session ID, or without a matching predecessor keeps verified reattachment. That path is unchanged. When it loses its runner, it still requests the user's Retry once (#19399). - A run that an older release left with its own runner start, controller checkpoint or provider event keeps verified reattachment, because its outcome may be ambiguous. - A profile without the new marker never changes acquisition. ## Verification - `npx vitest run server/src/services/heartbeat/recovery.test.ts -t "never launched its own runner|keeps verified reattachment"` (embedded PostgreSQL): - Staging shape (inherited session and process identity, no runner start of its own, saved remote workspace, active own lease, recorded PID). Before the fix: fails, because recovery requested reattachment. After the fix: passes. The PID fields are cleared, the marker names the predecessor and the retained lease, `executeRun` is called once with a plain `nativeLeaseOwner` claim, the run still has exactly one (active) lease, and no process is signalled. - Negatives, passing before and after: a provider turn of its own, a runner start of its own, a controller checkpoint of its own, no matching predecessor, a pending runtime request in the saved session, and a failure detail without `providerEventsExist`. Each keeps verified reattachment and gets no marker. - `npx vitest run server/src/services/heartbeat/workspace-preparation.test.ts -t "retained sandbox|retired inherited"`: - A retired inherited runner acquires its retained sandbox with `reattachRemoteLease`. Before: fails. After: passes. - When the retained sandbox is gone, it acquires as an ordinary resume. Before: fails. After: passes. - Without the marker, acquisition is unchanged (passes before and after). - A `nativeWorkspaceSync` reference also resumes its retained sandbox. A live lease's `lease_scope` failure is kept and acquires nothing. - `recovery.test.ts` also covers a second retirement call (returns `retired`, one event) and a held issue lock (defers; no reattachment, no dispatch, PID kept). - `environment-runtime.test.ts`: on exact resume, only a provider reply with no lease and `expired: true` (with or without a workspace sentinel mismatch) is `RetainedSandboxAbsentError`; a different lease or a reply without `expired` keeps the existing error. `environment-run-orchestrator.test.ts`: confirmed absence becomes `stage=environment_resume field=lease_expired` and acquires nothing. - Affected suites pass locally: `recovery.test.ts`, `workspace-preparation.test.ts`, `heartbeat-native-remote-failure-retention.test.ts`, `workspace-recovery-retry-copyback.test.ts`, `native-runner-restart-recovery.integration.test.ts`. `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - **No second executor.** The rule applies only when this run never requested a runner, so no runner of this run exists. The predecessor must be settled, and its identity is matched exactly. The resume then takes the normal same-run claim, which holds the issue lock and admission gates. - **No repeated side effects.** The run has no provider event, turn or tool start of its own. A saved session with a pending runtime request or an active turn keeps verified reattachment. The resumed provider session goes through the existing provider recovery, which refuses a pending side-effecting call with a permanent, named code. - **Bounded.** The retirement takes effect once per run, because afterwards the run has no recorded identity. A contended retirement waits for the next sweep. Further failures use the existing three-attempt provider budget. - I did not run a live Daytona sandbox, or a full preparation through a real sandbox plugin, through this path. The orchestrator's exact-lease checks are covered by its own unit tests. The evidence is recovery and preparation tests on a real database with the staging run shape. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file editing, read-only browser access to staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
68db7ed9e3 |
fix(recovery): never accept a dead local pid as proof for an owner on another host (#19868)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, recovery decides that an in-progress record's owner is gone and takes it over > - Two of those proofs use `kill(pid, 0)` on the local host. A pid that is dead here says nothing about a process on another host or in another PID namespace > - So a controller or runtime-control owner that is still alive elsewhere can be taken over while it still runs > - This pull request records the owner's PID space (the existing `controller-process-identity` host hash) and judges a pid only in that same space > - The benefit is that a different host is never proof. Those records follow their existing lease or TTL instead ## Linked Issues or Issue Description **Bug description** The Phase 0 ownership inventory (finding 2) found three proofs that accept "the pid is dead here" as proof for an owner on another host: `evaluateNativeControllerTakeover` (`native_run_finalizations` has no host column), the runtime-control sweep (`workspace-operations.ts`), and the orphan reaper `terminalizeOrphanedRunningRun`. Only `controllerProcessExited` checked the host. **Steps to reproduce** 1. Run two control-plane processes on different hosts, or in different PID namespaces, against one database. 2. Let host A own a native finalization lease or a runtime-control operation. 3. On host B, run restart recovery. If A's pid number is not alive on B, B takes over the live owner. **Expected behavior** An owner whose PID space differs from this process's, or cannot be read, is unprovable. Its record waits for its lease, fence, or TTL. Refs #19846 ## What Changed - Migration `0441`: add nullable `native_run_finalizations.controller_host_hash`. - `controller-process-identity.ts`: export `currentProcessSpaceHash()` and `sameProcessSpace()`. An unreadable space is stored as `"unknown"`, which never matches. - `native-restart-recovery.ts`: `currentNativeControllerIdentity` carries `hostHash`, which both coordinator claim writes (`native-restart-recovery.ts`, `native-session-executor.ts`) store. `evaluateNativeControllerTakeover` returns `controller_host_unprovable` before any pid probe when the recorded space is not this one. `native-dead-controller-result-lease.ts` selects the column. - `workspace-operations.ts`: runtime-control owner stamps record `hostHash`. `ownerProcessProvenGone` requires the same PID space (EPERM = alive), otherwise the existing staleness TTL applies. - Not in this PR: the orphan reaper `terminalizeOrphanedRunningRun`. `heartbeat_runs` has no host for `process_pid`, and its writers are spread across legacy and native paths. It follows as a separate PR with its own column. ## Verification - `npx vitest run src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/workspace-runtime-control-recovery.test.ts`: 55 passed. New cases cover another host and an unreadable space (dead and recycled pid) for takeover, plus another host, unreadable space, and legacy stamp for the sweep. The sweep cases also confirm that the TTL still reclaims after the heartbeat lapses. - Red before the fix: the 5 new cases fail on the old code. Green after. - `native-controller-recovery.integration.test.ts`, `native-failed-run-owner-release.test.ts`, `native-startup-cleanup-recovery.test.ts`: 71 passed. - `pnpm db:generate` produced `0441_lucky_chameleon.sql` (one `ADD COLUMN`) and pruned to five snapshots. `pnpm --filter @paperclipai/db check:migrations` passed. Server `tsc --noEmit` shows no errors in the changed files. ## Risks - Previous-release story: `native_run_finalizations` rows saved before this column have `controller_host_hash = null`. They keep the previous local dead-pid proof for this release. Making them unprovable would leave every native run in flight at the upgrade in `awaiting_evidence` with no TTL, because local takeovers have no lease fallback. Rows written by this release onward are strict. Runtime-control stamps without `hostHash` are strict now, because that path has a staleness TTL. - Owners built from other receipts (workspace finalization owner, workspace stop proof) pass no host and keep the previous behavior. Track 2's shared `OwnerStamp` should absorb them. - A new row on a host that really moved (redeploy onto a new machine) waits for its lease or an operator instead of being taken over by pid. That is the intended "different host is never proof" rule. - No new I/O before `turn.start`. The hash is computed once at module load. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a9ac2e2039 |
feat(native): split local runner startup into stage, launch and session-load spans (#19852)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a person messages an agent, the agent must start its turn fast. Time to first token (TTFT) and agent boot are first-class metrics. > - The Phase 0 baseline (#19846) found that local runner startup costs 1.6–17 s at p50 and up to 50 s at p95. It is the largest part of TTFT. > - The run log records that startup as one span (`runner.session.bootstrap` or `runner.session.resume`). We cannot see if the time goes to staging, to the `paperclip-runnerd` process launch, or to the provider session load. > - This pull request splits local runner startup into three sub-spans in the run log. > - The benefit: the follow-up work to cut follow-up-turn TTFT can measure each step before and after. ## Linked Issues or Issue Description Refs #19846 (TTFT baseline, finding 1 and gap "No sub-spans exist inside local runner startup"). Refs #19838. **What existing behavior does this improve?** The run-log performance spans for local native runner startup. **Current behavior** The local path records one `runner.session.bootstrap` or `runner.session.resume` span that covers runtime staging, the runner process launch and the provider handshake together. The remote path already records `runner.runtime.stage` and `runner.process.launch`; the local path does not. **Proposed behavior** The local path records three sequential child spans of `runner.session.startup`: `runner.runtime.stage`, `runner.process.launch` and `provider.session.load`. **Reason and benefit** The TTFT work needs to know which step dominates before it changes the runner lifecycle. Without these spans, a change to warm runner reuse cannot show which cost it removes. ## What Changed - `RunnerdCodexTransport` has a new optional `onStartupStep` callback. `#start` and `#resume` time three milestones and call it synchronously. A failing callback never changes startup. A nested startup (exact adoption from `thread/start`) keeps the outer clock. - `runner.runtime.stage`: launch preparation before the runner process exists. - `runner.process.launch`: spawn until runnerd accepts the run (`run.prepare`, or the recovery attachment on resume). An adopted live runner records no launch. This keeps the existing meaning in `summarizeStartupReuse`: a launch span means a fresh runner. - `provider.session.load`: `session.open` (or the recovery probe) until the provider identity is confirmed. - `recordNativeRunnerStartupStep` (`native-run-trace.ts`) records each step as a `run.performance.span` row with `mode` and `target: "local"`. It uses `trace.defer`, so the run-log write never delays `turn.start`. - The native executor passes the callback only for a local target. Remote launchers keep their own spans, so there are no duplicates. - `redaction.ts`: allowlist `runner.process.launch` and `provider.session.load` as native span names. - `doc/run-log-events.md`: new section "Local runner startup sub-steps". Data path (AGENTS.md §5.7): run log only, plus Observability spans when an operator sets an OTLP endpoint. The attributes (`mode`, `target`, `outcome`) are already in the closed span-attribute allowlist. No Telemetry change. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-startup-steps.test.ts`: a real runnerd and the fake Codex fixture. Bootstrap and a cold resume each report stage, launch and session load in order, with `ok` outcomes and no overlap. A throwing sink does not change startup. - `cd server && npx vitest run src/services/native-runtime/native-run-trace.test.ts src/__tests__/redaction.test.ts`: the rows have the correct parent, offsets, durations and failed level. A stalled run-log sink does not block the caller. The rows settle before `task.provider_session.measured`. The names survive redaction. - The existing `runnerd-codex-transport*` suites pass (412 tests). Under host load (load average 16–27), the first full run had 7 timing failures in suites that do not set the callback. Each failing file passed when I ran it again alone. `runnerd-codex-transport-settlement` passed 3/3 for the failing case alone. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - Low risk. Without `onStartupStep`, the transport behaves as before: the clock is not created and each milestone call is an optional-chain no-op. - More run-log rows: three per local runner startup. - The step boundaries are controller-side observations. `runner.process.launch` includes the runnerd authentication and the `run.prepare` acknowledgement, not only the OS spawn. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fa86162790 |
refactor(recovery): one process boot id and a cached controller identity (#19851)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Restart recovery decides when a record owned by a dead server process may be released > - One server process minted three different boot UUIDs, one per module (`controller-process-identity.ts:13`, `native-restart-recovery.ts:284`, `legacy-controller-lease.ts:8`) > - The controller identity also ran `ps -o lstart` on macOS at every native claim, preparation check and lease renewal, which is on the path before `turn.start` > - The durable ownership plan (Phase 1, step 1) needs one process identity before it can add one owner stamp and one prover > - This pull request makes the three modules stamp one boot id and reads the process start time from the OS once > - The benefit is one meaning of "owned by this process" across tables, and no `ps` spawn on the turn path ## Linked Issues or Issue Description Refs #19831 (plan: `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`, Phase 1). Refs #19846 (Phase 0 inventory, finding 1, and the TTFT baseline row "Controller identity"). **Problem** One process has three boot ids, so the same fact ("this record is owned by this process") has three encodings. The uncached controller identity spawns `ps` on macOS on every claim and renewal. **Expected behavior** One boot id per process. The OS start-time read runs once per process. **Impact** Phase 1 needs one identity to build `OwnerStamp` and `proveOwnerGone` on. The turn path loses one subprocess spawn per claim and renewal. ## What Changed - `controller-process-identity.ts` owns the single process boot id. It adds `currentProcessStartedAtMs()`, which caches a successful OS start-time read for the life of the process and retries a failed read. It also exports `currentProcessSpaceHash()` for the coming prover. - `native-restart-recovery.ts`: `currentNativeControllerIdentity()` and `isCurrentNativeControllerProcess()` use the shared boot id and the cached start time. Its own `randomUUID()` boot id is removed. The fallback to the server-info start time is kept. - `legacy-controller-lease.ts`: `legacyControllerBootId` is now the shared boot id. - Tests in `native-controller-identity.test.ts`: one boot id across native claims, legacy leases and creation intents; one OS read across many claims and renewals; a failed read is retried, not cached. ## Verification - `npx vitest run src/services/native-runtime/native-controller-identity.test.ts src/services/legacy-controller-lease.test.ts src/__tests__/heartbeat-pending-cleanup-sweep.test.ts src/__tests__/sandbox-creation-operations.test.ts src/services/heartbeat/upgrade-readiness-post-result.test.ts src/services/heartbeat/recovery.test.ts` (in `server/`): 6 files pass. - `npx tsc --noEmit -p server`: no errors. - I checked every reader of the three ids. Each one compares a stored column only with the id of the module that writes that column (`heartbeat_runs.controller_boot_id` with the legacy id, `native_run_finalizations.controller_boot_id` with the native id, metadata `bootId` with the creation id). No reader compares the ids of two modules, so making them equal changes no decision inside one process. ## Risks - Low risk. No schema change and no persisted shape change. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): rows saved by an older release carry that release's random per-module ids. They never equal the new process's id, exactly as a row from any earlier boot never did. They carry forward with no migration. - The start-time cache is correct because a process's own start time never changes. A failed read is not cached, so a transient `ps` failure does not pin the fallback value. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
824b25654c |
fix(ui): keep a Retry for retained-file holds whose source cannot be verified (#19840)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a native run finishes but its files cannot be saved, the task stays blocked behind a retained-file recovery hold. > - The server records that hold as blocked and retryable, and the task notice tells the user to "Retry to try again". > - The task page hides its own Retry for these holds and relies on the retained-file recovery panel instead. > - That panel only binds the recovery record when a read-only source inspection returns `verified`. > - Inspection can return `blocked` for normal reasons: a sandbox that predates ownership receipts (state saved by an older release), or a workspace that moved on after its files were recovered. > - Then neither surface offers an action, and "Check file recovery status" always answers "File recovery changed. Refresh the task", also after a reload. > - This pull request binds the record by its own identity, uses inspection only to gate the physical file-recovery request, and keeps the task Retry visible whenever the panel has no control of its own. ## Linked Issues or Issue Description Refs #18848 (the change that added the retained-file recovery panel and hid the generic Retry for these holds). **What happened?** A blocked task with an active `native_workspace_sync_out_retry_exhausted` hold showed "The saved files need recovery before work can continue", "The recorded source is unavailable or has changed", and "Saving the agent's files failed after automatic retries. Retry to try again". It showed no Retry and no recover button. "Check file recovery status" always answered "File recovery changed. Refresh the task to inspect the current recovery", and a reload did not change it. The hold was saved by an earlier release: `systemRecovery` exhausted with `allocation_ownership_ambiguous` / `service_required`, and `workspaceRecovery` blocked with `retryable: true`. The source inspection endpoint returned `status: "blocked"` for it. A second task whose files were already recovered showed the same "need recovery" and "has changed" text, because its inspection reported a reference mismatch after recovery. **Expected behavior** A retryable hold always shows a working Retry or recover control. A notice that says "Retry" has a Retry button. Status checks never loop on "changed, refresh". Recovered files show as recovered. When the source cannot be verified, the page names the reason and who must inspect it. **Steps to reproduce** 1. Use an active retained-export hold whose `workspaceRecovery` is blocked and retryable, and whose `systemRecovery` is exhausted. 2. Make the source inspection (`GET /api/issues/:id/recovery-actions?inspectSourceRunId=…&inspectSourceActionId=…`) return `status: "blocked"`. 3. Open the task as a board operator. 4. Before this change, the page shows no Retry and no recover button. "Check file recovery status" reports "File recovery changed. Refresh the task". **Paperclip version or commit** Observed on build `8812c19`. The regressions fail on master `0f5ce5050e` before the production edits. ## What Changed - `WorkspaceExportRecovery`: bind the recovery record by action, run and schema identity, not by inspection status. Inspection status now gates only the physical "Recover saved files" request (and the request re-checks it before POST). - `WorkspaceExportRecovery`: report through `onRecoveryControlChange` whether the panel offers its own control. It reports `true` only while the source is loading or verified and the files are not yet recovered, or while it shows the previous-server stop confirmation. - `IssueDetail`: hide the task Retry for retained-export holds only while the panel reports that it owns the control. Otherwise the existing task Retry (`POST /issues/:id/retry`) shows. The server renews a bounded recovery generation and re-verifies the source. - Show recovered files as recovered even when inspection no longer matches after recovery. - Name an unverifiable source in plain words (with the code for other reasons). State that Paperclip does not apply files it can't verify. - Recovery text never asks a person to inspect a sandbox or environment, and never names an owner or administrator as the one who must act. This covers the retained-file panel, the run source inspector, the copyback wait, and the native cleanup quarantine message. The execution blocker and the recovery action read model describe a spent recovery budget from its current state, so text saved by older releases ("If Retry keeps failing, inspect the run and its environment") is not shown. - Task Retry uses the existing `createUuid()` helper, so it works on plain-HTTP previews without `crypto.randomUUID`. An accepted Retry also refreshes the inspected file-recovery state. After a failed accepted request, the unverified-source reason stays visible; it hides only while recovery runs. - When the recovery record was replaced, "Check file recovery status" reloads the task detail instead of telling the user to refresh. - Tests: component regressions for an old-release hold, a recovered hold with a mismatched source, verified ownership, and a replaced record. A page regression proves Retry is shown and calls `retryTask` for unverified and stale sources. A server safety test proves that Retry for the old-release hold shape is accepted, opens one new bounded generation, queues file recovery again, keeps the hold active, and is idempotent. ## Verification - RED (master `0f5ce5050e` production files, new tests): 5 component tests fail. The failure output shows the exact staging text: "Check file recovery status", "File recovery changed. Refresh the task to inspect the current recovery.", "The recorded source is unavailable or has changed." 2 page tests (unverified and stale) fail with `expected [ 'Check file recovery status', … ] to include 'Retry'`. - GREEN: `cd ui && npx vitest run src/components/WorkspaceExportRecovery.control.test.tsx src/components/WorkspaceExportRecovery.test.tsx src/components/RunWorkspaceRecoverySurface.test.tsx src/components/ExecutionBlockerNotice.test.tsx src/pages/IssueDetail.test.tsx` passes 320 tests. - Server: `cd server && npx vitest run src/services/task-retry.test.ts` passes 81 tests, including the new old-release safety case. This is safety evidence, not a red: the server path was already correct, and the defect was in the UI. - Full UI suite: 11564 pass. 4 tests in `IssuesList.test.tsx` and `CompanyEnvironments.test.tsx` failed under full-suite load and pass in isolation (106/106). They are unrelated to this change. - `pnpm check:token-gates` is clean. `node scripts/run-tsc.mjs -p ui/tsconfig.json --noEmit` reports no errors in the touched files. The remaining errors are local artifacts that were not built (plugin SDK UI entry, generated Storybook catalog). - Staging was read only (GET requests). The inspect request did not loop in a fresh tab: two calls in 30 seconds. Most of the tab's request volume is periodic company issue-list refetches, which this change does not touch. ## Risks - Low risk. This is a UI-only behavior change. Server authorization, source verification, stop receipts, and the bounded recovery policy are unchanged and remain authoritative. - The task Retry is now visible for retained-export holds whose source does not verify. Retry renews one recovery generation. The worker re-runs the same source-identity checks, so Retry cannot bypass an ownership refusal. If the refusal is permanent (for example an ambiguous sandbox allocation), the hold returns to exhausted and the page names the reason. An automatic fallback for such sources is follow-up work. - Wording: a parametrized UI test renders every recovery state and asserts that no administrator, owner, or "inspect the environment" copy appears. A server test proves that a stored old-release message is replaced by state-derived text. Both failed before the change. - Previous-release story: the old-release hold shape (board-owned, `systemRecovery` revision 5 exhausted with `service_required`, `workspaceRecovery` blocked and retryable) is covered by component, page, and server tests. No persisted state shape changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (repository search, file edits, shell, test execution, and a read-only browser). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
630411af53 |
fix(recovery): require a 1 s start-time gap and treat EPERM as alive in owner proofs (#19848)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, recovery code decides whether the process that owns an in-progress run is gone > - Two proofs compare process start times to the millisecond, but `ps -o lstart` on macOS and BSD reports whole seconds > - So a live owner can look like a reused PID, and recovery can release it while it still runs > - Signal-0 handling also disagrees: one check treats EPERM as dead, while all others treat it as alive > - This pull request puts one start-time rule and one EPERM rule in `controller-process-identity.ts` and uses them at every site the Phase 0 ownership inventory named > - The benefit is that no proof releases a live owner because of clock precision or a permission error ## Linked Issues or Issue Description **Bug description** The Phase 0 ownership inventory (finding 1) found inconsistent "owner gone" proofs. `native-failed-run-owner-release.ts` (`recordedProcessMayBeLive`, `runProcessMayBeLive`) and `native-session-executor.ts` (`verifiedRecoveryProcessIsAlive`) treat any start-time difference of 1 ms or more as a reused PID. `native-restart-recovery.ts` and `controllerProcessExited` require a gap of at least 1 s. `local-service-supervisor.ts` (`isPidAlive`, `isProcessGroupAlive`) treats EPERM as dead. **Steps to reproduce** 1. On macOS, record a live process start time with millisecond precision. 2. Run the abandoned-owner release pass, or the recovered-runner liveness check, against that live PID. 3. The pass releases the owner, or the check reports the runner as gone. **Expected behavior** A sub-second start-time gap is ambiguous and keeps the owner. EPERM means the process exists. Refs #19846 ## What Changed - `controller-process-identity.ts`: export `processExists` (EPERM = alive) and `processStartsDefinitelyDiffer` (gap >= 1 s; a missing time proves nothing). `controllerProcessExited` uses the new helper. - `native-failed-run-owner-release.ts`: the controller and run-process checks use the shared helpers instead of exact millisecond equality. - `native-session-executor.ts`: `verifiedRecoveryProcessIsAlive` accepts a start time within 1 s as the same runner. Its signal path uses the same check. - `native-restart-recovery.ts`: `definitivelyDifferentProcessStart` delegates to the shared helper. No behavior change. - `local-service-supervisor.ts`: `isPidAlive` and `isProcessGroupAlive` treat EPERM as alive. ## Verification - New `controller-process-identity.test.ts`: the 1 s boundary, EPERM and ESRCH for every signal-0 check, and a live spawned runner whose recorded start is 500 ms off. - `native-failed-run-owner-release.test.ts`: two new cases (controller and run process) with a live child and a recorded start 500 ms off. The owner stays `observed`. - Red before the fix: with the old call sites, all 4 new cases fail (`releaseAbandonedNativeOwners` returns 1 and releases the live owner; EPERM returns false; the live runner reads as gone). Green after the fix. - `npx vitest run src/services/controller-process-identity.test.ts src/services/native-runtime/native-failed-run-owner-release.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/local-service-supervisor.test.ts`: 59 passed. ## Risks - Low risk. The change only makes proofs stricter: a sub-second gap or EPERM now keeps an owner that the old code released. Such a row waits for the existing lease, TTL, or real process exit. - A recycled PID whose new process started within 1 s of the recorded one now stays held until it exits. That is the same rule the takeover path already used. - Previous-release story: no persisted shape changes. Rows saved by the previous release are evaluated with the stricter rule. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e2d490c61b |
feat(observability): anchor native time to first token at the accepted wake (#19838)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When someone messages an agent, the reply must start fast. The durable ownership plan (#19831) makes time to first token (TTFT) a hard invariant. > - Phase 0 of that plan records a TTFT baseline per adapter and environment, measured from the moment the message or wake is accepted. > - Today `task.time_to_first_token` starts at the wake comment, the answered question, or the run row's creation. A wake request that was accepted earlier (deferred, coalesced or retried) is not part of the measurement, and the span does not say which start it used. > - This pull request moves the start back to the earliest accepted input and names it on the span. > - The benefit is that the baseline and later budgets measure what the user waits for, with no new work before `turn.start`. ## Linked Issues or Issue Description Refs #19831 **What is the problem?** `task.time_to_first_token` does not include the time between an accepted wake request and the creation of its run row. It also does not record which input it started from. **What is the expected outcome?** The span starts at the earliest accepted input and carries an `anchor` attribute: `comment`, `question_response`, `wake_request` or `run_created`. **Why does it matter?** Phase 0 of the durable ownership plan sets TTFT budgets from this baseline. A start that skips queue time hides the latency that deferred wakes add. ## What Changed - `resolveNativeTimeToFirstTokenStart` (pure) in `native-run-trace.ts` picks the start and the anchor. `anchor` joins the closed span-attribute allowlist. - `readNativeRunWakeRequestedAtMs` reads `agent_wakeup_requests.requested_at` for the run. It never throws. - `native-session-executor.ts` calls it only after the first deliverable text, so it adds no wait before `turn.start`. The span now ends at the text commit time, not after the trace write. - Updated `doc/run-log-events.md` and `doc/observability.md`. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-run-trace.test.ts src/services/native-runtime/native-run-wake-anchor.test.ts` (26 pass). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t first` (41 pass). The existing TTFT test now asserts the anchor for each ingress. ## Risks - Low risk. This changes only the run-log and OTLP span. It is not Telemetry (AGENTS.md §5.7), and it does not change runtime behavior. - One indexed primary-key read per native dispatch, after the first text. A read failure drops the anchor and keeps the old start. - TTFT values can grow for deferred wakes. That is the intended correction. `startOffsetMs` can be negative when the wake request comes before the trace start. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
84908d176b |
fix(recovery): name the failed check in native_remote_recovery_lease_mismatch (#19839)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote native runs execute in a sandbox, and recovery can reattach a run to its original sandbox runner > - Reattachment checks the saved lease, workspace, and runner identity in five places. Each failed check throws the same bare `native_remote_recovery_lease_mismatch` > - On staging, 34 OpenCode runs on 31 tasks failed with this bare error in one evening. The run records could not show which check failed, so the cause could not be confirmed from the run alone > - This pull request makes every producer name the failed step and field with closed values > - The benefit is that an operator and the next investigation can see which reattachment check failed, without new logging or identifiers in the error ## Linked Issues or Issue Description Refs #19399, Refs #19733, Refs #19755, Refs #19779 **What happened?** Remote same-run retries failed with `native_remote_recovery_lease_mismatch` and the task was blocked for reconciliation. The error is thrown by environment acquire, environment resume, workspace preparation, workspace sync, and remote runner verification. All of them used the same bare string. The run's `error` field and its run events did not show which check failed. **Steps to reproduce** 1. Start reattachment for a remote native run whose saved lease, workspace reference, or runner identity no longer matches. 2. Read the run's `error`. 3. It shows only `native_remote_recovery_lease_mismatch`. **Expected behavior** The error names the step and the field that differed, with no identifiers, and existing consumers of the code keep working. ## What Changed - New `server/src/services/native-runtime/remote-recovery-lease-mismatch.ts`: the code constant, a closed `stage` and `field` vocabulary, and `remoteRecoveryLeaseMismatch(stage, field)`. It builds `native_remote_recovery_lease_mismatch: stage=<stage> field=<field>`. - `environment-run-orchestrator.ts` (stages `environment_acquire` and `environment_resume`), `heartbeat/workspace-preparation.ts` (`workspace_preparation`), `native-workspace-sync.ts` (`workspace_sync`), and `verifyRemoteRunnerReattachment` in `native-session-executor.ts` (`runner_verification`) now throw the detailed form. Each check keeps the same condition and order. Only the message changes. - `restart-interrupted-turn.ts` uses the shared constant. It already matches the code exactly or as the `code: ` prefix, so automatic continuation after a reattachment loss is unchanged. No other consumer matches the code. - `doc/execution-semantics.md` documents the detail format. ## Previous-release state This change adds no persisted state and changes no saved identity. Runs that a previous release failed keep the bare `native_remote_recovery_lease_mismatch` in `error`. Runs failed by this release store the detailed form. The only consumer (`isReattachmentLoss` in `restart-interrupted-turn.ts`) accepts both forms, and `workspace-recovery-retry-copyback.test.ts` covers both. ## Verification - Exact-detail assertions in the producer tests: - `server/src/__tests__/environment-run-orchestrator.test.ts`: every scope, status, expiry, provider, lease-ID, cwd, plugin-task, and resume case asserts its stage and field. Against the previous producer, 21 of these cases fail (the message was the bare code). With this change, all 57 tests pass. - `native-session-executor.test.ts`: a replaced sandbox reports `runner_verification`/`provider_lease_id`. A missing durable runner identity reports `identity_run_id` before any sandbox read. - `native-workspace-sync.test.ts`, `heartbeat/workspace-preparation.test.ts`, `native-controller-recovery.integration.test.ts` assert their exact detail. - End to end on embedded PostgreSQL: `heartbeat-native-remote-failure-retention.test.ts` records `native_remote_recovery_lease_mismatch: stage=environment_resume field=lease` on the run when the provider no longer resumes the lease, and it still requests the user's Retry once. `workspace-recovery-retry-copyback.test.ts` adds a detailed-form row that still continues once. - The affected server test files pass locally. `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. Only the error text changes. Conditions, ordering, and fail-closed behavior are unchanged. - Logs and run errors show a longer message. The detail has closed values only, so it adds no identifiers or secrets. - A first version of this PR also changed remote retry selection. Review showed that a plain resume in a new sandbox hits the remote workspace's unsaved-work hold, and that a `destroy` disposition does not prove the old sandbox is gone. That change is removed. It needs a separate design. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file editing, read-only browser access to staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Dotta
|
86bc3db693 |
feat(runner): search and load unassigned company skills (#19676)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents use tools to read reusable company skills. > - The existing native skill tools read only skills assigned to the agent. > - An agent can stop when a task needs another skill that is already in the company library. > - This pull request adds tools to search and read accessible company skills on demand. > - Agents can use the saved instructions without a persistent assignment change. ## Linked Issues or Issue Description **Problem or motivation** An agent cannot use an unassigned company skill through the standard native skill tools. A generic API workaround needs extra discovery and can return the full library. **Proposed solution** Add `search_skills` and `load_skill` to the native tool catalog. Search returns small pages of stored metadata. Load returns text from a saved version and lists its files. Both tools use the current run identity and existing company and project access checks. **Alternatives considered** Assign each skill before a read, or use generic API discovery. Both add work to a read that the company already permits. **Roadmap alignment** This extends the existing skills discovery path. It does not add a store, permission model or installation flow. Related: #15409 adds an assigned-skill loader to a different adapter. ## What Changed - Add bounded company skill search. Search does not synchronize sources or fetch repositories. - Add versioned text reads for SKILL.md and supporting files. Page text and file lists. Reject unsafe paths and binary files. - Recheck access after storage reads. Keep the existing closed Muse, skill-test and completion-review tool scopes. - Add integration tests with isolated PostgreSQL and real local package storage. Update the native tool count assertion and skill documentation. - Limit text files to 16 MiB before body reads. Accept saved file paths up to 4,096 characters plus the repository prefix. Return the response digest only on the first page. - Add optional byte limits to the package readers. Select only the requested legacy file in PostgreSQL. Cast the size comparison to `bigint` so existing callers can omit the limit. - Cache decoded text and its digest for one run. Limit the cache to 64 MiB and 64 files. Check access on each page and clear the cache when the run closes. - Read repository source files and wrapper inputs through verified warm reads or isolated recovery. These reads do not hold the company startup lock. ## Verification - Local verification tested tree `7ba5828387ca0a9c2ee99655818a28de1235a854`, published as commit `c9d04cba601541cca506f5b72aa6f7272a25379f`. The published tree matches that tested checkout. Current head `33fa3d3d4c7988288685ce7c6499d064e0cb9079` (tree `8daaa2f3264d1cd65a9b6d85dbb9d3729ac4a29d`) removes six comment lines from three source files; its executable code is unchanged from the locally tested revision. - Four native skill and authority Vitest suites: 72 passed and 7 existing sandbox checks skipped. All 20 skill-library integration tests passed. - Skill-library and package-store suites: 65 passed. This includes the 45 package-store cases and the same 20 integration cases. Ran with umask `022` for executable-file tests. - Server source and test typecheck layers passed in low-memory mode. The affected source cycle also passed a final separate check. - `git diff --check` and `node scripts/check-module-boundaries.mjs` passed. - Red/green evidence: the new default inline-read case reproduced PostgreSQL error `22003`. Two max-size paging cases reproduced extra body reads. Warm and cold repository reads timed out while another connection held the company startup lock. All five cases pass after the fixes. - Both 16 MiB paging cases check that the second page performs no additional body read. They also check that source-access revocation blocks a cached page. Warm and cold lock cases load both the generated wrapper and a supporting file while the startup lock remains held. - The first cold-read test attempt had an incorrect helper import. Corrected the test import, reproduced the lock timeout, then verified the fix. - [CI on the previous revision](https://github.com/paperclipai/paperclip/actions/runs/42319652795) passed typecheck, build and policy. Serialized shard 2 failed in the unchanged test `schedules bounded retries for failed accepted interaction continuation wakes`. The same case passed locally: 1 passed; 416 cases were excluded by the filter. The cause of the earlier CI failure remains unknown. [CI for |
||
|
DottaandPaperclip
|
b8e02fc2b4 |
fix(native): preserve bounded larger web inventories (#19832)
## Thinking Path > - Paperclip manages AI agents and preserves their work across server changes. > - Native harness preservation stores retained web-server files in authenticated archives. > - A retained skill can contain hundreds of valid local files. > - The WEB evidence owner rejected any directory or inventory above 128 entries. > - This blocked preservation of two 462-file skills even though their payload fit the existing byte limits. > - This pull request shares finite inventory bounds between capture and archive validation. > - It preserves every file and keeps the execution, identity, writer, and publication guards. ## Linked Issues or Issue Description Refs: #19814 **What happened?** WEB evidence preservation rejected a retained source with 927 local skill files. The source had settled execution ownership. The files fit the existing payload limits. The directory enumeration failed before archive publication because its entry limit was 128. **Expected behavior** Preserve a valid finite inventory within explicit file, payload, path, and metadata limits. Keep remote recovery obligations unchanged. **Steps to reproduce** 1. Retain two 462-file skills and one three-file skill under a settled native source. 2. Request canonical WEB evidence preservation. 3. The old owner rejects directory enumeration. The added integration test reproduces this refusal with real files and PostgreSQL. ## What Changed - Share the WEB limit through the existing controller archive owner: 10,000 total entries, 4 MiB of manifest metadata, and 4,096 bytes per path. - Apply the limits to filesystem capture, published binding validation, and archive descriptor validation. - Keep the existing 16 MiB per-file and 64 MiB payload bounds. Keep authority history at 128 entries and 32 epochs. - Add real archive and cold-restore coverage for 927 files, legacy manifest compatibility, and bound refusals. Document the limits. ## Verification - Genuine baseline RED: the new 927-file integration case reaches the original `captureWebEvidence.directoryNames` limit and fails before publication. - The initial setup attempt failed before test collection because the generated plugin SDK was absent. The required SDK build fixed setup; it was not counted as a regression result. - The first affected three-suite run passed 93 cases. The large roundtrip exceeded the generic 15-second case deadline. That failure is retained. The isolated large case passed with a 60-second case limit, but full CI later exceeded that limit while the other 4,577 tests in its shard passed. Both timeout failures are retained. The final fixture keeps all 927 files and 18.5 MB, uses bounded 16-way independent comparisons, and has a 120-second whole-case limit. The canonical preservation signal remains 60 seconds. No production deadline or assertion changed. - The 927-file case then passed: encrypted archive, authenticated cold restore, all bytes/hashes/modes, unchanged remote leases, and database-only validation with source volume absent and storage reads forbidden. - The final metadata repair adds a genuine RED case: an entry list below 4 MiB previously emitted a larger complete manifest without rejection. The owner now checks the complete serialized manifest. All ten focused cases passed after that repair. The final bounded-I/O fixture passed the same ten cases in 37.26 seconds, including the full 927-file roundtrip. All 93 other affected cases passed in the preceding three-suite run. Fresh required CI supplies broad final-head coverage. - Server TypeScript, module boundaries, and whitespace checks pass. Run the focused PostgreSQL checks from `server/` with the repository wrapper: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts --testNamePattern 'preserves all 927|existing 128-entry WEB|WEB manifest beyond|oversized WEB filesystem' --hookTimeout 90000 --maxWorkers 1 --fsModuleCache ``` ## Risks The owner can hold more entries in memory. The fixed 64 MiB payload, 4 MiB metadata, path, and entry bounds constrain that cost. The archive format and old receipt bytes do not change. Older builds still have the 128-entry limit and cannot qualify a larger new receipt. This change does not settle remote recovery, grant execution authority, alter finalizer locks, or change publication CAS rules. No hosted preservation or deployment was performed for this PR. ## Model Used OpenAI `gpt-6.1-sol`, with code execution and repository tools. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
a94881d1a7 |
fix(heartbeat): save a comment refused by a temporary hold and route post-quarantine input to its own gate (#19829)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user's follow-up comment or Retry must be delivered once, eventually, even when a temporary company admission hold (accounting still settling) is active > - #19741 kept claimed runs queued through such a hold and woke held work when it cleared > - A restart-chaos wave still lost a comment, because its wake was refused before admission and written as a terminal skip > - The same wave left a Retry waiting forever for a stop record of a process that never existed > - This pull request saves the refused comment as a deferred receipt, and routes input after a pre-launch quarantine failure to the quarantine's own gate > - The benefit is that the comment is delivered after the hold, and the task asks for the action that actually unblocks it instead of offering a Retry that can never start ## Linked Issues or Issue Description Refs #19741 and #19702. **What happened?** In a restart-chaos wave, a failed run's accounting held the company for 600 s. 1. A user commented while the task's run was live. The enqueue budget gate refused the comment's wake before it reached the deferral behind the live run. Because it was a user request with no saved receipt, it was written as `skipped` (`budget.blocked`). The live run finished without it and the task ended `done`, so the comment was lost. 2. A Retry created a run that the native cleanup-quarantine fence refused before it requested any runner launch (`native_session_cleanup_quarantined`, no process events). A second Retry then waited with `process_identity_missing` ("The previous run has no verified stop record") for a process that never existed. The task kept offering Retry, and that Retry could never start. **Expected behavior** A comment refused by a temporary hold stays saved and is delivered once later. Input after a run that was refused before launch is handled by the quarantine gate: Retry is not offered (send a new message), and a new message gets the verified-stop check of the previous provider session. ## What Changed - `heartbeat/queue.ts`: when the enqueue budget gate refuses a user's comment wake and the block is temporary (not a reached hard stop, per `isBoardOwnedBudgetBlock`), the wake is saved as a `deferred_issue_execution` receipt. The receipt carries the comment ids and an `executionWait` of `budget.blocked`, and the gate notifies the deferred-receipt worker and throws `PersistedExecutionWaitRefusal`. It is not written as `skipped`. Normal promotion (when the live run ends) and #19741's hold-release wakeups deliver it. - `native-local-process-stop.ts`: new `nativeRunRefusedBeforeLaunch`. It returns true for a failed native run with `native_session_cleanup_quarantined`, no recorded process, and no launch-request or identity event. - `explicit-native-continuation.ts`: such a run passes the process-identity check, so the existing quarantine branch decides. A Retry gets `cleanup_quarantined` (shown with `canRetry: false` and "send a new message"). A new message gets `verifyStoppedNativeSessionForContinuation`. Any launch request or identity record keeps the original stop proofs. - `native-session-executor.ts` (`verifyStoppedNativeSessionForContinuation`): when the source has no process record and no launch event, the verifier relies on the previous owner's stop evidence. Before this change it required the source to share the owner's process identity, so a new message stayed held even after the owner stopped. A source that carries a process identity must still match the owner exactly. ## Verification Each new test fails on `master` and passes with this change, on embedded Postgres: - `server/src/__tests__/heartbeat-held-comment-wake.test.ts`: a comment whose wake the hold refuses at enqueue is saved with a `budget.blocked` wait and is not skipped. It is delivered exactly once after accounting settles. - `server/src/services/explicit-native-continuation.test.ts`: - A run refused by the quarantine before launch routes a new message to the quarantine's stop verification and a Retry to `cleanup_quarantined`. These failed on `master` with `process_identity_missing`. - Guards (green on both): a quarantined run that requested a launch, and another failure without a process record, still require `process_identity_missing`. - `server/src/services/quarantined-task-retry.test.ts`: a never-launched quarantined source is certified once the previous owner has stopped (red on `master`). It stays held if the source requested a launch or the owner is still running. All 31 tests pass, and the `stopped native conversation physical cleanup` suite passes (46). - Related suites pass locally (6 files, 620 tests): explicit native continuation, held comment wake, queued comments routes, comment wake batching, refused assignment wake, deferred receipt work. Two `deferred-execution-receipt-work` tests failed once under parallel load and passed when run alone. ## Risks - The saved-comment path applies only to user comment wakes refused by a temporary budget hold. Hard stops, system wakes, case handoffs, durable chat requests, and existing saved receipts are unchanged. - The pre-launch rule is limited to the quarantine failure code with no launch evidence. Every other missing process identity still requires the existing stop proof. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, kept restart-chaos instance inspection). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
45c2bd94c6 | fix(native): name the interrupted attempt when an exact resume is refused (#19709) | ||
|
DottaandClaude Opus 5.5
|
b2af45f0f1 |
fix(runner): let OpenAI API-key Codex agents generate images with their existing key (#19654)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - New users often connect a Codex agent with an OpenAI API key during
onboarding.
> - A user asked such an agent for a logo. The agent said that the
built-in image generator was not available and that its fallback needed
a separate API key. It then drew an SVG by hand.
> - #19516 removed `--disable image_generation`. That change is correct,
but it does not reach this path. The pinned Codex 0.160.0 shows its
`image_gen` tool only when Codex signs in with ChatGPT. The check is
`image_generation_available` in `codex-rs/core/src/tools/spec_plan.rs`.
An API key never gets the tool.
> - The runner keeps the provider key out of the agent's shell on
purpose, so the skill's `scripts/image_gen.py` fallback cannot run.
> - This pull request adds a Paperclip `generate_image` tool. The server
makes the image with the agent's existing OpenAI key. The key never goes
to the agent process or the shell.
> - The benefit is that API-key Codex agents can make images with no
second key. The spend goes to the agent's budget and the budget can stop
it.
## Linked Issues or Issue Description
Refs #19516
**What happened?**
An agent connected through onboarding with an OpenAI API key (Codex
harness, `gpt-5.6-sol`) could not generate images. The agent said: "The
image skill's built-in generator isn't available in this run, and its
fallback would require a separate API key." It then drew an SVG instead.
**Steps to reproduce**
1. Create a company with a Codex agent that uses an OpenAI API key.
2. Ask the agent for a raster image, for example a logo PNG.
3. The agent reports that image generation is not available. It does not
make the image.
**Expected behavior**
The agent makes the image with the OpenAI key that already powers it.
**Root cause**
The agent ran on the native runner's Codex app-server path, with
`auth.json = {"OPENAI_API_KEY": ...}`. Codex 0.160.0 removes
`image_gen.imagegen` from the model's tools unless
`current_auth_uses_codex_backend()` is true. That function is true only
for a ChatGPT sign-in.
I proved this with the real pinned binary. I gave it the runner's
isolated launch arguments and the API-key auth file, and pointed it at a
local Responses endpoint. The first model request offers only `exec`,
`wait`, `request_user_input` and the collaboration tools. #19516's test
checked only `codex features list`, so it could not see this gate.
## What Changed
- **New `server/src/services/native-runtime/generate-image-tool.ts`:**
- The tool schema and definition.
- `imageGenerationBindingForRun`. It applies only to native Codex runs
on a direct OpenAI `api_key` connection with no router selection.
- The OpenAI Images API call (`gpt-image-2`, `b64_json`, PNG check, 180
s timeout). The reply is read as a stream with a byte cap before it is
parsed. A decoded image larger than 20 MiB is refused.
- A reviewed rate card for `gpt-image-2`: text input $5, image input $8
and image output $30 per 1M tokens, from
developers.openai.com/api/docs/pricing, retrieved 2026-11-06.
- A create-only, symlink-confined local write.
- **`paperclip-runner-tool-authority.ts`:**
- Adds the `imageGeneration` binding.
- Advertises the tool only for standard mode with a workspace. A remote
target must also have an installer. Dot, Muse, Grok and native review do
not get the tool.
- Adds the `#generateImage` handler. Before any spend, it checks again:
the run and task, budget authority, `issue:mutate`, case handoff, and
the connection.
- Stores a claim-before-spend per-call receipt in
`resultJson.imageToolReceipts`. A retry or recovery replays the receipt
and does not pay again.
- Saves a `charged` receipt (usage, price, request ID and charge time)
as soon as OpenAI returns. A failed save, failed accounting write or
crash can then still settle the spend. A retry or recovery records the
cost once and never buys again.
- Records one `cost_events` row through `createCostEventInTransaction`.
The row is `metered_api` with `costStatus: "estimated"`, and the budget
is evaluated. The row is recorded even if saving fails.
- Checks the task's project budget, as well as the company and agent
budgets, before spending.
- Writes content-free `runner.image_generate` activity entries: started,
completed, failed and unconfirmed.
- **`ai-connections.ts`:** adds `selectImageGeneration`. It checks again
that the connection is an active, healthy OpenAI `api_key` connection
with no custom routing. It also checks the responsible user's
membership, the grant audience and the agent or company installation.
These are the same rules that runtime selection uses.
- **`native-session-executor.ts`:** gets the binding from the run's
admitted `contextSnapshot.aiConnection`. The value comes only from the
server, never from tool input.
- **`native-session-resume.ts`:** the native tool contract changes from
`v18` to `v19` and now lists `generate_image`. A persisted provider
thread cannot add tools when it resumes, so older threads must rotate.
- **`doc/runner-api-tools.md`:** a new section about image generation
for OpenAI API-key Codex agents.
- **#19516: kept, not reverted.** Its changes have no effect on the
API-key path. They are still correct for Codex agents that sign in with
ChatGPT, because `image_gen` works there and saves to
`CODEX_HOME/generated_images`. Those agents need the read grant that
#19516 added. On the API-key path, the new tool replaces the built-in
tool. The tool's description tells the agent to use it instead of
`image_gen` or `scripts/image_gen.py`.
## Verification
**Red on `master` (
|
||
|
DottaandPaperclip
|
ae1f781cac |
fix(native): preserve web files without settling remote recovery (#19814)
## Thinking Path > - Paperclip manages work that AI agents perform for users. > - Native sessions retain files on the web server and state in a remote provider. > - Replacing the web server must preserve its files without changing remote recovery authority. > - The existing maintenance owner requires destroyed-provider receipts even when all web writers have settled. > - This can keep the old web volume required only because a remote recovery obligation remains unresolved. > - This pull request adds a closed web-byte evidence mode to the existing preservation owner. > - The benefit is durable web-file coverage with unchanged remote references, failures and continuation policy. ## Linked Issues or Issue Description Refs #19748, #19806. A search for native harness filesystem evidence and open web/harness changes found no duplicate implementation. **What happened?** A settled successful native source can retain a raw controller, a separate warm supervisor and provider-home files while its historical external lease has no termination receipt. The existing filesystem-independence owner refuses before it can preserve these web files. One retained skill file can also remain physically present after its version row was deleted. **Expected behavior** Preserve every nonexcluded web file as authenticated evidence. Keep missing files absent and deleted skill versions deleted. Leave VM references, recovery actions, failures and execution authority unchanged. Validate the published web-byte receipt on an empty replacement using database bindings only. **Steps to reproduce** Use the included successful-source fixture with a settled web owner, an unresolved external lease, real controller/supervisor files and a retained skill whose version row is deleted. The baseline rejects at its destroyed-provider guard. The new explicit maintenance mode preserves the files and validates on a replacement. A separate fixture uses a prior successful source, a failed retained last run, a genuinely absent controller, a warm supervisor and 267 durable checkpoint parts. **Paperclip version/commit** Baseline `384fa2b7be`; final branch rebased onto `585c885168`. Current head: `990210d7e9`. **Deployment mode** Original-volume maintenance followed by an ephemeral replacement. No hosted operation or provider request is part of this PR. ## What Changed - Add explicit `web_evidence` maintenance mode and a separate `paperclip.native-harness-web-evidence.v1` receipt. Keep the existing terminated-provider mode and old manifest shapes. - Archive the closed canonical session-root inventory, the separate warm supervisor and exact admitted skill manifests through the existing encrypted `original_volume_source_evidence` owner. Preserve bytes and modes. Keep canonical credential exclusions. - Preserve local skill edits and deleted-version files as forensic evidence. Create no skill version, executable checkpoint, provider backup or recovery authority. - Require original-volume capability, settled web writers, physical/finalizer locks, exact task/source/cohort/input bindings and final byte/mode/membership comparisons. Keep genuinely absent raw controllers absent. - Bind allocation and stamp identity while allowing later external status and cleanup observations. Validate replacements from durable database and archive bindings without the original anchor, filesystem or storage reads. ## Verification - Node 24.22.2, PostgreSQL 18.1, macOS arm64. The unchanged baseline fails at the real destroyed-provider guard. The final revision passes all 85 owner, original-volume capability and unchanged continuation-policy cases in 21.14 seconds. - Tests perform real encryption, persistence through the S3 storage interface backed by the owned local fixture, authenticated restore and exact byte/mode comparisons. The 281-byte file remains present while its version row remains absent. The evidence archive stays separate from the provider-session backup scope. - The supervisor-only case preserves 267 database checkpoint parts, keeps the raw controller absent and validates without original-volume identity. Existing strict-mode and old receipt controls pass. - Negative controls cover active web writers, expired but owned finalizers, physical owners, foreign controller/supervisor identities, missing manifests, symlinks, changed bytes/modes/membership/source during upload, changed durable bindings, incomplete receipt coverage and invalid late publication. Concurrent attempts retain one valid winner and one audit event. - Server TypeScript, module boundaries, diff checks and private-data scans pass. The verification planner's first invocation used a wrong script path; its canonical invocation refused a plan because fixture inputs changed during selection. Neither attempt counts as a pass. Required CI will provide broader application coverage. - Earlier fixture and treatment failures remain recorded. They exposed a missing helper, JSONB property-order instability, missing separate-supervisor mode coverage and fixture timing/source/count errors. The final same owner suite passes without weakened assertions. - A canonical cleanup audit found two additional observation fields: the cleanup controller and manual-attempt ID. Their real publication updates reproduced false receipt invalidation on the first PR head. The narrow correction keeps WEB coverage valid through those observations while changed allocation identity still fails. The exact same 85-case suite passes afterward. The final `990210d7e9` head has fresh Greptile 5/5 with no comments or unresolved threads. Required CI run [42370607882](https://github.com/paperclipai/paperclip/actions/runs/42370607882) passes: 55 successful checks and four intended skips, plus successful Snyk. The old head's CI was superseded by this correction and is not counted as a pass. From `server/`: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts src/services/native-runtime/native-harness-original-volume.test.ts src/services/native-runtime/native-harness-filesystem-independence-continuation.integration.test.ts --hookTimeout 90000 --maxWorkers 1 --fsModuleCache ``` ## Risks This changes a maintenance trust boundary. It proves web-byte preservation, not provider recovery or termination. External recovery obligations and existing resume failures remain. Unknown layouts, missing manifests, active local owners and changed authority fail closed. Invalid receipts are not automatically replaced. The inventory has independent limits of 128 entries, 16 MiB per file and 64 MiB total. Larger valid data stays on the original volume and blocks this mode. Canonical credential exclusions remain excluded. Evidence uses the existing encrypted ledger and retention roots; it does not become an executable backup. Old receipts remain byte-compatible. Startup adds bounded per-retained-scope database reads. Measure the affected cohort's actual serving startup cost before enrollment. Validation adds no filesystem or object read, worker or recurring poll. Live source qualification and publication remain separate after review, merge and normal delivery. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and PostgreSQL/filesystem test execution. The runtime does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d2efe3392b |
fix(native): preserve retired authority evidence (#19806)
## Thinking Path > - Paperclip manages work that AI agents perform for users. > - Native sessions retain control-plane files when execution authority moves to a later run. > - Volume migration must preserve these historical bytes before a server can stop depending on its old volume. > - The existing preservation owner archives the current controller and supervisor, but refuses a nonempty authority-epoch directory. > - A settled four-run session can therefore remain blocked after its current controller archive is complete. > - This pull request adds a closed historical inventory to the existing encrypted evidence archive. > - The benefit is preserved source evidence and database-only replacement validation without granting execution or recovery authority. ## Linked Issues or Issue Description Refs #19748. A search of pull requests for filesystem independence and issues for authority-epoch history found no duplicate change. This fixes its uncovered historical-file case. **What happened?** A settled native session retained three prior controller files in authority-epoch directories. Its three historical database checkpoint headers were absent. Maintenance preserved the current controller, then refused filesystem independence because those historical files remained uncovered. **Expected behavior** Preserve the exact historical bytes and modes through the existing encrypted evidence owner. Keep original files, prior archive rows, run outcomes, warnings, stamps, session pointers, and recovery policy unchanged. Validate the resulting receipt on a replacement without the source volume. **Steps to reproduce** Use the included four-run fixture with a current-controller-only archive, three prior authority epochs, destroyed provider receipts, released ownership, and absent historical checkpoint headers. Call the canonical filesystem-independence owner. The base rejects the nonempty directory; the treatment preserves the complete evidence and validates it after removing source-volume identity from the test process. **Paperclip version/commit** Baseline `ef83d566cacf1d5f740d701560597dad2bfa4619`; final branch rebased onto `27170dca49`. **Deployment mode** Original-volume maintenance followed by an ephemeral replacement. This PR performs no hosted migration or provider operation. ## What Changed - Add optional closed authority-epoch entries to the existing `original_volume_source_evidence` archive. Preserve nested file bytes and modes through the existing encryption, readback, ledger, audit, and retention owners. - Bind each epoch to its retained run, canonical checkpoint identity when present, or explicit header absence plus the exact admitted authority binding. Keep current-source checkpoint and artifact checks strict. Historical evidence never becomes an executable checkpoint. - Recheck the complete historical inventory at archive and receipt publication fences. Keep physical locks, all-run ownership and termination checks, source capability checks, and final CAS. - Bind the history inventory and its database facts into the authenticated archive manifest. Replacement validation uses database bindings only. Later header presence, removal, retirement, or changed identity invalidates the receipt. - Leave history-free manifest bytes and receipt shapes unchanged. Keep an existing current-controller-only archive when a later complete archive is added. Unknown children and other nonempty history directories remain blockers. ## Verification - Node 24.22.2, PostgreSQL 18.1, macOS arm64. The final four-run fixture fails on the baseline owner at its nonempty-history guard. The same final fixture and treatment pass all 51 focused PostgreSQL/filesystem cases. - The positive case performs actual encryption, object-store persistence, authenticated restoration, and exact byte/mode comparisons. It proves replacement database-only validation with absent source paths and storage reads refused. - Controls cover foreign identities, unknown children, owned prior runs, changed bytes or new files after upload, missing or newly present historical headers, malformed and retired headers, an unknown authority override, receipt-identity tampering, and a late concurrent winner. Existing current-controller evidence and history-free manifest bytes remain intact. - The baseline/treatment driver restored only its two owned source files in `finally` and verified their exact hashes. Earlier setup and treatment failures remain recorded: missing bundled PostgreSQL library links, an overly narrow raw-schema check, and an incorrect synthetic transient-workspace binding. The final baseline and treatment use the corrected same fixture; those earlier attempts are not counted as passes. - Review follow-up `21183b75f4` closes the missing-history-field gap. Deleting history from an actual published receipt reproduced an invalid true result before the fix. Unconditional manifest reconstruction now rejects it; all 46 owner cases pass. Genuine history-free manifests retain their exact bytes. - Review follow-up `f3d305dd36` adds suspended historical Runner-state bytes and mode checks through authenticated restore, active/foreign state refusals, exact 16 MiB and 64 MiB boundaries, and retention of an oversized file without publication. All 51 cases and server TypeScript pass; production behavior is unchanged. - CI run `42362784772` on the previous head had one unrelated 30-second timeout in `manual-private-cell-report-assets.test.ts`, with 6,864 support tests passing. The failure remains recorded. The prior head `21183b75f4b223cd3ec28124cdc194f24b4f79ba` passes [required CI run 42363408227](https://github.com/paperclipai/paperclip/actions/runs/42363408227), including Runner support: 55 successful checks and four intended skips, plus successful Snyk. The final test/documentation head `f3d305dd3690a7db38b57c4a77389c17206cb76b` passes [required CI run 42364958386](https://github.com/paperclipai/paperclip/actions/runs/42364958386): 55 successful checks, four intended skips, one neutral review check, and successful Snyk. Fresh Greptile is 5/5; all four review threads are resolved. No timeout or assertion was weakened. - The count warning is resolved as independent bounds: the actual descriptor accepts 32 control-plane-only epochs (97 entries) and 32 epochs with 31 Runner states (128 entries), and rejects 129 entries. The 32-epoch and 128-entry maxima both apply; the capture budget is unchanged. - Server TypeScript and module-boundary checks pass. The local verification planner selects ten broad application lanes; required CI supplies that wider coverage instead of duplicating it locally. - Reproduce the focused owner suite from `server`: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts --hookTimeout 90000 --maxWorkers 1 --fsModuleCache ``` ## Risks This changes a maintenance trust boundary. A source capture is limited to 128 entries, 16 MiB per file, and 64 MiB total. These limits are intentionally below the Runner's 256 MiB state maximum. Larger valid epochs remain on the original volume and continue blocking filesystem independence. It uses bounded directory reads and file descriptors, rejects aliases and unknown paths, and rechecks bytes before publication. Storage failures retain original files and ledger evidence; they do not settle recovery or authorize a retry. Startup adds bounded database reads for retained historical header bindings or their absence. It adds no source filesystem or storage reads. The selected cohort's serving startup cost still needs measurement before enrollment. Other historical layouts require their own canonical source qualification. No missing provider history is declared recovered, and no automatic corruption repair is added. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source editing, shell tools, and PostgreSQL/filesystem test execution. The runtime does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
1674902b4f |
fix(native): settle a crashed controller's running workspace finalize once a later attempt finishes (#19807)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run saves its workspace in a `workspace_finalize` operation after the agent gives a result > - A crash loop can kill the server after it records this operation as `running` and before it finishes > - The next server resumes finalization and records a new attempt, which succeeds > - Nothing settles the first attempt, so it stays `running` forever and looks like live work > - This pull request settles the old attempt when a later attempt for the same run finishes, under the proof that the old controller is gone > - The benefit is that no workspace operation stays `running` after its owner is dead ## Linked Issues or Issue Description Refs #19666, #19759, #19721, #19717 **What happened?** A control-plane restart chaos test sent SIGKILL to the server while a native run finalized its workspace. The server crash-looped. One boot recorded a `workspace_finalize` operation as `running` and then died. A later boot resumed finalization. Its new operation (with `resumedFromOperationId` set to the old one) succeeded, and the run succeeded. The old operation stayed `running` at rest. **Expected behavior** When a later finalization attempt for the same run finishes, and the owner of the earlier attempt is proven gone, the earlier attempt gets a terminal status with a reason. A live owner keeps its attempt. **Steps to reproduce** 1. Start a native run that finalizes a workspace. 2. Kill the server with SIGKILL after it inserts the `workspace_finalize` row, and before it updates the row. 3. Restart the server and let it finalize the run again. 4. See that the first `workspace_finalize` row stays `running`. **Paperclip version or commit** `ef83d566ca` (also on `27170dca49`). ## What Changed - Root cause: `attemptNativeWorkspaceFinalization` in `server/src/services/native-runtime/native-workspace-finalizer.ts` reads the previous attempt only to set `resumedFromOperationId`. It records the new attempt but never settles the previous `running` attempt. The early "already succeeded" return path also left it alone. Only the recorder that started a row can settle it (`workspace-operations.ts` `recordOperation`), and a killed process never does. - Add `settleSupersededNativeWorkspaceFinalizations` in `native-workspace-finalization-supersession.ts`. It sets earlier `running` native finalize attempts for the run to `failed`. It writes `metadata.supersededFinalization` (reason `superseded_workspace_finalization`, superseding operation id and status), an stderr note, a terminal-workspace wake (registered before the write), and a `workspace_operation.reconciled` activity entry. - The proof is the run's workspace-finalization ownership, not age. The caller holds the advisory lock and the durable owner receipt. That claim requires the previous controller to have released it or to be proven stopped (same-host process proof, platform proof, or a completed copyback barrier). The function also skips an attempt that this process is still running, and attempts with independent obligations (`uncertain`, `saveWork`). It only touches rows that `native_workspace_finalizer` recorded and that are older than the superseding attempt. - The finalizer calls it after a new attempt reaches a terminal status, and on the "already succeeded" path. A settlement error is logged and does not change the finalization result. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-workspace-finalization-supersession.test.ts src/services/native-runtime/native-workspace-finalizer.test.ts` on embedded Postgres: 72 passed. - New tests in `native-workspace-finalization-supersession.test.ts` (a separate file, so `native-workspace-finalizer.test.ts` stays under 1,000 lines): - A `running` finalize from a dead controller plus a resumed attempt that succeeds: the old row becomes `failed` with the supersession reason, no `running` rows remain, and one activity entry exists. - The resumed attempt fails: the old row is still settled, and `supersedingStatus` is `failed`. - A `running` finalize older than an already succeeded attempt: the old row is settled on the next sweep. - While another controller holds ownership, the `running` row stays `running`. - A `running` finalize recorded by another service stays `running`. - Attempts with `saveWork` or `uncertain: true` stay `running`. - Red/green: with the settle call disabled, the three settle tests fail (row stays `running`). With the fix, all pass. - `npx tsc --noEmit -p server` passes. ## Risks - Low. The change only writes rows that are already proven abandoned, and only under the finalization ownership that already fences copyback. - A settled attempt now reads as `failed`. Readers pick the latest attempt by creation time, and the superseding attempt is always later, so the run's finalization outcome does not change. - Existing leaked rows on runs that never finalize again stay `running`. This PR does not add a backfill. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit) and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
bf12b2d8ad |
fix: retry reviewed chat setup after temporary contention (#19787)
Retry reviewed chat preparation with the original accepted input. Bound retries for authorization proof waits that expire before provider work. Preserve the original answer receipt and queue later wakes separately. Keep plan confirmation semantics by assigning the interaction marker only to question answer delivery and legacy question payloads. Verification: 433 tests across four backend suites, all 17 project-chat browser tests, and all server source/test typecheck layers passed. The plan approval regression was reproduced on the previous head before the fix. All applicable CI passed and exact-head Greptile Apex scored 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
ef83d566ca |
fix(native): ask for the result once after a restart continuation ends without one (#19771)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs continue a turn that a control-plane restart lost, in the same conversation (#19501, #19699) > - In restart storms 5, 6b and 8, ACPX Claude runs crossed restarts, ran the continuation, and then failed with `native_session_retry_exhausted` > - The continuation completed, but Claude ended it with "I'll wait for the notification when it completes" for a background command, and did not call `paperclip_finish` > - The next same-run attempt then sent no turn at all and failed with "session returned no semantic result" > - This pull request makes the continuation ask for the result, and asks once more in the same attempt if the continuation still ends without one > - The benefit is that a restart-interrupted task settles without a Retry, and the restarts that the run survived cannot block the request ## Linked Issues or Issue Description Refs #19501, #19699, #19720 **What happened?** Storm 8, run 84c532f7 and its Retry 0faead66 (ACPX Claude, task CHA-5). The steps are the same in both runs: 1. `resume_dead_runner`, then `session.resumed`, then `turn.failed` with `provider_turn_lost_on_restore` (recoverable). 2. The restart continuation runs. Claude starts the bench in the background and ends the turn with "I'll wait for the notification when it completes". The turn completes with no `paperclip_finish`. 3. Lifecycle: "provider turn completed without a semantic result; same-run disposition recovery persisted" (attempt 2). 4. 30 s later, attempt 3 restores the session. It sends no turn. After about 2 s it fails with `native_finalization_missing: session returned no semantic result`, and the attempt count makes it `native_session_retry_exhausted`. **Expected behavior** The continuation tells the agent to finish and report its result. If the continuation still completes without a result, Paperclip asks for the disposition one time. Restarts that the run already survived do not block that request. An agent that never reports a result still fails closed. **Root cause** - `packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts` `startTurn` (lines 224 and 248): the restart continuation uses the driver's one-shot disposition marker (`dispositionOnlyRecoveryAvailable`). #19699 uses the marker on purpose, so that a crash can adopt the accepted continuation turn. The marker then stays consumed with the continuation turn id. - `packages/paperclip-runner/src/drivers/codex/codex-session-terminal.ts` (line 226): only a *lost* continuation gives the marker back. A *completed* continuation with no result does not. - `packages/paperclip-runner/src/native-session-runtime.ts` (`dispositionRecoveryWasSubmitted`, about line 2761): on the next attempt, the consumed marker looks like a disposition request that already ran. Recovery replays the continuation's terminal and starts no turn. So the disposition request never happens. - `server/src/services/native-runtime/native-session-executor.ts` (lines 9755-9761 and `nativeSessionFailureDisposition` at 7768): the `semantic_result_missing` same-run retry charges a provider attempt on the same counter as restart resumes. In this run, attempt 3 was the last allowed attempt. - The continuation prompt (`restartContinuation`) did not tell the agent to report a result, or not to stop and wait for a background notification. ## What Changed - Codex-protocol driver (used by Codex and by ACPX through runnerd): when a restart continuation completes without a semantic result, the driver gives back the one-shot disposition marker. The disposition request is not a continuation, so it consumes the marker for good. - Native session runtime: after that, the same attempt sends one disposition request (`Paperclip semantic-result recovery after a restart continuation ... invoke exactly one paperclip_finish or paperclip_block`). It does this only when the driver shows that the continuation borrowed the marker and gave it back. Other drivers keep the current behavior. - Restart continuation prompt: it now says to finish in this turn, not to end the turn to wait for a background command or a notification, and to report one `paperclip_finish` or `paperclip_block`. - `doc/execution-semantics.md`: one additive paragraph. - No server change. The disposition request runs in the same attempt, so it uses no provider attempt. This avoids `native-restart-recovery.ts` (#19747). ## Verification Red before the fix, green after: - `codex-app-server-driver.recovery.test.ts` "keeps one disposition request after a restart continuation completes without a result". Red: marker stays consumed on `turn-2`. Green: the marker is given back, one disposition turn starts, and a resultless disposition turn does not re-arm. - `native-session-runtime.test.ts` "settles a restart continuation that ended without a result through one disposition turn (Codex driver)". Real Codex-protocol driver with a fake transport. Red: only one `turn/start`, then `native_finalization_missing`. Green: the disposition turn `turn-3` reports the result and the run completes. - `native-session-runtime.test.ts` "asks once for the disposition when a restart continuation ends without a result" and "fails closed when the driver keeps the continuation's disposition marker". Also run: - `npx vitest run src/native-session-runtime.test.ts src/drivers/codex/` in `packages/paperclip-runner`: 593 passed. - `src/live/runnerd-restart-continuation.test.ts` (real runnerd and fake Codex, killed runner): 5 passed. - `tsc --noEmit` for `packages/paperclip-runner`. - `pnpm verify` exits 2 because the doc change is a global CI input. Required CI supplies the broad coverage. Not run: a live replay of storm 8 t05 with Claude. The chaos harness needs provider keys from the operator's shell. ## Risks - Low to medium. A run whose continuation ends without a result now gets one more provider turn in the same attempt. The bound is one disposition request per continuation. Each continuation still needs an admitted restart, and the provider-attempt budget bounds those. - If the controller crashes after the continuation completes but before its checkpoint, the checkpoint keeps the stricter consumed marker. That case fails closed as before. - The prompt text changes for restart continuations. It does not change a protocol schema or a profile digest. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6adc8b1049 |
fix(recovery): keep restarted Retry attempts as task directories and settle failed re-preparation (#19759)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task recovery lets a user click Retry after a run fails, and a native run restarts its own preparation after a control-plane restart > - A Retry for a task without a project runs in a fresh task directory under `isolated-workspace-attempts` > - When a restart hit that Retry during bootstrap, preparing it again treated the task directory as the base of a new git worktree and failed > - The failed run then kept its native recovery claim open, so the task showed Retry but every Retry click created no run > - This pull request keeps the restarted attempt in its task directory and settles a failed re-preparation > - The benefit is that a Retry finishes after a restart, and a failed one can be retried again ## Linked Issues or Issue Description Refs #19666 **What happened?** A task without a project failed after a restart (`native_session_interrupted`). The user clicked Retry. The Retry run started in a fresh task directory. A restart hit it during bootstrap. Restart recovery prepared the same run again and failed with `workspace_validation_failed`: "requested isolated_workspace with git_worktree, but base workspace `.../isolated-workspace-attempts/<company>/<issue>/<run>` is not a git checkout". The task rested blocked with Retry shown. A later Retry click created no run: the receipt stayed deferred with "New work must wait while a native run recovers its unfinished accounting." A second storm wave showed the same failure on a Claude C1 task. **Expected behavior** Preparing a run again after a restart reuses the task directory that run already bound. A git worktree base is only the project's checkout or the task's own worktree. A run whose re-preparation fails leaves a working Retry. **Paperclip version or commit** `master` at `51eb4ff705` and `131b566ca4` (both include #19666). Restart-chaos waves on 2026-11-07. **Steps to reproduce** 1. Use a task with no project and isolated workspaces enabled. Let a native run fail after a restart. 2. Click Retry. The run starts in `isolated-workspace-attempts/<company>/<issue>/<run>`. 3. Restart the control plane while the run bootstraps. 4. The re-preparation fails with `git_worktree_base_not_git_checkout`. Click Retry again: no run starts. Root cause: - `server/src/services/fresh-workspace-continuation.ts` `readFreshWorkspaceReadiness` returns `null` for a native run that already has admitted input and a bound workspace (from #19693). This is correct: the restart must resume the admitted workspace. - `server/src/services/heartbeat/environment-selection.ts` (`boundTaskDirectory`) excluded every native recovery binding from the task-directory path. With readiness no longer fresh, `useIsolatedTaskDirectory` was false. The default `isolated_workspace` strategy (`git_worktree`) applied, with the task directory as its base. - The restart recovery claim set the native coordinator to `observed` with its lease owner. `failRunSetup` failed the run, but nothing released that claim. `budgets.ts` reads an open coordinator as unfinished accounting, and Retry admission reads it as `controller_settling`. ## What Changed - Environment selection treats a native recovery binding as the task directory when it is the task's current binding and a bound isolated task directory. The directory strategy is kept; it never becomes a git worktree base. A recovery binding the issue no longer points to keeps its existing path. - New `settleFailedNativeRecoveryPreparation`. When a `bootstrap_incomplete` or `resume_dead_runner` restart claim fails its re-preparation before any provider dispatch, it settles the claimed coordinator as `terminal_failure` and wakes deferred work. It only changes a coordinator that is still `observed`, owned by that claim, without a result, for a failed run. Reattach claims (live runner) are not changed. - `doc/task-recovery.md` describes both rules. ## Verification - `environment-selection.test.ts`, RC-58 cases: a restarted fresh attempt bound to the task re-prepares as its task directory (`useIsolatedTaskDirectory` true). It failed before the fix and passes now. A binding the issue moved away from keeps the old behavior. - `native-runtime/native-recovery-preparation-failure.test.ts` (embedded Postgres): before settlement, Retry admission reports `controller_settling` (the dead Retry). After settlement the coordinator is `terminal_failure` with the run's error code, and admission no longer waits on the controller. Settlement is idempotent. Another owner, a reattach claim, a run that is not failed, and a coordinator finalizing elsewhere are left unchanged. - Local regression: `environment-selection`, `workspace-preparation`, `retry-workspace-continuity`, `native-recovery-preparation-failure`, `explicit-native-continuation` (588 tests passed). - Restart-chaos: local restart-chaos runs on `3a7abef4e5` passed, but neither one reproduced the exact failure (a restart during the Retry's bootstrap). The race depends on timing, so the unit tests above are the red/green proof for both defects. - `--replay w-20261107T011010-s7403/t02` (codex C1 K4 with the original fault sequence): PASS, R1-R6 and T. The replay's Retry did not cross a bootstrap restart. - Wave `w-20261107T064432-s7210` (codex+claude, C1+B1, K1/K4/K6, `--triggers retry`, concurrency 2): 8/8 PASS. It had 4 Retries, including C1 tasks with a fault fired right after the Retry click. No `git_worktree_base_not_git_checkout` and no dead Retry. The server log has no "incomplete native runner bootstrap" restart in this wave either. - `ce7e3f9b19` registers execution work before the settlement writes (Greptile finding). ## Risks - A native recovery now re-prepares a bound task directory through the task-directory path. That path still validates the binding, owner, status and directory identity before use. - Settling a failed re-preparation ends that run's recovery claim. It applies only when the previous runner is verified gone (`bootstrap_incomplete`, `resume_dead_runner`) and no provider dispatch started in this attempt. A Retry still passes its normal stop, ownership and admission checks. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
909f4d1a33 |
fix(native): recover exact historical provider pack code (#19779)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native sessions retain the provider pack that admitted their saved
state.
> - An image update can replace that pack's executable, proxy, or
contract.
> - Older sessions can predate the durable three-file launch artifact
receipt.
> - The current image's archive cannot satisfy a different pinned
manifest.
> - This pull request obtains the exact public code from its recorded
release image.
> - Full source, archive, and installed-file checks still control
provider admission.
## Linked Issues or Issue Description
Refs #19558, #19622, #19755.
A saved OpenCode session can fail with
`runner_remote_provider_archive_invalid` after an image update. Its
original provider state and checkpoint remain valid, but its pinned pack
differs from the controller's pack. The old manifest can name a release
that still has the exact code in GHCR. Matching the OpenCode version
does not prove matching proxy or contract bytes.
## What Changed
- Resolve a pinned historical pack from the recorded source-SHA image in
the public Paperclip GHCR repository. Verify the platform manifest, raw
config, source identity, dedicated code layer, selected pack manifest,
archive contents, paths, and hashes.
- Keep a private temporary cache of verified public code. Rebuild it on
an empty host. Bound each source to 4 GiB and three minutes, with two
active imports or cache verifications, 32 distinct queued sources, and
disk-headroom checks. Share warm-cache verification by manifest while
each caller keeps its own cancellation.
- Select this verified archive through the existing uploader. Preserve
installed-pack and preinstalled-pack checks. Refuse unavailable
historical transport before clearing the target. Keep the current-source
identity guard from #19755. Refuse an oversized layer argument before
any target effect.
- Attach the source wait to the actual per-turn startup. Stop, lease
loss, and startup completion cancel it. Keep existing Runner admission
and turn limits.
- Document the registry retention requirement and the distinction
between public code recovery and encrypted provider-state backup.
## Verification
- Genuine baseline RED: the real uploader rejects the selected old pack
before any remote effect. Treatment installs the exact old code through
that uploader.
- Current head: all 19 source/integrity/cache/deadline controls, all 45
archive controls, and 29 actual-caller and manifest/startup controls
pass (93 total). Four review regressions fail against the prior head and
pass with the fixes. The server and colocated test TypeScript graph
passes. Module-boundary and whitespace checks pass.
- Actual public GHCR cold reconstruction: 1,054,587,008 compressed code
bytes, 115.4 s, about 248 MiB maximum owner RSS. All 114 file layers and
the overlay passed source validation.
- Linux Node 24.22.2 fixture: the real historical pack passes the normal
uploader, full installed-file verifier, and pinned OpenCode version
check. It used one transfer, no application startup, no network, and no
provider calls. Revision12b0d2ba passed in 220.2 s with about 282 MiB
owner RSS. The earlier prototype passed in 108.5 s; both measurements
are retained. The three-minute source limit and the fixture's
four-minute limit were unchanged. The later review fixes add shared
cache verification, hash cancellation, and an early size guard. They do
not change extraction or the accepted Linux fixture input.
- Earlier fixture failures are retained: unsafe hardlink classification
was corrected with confined-link controls; a cached image lacked the
required zstd CLI; Docker Desktop bind storage caused GNU tar
directory-identity errors. The passing fixture uses the official Debian
zstd binary and an owned Linux tmpfs target. Production extraction code
is unchanged.
- Fresh Greptile review is 5/5 on
|
||
|
DottaandPaperclip
|
ae2d2956b3 |
Reject relative local Runner paths before native admission (#19776)
## Thinking Path > - Paperclip manages AI agents and their work. > - Heartbeat selects the runtime that will execute a task. > - A new local Native run uses the controller's Runner binary setting. > - A relative binary path is invalid. The resolver already rejects it, but that check occurs after Native admission. > - This pull request checks the same setting before it creates Native execution authority. > - Remote providers and admitted Native recovery keep their existing behavior. ## Linked Issues or Issue Description **What happened?** A relative `PAPERCLIP_RUNNER_BINARY` setting can reach local Native admission before the binary resolver rejects it. The run has transferred preparation ownership even though this setting cannot start the local Runner. **Expected behavior** Reject this invalid setting before creating a completion contract or Native coordinator. Keep the existing preparation owner and the run row unchanged. **Steps to reproduce** Set `PAPERCLIP_RUNNER_BINARY=relative-runner`. Select a new local `paperclip_runner` Native run. The selection must report the existing absolute-path error before Native admission. Refs: #19466 and #19340. Those merged changes repair Retry authority and saved-workspace recovery. This change only moves local path validation before new admission. ## What Changed - Add a pure absolute-path check beside the existing binary resolver. Both use the actual controller environment setting. - Run that check before a new local Runner admission. Skip remote execution, API-only providers, and an already admitted Native run. - Test the rejection, unchanged preparation ownership, controller setting selection, remote behavior, and immutable recovery. Keep the existing resolver checks. ## Verification - The full binary test file passed: 32 tests. - The full runtime-selection file passed against a dedicated PostgreSQL database: 62 tests. It exercises real admission transactions, contracts, coordinator rows, and instruction-copy controls. It replaces context materialization and remote workspace synchronization at their existing test boundaries. - After normal integration with current master, the normal Shared/SDK, database, eval-kernel and Runner builds passed, together with all seven normal application TypeScript programs. The original recovery-preparation case passed again on this integrated revision against a fresh dedicated database (1 passed; 61 filtered). The earlier full 32-test binary and 62-test selection results carry only where their source and executed boundaries are unchanged. - The first private selection wrapper failed during module resolution, before any case or database use. A TypeScript wrapper with the same body recovered that setup failure. This was not a product regression. - Reproduce with `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-codex-runner.test.ts --maxWorkers=1 --no-file-parallelism`. - Set `PAPERCLIP_RUNTIME_SELECTION_TEST_DATABASE_URL` to a dedicated disposable test database before running `pnpm --filter @paperclipai/server exec vitest run src/services/heartbeat/runtime-selection.test.ts --maxWorkers=1 --no-file-parallelism`. ## Risks - This check validates path syntax only. An absolute path can still fail the existing file-access or artifact checks later. - A wrong branch condition could reject remote or existing Native work. The tests cover these exclusions. - This does not prove that a real provider can start or that a deployed service is healthy. It changes no database schema or retry authority. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with source inspection and code-execution tools. The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
2d4f80504e |
fix(runner): retain exact ACPX checkpoints after rejected turn starts (#19762)
Preserve the exact idle ACPX conversation when a matching turn.start response rejects work with retryable SESSION_RESUME_REQUIRED. Require verified identity, no pending work, confirmed process cleanup and the provider lifetime fence before normal drain and suspension can publish a reusable checkpoint. Unknown acceptance and arbitrary closed states remain closed. The real Runner regression completes the next turn on attempts 2 and 3, requires successful terminal status for the exact submitted turn, and checks expectedIdentity on both resume requests. Controlled wire faults reject failed, cancelled and missing-identity recovery. Validation: 193 affected Rust tests on unchanged Rust source/fixtures; 134 recovery integration tests, 14 environment-gated skips; 21 Claude effort tests; all server typecheck layers. Exact-head CI: 59 successful checks and statuses, 6 intentional skips. Greptile Apex 5/5, no unresolved findings. Hosted deployment and task acceptance remain separate. Fixes: #19751 Refs: #19366 Refs: #19745 Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
beb3e55072 |
Continue a released Runner after confirmed graceful eviction (#19760)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core can release a completed Runner while its host retains the session. > - A host can later retire that Runner under its retention policy. > - Graceful retirement saves a suspended session before physical shutdown. > - A cached Core owner still expects the old physical task and rejects a new allocation. > - This pull request retires that dead local controller after verified stop. > - The benefit is ordinary session continuation for Codex and qualified Claude. ## Linked Issues or Issue Description Refs #19611, #19683, #19196, #19733 and #19157. After a host gracefully evicts a released task, the next run can receive a new allocation. A cached controller rejects it with `native_session_released_owner_scope_mismatch`. The detached-controller path can already recover the suspended session. ## What Changed - Require the exact source, old task and current account/workspace scope before releasing a cached controller. - Use the existing physical-stop receipt checks. Detach only the local controller, then use the same suspended-state admission as the detached path. - Preserve the current run's instruction reservation. Retain a failed detach for cleanup. - Add actual Core, PostgreSQL and native Runner tests for Codex and qualified Claude through ACPX. Route them once through the native CI lane. ## Verification - The original real-entry Codex case failed at the cached-task scope check after native retirement, confirmed process exit and a recorded stop receipt. - The current head's native CI lane passed all 16 eviction cases for Codex and qualified Claude. These check cached and detached owners, current instruction preparation, reused-allocation inspection, missing stop/state, foreign identity and non-suspended state. They verify the same provider session, two ordinary turns, one accepted source result and no replacement submit on refusal. - Local checks passed the composition matrix, 12 targeted executor controls, 43 CI lane-selection checks and server TypeScript. All eight refusal cases now assert the expected error and whether the checkpoint request or host read was reached. This addresses the first standard review's test finding. - Commands: `pnpm exec vitest run src/services/native-runtime/plugin-runner-eviction.integration.test.ts`, focused `native-session-executor.test.ts` controls, `node ../node_modules/typescript/bin/tsc --noEmit -p tsconfig.json`, and `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/42349579107) passed on head `9a25709ef43365b1f1849413b935785838eca19b`. [Standard review](https://github.com/paperclipai/paperclip/pull/19760#issuecomment-6703629255) is 5/5 on this head, with no unresolved findings. Whitespace and added-line secret/PII checks passed. ## Risks - Controlled provider processes and an authenticated local host fixture model the Launcher edge. This does not qualify live vendor transcript storage, mounts or a deployment. - This covers graceful eviction before allocation or before stopped-task inspection during `describe`. Loss after that inspection or an ambiguous claim remains held for reconciliation. No unavailable reply proves physical stop. - Missing or invalid suspended state still refuses continuation. The deterministic refusal rule from #19733 remains unchanged. Abrupt death without a suspended checkpoint stays with the separate #19157 work. - No new capability, mode, schema, dependency, deployment or activation is added. The existing draft #19157 and dependent branches remain unchanged. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with repository tools and code execution. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
3895195f4b |
fix(native): report a retained provider pack the controller does not ship as incompatible (#19755)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote OpenCode runs stage a controller-owned provider pack into the sandbox before the provider starts > - Since #19558, an ordinary OpenCode continuation retains the pack that an earlier release admitted for its saved provider session > - After a controller release changes the pack, the controller no longer ships those bytes, and the continuation must stay blocked > - The configured-pack upload path did not detect that case: it read the build-time archive receipt for the old digest and failed with `runner_remote_provider_archive_invalid`, a generic, retryable error that suggests corrupt image files > - This pull request compares the configured pack's identity with the selected identity before any upload and fails as `runner_remote_provider_artifact_incompatible` > - It also puts a fixed reason in each real archive integrity error > - The benefit is a correct, permanent classification that matches the bundled-image path, no wasted retries, and a diagnosable message for real archive corruption ## Linked Issues or Issue Description Refs #19558, #19743 **What happened?** On a hosted deployment, two remote OpenCode runs of a review agent failed before the provider process started. The error was `runner_remote_provider_archive_invalid` (classified as a generic `native_session_interrupted`). Both runs continued an issue whose provider session was admitted by an earlier release. In both runs, the run log shows `provider_pack.compatibility`, `provider_pack.verify`, and `provider_pack.verify_preinstalled` fail, then `provider_pack.archive.verify` fails after 4–6 ms. Fresh runs of the same agent on the same controller passed pack preparation with the current pack. The cause is not a cache race. The archives are created at image build time (`Dockerfile` `runner-provider-pack` stage) and the server never rewrites them. The retained authority selects the old pack digest. `readProviderPackLayers` then compares the receipt's `providerPackDigest` (current release) with that old digest and throws `runner_remote_provider_archive_invalid`. Directory staging (no archive) instead clears the staged remote pack, uploads the different pack, and fails only at the final verification. **Expected behavior** A continuation pinned to a pack that the controller does not ship fails as `runner_remote_provider_artifact_incompatible`, as the bundled-image path already does ("the image provider pack did not match the bundled release identity"). It does not clear or upload anything. Real archive corruption keeps `runner_remote_provider_archive_invalid` with a fixed reason that the run report shows. **Steps to reproduce** 1. Build a provider pack and its archives for digest A. 2. Call `uploadVerifiedProviderPack` with that pack and a selected (retained) digest B. 3. Before this change: the call fails with `runner_remote_provider_archive_invalid` (v1 and v2 archives), or resolves after it clears and re-uploads the remote pack (directory source). ## What Changed - `uploadVerifiedProviderPack` takes the configured pack's `sourceDigest`. If it is not the selected `expectedDigest`, the call fails with `runner_remote_provider_artifact_incompatible: selected provider pack <12 hex> is not the controller's pack <12 hex>; ...`. The check runs before the remote clear, the archive read, the directory hash and any upload. - The executor passes the configured pack's identity (`readRemoteProviderPackIdentity(root, false)`, the same reader that selects the current pack). - Archive integrity errors keep the `runner_remote_provider_archive_invalid` prefix and add a fixed reason: `receipt_unreadable`, `receipt_identity_mismatch`, `receipt_invalid`, `layer_invalid`, `layers_unreadable`, `archive_unreadable`, `archive_not_file`, `size_mismatch` or `sha256_mismatch`. The run report keeps only the outer message, so the reason must be in the message. Filesystem causes stay on `cause` and do not put private paths in the message. - The v1 reader now uses the shared `verifyArchive` helper. - `doc/execution-semantics.md` documents the rule. ## Verification - Red before the fix (production files unchanged, new tests only): `npx vitest run src/services/native-runtime/provider-pack-archive.test.ts` gave 5 failed / 38 passed. The v1 and v2 cases failed with exactly `runner_remote_provider_archive_invalid`, which is the hosted error. The directory case resolved after it cleared and uploaded the remote pack. The reason tests got a bare `runner_remote_provider_archive_invalid`. - Green after the fix: the same file gave 43 passed. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1245 passed. This includes the #19558 continuation cases. - `remote-provider-pack`, `remote-provider-pack-authority` and `remote-provider-pack-verification` tests: 69 passed. - `pnpm --filter @paperclipai/server typecheck`: all layers passed. - Review follow-up (`c8156841a9`): an unreadable numbered layer now reports `archive_unreadable`. The layered-corruption test deletes `0.tar.zst` and checks for that reason before any transfer. The archive file gave 43 passed. - Hosted evidence came from read-only run, event and issue-run reads only. No hosted retries, recovery or deploys. ## Risks - This change does not unblock an affected session. It makes the failure honest and permanent: `runner_remote_provider_artifact_incompatible` is already a permanent, pre-session failure code, so the run is not retried with the same result. To recover, the session needs its original artifacts (the captured provider-artifact path, which #19743 repairs for new admissions) or an authorized continuity break. - Low risk for other paths. Fresh launches select the configured pack's own identity, so the digests are equal. Restored provider artifacts and compatible installed packs return before this upload path. - Callers that match `runner_remote_provider_archive_invalid` with a substring still match. No code compares the exact message. - Not tested end to end against a hosted sandbox. The executor-level wiring of `sourceDigest` has typecheck coverage but no dedicated executor test. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic tool use with shell, code editing, test execution and read-only authenticated browser diagnostics. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
167e39c4ef | fix(runner): keep warm run.attach events on ordinary accounting (#19707) | ||
|
DottaandClaude Opus 5.5
|
6d112c74cf |
fix(native): do not charge a restart resume that never started the provider (#19747)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs have an execution recovery budget of three provider attempts, so a run that keeps failing stops retrying > - #19720 stopped charging a crash restart when the previous restart resume made progress > - In a crash loop (several kills within seconds), the controller can die again before the resumed execution starts its runner > - The next claim then charged that resume, although it used no provider work > - This pull request does not charge a resume that never started; the next claim replaces it > - The benefit is that fast restart bursts do not exhaust a healthy run's budget, and the generation cap still bounds real loops ## Linked Issues or Issue Description Refs #19720, #19625 **What happened?** In a restart-chaos storm wave, 8 runs still reached `execution_recovery_budget_exhausted`, mostly in K6 crash loops (three kills in about 20 seconds). Example: a Codex run was claimed at generation 9, attempt 3. The controller was killed 6 seconds later, before the executor recorded `native.process_start_requested`. The next claim found no progress after generation 9's position, charged, and blocked the run at attempt 3. The recorded history of the blocked runs shows that almost every charge after the first restart was for a resume with no start evidence (no runner request and no runner event) before the next claim. Two runs were blocked at generation 12, the existing cap. **Expected behavior** A resume that never started the provider consumed nothing and must not be charged. A resume that started and then made no progress must still be charged. The generation-12 cap stays. **Steps to reproduce** Start a native run. Kill the server during the run. Let a restart resume do work. Then kill the server repeatedly within seconds, before each new controller starts the runner. After one or two such bursts the run is blocked with `execution_recovery_budget_exhausted`. ## What Changed - `native-restart-recovery.ts`: `restartResumeKeepsAttempt()` replaces `previousResumeMadeProgress()`. It reads the previous claim's recorded `progressWatermarkSeq` and `progressAttempt` (`lastRecoveryClaimResume()`). - New `nativeExecutionStartedSince()`: after the watermark, it looks for `native.process_start_requested`, `native.process_identity_recorded`, `adapter.invoke`, `tool.execution.started`, or any event with a runner `source_event_id`. This is the same start evidence as `hasNativeExecutionStartEvidence()`, limited to events after the watermark. - When the previous resume never started, the claim keeps the attempt if the run is already at the attempt that resume was to run as. If the executor never applied that charge (killed before its execution lease), the claim charges once, so the run reaches that attempt and never goes above it. - When the previous resume started, the #19720 progress rule applies unchanged. - The generation-12 cap applies to both rules. - `doc/execution-semantics.md` states the rule. ## Verification New tests in `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` (embedded Postgres, real `claimNativeRestartRecoveries`): - `does not charge for a resume killed before it started the provider (after the executor took its lease)`: one progress resume, then four crash claims with no start evidence. The attempt stays 2. A resume that starts and fails then charges to 3. Red on master (charged to 3, then blocked). - `... (before the executor took its lease)`: the same with the lease never taken. Green on master and with the fix, as a guard. - `charges the owed attempt once when an unstarted first resume is replaced`: the replacement claim charges once (attempt 2). - `bounds a crash loop that never starts the provider`: blocks after the generation cap. Red on master (blocked at the third claim). - The #19720 tests now record `native.process_start_requested` in their resumes, as a real resume does. - `native-runner-restart-recovery.integration.test.ts`, `heartbeat-native-restart-suspension.test.ts`, `native-restart-recovery-isolation.test.ts`, `native-remote-retry-admission.test.ts`: 231 passed, 14 skipped. - `tsc --noEmit` for `server` has no errors in the changed files. ## Risks - A crash that happens before the runner start, every time, gets free resumes until generation 12. Then the budget applies. This is the same bound as the progress rule. - Start evidence is the executor's durable `native.process_start_requested` record, written before any backend may spawn a runner. If an older path launched a provider without it, that resume would be treated as unstarted until the generation cap. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
f673bd3cb5 |
fix(native): verify legacy harness filesystem independence (#19748)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native sessions retain provider checkpoints and recovery state across turns. > - Server replacement must preserve every remaining local data dependency. > - A legacy backup stamp can outlive a destroyed provider and an absent local backup. > - Surviving skill files can already have exact immutable package copies. > - Raw controller files also contain stream events that database checkpoints omit. > - This pull request records verified filesystem independence under the original-volume maintenance owner. > - Replacement startup checks durable bindings while existing resume failures and policies remain intact. ## Linked Issues or Issue Description Refs #19715, #19729, #19733. **What happened?** Startup treats every retained legacy backup stamp without a current backup row as a local filesystem dependency. This also blocks sessions whose provider was already destroyed and whose remaining local skill bytes have exact durable copies. **Expected behavior** An authorized maintenance owner can prove that an exact retained source has no uncovered local harness payload. Startup can use that proof after replacement. The proof must preserve failed recovery state and must not invent a provider backup. **Steps to reproduce** 1. Retain a terminal native session, its checkpoint, and its legacy backup stamp. 2. Record exact canonical destroyed-provider receipts. Keep only admitted immutable skill files on the original host. 3. Publish the owner-verified independence receipt. The old startup predicate still blocks; the updated predicate validates its durable bindings. ## What Changed - Add an internal original-volume capability. It requires a private nonce anchor, independently reviewed attachment evidence, current reserved runtime identity, and a witness that is at most 120 seconds old. - Verify retained and selected-source inputs, checkpoints, leases, destroyed-provider receipts, released ownership, and exact immutable skill references. - Inspect canonical persistence roots and compare surviving skill bytes and modes. Refuse any provider directory present outside the current persistence profile, including an old Codex home after a preconfigured-home switch. Unknown data, backups, history, owners, or source changes still block publication. - Preserve raw controller and supervisor bytes through the existing encrypted evidence archive owner. Match recovery records to the database checkpoint, authenticate the exact archive row, and compare the original bytes again before publication. The archive grants no executable authority. - Publish one audited receipt with source locks and a final comparison. Keep original files, stamps, warnings, statuses, and session pointers. A late capture retains a valid published winner and refuses an invalid receipt. - Validate published receipts from current database bindings on replacement hosts. Add no startup filesystem probe, object read, provider call, or recurring worker. ## Verification - Genuine baseline regression: restore the original startup predicate while keeping the actual receipt owner. The positive PostgreSQL/filesystem case fails with expected `false`, actual `true` after valid publication. Restore the changed predicate; the case passes. - Final rebased head `e78f4b4424`: 42 affected PostgreSQL/filesystem cases pass. This includes 32 owner guards, three paired continuation cases, six existing failed-source archive controls, and one existing lease-verified backup migration control. The other 113 tests in the selected existing files were explicitly unselected. - Genuine controller baseline: the unchanged prior owner publishes no raw archive and accepts missing checkpoint parts. Both new regressions fail. The repair passes encrypted byte-for-byte controller/supervisor restore, missing checkpoint, foreign source, active source, and changed-after-upload controls. - Genuine omitted-provider baseline: the unchanged owner accepts old Codex, OpenCode, and ACPX history after a preconfigured-home switch. All three refusal cases fail. The fix passes those cases and permits preconfigured execution with no omitted directory. - Genuine late-publication baseline: both concurrent winner tests fail before the repair. The repair keeps the existing valid receipt, refuses an invalid one, and keeps one audit event. A delayed raw upload also retains the first authenticated archive and its receipt. - Capability controls: 12 passed, including empty replacement refusal, stale or future evidence, changed deployment, missing volume, path aliases, changed anchor, and current remount identity. - The actual server-to-Runner boundary preserves explicit recovery policy on original and empty homes. The separate actual CREATED-sandbox preparation callback preserves checkpoint identity and policy without a backup. Provider I/O and final transport construction are controlled. - Warm sessions retain the same canonical refusal on both homes, make no backend/provider attempt, and retain the saved checkpoints. - Server TypeScript, module boundary checks, diff checks, and a private-data scan pass. The Runner TypeScript build was refreshed after rebase to include two new upstream exports. - Fresh Greptile review is 5/5 with no unresolved threads on `e78f4b44240b0fc33c2fd7cae2d5b4a2578e3153`; check `128330103712` completed successfully. Required CI remains pending. No live preservation or enrollment action ran. Focused command from `server/`: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts src/services/native-runtime/native-harness-filesystem-independence-continuation.integration.test.ts src/services/native-runtime/native-controller-migration.integration.test.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts --testNamePattern 'owner-verified|legacy harness continuation|preserves a failed source online|migrates a lease-verified' --hookTimeout 90000 --maxWorkers 1 --fsModuleCache ``` ## Risks - This proves that no uncovered harness payload remains local. It does not recover provider history or certify that no other copy exists. Existing governed-wait and warm-session failures remain. - Unknown providers, retained owners, extra payloads, missing references, or changed source bindings fail closed. Invalid receipts require investigation; this change does not repair or replace them automatically. - The maintenance capability requires independently reviewed Railway attachment evidence. The original anchor remains stable across normal releases; each new staging invocation needs current deployment and mount evidence. - Startup validation adds bounded per-scope database reads. It pages candidate scopes and caps each source history at 500 rows. Measure the affected cohort's actual startup latency on the qualified serving build before enrollment. - Raw archives use the existing backup ledger, encryption, retention, and restore machinery. They preserve evidence only. They do not repair missing provider history or grant a new resume policy. - Local tests control provider and transport boundaries. Live staging remains a separate step after review, merge, normal delivery, and a fresh original-volume capability. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and local code/test execution. The exact runtime variant and context-window limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
96bc021edb |
fix(server): certify settled legacy processes for future retries (#19377)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Bounded retries must preserve current authority, budgets and deadlines. > - The tenant can wait for supported future work after the source process and its cleanup stop. > - Retained process identity alone cannot prove that cleanup finished. > - This change records host-owned settlement and checks its exact identity before a retry can cover its wait. > - Uncertain process state, changed authority, live claims and cleanup debt keep the tenant awake. ## Linked Issues or Issue Description Refs #19001. Related dependencies: #19019 and #18988. Related #19340 covers explicit Native task Retry; this change covers future Legacy process retries. Current integrated source: `1645970a9322d1213ba25fc7b52366c285682977`, normally merged with actual master `131b566ca4fa866c4348739c75cfe48c805b3398`. The prerequisites above are merged. This PR targets `master`. **What happened?** A future issue, company or review retry can retain its failed source process identity. The deadline reader could not prove that host cleanup finished. Supported waits stayed blocked. Ordinary issue retries also needed an exact source-settlement check, and represented coalesced requests needed exact input coverage. **Expected behavior** A supported future retry covers only its existing wait, under current company and actor authority, after exact host settlement. Missing, malformed, stale, cancelled, Native or unfinished source state stays blocked. No retry gains new execution authority. **Steps to reproduce** Fail an ordinary process or fail environment preparation before adapter entry. Let normal cleanup schedule the existing retry. Inspect its settlement and deadline, stop and reconstruct the owner, and admit one useful successor at the saved deadline. Hold cleanup or change authority to verify that the reader refuses coverage. ## What Changed - Add nullable internal `legacyProcessSettlement` state through normally generated migration `0439_cultured_impossible_man`. No backfill or adapter result creates the record. - Write exact host settlement after stopped-process and cleanup checks. Compare the current run and reject live claims or cleanup debt. Optional persistence failure preserves normal cleanup and keeps sleep coverage blocked. - Add a separate preparation receipt for a claimed Legacy run that failed before any adapter or Native dispatch. Require the same boot, run times, preparing stage, released claim and completed cleanup. A null PID alone proves nothing. - Clear settlement on a new claim, process metadata or Native selection. Omit the internal record from public serialized data. - Check exact source lineage and current authority for ordinary issue, company and taskless retries. Cover completed coalesced requests only when the admitted retry represents their exact inputs. - Add the closed `current_review_retry` wait with current reviewer, stage and existing issue authority checks. Require a nonempty current stage ID. - Add actual producer, cold reconstruction, SQL negative, redaction and optional-persistence controls. - Repair CI fixture expectations for the additive column, startup mock, budget owner and popup return URL. Bind the controlled Native comment server explicitly to loopback. - Replace temporary Slack clock logging with an exact persisted-deadline assertion. Wait until that real deadline before the existing single default poll. Keep both forced polls, business assertions and timing limits. ## Verification - On final integrated source `1645970a`, normal Shared/SDK prerequisites, Runner TypeScript and the entire normal App layered typecheck pass. Tracked source and lockfile remain unchanged. - All 26 whole-file settlement, recovery-reader and bootstrap controls pass. Seven separate real PostgreSQL producer/negative cases pass at `cf16c6a`; each has one selected passing case and 18 filtered registrations. Filtered registrations are not acceptance. - Master then changed the preparation source-lock scope. Its three intersecting cases were rerun on final `1645970a`: an actual failed company process, an actual failed ordinary issue process, and held preparation cleanup. Each preserves its source identity and original deadline and admits one useful successor. Issue and preparation cases exercise normal automatic retry; the company case uses the normal explicit bounded-retry producer. - The unchanged 26 controls and four other PostgreSQL cases retain their `cf16c6a` proof after a scoped source comparison. Optional settlement/preparation persistence failures keep certification blocked while normal cleanup completes. An entered adapter without process identity stays uncertified. This is scoped carry, not a claim that every earlier case ran again on the final head. - All recorded test owners and controlled groups closed. Final read-only SQL audits returned no test sessions. One earlier postcase resource barrier observed a transient unattributed database backend; that database and original evidence remain preserved. Its later disappearance is recovery, not an explained cause. No assertion, deadline or test body was relaxed. - Migration0439 was generated normally after master history. All executable SQL and journal history are preserved; the five retained snapshots are0435–0439. Every Legacy-owned patch and unrelated actual-master blob/mode is preserved. - Published exact head `1645970a9322d1213ba25fc7b52366c285682977` passes all current required checks: 68 successful contexts and eight expected skips after exhaustive pagination and superseded-run accounting. The fresh complete Greptile review scores this exact head 5/5. All four inline review threads are resolved. The latest Ponytail turn stopped without a verdict; one normal retry was requested, with no active turn or new finding observed. No old-head score is used. The original old-base policy failure, its dependent aggregate failure, and the corrected description-gate failure remain recorded. No full local repository suite is claimed; remote typecheck, test and build gates pass. To repeat the real database coverage, set both `PAPERCLIP_RECOVERY_DEADLINES_TEST_DATABASE_URL` and `PAPERCLIP_RECOVERY_DEADLINES_TEST_ADMIN_DATABASE_URL` to the same fresh dedicated test database and run the selected `recovery-deadline-acceptance.test.ts` cases with one worker. An unset URL, skipped case or filtered registration is not acceptance. The fixtures use controlled local processes and real PostgreSQL. Reconstruction creates new service owners inside the same test process. They do not prove deployed Cloud alarms, physical host/database suspension, new-host recovery or Native capacity settlement. The original30-second retry remains below the Cloud270-second minimum wake lead. The merge projection also includes actual master `c3180398faca815fed7a62674bd4b0c4e88884ae`, whose common completion path now sends deferred/recovery notifications. Its unpublished normal merge tree is `16b028d57150f8caa576e913f38343a365b542dd`. Normal Shared/SDK, Runner TypeScript and the full App graph pass on that combined source. One unchanged ordinary-issue acceptance case was rerun there: an actual failed process retains the original automatic retry deadline, admits one useful successor, and passes the settlement, authority and no-duplicate assertions. It passed in 45.055 s; 18 filtered registrations are not acceptance. All known owners and groups closed, and the independent final database audit found zero remaining test sessions. The successful database was removed normally. Recorded worker socket peak is unavailable for this run; parent-process zero values are not used as worker measurements. All 29 Legacy-owned paths and every unrelated master blob and mode are preserved. Two outside-diff review observations concern existing waiting-chat compaction and cross-host Native result-lease recovery. Their files are byte-identical to current master. This focused Legacy change does not fix or qualify those separate paths. ## Risks Settlement proves exact local closure. It does not cover live provider work, Native ownership, cancellation, costs, claims, cleanup debt or changed authority. Independent checks remain. Ship normal migration0439 with the reader and writer. Preserve executable history and the newest five snapshots0435–0439. Generate the MCP child's next migration normally after this parent. The prerequisites are merged; the final remote checks must use this integrated exact head. This proof uses controlled local processes and external PostgreSQL. It does not establish deployed Cloud alarm acceptance, physical host or database suspension, cold host recovery, or R5 Native capacity settlement. The30-second retry is below the Cloud minimum wake lead. This focused repair follows the roadmap's bounded recovery and Cloud deployment work. It adds no new provider policy. ## Model Used OpenAI `gpt-6.1-sol` assisted with readers, fixtures, source review and verification through Codex. A coordinating GPT-6-family agent implemented and reviewed shared lifecycle wiring and integration. Tools included Git, native TypeScript checks and controlled tests. Exact served revisions and context limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and its available capability details - [x] I have checked ROADMAP.md and related work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked existing public work and described the issue - [x] I have not included internal issue IDs or private links - [x] My branch name describes the change - [x] Required focused checks pass at their explicitly recorded source/carry scopes above - [x] I have added or updated meaningful tests - [x] I have recorded the internal contract and migration requirements - [x] I have documented risks and evidence limits - [x] All required published exact-head Paperclip CI gates are green - [x] Published exact-head Greptile is 5/5 with no open findings in this focused diff; separate unchanged master observations are identified above - [x] I will address reviewer findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
c3180398fa |
fix(native): resume a remote Runner after a settled suspension
fix(native): resume a remote Runner after a settled suspension A scheduled checkpoint retry can retain a remote reattach claim after closing its failed predecessor cleanly suspends and exits the Runner. Resume only that exact settled source after confirmed exit and a fresh identity/state check. Preserve pending work and uncertain-exit fences. Admit the last already-charged provider attempt and share one bounded recovery deadline with the final state read, retaining closed diagnostic labels on read failures. Cover fresh ACPX Runner relaunch on attempts two and three through actual processes and the durable close barrier. Six affected suites pass 1,579 tests, with 14 environment-gated skips; all server typecheck layers pass on Node 25.6.1. The complete provider turn continuation failure remains preserved as follow-up #19751. Refs: #19366 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing> Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> |
||
|
Nicky Leach
|
08b049f381 | Admit qualified Claude sources through the existing Core claim (#19196) | ||
|
DottaandClaude Opus 5.5
|
e620ce3678 |
fix(heartbeat): deliver each accepted follow-up exactly once through restarts and admission holds (#19741)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can comment, click Retry, or rely on automatic recovery while an agent works on a task > - Restart-chaos waves showed accepted follow-ups being dropped, or delivered twice, around restarts and temporary company admission holds (accounting still settling after a restart) > - A dropped comment means the user's request is never acted on. A duplicate reopens work that is already done > - This pull request makes accepted work wait through a temporary hold and resume when it clears. It also stops a comment from being promoted again after a completed run already received it > - The benefit is that each accepted follow-up is delivered exactly once, eventually, unless the task is cancelled or a reached hard stop applies ## Linked Issues or Issue Description Refs #19702 (never-started tasks during a temporary hold), #19700 (bounded accounting hold), #14910 (stale reopen from a deferred wake; no file overlap). **What happened?** Restart-chaos waves found these shapes: 1. **Comment wake cancelled.** A comment was saved behind a run and promoted when the run ended. A temporary company hold refused the unstarted queued run, and claim cancelled it terminally. The recovery wake that followed was cancelled the same way, and the task rested `blocked` without the comment. 2. **Dead Retry.** A Retry click during the hold stayed `deferred_issue_execution` after the hold cleared. Nothing woke it, and the issue held no execution lock. 3. **Refused assignment.** A new task's first wake was refused during the hold. Nothing woke stranded-work recovery when the hold cleared, so the task stayed `todo` with no run. 4. **Silent comment after recovery.** A comment was saved behind a run that restart recovery finalized. A company-scoped hint waited out the 30 s deferred-receipt rescan, so the task looked done with the message unread. 5. **Comment delivered twice.** A run whose provider process was lost had a saved comment. Its retry started after the comment, received it in its prompt (task history), did the work, and finished the task. The saved wake was then promoted anyway, which reopened the done task. **Expected behavior** An accepted follow-up (comment, Retry, recovery) is delivered exactly once, eventually, unless the task is cancelled. A temporary refusal never ends in a terminal cancel, skip, or block. Reaching a hard stop remains the board's decision. ## What Changed - `heartbeat/queue.ts`: on a budget block at claim, an unstarted queued run stays queued when the block is temporary. A reached hard stop or blocked unpriced usage still cancels it. - `recovery/service.ts`: stranded-work recovery waits out a temporary hold for any task. It does not escalate to `blocked`. This extends #19702 beyond never-started tasks. - `budget-block-ownership.ts`: a shared `isBoardOwnedBudgetBlock`, moved out of the recovery service. `budgets.ts` is unchanged. - `run-cost-accounting.ts`: settling a run's accounting wakes queued-run dispatch, the deferred-receipt worker, and stranded-work recovery. Held work waits passively; this is what resumes it. - `deferred-execution-notifications.ts`: new `notifyDeferredWakesForIssue`. `native-run-projection.ts` (a run reaching a terminal status) and `environments.ts` (lease release) now hint each saved wake of the issue by id, so delivery does not wait out the company rescan interval. - `wake-queue/adapters/postgres.ts`: release treats a saved comment as delivered only when a succeeded run of the wake's agent on that issue started after the comment's latest revision, and either received the comment as its own wake input (`wakeCommentIds`) or was a retry of the interrupted turn whose task history included it. In that case it cancels the saved wake with "Queued messages were already delivered to a completed run" and leaves the task as it is. Task history alone never consumes a message saved for its own turn. One query checks the whole batch. - `run-cost-accounting.ts`: settling late receipt debt also sends the hold-release notifications. - `doc/execution-semantics.md`: documents the rule. ## Verification Red on `master` and green with this change, on embedded Postgres: - `server/src/__tests__/heartbeat-held-comment-wake.test.ts`: - A promoted comment refused by the hold stays pending, with no cancelled run and no `blocked` task. It is delivered once after the hold clears through real `accountRunCost`. - A Retry clicked during the hold starts exactly once after accounting settles, which wakes its owner. - A new task whose assignment was refused during the hold starts once accounting settles, which wakes stranded-work recovery. - A comment that a later succeeded retry already received as history, or that a later run received as wake input, is consumed without reopening the task. - Guard tests (green on both): - A comment is still delivered in each of these cases: the later run never received it, it was edited after that run started, it was posted while the run was already running, or a non-retry run only saw it as history. - A task whose continuation was refused waits out the hold. - A reached hard stop is still refused. - `server/src/__tests__/native-run-projection-work.test.ts`: projecting a recovered run terminal hints the saved comment's own wake (red on `master`). - Related suites pass locally (14 files, 482 tests): wake-queue, queued comments routes, refused assignment wake, native run projection. - Live replays (local, unmasked, kept instances): - `--replay w-20261107T031005-s7501/t04` (Claude C1, K1): PASS. The comment's run was created at 03:56:24 during the hold, stayed queued, started at 04:06:24 when the hold cleared, and succeeded. It is the only run carrying the comment. - `--replay w-20261107T002911-s7401/t01` (before the per-wake hint): it reproduced shape 4. Run 592d772f was killed 250 ms after `run.result.accepted`, released and finalized at restart, and the task rested `done` with the comment still saved. Calling the existing sweep on the kept database promoted it, which confirmed that the missing piece was the trigger. ## Risks - A queued run can now sit queued for the length of a temporary hold, which #19700 bounds. It does not hold the issue's execution lock, so later comments still coalesce into it. - The delivered-comment rule depends on the run's recorded prompt context. A succeeded run that received the comment in its history counts as delivery, even if the agent did not act on it. This matches exactly-once delivery semantics. - Settling accounting now emits three extra notifications per run. They are coalesced per company. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, live restart-chaos replays). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
18b6b71d83 |
fix(native): refresh per-turn projections from the selected checkpoint (#19744)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Per-turn native sessions save controller checkpoints and suspended Runner backups under separate owners. > - A local controller projection can still name the first turn after the next turn has saved its checkpoint and backup. > - The old per-turn restore path selected that stale local reference instead of the newer canonical source. > - Admission then saw different turn identities and archived the valid backup with the old projection. > - This change uses the canonical selected checkpoint and the existing finished-owner proof before it replaces a stale projection. > - The benefit is that a third turn keeps the valid saved conversation and backup. ## Linked Issues or Issue Description **What happened?** After two successful per-turn remote runs, a third turn could fail before provider execution with unavailable provider resume state. The selected database checkpoint and suspended backup belonged to the second run. The local controller projection still belonged to the first run. **Expected behavior** The selected checkpoint must govern controller rehydration. Preserve the older projection and the selected run's backup. Do not overwrite current, newer, foreign, changed or unsettled ownership evidence. **Steps to reproduce** 1. Save two real controller checkpoints for one per-turn conversation and finish both owners. 2. Leave the first checkpoint's readable projection or external marker beside the second run's suspended backup. 3. Admit the third run through the real canonical selector and backend owner. 4. Before this change, admission archives the active backup with the stale projection and leaves no active controller state. Related: #18018 added per-turn controller restoration. #18718 refreshed warm projections. #18846 added finished-owner retirement. This change keeps the selected per-turn source from entering that retirement path because of an older projection. The open warm-only change #18436 concerns a different restart branch. ## What Changed - Resolve the canonical source before refreshing a prior per-turn controller projection. - Re-read it under the existing sorted coordinator locks. Replacing an older owner reuses finished-owner checks for both runs and verifies their full session scope. A same-source marker only rehydrates its exact selected identity before the normal retained-sandbox verification. - Recheck the exact initial local bytes after asynchronous reads and lock waits. Preserve the old control-plane directory in the existing archive, while keeping the newer backup and provider files active. - Restore a current-run external marker only from its own checkpoint before the existing bootstrap retry proof. A missing current checkpoint cannot select a previous turn. Retain readable current-run, legacy file-backed, backup integrity and provider resume checks. - Stage the existing OpenCode proxy bytes into the controlled fake-sandbox filesystem in the real ingress-retry fixture. Keep remote artifact path and digest checks unchanged. - Add real PostgreSQL/filesystem coverage for Codex and ACPX, readable projections and external markers, missing or corrupt parts, held owners, foreign scope, newer history and a current owner arriving during a real coordinator-lock wait. ## Verification - Baseline RED: the two new readable-projection cases failed against unchanged production code because active control-plane state was absent after admission. The real selector selected the second run. Setup failures were separate and are not counted as RED. - Exact final existing per-turn restore, active-owner refresh and retained-owner wait controls: 19/19 passed in 5.91s. - Exact final local source (`fd8d9d9021`, based on `131b566ca4`): 58/58 real PostgreSQL/filesystem and native Runner tests passed in 31.58s. This includes both existing ingress bootstrap retries through the real Runner and fake Codex/OpenCode providers, plus 56 focused owner cases. Cargo validated and rebuilt current source from a private APFS clone of an unqualified local cache; the original cache was untouched. A private local configuration resolves workspace DB, Shared and Runner sources without replacing behavior. The local CLI allowed a 90s setup hook; no default timeout changed. - Exact final full server and test-tree TypeScript check with Runner source paths: passed. The stock fresh-worktree check first failed on unbuilt Runner declaration entries; no product types were changed to hide that setup gap. - Exact final module-boundary and diff-whitespace checks: passed. - Native-server CI RED on the previous head: both real ingress retry cases failed because a current-run marker was returned without hydration. The aggregate gate failed from that one underlying lane. The correction preserves the own-checkpoint bootstrap path. The first local OpenCode attempt hit a controller-host fixture path guard before its intended failure; staging the same proxy bytes into the fake sandbox made both native cases pass. - Review regression RED: the real retained-sandbox fixture failed at `sandbox_resumed` on the first PR head. GREEN: the corrected owner hydrates the same-source marker and reaches the existing exact retained-lease/suspended-runner read. A mismatched item identity still refuses before any remote read. - Local environment: macOS arm64, Node 26.4.0, PostgreSQL 18.1. Provider execution is deliberately outside this admission fixture. The real three-turn workflow still needs verification on the normally deployed fixed source. ## Risks - Refresh requires settled, committed, released and stopped or fenced owners. Missing or changed evidence remains blocked. - The refresh holds existing coordinator locks while it validates bounded controller state. It performs no provider calls. - The old control-plane bytes remain in the archive. A restore write failure still fails admission; it does not reset provider state. - Final-head full CI is green, including typecheck, build, Rust, native-server, server and browser suites; Greptile is 5/5 with no unresolved threads. Hosted deployment and continuity verification remain separate acceptance steps. A successful replacement conversation is not proof of provider continuity; hosted continuity acceptance is still pending. ## Model Used OpenAI Codex, GPT-6 family, with code execution and repository tools. The host does not expose the exact backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
24cbea136f |
fix(native): export OpenCode launch artifacts through a confined workspace copy (#19743)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native recovery saves the qualified OpenCode launch files so a cold allocation can resume the same session (#19622) > - Core reads those files from the sandbox with the provider's bulk `syncOut` > - The launch root is under the sandbox account home, not under the workspace remote dir > - Daytona, Kubernetes, and CreateOS confine every export source to the workspace remote dir, so the export always fails > - Every remote OpenCode run on a durable-storage instance now fails 40-90 s after start with `native_session_interrupted` > - This pull request exports a confined scratch copy of the three launch files from inside the workspace runtime dir > - The benefit is that remote OpenCode runs work again, and the provider confinement guard stays unchanged ## Linked Issues or Issue Description Refs #19622 **What happened?** After #19622, every remote OpenCode run on Daytona fails at provider admission. The run error is `Daytona sync source path escapes the workspace remote dir: /home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/<digest>`. The error code is `native_session_interrupted`. The last successful run of the affected agent was immediately before #19622 reached the deployment. **Expected behavior** Core saves the qualified launch files and the run continues. The provider confinement guard still refuses any export source outside the workspace remote dir. **Steps to reproduce** 1. Configure an OpenCode agent on a Daytona environment with durable harness storage. 2. Start a run. 3. The run fails when Core captures the provider artifacts, before the first turn. ## What Changed - `captureRemoteNativeProviderArtifacts` now takes the workspace `remoteCwd`. - Before the export, a sandbox command copies the three launch files into a new scratch directory at `<remoteCwd>/.paperclip-runtime/paperclip-runner/provider-artifact-export/<uuid>/`. The command refuses a symlink or a non-regular file. It uses `cp -P`, so it never follows a link. - `syncOut` reads from that scratch copy. A `finally` step removes the scratch copy after success or failure. A failed removal is logged and does not replace the transfer result. - The launch root does not move. OpenCode still binds to the same absolute paths. Core still verifies each downloaded byte against the manifest SHA-256 values. - `.paperclip-runtime` is already excluded from workspace sync, so the scratch copy never enters the user workspace. - The Daytona `assertConfinedSandboxPath` guard is not changed. ## Verification - New regression `server/src/services/native-runtime/native-provider-artifact-capture-confinement.test.ts`. It runs the real sandbox commands on a local directory tree that has a home-rooted launch root and a separate workspace. Its `syncOut` applies the same workspace confinement rule as the Daytona provider. - Before the fix: it fails with `Daytona sync source path escapes the workspace remote dir: .../home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/dfe9e4b1d660/bin/nodeCommand`. This is the staging error. - After the fix: it passes. The published bytes match, every export source is under the workspace runtime dir, and the scratch directory is empty after the capture. - Safety cases: a failed transfer removes the scratch copy and does not publish. A launch file replaced by a symlink is refused before any export, and nothing is published. - `native-session-executor.test.ts`: the durable OpenCode admission case now asserts that the capture receives `remoteCwd` and that every `syncOut` source is under `/workspace/`. - `pnpm vitest run` for the capture, confinement, remote OpenCode artifact, and session root entries tests: 14 passed. - `vitest run native-session-executor.test.ts -t "opencode|provider artifact|durable"`: 62 passed. - `pnpm --filter @paperclipai/server typecheck`: passed. - Not run: a live Daytona run with this revision. The fix is not deployed to staging. ## Risks - Low risk. The capture now runs two short `sh` commands in the sandbox and holds a second copy of the launch files during the transfer. The files are bounded by the existing 512 MiB limit and are deleted afterwards. - The scratch copy uses `cp -P`, `mkdir`, and `rm -rf` from POSIX `sh`. The Daytona image already uses these tools for the workspace sync. - If the sandbox stops between the copy and the removal, a scratch directory can remain under `.paperclip-runtime`. Workspace sync excludes that directory. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, and read-only browser access to staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9f0ebf6f70 |
fix(accounting): close a run whose turn was lost on restore, and stop succeeded runs holding all work (#19724)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budgets hold new work while the cost of completed runs is unknown > - In a restart chaos wave, a native ACPX Claude run crossed a server kill, resumed, and succeeded. Its usage receipt never became complete. > - For the 10-minute closure bound, admission refused all work in the company. This included tasks that never crossed a fault. > - This pull request fixes the receipt for a turn that was lost on restore. It also stops a succeeded run from holding all new work. > - The benefit: one late usage report no longer freezes a company, and hard stops still apply. ## Linked Issues or Issue Description Refs #19700 (the accounting closure bound that this follows). **What happened?** A native ACPX Claude run crossed a server SIGKILL in the middle of a turn. The Runner restored the session, and the run succeeded 54 seconds after it started. All 17 usage receipts stayed incomplete. Accounting settled the run only at the 10-minute closure bound. During those 10 minutes, admission refused all company work with "Company cannot start work while completed runs await accounting". **Root cause** ACPX reports provisional usage (`runDeltaComplete: false`) until a turn ends. The kill lost the turn that was active, so that turn never got a complete report. The restored Runner failed that turn with `provider_turn_lost_on_restore`. Native turn accounting (`createNativeTurnAccounting`) required every terminal turn to have usage, so this turn kept every later receipt incomplete. This was true even after the continuation turn completed with full usage. The admission gate then counted the succeeded run as a pending terminal run for the whole company. **Expected behavior** The receipt closes when the run's turns end. A turn that was lost on restore makes the usage a lower bound. A succeeded run with only usage outstanding does not hold other work. ## What Changed - `native-usage-accounting.ts`: a turn that failed with `provider_turn_lost_on_restore` and has no complete usage no longer holds the receipt open. The snapshot marks `accountingUsageLost`. - `native-session-executor.ts` and `native-usage-delivery.ts`: such a receipt carries the provenance note "a provider turn was lost on restore and never reported its usage; observed usage is a lower bound". - `anthropic-pricing.ts` and `codex-pricing.ts`: the rate-card pricers keep a producer's provenance note. - `native-usage-delivery.ts` and `usage-receipts.ts`: remote producer delivery keeps a `lostOnRestore` marker on the receipt's producer event, so replay also excuses the lost turn. Receipts saved without the marker still replay. - `budget-exposure.ts` and `budgets.ts`: admission waits only for pending runs that ended without succeeding (`waitingRunCount`). A succeeded run whose provider reported observed usage is bounded like a running run: its observed cost counts toward spend and reservation capacity, and its settled charge can still reach the hard stop. - `budget-exposure.ts`: a closure that timed out because of a control-plane restart keeps the unpriced-usage exemptions, whatever its token count. Restart evidence is a replaced receipt source, a native recovery transition that is not an ordinary same-run retry, or a turn lost on restore. A second chaos run showed the need. It recovered and succeeded, but its final usage report was lost with the restart, so it settled as unpriced 0 cents. The blocking policy then opened a hard incident and paused the company. A timed-out closure with no restart evidence still counts. - `run-cost-accounting.ts`: a lost receipt that carries a positive provider-reported cost settles at that cost as an estimate marked as a lower bound, not as unpriced. - `paperclip-runner-tool-authority.ts` and `budgets.ts`: a running run's own tool mutations are evaluated as admitted work. Hard stops and pauses still refuse them. Accounting holds, recovering runs and budget inputs that await evaluation do not. In a third chaos run, a live run's write to its task document was refused during a temporary hold, so the run proposed a blocked result. - `doc/SPEC-implementation.md`: documents these rules. Policy choice: a succeeded run's provider work is over, so it has less unknown spend than a running run. Running runs never held admission. Failed, cancelled, timed-out and interrupted runs still hold new work, because their provider may not have stopped. A per-agent gate would still freeze unrelated work of the same agent. A per-reservation gate does not help when `reservationCents` is 0, which was the case in this wave. ## Verification - Live: a restart chaos wave ran master plus this PR (c2a48fb1e0) with no masking. It passed 6/6, and every task crossed a restart. There were 0 accounting holds, 0 refused agent writes, 0 company pauses, and every cost settled as estimated. - Red then green: `native-usage-accounting.test.ts`, "closes a run whose earlier turn was lost on restore, as a lower bound". It uses the event sequence of the chaos run. - Red then green: `closed-native-receipt-accounting.test.ts`, "does not hold other work for a succeeded run with only usage outstanding". It also checks that the hard stop still applies. - I changed "neither pauses nor holds admission while a succeeded run's charge commits" to the new policy. A failed run in the same state still holds new work. - Red then green: `closed-native-receipt-accounting.test.ts`, "never hard-stops a blocking policy for a restart-derived timed-out receipt". It uses the zero-token row shape of the chaos run and a case with unpriced tokens. Without the fix, the company was paused. - Red then green: `runner-api.integration.test.ts`, "lets a running run write during a temporary accounting hold, but not after a hard stop". - Red then green: `closed-native-receipt-accounting.test.ts`, "does not hold other work for the s7504 succeeded row shape". - Red then green: `closed-native-receipt-accounting.test.ts`, "settles a timed-out restart receipt at its provider-reported cost, without holding or pausing". It uses the OpenCode/OpenRouter row shape from a fourth chaos run. - `anthropic-pricing.test.ts`: the rate-card provenance keeps the lower-bound note. - Local: `closed-native-receipt-accounting`, the budget suites, `native-usage-accounting`, `native-usage-delivery.integration`, `native-runner-restart-recovery.integration` and `anthropic-pricing` all pass (187 tests). Server `tsc --noEmit` is clean. `pnpm verify` asks for CI coverage because the spec changed. ## Risks - A succeeded run whose final charge is higher than its observed cost can let one more task start before the charge commits. The hard stop then pauses the scope as it does for a running run that overruns. - A run that includes a lost turn settles with a lower-bound usage figure. The provenance says so, and a later receipt still reconciles through the late-receipt path. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
464e6e16e0 |
fix(native): resume copyback after a restart kills its same-host owner; one board action otherwise (#19717)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs save the agent's workspace files after the provider turn finishes (copyback) > - One controller owns copyback at a time. The run records that owner in `runnerProfileJson.nativeWorkspaceFinalizationOwner` (host, PID, token) > - A control-plane restart can kill the controller during copyback. The dead controller can never confirm its own stop > - Recovery then waited for a board confirmation. No surface offered that confirmation, so the run stayed `running` and the task stayed `in_progress` with no visible action > - This pull request lets the next controller on the same host prove the stop by itself. It also gives the board one clear action when that proof is not possible > - The benefit is that a restart during copyback resumes by itself, and a task never waits silently for a person ## Linked Issues or Issue Description **What happened?** In a Daytona restart-chaos wave, a kill during workspace copyback left the run `running` and its task `in_progress`. Recovery reported `native_workspace_finalization_owner_unverified`. The run was still `running` when the wave timed out. **Steps to reproduce** 1. Start a native remote (sandbox) run. 2. Kill the control-plane process (SIGKILL) after the provider result is accepted, during copyback. 3. Restart the control plane on the same host. **Expected behavior** If the previous controller and its copyback children are proven stopped on this host, recovery resumes copyback without a new agent turn. Otherwise, the task shows one board action to confirm that the previous server stopped. **Actual behavior** The recorded owner was cleared only by `resumeNativeWorkspaceAfterOwnerStop`. Same-host recovery also required a completed-copyback barrier, which an interrupted copyback never has. The task presentation showed `working`, and the UI had no control for the confirmation. Refs #19578 (boot-ID fencing for unstarted runs; same proof style). ## What Changed - Root cause: `server/src/services/native-runtime/native-workspace-finalization-ownership.ts` sent a dead same-host owner to `requireStopVerification` when no `workspace_finalize` operation had succeeded. That case is exactly a restart during copyback. - New `native-workspace-owner-stop-proof.ts`. The owner receipt now records the controller's process group (`processGroupId`). Copyback children (tar, Git) are not detached, so they keep that group. A successor on the same host releases the receipt only when the receipt names another controller boot, the PID is dead or recycled (process start time), and the group is empty or has no member outside the successor's own process tree that started after the old controller. Age alone never proves a stop. - A still-live group member returns `local_stop_pending`. The finalization worker rechecks it on its existing ownership recheck interval. The board action stays visible meanwhile. - A successful local proof writes `issue.workspace_owner_stop_verified` with the proof and marks the recovery `resuming`, like the existing platform proof. - A receipt from another host, a receipt without a process group, or a host without `ps` keeps the fence. The task interruption is now `waiting_for_user` with `userAction: "confirm_owner_stop"` (new shared enum value), not `working`. - UI: the task chat shows one **Confirm the previous server stopped** action. It asks for stop evidence and calls the existing `resume-workspace-finalization` API. It is shown only to roles that hold `runtime:confirm_owner_stop` (company owner or administrator, local board, instance administrator). The inbox names the same action. - The copyback advisory lock, the coordinator finalization lease, and remote save fencing are unchanged. - Docs: `doc/native-workspace-finalization-recovery.md`. ## Verification - Red then green on embedded Postgres (`server/src/services/native-runtime/native-workspace-finalizer.test.ts`): - same-host controller killed with SIGKILL, its process group empty: before `owner_unverified`, after `acquired` with the proof logged and no agent wake - a non-detached orphan child of the dead controller still runs: `local_stop_pending`, no copyback; after the child exits, `acquired` - a receipt without a process group: still `owner_unverified` - a foreign-host owner: the task presentation was `working` (red), now `waiting_for_user` / `confirm_owner_stop`; after the board confirmation it is `working` again - The existing orphan test (a detached child) and all other finalizer tests pass: `npx vitest run src/services/native-runtime/native-workspace-finalizer.test.ts src/__tests__/native-workspace-finalization-work.test.ts src/services/task-interruption-presentation.test.ts` (135 passed). - UI: `npx vitest run src/components/WorkspaceExportRecovery` (34 passed), including the action call body and the non-administrator case. - `pnpm check:token-gates` clean. Server and UI typecheck clean for the changed files. - Live Daytona restart-chaos wave: results are added in a comment below. ## Risks - A process-table check that reports a member wrongly as live only keeps the existing fence. The board action is still available. - A subreaper that adopts orphans does not change their process group, so the proof does not depend on the parent PID. - A child that a future change spawns with `detached: true` would leave the group. Copyback paths do not do that today. The existing detached-orphan test still fails closed, because a receipt without a process group is not released. - Old receipts have no process group. They keep the board action. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
7a72caeda1 |
perf(workspaces): batch online companion discovery (#19735)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace migration checks all local aliases before it captures a source. > - Each alias must retain its prepared runs and recovery holds in that check. > - The current reader scans the same company run table once for every alias. > - Shared Git sources repeat that scan even after the complete alias group is known. > - This pull request reads those narrow companion rows once per inspection. > - The existing owners still validate every source and recovery hold before migration can proceed. ## Linked Issues or Issue Description **What happened?** Online source qualification repeats the prepared and finalized run query for each workspace alias. A shared Git source with four aliases makes four copies of this query. **Expected behavior** One inspection should read the complete alias group once. It must return the same source and recovery proofs, and a later inspection must read fresh rows. **Steps to reproduce** 1. Create a shared Git source with four aliases and a valid settled prepared run. 2. Add a finalized companion and a recovery hold. Add a foreign-company run that names the same workspace ID. 3. Inspect the source and count companion discovery queries. Before this change, the regression sees four queries instead of one. Related public work: [source ownership guards in #19688](https://github.com/paperclipai/paperclip/pull/19688) and [bounded accounting closure in #19700](https://github.com/paperclipai/paperclip/pull/19700). ## What Changed - Read prepared and finalized companion rows in one company-scoped query after the complete alias group and foreign aliases pass validation. - Group the narrow rows by workspace ID. Keep run order and all existing source and recovery owner checks. - Add a PostgreSQL regression for proof equivalence, one scan, company isolation, and a new source obligation between inspections. - Align an existing Stop-accounting test with the durable closure deadline from #19700. Keep the incomplete debt, quiet database, cancellation, and single-charge checks. Product workers and timers are unchanged. ## Verification - Before the source change, the new PostgreSQL test passed its proof comparisons and failed because four companion scans ran instead of one. - The two workspace suites pass: 77 tests in `remote-workspace-online-source.test.ts` and `online-legacy-workspace-recovery.test.ts`. - CI exposed an old Stop-accounting assertion that expected no timer while an incomplete receipt had an intentional closure deadline. Both automatic CI attempts failed. The unchanged local variants passed, so that run did not reproduce the failure. Waiting for the real owner deadline then proved the stale assertion red. - After the test alignment, both Stop-accounting variants and all 77 workspace tests passed together: 79 tests across three files. The three unrelated Stop-recovery cases were not selected locally. Fresh CI covers the complete suites. - The affected suites cover retained recovery owners, foreign aliases, physical source locks, and changes before publication. - Tests use a disposable PostgreSQL 18.1 database. The local hook allowance is 90 seconds. Product timeout settings are unchanged. - Final head `0eed2e5062cbfb62df7f76168c27d2172fc2d7de`: [full CI](https://github.com/paperclipai/paperclip/actions/runs/42341640259) passed, with 56 successful checks and four configured skips. [Fresh Greptile review](https://github.com/paperclipai/paperclip/pull/19735#issuecomment-6702476739) is 5/5 with no open threads. This PR makes no claim about deployed latency. ## Risks - Each inspection now holds the narrow companion rows for the complete alias group in memory. The existing alias group bound still applies. - The query keeps the same company, workspace, and sync-state predicates. There is no runtime filter, row limit, schema change, or cache between inspections. - No public command or API changes. Public documentation does not need an update. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`. Used reasoning, code editing, shell tools, and test execution. The context window size is not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
fc7e3a6503 |
fix(recovery): never replace a native session after a deterministic refusal (#19733)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs keep a saved provider session, and same-run
recovery resumes it after a failure
> - When resume fails and the checkpoint allows it, recovery archives
the saved continuity state and opens a fresh replacement session
> - Recovery used that path for every resume failure, including
deterministic refusals that left the saved state unchanged and repeat on
every retry
> - In five staging tasks a refusal (a protected launch argument, or a
predecessor tool call with no result) escalated into a replacement. The
remote archive move failed part way, the next attempt found no runner
state, and each task was blocked for workspace repair
> - This pull request stops before replacement after a closed-class
refusal, keeps the saved state, and hands the run to an operator with a
named reason
> - The benefit is that a deterministic refusal can no longer damage
saved state or open a replacement over an unsettled external action
## Linked Issues or Issue Description
Refs #19659, Refs #19516, Refs #19334
**What happened?**
Five Codex tasks on Daytona sandboxes failed on staging. The first
attempt failed with a deterministic refusal:
- `run.attach cannot change protected launch arguments` (four tasks,
fixed separately in #19659), or
- `native_runner_source_semantic_reconciliation_required` with a
predecessor `paperclip_call_assigned_tool` call still pending (one
task).
The same-run retry then opened a replacement session.
`archiveContinuityState` failed with
`runner_continuity_break_archive_failed` and no detail. The third
attempt failed with `runner_remote_state_unavailable: exit=1`, and the
task was blocked with "Workspace repair required".
**Steps to reproduce**
1. Save a native session whose checkpoint policy is
`allow_replacement_after_resume_failure`.
2. Make the resume fail with a refusal that does not change state (an
attach refusal, or a pending tool call).
3. Same-run recovery opens a replacement session and archives the saved
continuity state.
**Expected behavior**
A deterministic refusal stops the run for an operator. The saved state
stays intact. A run with a pending side-effecting call never reaches
replacement. Transport drops keep the existing retry behavior.
**Root cause**
`executeNativeSession` checked only the checkpoint's replacement policy.
The closed failure class (#19334) only had an effect when replacement
was forbidden. Attach refusals and pending tool calls had no class at
all, and the source-reconciliation refusal was a plain `Error`.
## What Changed
- Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach`
refusals that would change the durable Codex profile or a protected
launch argument carry the closed classification
`provider_attach_refused`. An attach blocked by
`durable_pending_tool_calls` carries `provider_side_effect_pending`.
These refusals already leave the saved state unchanged.
- Contracts (`harness-driver.ts`): new closed classes
`PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed
`NativeRunnerSourceReconciliationRequiredError` replaces the plain
`Error` at the five `native_runner_source_*_reconciliation_required`
throw sites. `runnerRecoveryFailureClass` reads the authenticated
command-result classification or the typed error, never message text.
- Runtime (`native-session-runtime.ts`): after either class, recovery
never opens a replacement session, whatever the checkpoint policy.
- An attach refusal ends with the existing
`NativeProviderStateIncompatibleError`, which maps to the existing
operator path `native_provider_state_incompatible`.
- A pending side effect ends with the new
`NativeProviderSideEffectPendingError`.
- Server (`native-session-executor.ts`, `native-session-resume.ts`,
shared `EXECUTION_RECONCILIATION_CAUSES`):
`native_provider_side_effect_pending` is a permanent failure and an
execution-reconciliation cause, so the board gets a reconciliation
(record whether the action ran) instead of a generic Retry. The
reconciled continuation does not rebind the saved session, which still
refuses attachment. It starts a fresh provider session in the same
workspace. The saved session and workspace are unchanged.
- Archive (`archiveContinuityState`): a failed remote move now records
the bounded (512 characters), redacted exit status and output in
`runner_continuity_break_archive_failed: exit=… timedOut=… <output>`.
The remote move already ran before the local move on `master`, so a
failed remote move leaves the local state in place.
`restart-interrupted-turn.ts` accepts the detailed message for its
reattachment-loss retry.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe the new
rule.
## Verification
Red before the fix, green after (each test first run against unchanged
source from `origin/master`
|
||
|
DottaandPaperclip
|
1c91bacf97 |
fix(native): admit independent replies during contended publication (#19726)
## Thinking Path > - Paperclip manages work done by AI agents. > - A completed native run must deliver its saved final response. > - The response worker also recovers retained replies that can wait on publication locks. > - A pass waited for its old publication groups even when other check slots were free. > - This could delay an independent final reply beyond a full page of retained candidates. > - This PR enables the existing shared coordinator option that admits independent due work during an active pass. > - Existing authority checks, resource keys and the four-check limit still control publication. ## Linked Issues or Issue Description **What happened?** A native run completed after replacement recovery. Its final response and accounting were saved, but its final issue reply was missing. The response queue contained more than 100 retained chat replies and repeated endpoint-lock failures. The existing worker did not admit independent due entries until its current groups finished. **Expected behavior** A contended publication owner must not hold free worker slots. An independent accepted final reply must publish once, including when its candidate is on a later SQL page. **Steps to reproduce** Create 100 accepted, committed replies that share one publication owner. Create an independent reply after them in candidate order. Hold the old coordinator rows with a database transaction and start the real response worker. Before this change, the independent reply stays unpublished until the old group is released. Related: #18977 changes response selection in standing conversations. This PR changes only worker admission and preserves response selection. ## What Changed - Enable `admitDueWhileActive` in the native response worker's existing shared coordinator configuration. - Add a real PostgreSQL regression with the full 100-row SQL page, the default dispatch batch, real row-lock contention and a later-page independent reply. - Verify one comment per reply after settlement and no further SQL or scheduled deadline across 60 seconds of idle time. - Update one existing principal/rescope integration assertion to observe the reconciled invalidation revision. Keep its actual per-run replies, authorization, publication and idle checks. ## Verification - Regression on the unchanged worker: the independent reply remained unpublished while the first 100 replies were locked. The real PostgreSQL test failed as expected. - Final head `748fb9609104c016c33ff7caf282ba9e15e88495`: all six publication regressions and both principal/rescope controls passed (8/8, 47.96 seconds). Checks cover later-page admission, exact per-run comment counts, retry and rollback behavior, authorization, and 60 seconds with no idle SQL or scheduled deadline. - The tests used the canonical PostgreSQL wrapper and one owned disposable PostgreSQL 18.1 fixture. All clients and test databases were closed before fixture cleanup. - The current-head [full CI run](https://github.com/paperclipai/paperclip/actions/runs/42338953241) supplies typecheck, build and broad test coverage. All current-head checks passed; the merge gate is green. - Local full typecheck was blocked by unbuilt dependencies; it is not counted as a pass. The prior timer-state CI failure and the final successful rescope-generation regression remain recorded. - Greptile reviewed the final head at 5/5 with no unresolved threads. `git diff --check` passed. - Deployed acceptance remains separate. No serving instance was changed by the PR tests. ## Risks The worker can now use free slots during a slow pass. The shared coordinator still limits checks to four and preserves task, endpoint and principal exclusions. Authorization, result identity, retry limits, response selection and transaction ownership are unchanged. The regression checks idle quiet after all concrete obligations settle. ## Model Used OpenAI GPT-6.1 (`gpt-6.1-sol`), with reasoning, code execution and tool use. The runtime does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
b21884bdcf | Release and resume retained plugin Runner sessions in Core (#19611) | ||
|
DottaandPaperclip
|
9efee7ee16 |
fix(runtime): save dynamic skill bundles before native admission (#19729)
## Thinking Path > - Paperclip manages agents, tasks, and their saved execution state. > - A replacement server must restore each admitted skill from shared storage. > - Installed company skills already have immutable versions. > - Connector skills add assigned resources and tool definitions for each run. > - Those dynamic bytes keep a null version ID and differ from the static skill. > - Native admission saved their input before it saved a durable package receipt. > - This change stages the exact packages first and publishes the input and receipt together. > - A new connector turn can then survive an empty server home. ## Linked Issues or Issue Description Refs #19675 for the existing runtime asset preservation owner. **What happened?** A new native run with an assigned Slack connector creates a concrete local skill bundle after the old runtime assets were migrated. Admission saves its execution input without a preservation receipt. The new run creates a local durability dependency again. **Expected behavior** With S3 configured, the exact dynamic bundle must be durable before the execution input becomes visible. Its null version ID and admitted bytes must remain unchanged. Local storage must keep its volume-backed behavior. **Steps to reproduce** 1. Configure S3 and a native agent with an assigned Slack resource. 2. Prepare a new native task through the normal runtime selection owner. 3. Inspect its runtime skill receipt or restore its skills with an empty server home. The previous owner saved no receipt. **Paperclip version or commit** Reproduced on master `d1ee25d0f7`. The fix is tested on `4a3396d9b9`. **Deployment mode** S3-configured server. Local tests use PostgreSQL 18.1 and a private disk transport with the S3 receipt contract. ## What Changed - Extend the existing runtime asset owner with package staging and transaction-bound receipt validation. - Stage exact unversioned skill packages before native admission. Keep storage I/O outside the run transaction. - Publish the frozen input and receipt together under the existing run, cancellation, and package collection locks. - Reject changed input bindings and retired package metadata. - Reuse valid same-run receipts without reading the previous host's bundle path or requiring a new object read. - Keep existing online preservation and local storage behavior. - Add a bounded preparation span and document its work. ## Verification - Genuine unchanged-production RED: the actual Slack overlay and real admission transaction save no preservation receipt. - Final rebased head: 68 tests pass in three suites in 25.36 seconds. These include all runtime-selection, runtime-preservation, and connector controls. - The new positive checks that no execution input or coordinator is visible during package readback. It restores exact skill and tool bytes, modes, digests, and null version IDs in a cold home. - Guards cover an uncertain storage failure, cancellation during upload, a changed binding, retired package metadata, and local mode. An existing receipt remains usable after removing the original bundle and during a storage outage. - The canonical PostgreSQL wrapper uses one owned disposable PostgreSQL 18.1 container. Each suite's database is cleaned up. Real package archives, streams, and readback run locally; no external provider or model is called. - Strict server TypeScript, module boundaries, changed-line private-data checks, and `git diff --check` pass. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/42337473062) passes all required build, typecheck, test, and Runner gates. Fresh Greptile is 5/5 with no open review threads. ## Risks New S3-backed admission now requires its dynamic packages to be saved before execution can proceed. A storage failure can therefore stop preparation. Existing resource cleanup keeps its authority. Failed or cancelled staging retains original files and the package owner's pending receipts. Same-run recovery retains its original capture provenance and cache behavior. The tests replace the external S3 transport; they do not qualify live S3 permissions or provider execution. No schema, API, workflow, company skill version, or provider protocol changes are included. ## Model Used OpenAI Codex (GPT-6 family; the exact runtime model ID and context capacity are not exposed in this session), with reasoning, repository editing, code execution, and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
db41bfcb38 |
fix(recovery): let an accepted Retry finish after a killed copyback owner (#19721)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run saves the agent's workspace files after the provider turn (copyback). One controller owns copyback at a time and records a durable owner receipt on the run (`nativeWorkspaceFinalizationOwner`) > - When copyback of an accepted result fails, retained saved-work recovery exports the files from the kept sandbox. A task Retry waits for that recovery before it continues > - A control-plane restart can kill the controller while it owns copyback. The dead controller can never confirm its own stop, and nothing in the retained recovery path could release its receipt > - The recovery worker then spent one attempt on each recheck of that receipt until it was exhausted, and the accepted Retry kept saying "Paperclip is finishing the saved result" with no run. The task waited forever > - This pull request stops spending attempts on that recheck, names the real wait, and lets the board release the exact receipt so the same recovery and the same Retry continue > - The benefit is that an accepted Retry never waits silently: it finishes, or it shows one honest action that makes it finish ## Linked Issues or Issue Description Refs #19717 (same-host automatic proof for a killed copyback owner; this PR covers the retained-recovery path and the case where no automatic proof is possible) Refs #19713, #19609, #19694 (related restart-recovery fixes; no overlap in files except this doc) **What happened?** Two Daytona restart-chaos waves. In each, a native run's copyback failed and the run failed with `native_workspace_sync_out_retry_exhausted`. The agent's result was accepted. Retained recovery started. A SIGKILL of the control plane landed while a recovery attempt owned copyback: - Wave s13, task t06 (Claude, crash loop): Retry was accepted at 02:30:35. A SIGKILL landed 2 s later. All later attempts failed with `copyback_stop_unconfirmed`. After 6 attempts the recovery was exhausted. The Retry showed `retryWait.reason = finalizing_saved_result` for more than 15 minutes. No run was created. - Wave s12, task t01 (Claude): attempt 1 failed with "Workspace checkpoint payload mismatch". The SIGKILL at 01:43:54 killed the owner of attempt 2. Attempts 3 to 5 failed with `copyback_stop_unconfirmed`. The Retry was accepted with HTTP 2xx and produced no run. In both kept databases the run carries an owner receipt for the killed process. **Steps to reproduce** 1. Accept a native result and make its workspace copyback fail, so that retained recovery owns the export. 2. Press Retry on the task. 3. Kill the control plane while a recovery attempt owns copyback, so that its owner receipt stays on the run. Use another host, or a receipt without automatic stop proof. 4. Let recovery run. **Expected behavior** Rechecking the receipt does not spend attempts. The task names the wait and does not offer a Retry that cannot help. A board member can confirm that the previous server stopped. The recovery then finishes, and the waiting Retry continues by itself. **Actual behavior** `withNativeWorkspaceFinalizationOwnership` returned `owner_unverified` for a retained repair without creating any board action (`native-workspace-finalization-ownership.ts`, `requireStopVerification`). `recoverRetainedWorkspace` reserved an attempt for each recheck and failed it as `copyback_stop_unconfirmed` (`workspace-recovery.ts`), until the budget was exhausted. The Retry gate in `explicit-native-continuation.ts` returned `finalizing_saved_result` for an accepted result with no verified repair, with no check for this hold. `resumeNativeWorkspaceAfterOwnerStop` accepted only an `owner_unverified` action on a `running` run, so no route could release the receipt of a finished run. The exhausted handoff told the board to press Retry again, which repeats the same failure. ## What Changed - `server/src/services/workspace-recovery.ts`: when retained recovery finds an unverifiable copyback owner, it records the recheck without reserving an attempt. The 30-minute recovery window still bounds it. The retained action records the exact receipt as `workspaceOwnerRecovery: { state: "operator_required", owner }`. A newer receipt replaces the record and clears an older confirmation. - `server/src/services/explicit-native-continuation.ts`: a Retry that waits for an accepted saved result reports `workspace_owner_stop_required` when the run's active retained action has that hold. It no longer reports `finalizing_saved_result` for this case. - `server/src/services/task-interruption-presentation.ts`: the new wait has a fixed message, and Retry is not offered for it, because another click cannot release the receipt. - `server/src/services/native-runtime/native-workspace-owner-recovery.ts`: the existing board route `POST /issues/:id/recovery-actions/resume-workspace-finalization` (permission `runtime:confirm_owner_stop`) also accepts a retained recovery action with that hold. The confirmation must name the receipt that is still on the run. A live owner on this host still rejects it. It clears only that receipt, records the stop proof and an activity entry, and returns the same recovery to the worker. A spent budget gets one new generation, as a deliberate recovery request does. It accepts no result and starts no agent turn. The owner check is shared with the existing path. - `server/src/services/system-recovery.ts`: an exhausted `copyback_stop_unconfirmed` handoff names the board confirmation instead of another Retry. - `ui/src/api/issues.ts`: the route receipt's `resultId` can be `null` (a retained recovery has no live result binding). - `doc/native-workspace-finalization-recovery.md`: an additive paragraph describes this path. ## Verification - Red then green on embedded Postgres: `server/src/services/workspace-recovery-preparation-gates.test.ts`, "lets an accepted Retry finish after a killed copyback owner instead of waiting forever" with one case per wave (s13, s12). Each case uses a real accepted remote export, a Retry through the HTTP route, a killed-owner receipt, and 8 recovery passes. - Before the fix (both cases): `attempts: 6, state: "exhausted"`. That is the wave state. - After the fix: no attempt is spent on the recheck. The hold is recorded. The Retry wait is `workspace_owner_stop_required` with `canRetry: false`. The board confirmation through the route returns 202 and clears the receipt. The same recovery completes. The same accepted Retry then finishes the saved result (s13: the task is `done`) or continues in exactly one new run (s12). - `npx vitest run src/services/workspace-recovery-preparation-gates.test.ts -t "killed copyback owner"`: 2 passed. - `pnpm verify` (affected selection): see the comment below. Server `tsc --noEmit` is clean. - No live Daytona replay. Both waves ran on old signed images. The local proof uses the same accepted-export fixture as the existing Retry tests, with the receipt state copied from the kept databases. ## Risks - A copyback owner on another host that is still alive is now rechecked within the recovery window without spending attempts. Before this change it spent them. The window still bounds the wait. - The board confirmation for a retained action is an operator statement, the same as the existing owner-stop path. It is limited to roles with `runtime:confirm_owner_stop`. It never clears a receipt other than the one it names, and a live same-host owner still rejects it. - #19717 changes the same ownership area and adds a UI action for the running-run case. That UI does not show a retained action yet. After both merge, a small follow-up can show its button for this hold. Until then the API route and the task's wait message cover it. - Not addressed here: the "Workspace checkpoint payload mismatch" that started wave s12. The checkpoint outbound transfer extracts without `-p` (`sandbox-managed-runtime.ts` checkpoint sync-out, `extractHostTarball` in the Daytona provider), so the host umask can change file modes that the remote manifest recorded. That is a separate signature. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test runs, embedded Postgres). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5cc89b1e4d |
fix(native): do not charge a restart resume that made progress before the next crash (#19720)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs have an execution recovery budget of three provider attempts, so a run that keeps failing stops retrying > - A control-plane crash (no graceful drain) makes the next server resume the run, and that resume charges one attempt > - #19625 stopped charging a resume after a graceful drain, but crash restarts still charge, even when the previous resume did real work > - On a deploy day with several restarts, a healthy run used up its budget and the task needed a manual Retry > - This pull request does not charge a crash resume when the previous restart resume made recorded progress > - The benefit is that restarts alone do not exhaust a run that keeps making progress, and genuine loops stay bounded ## Linked Issues or Issue Description Refs #19625 **What happened?** In restart-chaos testing (storm wave, OpenCode run), a run crossed four control-plane restarts: K1, K1, K2 (drain), K1. Each resume was clean: the runner restored the session, recorded `provider_turn_lost_on_restore`, and the #19699 continuation turn then ran tools for minutes. The two K1 resumes each charged one attempt (attempt 1 to 3). The drain resume did not charge (#19625). The fourth restart found attempt 3 and blocked the run with `native_restart_recovery_blocked` / `execution_recovery_budget_exhausted`. Seven runs in the same wave failed in the same way. In every case, each resume had accepted a new turn and completed tool work before the next restart. **Expected behavior** A run must not fail because it survived several controller restarts that each resumed cleanly and made progress. The budget must still stop a resume that fails each time, a crash loop that makes no progress, and a provider that fails repeatedly. **Steps to reproduce** Start a native local run. Kill the server (SIGKILL) while the run is active and let it resume. Let the resumed turn run tools. Repeat three times. The third crash resume is blocked with `execution_recovery_budget_exhausted`. **Paperclip version or commit** `master` at `8a34c1b62c`. ## What Changed - `claimNativeRestartRecoveries()` (`native-restart-recovery.ts`) records `progressWatermarkSeq` (the run's highest `heartbeat_run_events.seq`) in the history entry of each recovery claim. - New `previousResumeMadeProgress()`: the next claim reads the watermark of the most recent claim. It finds progress when, after that watermark, a `turn.accepted` event names a turn, and a later `tool.execution.completed` or `item.completed` event (not a `usage` item) belongs to the same turn. The evidence is committed run events, not timing. - When that progress exists, the claim is not charged (`chargesProviderAttempt: false`, and the `>= 3` exhausted gate is skipped). This uses the same path as the clean-drain rule from #19625. The history entry records `attemptKeptForProgress: true`. - Restore activity does not count: lineage items without a turn, `turn.failed`, work on the interrupted turn, `turn.submitted`, and an accepted turn with only usage or a failure. So a resume that dies or fails before it does work still charges. - The credit stops at controller generation 12 (`NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT`). A run that does some work and then crashes the controller every time still reaches the budget. - The first restart (which interrupts the original execution, not a resume) still charges. A history entry from an older release has no watermark, so it gets no credit. - `doc/execution-semantics.md` states the rule next to the three-attempt budget. ## Verification New tests in `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` (embedded Postgres, real `claimNativeRestartRecoveries`; between claims the test applies the executor's attempt charge and marks the controller dead): - `keeps recovering a run whose restart resumes each completed work before the next crash`: five crash resumes, each followed by restore noise and one completed tool in the continuation turn. Charges are `[true, false, false, false, false]` and the attempt stays 2. Red on master (attempt 2, 3, then blocked). - `still exhausts the budget when restart resumes complete no work between crashes`: resumes followed only by restore noise and a failed continuation turn. Attempt 2, attempt 3, then `execution_recovery_budget_exhausted`. Green on master and with the fix. - `bounds a crash loop that makes progress before every crash`: credit stops at the generation limit, the last attempt is spent, and the run blocks. Red on master. - `native-runner-restart-recovery.integration.test.ts`, `heartbeat-native-restart-suspension.test.ts`, `native-restart-recovery-isolation.test.ts` and `native-remote-retry-admission.test.ts`: 218 passed, 14 skipped. - I ran the progress query against the live database of the failing wave. For all seven blocked runs, every resume had progress before the next restart, so this rule would have kept each of them recovering. - `tsc --noEmit` for `server` has no errors in the changed files. ## Risks - A run that makes progress and then makes the controller crash each time gets up to about ten extra resumes before the budget applies. The generation limit bounds this. - The rule depends on the PRP event shape (`payload.prpEvent.turnId` on `turn.accepted` and on completed work). Events without that shape get no credit, so the failure mode is the old behavior (charge), not an unbounded loop. - The query runs once per claim, on the `(run_id, seq)` unique index range after the watermark. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
1a60d47804 |
Preserve settled native log files during online migration (#19719)
## Thinking Path > - Paperclip manages work for AI agents. > - The web server must retain run history across replacements. > - Older native runs can have local log files with no final size or digest. > - The normal importer correctly refuses to call those files complete. > - The existing raw archive mode requires the entire instance to stop work. > - This change gives settled sources a separate, guarded online archive path. > - Active work can continue while old evidence moves to object storage. ## Linked Issues or Issue Description Refs #18864, #19678, #19688. The offline historical log archive path cannot migrate a busy instance. This change preserves exact surviving bytes without claiming complete native-event coverage or changing the original run outcome. ## What Changed - Add an explicit online option for settled native log files. Keep the default importer and offline option unchanged. - Require a released, closed native finalizer and no pending attempt, allocation, physical owner, or task binding. A stale lease or terminal run status alone is insufficient. - Use the canonical physical finalization lock and row locks. Check the source generation before claims and publication. - Read only the canonical, private local file. Reject aliases, symlinks, hard links, changed files, and changed owners. - Verify remote bytes before publishing a read-only archive and activity receipt in one transaction. Keep original files, errors, run status, event rows, and completeness metadata. - Document the command and its authority limits. Add real PostgreSQL and loopback S3 transport tests. ## Verification - The original implementation failed the new archive regression. It returned blocked instead of migrating the settled source. - Final source integration suite: 15 tests pass on disposable PostgreSQL and the bounded loopback S3 fixture. Coverage includes active unrelated work, failed and successful sources, a 4 MiB segment boundary, owner changes, file changes, unresolved owners, task bindings, allocations, and filesystem aliases. - The qualified source TypeScript check and module boundary check pass. - All 73 tests in five affected existing log-store, object-journal, cancellation, and migration-worker suites pass. The first borrowed dependency setup loaded work contexts from another worktree and failed four tests. Explicit database subpath aliases resolved that setup fault; no worker code changed. - Earlier test fault injection used the wrong storage method, and the first segment test reused a closed fixture endpoint. Those fixture defects were corrected before the final 15-test pass. - Local tests use Node 26. CI supplies the standard full typecheck, build, and test gates. Production object-storage credentials, live Railway replacement, and provider continuation are separate deployment acceptance steps. - Final exact-head CI: 59 successful contexts and 3 skipped, with all typecheck, build, test, E2E, security, and review gates complete. [Workflow evidence](https://github.com/paperclipai/paperclip/actions/runs/42332349644). - The first CI attempt had 12 jobs interrupted by runner shutdown. Their logs end in cancellation, with no test assertion found. The failed/cancelled-only retry passed all 12 jobs and recomputed aggregates. The 38 original successful job records kept their original execution times and were not rerun. Original interruption logs remain retained. ## Risks - This path preserves raw history. It cannot recover missing source bytes or prove a complete native transcript. - Uploads can finish before a later source check fails. The run reference stays local in that case; a retry must still satisfy all authority checks. - The new option is explicit. It adds no startup worker, recurring poll, or background scan. - Source transactions hold scoped locks only for ownership checks and publication. Streaming and remote verification run outside those transactions. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use, and code execution. The host does not expose the exact model ID or context window to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
51eb4ff705 |
fix(recovery): finalize an accepted native result after a hard kill (#19713)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run hands its accepted result to a controller lease, and the controller finalizes the task from that result > - A graceful shutdown releases that lease for the next server (#19591), but a hard kill (SIGKILL, OOM, host crash) cannot > - After a hard kill between result acceptance and finalization, the next server waited for the dead controller's 20-minute lease, so the run stayed `running` and the user's follow-up comment waited with it > - This pull request releases that lease at startup when the controller is proved dead by its boot ID and process identity, never by age > - The benefit is that the saved result finalizes within seconds after a restart, without another provider turn, and queued follow-up work continues ## Linked Issues or Issue Description Refs #19591 (the graceful-shutdown counterpart) and #19578 (boot-identity proof for legacy runs). **What happened?** A restart-chaos wave sent K1 (SIGKILL) 20 ms after a Claude run recorded `run.result.accepted` and `run.terminal`. The run then stayed `running` for more than 10 minutes, through four more restarts. The finalization row was `workspace_finalizing` with the stored result. Its lease belonged to the killed server: a different `controller_boot_id`, a `controller_pid` with no process, and `lease_expires_at` 20 minutes ahead. Startup recovery skips runs that already have a result. The result-commit worker waits until the lease expires. The task's follow-up comment could not run until then. **Steps to reproduce** 1. Let a native local run accept its result. 2. SIGKILL the server before finalization finishes. 3. Restart the server. The run stays `running` until the old lease expires. **Expected behavior** After a restart, a result lease held by a stopped controller of an earlier boot is treated as released at once. The stored result finalizes once, with no new provider turn, and the follow-up wake runs after it. ## What Changed - New `server/src/services/native-runtime/native-dead-controller-result-lease.ts`: `releaseDeadControllerResultLeases` finds `running` native runs with an accepted result and an unexpired lease. It requires the lease to belong to a different controller boot. It proves the controller stopped with `evaluateNativeControllerTakeover` (`requireStoppedController`): the process is dead, or its PID now names another process. It then expires that lease with a compare-and-set on the exact owner token, boot ID, and PID, and notifies the result-commit and finalization workers. - `server/src/services/heartbeat/recovery.ts`: `recoverNativeRunsAfterRestart` calls this first and records a `native.recovery.transition` event (`awaiting_result_finalization`) for each released run. - Only executor leases are released, because only the executor claim writes the controller identity together with the lease. Finalizer (`native-finalizer:`) and cleanup (`native-cleanup:`) claims keep an earlier executor's identity, so they are never released this way and expire on their own short leases. A finalization-resume token (`:resume:`) on a result row always belongs to the executor that adopted it, because a pending resume claim is taken only on a row without a result. - Also not released: a live owner, an owner without a recorded process, a lease of the current boot, a run this process is executing, and remote workspaces (`nativeWorkspaceSync` or `remoteWorkspace`), which keep their fenced lease. - The compare-and-set does not compare the stored expiry. Postgres stores microseconds that a JavaScript `Date` drops, and an exact-expiry comparison silently skipped the real lease in the live reproduction. - `doc/execution-semantics.md`: documents the rule. ## Verification - Red then green on embedded Postgres (`server/src/__tests__/heartbeat-native-restart-suspension.test.ts`): - A hard kill with a dead controller's executor lease (first claim, restart claim, or resumed claim) now finalizes on the next startup: run `succeeded`, finalization `committed`, task `done`, one result, one run, no provider dispatch. These tests failed before the fix with `status: running`. - A lease expiry stored with microsecond precision is released. This test failed against the first version of the fix and reproduces the live miss. - A live controller of another boot keeps its lease, and so does a lease of the current boot. - A live claimant's finalizer or cleanup lease is kept when the recorded earlier executor is dead. - `npx vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts`: 20 passed. - Related suites pass: `native-session-resumption`, `native-runner-restart-recovery.integration`, `native-run-projection-work`, `live-recovery-work`. One `live-recovery-work` legacy case timed out once under load and passed when run again by itself. - Live, local: `tests/restart-chaos/main.ts --workloads C1 --triggers result --faults K1 --rc06-masked`. The failing case above came from such a wave. I started its kept database and confirmed the stored lease state and the compare-and-set miss. Against that same database, the fixed release expires the stuck executor lease. Two later waves (12 tasks, `--triggers result`) passed every task. Run events were not kept for passing tasks, so I cannot confirm that a kill landed in the narrow pre-finalization window in those waves. - Live proof on head `8971874654`: local ACPX Claude, workload A1, seed 7411, `--triggers result` (a local variant that fires on A1 2.5–3.5 s after the run's result proposal), `--keep`. The run passed 4 of 4 tasks. In two tasks, K1 landed 1.5 s and 2.0 s after `run.result.accepted`. On the next startup, each run recorded "Server restart released the accepted native result of a stopped controller for finalization" (`controller_process_dead`). Each finished `succeeded` about 1 s later, with its task `done` and exactly one run per task (no extra provider turn). Before this fix, the same shape stayed `running` until the 10-minute budget ran out (seeds 7411, 7422, and 7423 from the RC-33b investigation in #19691). ## Risks - Startup could take a lease from a finalizer that is still running. Mitigations: the lease must belong to another boot, the controller process must be proved stopped on this host (the same proof the existing runner takeover uses), and the update matches the exact owner. Remote workspaces are excluded. Multi-host deployments keep the existing PID-liveness assumption that `claimNativeRestartRecoveries` already makes. - Low behavioral risk otherwise: finalization uses the existing reconciler path, which #19591 already exercises after a graceful drain. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, live restart-chaos waves). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
d1306f338b |
fix(runtime): preserve settled failed harness backups online (#19715)
## Thinking Path
> - Paperclip coordinates agents, tasks, and their saved execution
state.
> - A replacement server needs native provider backups in shared
storage.
> - The online pre-copy owner accepts successful runs with a committed
result.
> - A sealed backup from a failed or cancelled run can still contain
user history.
> - The existing owner defers those files even after their source
ownership is settled.
> - This change adds a separate preservation owner with exact source and
publication checks.
> - The copy retains the failed outcome and original files while making
the backup readable from a fresh home.
## Linked Issues or Issue Description
Refs #19137 and #19108 for the existing pre-copy and encrypted backup
owners.
**What happened?**
Online pre-copy reports `source_not_successful` for a sealed legacy
backup from a settled failed or cancelled run. The provider files remain
tied to the old disk.
**Expected behavior**
Copy the exact recorded backup when the source has no remaining writer
ownership. Preserve the run status, warnings, result, allocation,
selected session, and local files.
**Steps to reproduce**
1. Create a lease-stamped suspended native backup.
2. Settle its source as a failed run with a terminal-failure coordinator
and no owner.
3. Keep an unrelated execution active.
4. Run canonical online pre-copy. The previous owner defers the source.
## What Changed
- Add a narrow preservation owner for terminal source files. It checks
the physical finalization lock, task, coordinator, run, selected
session, source lease, and source fingerprints.
- Recheck that owner before the encrypted upload receipt and
current-backup publication.
- Reuse the finite workspace keepalive during long source checks and
defer source-local invalid archives.
- Add optional exact-root stamp verification. A valid previous backup
cannot authorize a changed current payload. Existing callers keep their
fallback behavior.
- Bind the archive input to the checked owner and record an activity
event after publication.
- Keep pending object receipts after an uncertain or failed publication.
- Add focused PostgreSQL and filesystem controls. Update the existing
migration runbook.
## Verification
- Real PostgreSQL baseline on unchanged production: the new positive
case fails with `source_not_successful` instead of publishing the
backup.
- Same case passes after the change and decrypts/restores the exact
provider history.
- Final head
|
||
|
DottaandClaude Opus 5.5
|
4612fa9fb4 |
fix(native): restart a drained Daytona run whose runner opened its session (#19711)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a `paperclip-runnerd` process in a sandbox. A graceful shutdown (every rolling deploy) suspends live native runs "without cancelling provider work". The next boot then decides how each run continues. > - #19583, #19616 and #19684 let restart recovery start a remote bootstrap again when durable evidence proves that no provider work exists. > - A chaos replay found another window. The drain suspended a remote run just after its sandbox runner opened the provider session, before the controller sent a turn and before a session snapshot existed. > - Recovery accepted only an unfinished `session.open` in the controller checkpoint, so it blocked the task as ambiguous. A Retry recovered it, but automatic continuation was lost. > - A related replay showed that the same-run bootstrap reuses the sandbox. The interrupted runner can still hold the runner ingress port there (`runner_ingress_bind_conflict`). > - This pull request proves the "session opened, no turn" case and stops the interrupted runner before relaunch. > - The benefit is that a deploy during Daytona startup continues automatically, without a port conflict. ## Linked Issues or Issue Description **What happened?** A chaos replay (ACPX Claude on Daytona) sent SIGTERM while a remote run was starting. The drain recorded `restart_suspended`. The dying controller still launched the runner and finished `session.open`. On the next boot, restart recovery returned `ambiguous_remote_recovery_evidence` with disposition `blocked`. The decrypted controller checkpoint of that run showed: - `run.prepare` completed and `session.open` completed (provider session opened). - Committed events `session.started`, `session.capabilities.updated` and `session.goal.snapshot` (`ackedSourceSeq: 3`). - No turn command. No run-profile session snapshot and no projected provider events. On the next run in that replay, a same-run bootstrap relaunched the runner in the same sandbox (one lease for all attempts). It failed with `runner_ingress_bind_conflict: failed to bind listener: Address already in use`, because the interrupted runner from 51 s earlier was still listening there. The bounded same-run retry recovered it on its last attempt. **Expected behavior** When the durable controller record proves that no turn was ever sent, recovery starts the bootstrap again. It first stops the interrupted runner in the reused sandbox. **Steps to reproduce** 1. Start a native run on a Daytona environment. 2. Send SIGTERM to the control plane while the runner opens its provider session. 3. Restart the control plane after the controller lease expires. 4. See that restart recovery blocks the task with `ambiguous_remote_recovery_evidence`. Root cause: in `server/src/services/native-runtime/native-restart-recovery.ts`, the remote checkpoint proof (`remoteBootstrapProvenIncomplete`, #19583) required an unfinished `session.open` with no committed events, and no provider events at all. A completed `session.open` with only session-open events had no rule, so it fell through to `ambiguous_remote_recovery_evidence`. Refs #19583, #19616, #19684, #19524 ## What Changed - `native-bootstrap-proof.ts`: new `runnerdStateHasNoProviderTurn` proof. The state must hold only `run.prepare` and `session.open` (pending or completed) and no warm, staged or adopted state. Every committed event must be a session-open event, with contiguous sequence numbers from 1, so no event is hidden. Only the controller issues turn commands, and it writes each one to Postgres before sending it. - `native-runner-checkpoint.ts`: `nativeRunnerCheckpointRemoteBootstrapProof` replaces `nativeRunnerCheckpointProvesIncompleteBootstrap`. It also reads a bounded number of `event:N` parts and returns which proof holds. It still rebuilds the state in memory. - `native-restart-recovery.ts`: a remote run can now claim `bootstrap_incomplete` with reason `remote_session_opened_without_turn`. Projected provider events may only be session-open events (`harness.ready`, `session.started`, `session.resumed`, `session.updated`), and only together with the stronger session-opened proof. A turn or provider event still blocks. - `native-session-executor.ts`: - For a remote sandbox target, the executor accepts the same proof on the restored record. - Before the same-run bootstrap relaunches, `stopStaleRemoteRunnerBeforeBootstrap` stops the interrupted runner in the reused sandbox. It uses the verified identity marker and process check from reattachment, which signals only a process whose identity matches this run: SIGTERM, a wait of up to 10 s, then SIGKILL. - The stop is best effort. If no prior runner can be verified, nothing is stopped, and the bounded same-run retry still handles any conflict. - `doc/run-log-events.md`: document the reason and the stop. ## Verification - Red, then green: `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "drained remote"`. Before the fix, the opened-session case (with a projected `session.started`) returned `blocked` / `ambiguous_remote_recovery_evidence`. - These keep the hold: a `turn.start` command, a committed `turn.started`, an acknowledged event that is missing from the record, and a projected `turn.started`. The executor test restores the checkpoint and checks that the local proof is false and the remote proof is true. - Regression: `native-runner-restart-recovery.integration`, `native-session-executor`, `native-restart-recovery`, `native-bootstrap-retry.integration` and `heartbeat/recovery`: 1460 passed, 14 skipped. - Stale runner stop: controlled sandbox-runner tests (`native-session-executor.test.ts`, "stale runner stop before a same-run remote bootstrap") run the real reattachment identity and process scripts against a fake sandbox runner: - a verified runner that exits on SIGTERM gets SIGTERM only; - a runner that ignores SIGTERM gets SIGTERM, then SIGKILL; - a runner with another identity gets no signal; - no live prior runner means no signal. - Live Daytona proof (wave `w-20261107T021837-s29`, ACPX Claude, `--faults K2 --triggers time,result`, 4 tasks, concurrency 1). The image was built from current master (`paperclip-daytona-runner@sha256:71f711ab…`), and its content id matches this branch. - All 4 tasks crossed a graceful restart, and all passed R1. Two of the K2s fired on the `result` trigger, during a running remote turn. - The K2 during a turn reattached to the live remote runner (`reattach_remote_runner`, generation 2), as designed. - Task t01 failed R6 with an orphan Daytona sandbox that no lease references. The graceful restart came during sandbox creation. This is a separate signature, outside this change. - The timed faults did not land in the exact "session opened, no turn" window. The integration tests cover that window with the evidence from the saved replay database. ## Risks - The proof relies on two facts: only the controller issues turn commands, and the controller writes each command before sending it (`#sendJson` waits for the checkpoint flush). A turn always leaves a `turn.start` command in the record. - The provider session opened in the old sandbox runner is not resumed. The new bootstrap opens a fresh session. No turn ran in the old one, so no work is lost. - The runner stop sends signals inside the run's own sandbox. It signals only after the existing reattachment identity and process checks pass. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
18be7ac696 |
fix(server): retire idle native sessions before tenant sleep (#18988)
## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Native agents can retain a warm session between useful turns. > - That owner must close and save its files before the tenant can sleep. > - A failed close must keep the owner visible until retirement succeeds. > - This PR prepares idle resources under the exact current admission hold, then takes a fresh safety snapshot. > - Busy turns, uncertain closure and changed holds still prevent sleep. > - CI also exposed a chat authorization race. The fix repeats full authorization after the exact delivery commits. ## Linked Issues or Issue Description Refs #18639 for local work and incoming wake boundaries. Related #18829 covers suspension receipts after admission. **What happened?** Idle preflight did not retire warm Native owners. Three failed-close paths could forget an owner without proving that its provider stopped. CI also found a false chat rejection when a delivery changed from processing to processed between two validation reads. **Expected behavior** Retire only idle owners under the exact bounded hold. Wait for closure and file collection. Retain failed owners. Scan safety after preparation. Keep diagnostic reads passive. A newly committed chat delivery must pass the full current authorization checks before attestation succeeds. **Steps to reproduce** Finish a warm Native turn. Start an idle drain and request safety with its owner ID. Delay or reject provider closure, or replace the hold during preparation or scanning. The report must keep the instance awake. For the chat race, commit the exact delivery between the first binding read and its fallback read. **Paperclip version or commit** Current head `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` normally integrates actual master `1aa0c3639714f4976cfa56f95aae6e2b7d1235ac`. The merge preserves every unrelated master path. Fresh exact-head CI and complete review remain required. ## What Changed - Retain uncertain Native owners after failed restart, idle timeout and failed-turn closure. - Prepare Native retirement through the existing heartbeat facade. Reuse provider closure and stopped-file collection. - Track preparation under the exact idle owner and expiry. Reject new activity, changed holds and expired holds before and after awaited work. - Take the final durable and local safety snapshot after preparation. Keep diagnostic inspection passive. - Cover ephemeral workspace retirement and useful next use, supported plugin drain and SDK persistence, busy owners, failed closure and hold replacement. - Keep dedicated recovery storage explicit. Use the same validated options for support and storage. Probe-free helpers retain the original implementation bodies and preserve default callers. - Complete the Native fixture through real retirement and budget evaluation. Keep its rejection, file, credential, accounting and strict NONE assertions. - Repeat the complete chat authorization once if the fallback sees the exact batch now processed. Preserve owner locks, principal and binding checks, denial handling and the original retry deadline. ## Verification - The support-mode regression reproduces one failure with four passes under the old policy, then passes all five controls. It retains prior allocation and cleanup cases. The explicit mode does not probe embedded PostgreSQL or use an admin alias. - The deterministic chat race baseline reproduces two authorized failures with all four current-authority denial controls passing. Both Slack and Discord use real PostgreSQL and the real authorization function. The negative setup commits before the attestation transaction; it does not claim to revoke access through a row lock. - The product fix passes normal own App declaration emission with no public declaration changes, plus both current normal owning compiler programs (399 and 611 roots). The same six controls pass after the fix. The full original file plus these controls passes all 30 tests with no skips. - The original controlled Native continuation passes: one test, no failures or skips, 1.718 seconds in its body and 23.138 seconds in its launcher. The original 40-second case and 180-second outer bounds are unchanged. It verifies held closure, strict NONE, an ordinary authorized comment, saved session/files, useful output, encrypted credentials, accepted results and settled accounting. All ten observed processes and groups close; worker connections and listeners reach zero. A separate post-owner audit finds zero sessions; the successful owned database is normally removed. - Earlier preparation, HTTP, lifecycle, workspace and plugin controls remain source-specific evidence. They are not a claim that every final-tree test ran locally. - The original CI assertion log is retained. The controlled race proves a possible cause; the original CI SQL schedule was not captured. Two other jobs lost their self-hosted runners and provided no test verdict. Earlier asset timeouts, readiness failures and unexplained later recovery remain separately recorded. No timeout, failure predicate or authority check was weakened. Focused public commands: ```sh pnpm exec vitest run server/src/__tests__/idle-sleep-preparation.test.ts server/src/__tests__/instance-settings-routes.test.ts --maxWorkers=1 --no-file-parallelism pnpm exec vitest run server/src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts --maxWorkers=1 --no-file-parallelism ``` Local database qualification uses fresh owned test databases on existing PostgreSQL, bounded pools and one worker. The local allocation adapter changes only database allocation for this test module; it runs normal migrations and client cleanup. It is not part of the published source. No new cluster is created. Local acceptance measured the `0a69efa6` integration with the same committed authorization bytes. The later normal master merge adds startup-timeout classification and recovery changes. Those paths are outside the controlled healthy-backend witness; retirement and authorization hunks are preserved. Exact-head remote CI at `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` has completed with 56 successful contexts and four skips. The complete current Greptile summary is5/5 with no new actionable findings; all review threads are resolved. Ponytail’s current turn is explicitly paused without a current-head verdict; earlier actual findings are fixed and accounted for. No whole final-tree local repository build/test run, live-provider restore or deployed suspension is claimed. Actual master subsequently advanced to `d5c4f7dd02d408a261cedd9a1926b9900a4aedc4`. Its local normal-merge projection with this reviewed head preserves all20 owned paths and every unrelated master blob and mode. Current Runner TypeScript, App declarations and their full owning compiler programs pass. Focused Native cleanup and warm-continuation composition checks are being completed before merge; the new upstream suspension-retry behavior is not inferred from the older controlled-backend witness. The remote PR head remains unchanged. ## Risks - Privileged idle preparation can close idle warm Native sessions. It requires the exact current hold. - Failed closure retains its owner and can keep the tenant awake until normal retry succeeds. - A changed hold cannot authorize sleep. Started cleanup remains tracked until it settles. - The chat repair can perform one extra full authorization read. It keeps the existing deadline and does not grant authority from the fallback query. - Unsupported resources and active work retain their existing blockers. This PR does not certify every transport, plugin or deployed host. ## Model Used OpenAI `gpt-6.1-sol` produced the delegated changes and verification. OpenAI Codex, based on GPT-6, performed managing review and integration. Both used repository tools and code execution. The coordinator's exact backend snapshot and context-window limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green at exact head `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups at the exact current head - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandPaperclip
|
55c9c44e41 |
fix: preserve database-only failed controller evidence (#19714)
## Thinking Path > - Paperclip preserves failed work as well as successful work. > - Ephemeral enrollment must retain each selected controller source. > - A failed source can already exist entirely in an encrypted database checkpoint. > - Its original inline snapshot does not create a local filesystem dependency. > - The migration owner must prove that both legacy files are absent before accepting that case. > - This change preserves the failed evidence without creating execution authority. ## Linked Issues or Issue Description Refs: #19678. **What happened?** Startup still reports `native_controllers_require_migration` for a failed retained source with a valid database checkpoint, its original inline snapshot, and no local controller or supervisor. The online archive owner reports that the source is already absent. The existing database-only preservation case accepts only a missing inline snapshot. **Expected behavior** Accept the exact database-only failed evidence after full checkpoint and identity validation. Keep a present local payload blocked. Retain the failed run, original snapshot, coordinator, task pointer and queued work. **Steps to reproduce** 1. Store a valid encrypted checkpoint for a failed native run. 2. Keep its matching inline session snapshot and terminal-failure coordinator. 3. Remove both legacy controller files. 4. Check local controller obligations. 5. The old owner reports an obligation even though the evidence is already in the database. **Paperclip version or commit** The fix builds on current master `d5c4f7dd02`. **Deployment mode** Hosted native Runner with durable controller checkpoints. The local regression uses the existing real PostgreSQL and encrypted checkpoint fixture. ## What Changed - Add a narrow database-only failed-source preservation case to the existing migration owner. - Require the decoded checkpoint, protocol header, admitted execution, selected run and inline snapshot to match their exact scope. - Require both the controller and supervisor paths to be absent. A file or link keeps the local obligation blocked. - Retain the earlier null-snapshot predicate and all original failed execution evidence. Create no recovery metadata or executable authority. - Exercise the actual startup and maintenance consumers. Document the preservation boundary. ## Verification - Final revision `1b1da543dafd576e3c420657e3c2c6d0c882172d`: all checks are terminal, with 56 successful and four skipped, zero pending and zero failures. The [required CI aggregate](https://github.com/paperclipai/paperclip/actions/runs/42329501690/job/128275664414) passed. Build, types, server/Runner/serialized suites, end-to-end aggregate and Snyk passed. [Fresh exact-head review](https://github.com/paperclipai/paperclip/pull/19714#issuecomment-6701018015) is 5/5 with no open threads. - Review found that the first non-recovery assertion stopped at the queued fixture's missing current binding. The fixture now binds the exact native issue/session before calling the real resolver and asserts `unsupported:metadata_format` from the selected failed source. That corrected case passes in 4.58 seconds, with wrapper exit 0 and cleanup complete. The production predicate is unchanged. The original startup RED remains valid. The final source/test type graph, module boundaries and diff checks pass. Fresh revision CI and review remain required. - The actual startup regression failed before the source change: one case, 16.62 seconds, expected the absent-files obligation to be false but received true. The wrapper exited 1 and completed cleanup. This failure is retained. - The final focused owner run passed nine cases in 10.79 seconds. It covers the new DB-only case, controller/supervisor presence, every inline scope field, foreign checkpoint Runner/session identities, the unchanged null-snapshot case, and existing online/offline archival controls. The wrapper exited 0 and completed cleanup. - The new case verifies unchanged run, coordinator, checkpoint and task-session rows, retained queued work, no upload, and no invented recoverable session. - The full qualified server/test source type graph passes with db/shared/Runner source aliases. Module boundaries and diff checks pass. Required broad package builds and regression suites remain in CI. - Read-only source diagnosis validated one exact existing checkpoint with the canonical decoder and matching header, run, Runner and session bindings. No source profile or secret values were emitted. No hosted mutation, admission hold, cancellation, status change, or writers-stopped claim was performed. ## Risks - A false database-only classification could hide local bytes. Both exact legacy paths must be absent, and every protocol and database binding must match. - Corrupt or mismatched checkpoints still fail closed. Existing failed evidence does not become successful or executable. - The original null-snapshot case is unchanged. Present local files still require the existing preservation path. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |