mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
ba3fe09954340ca080307a73c857444cd74fcdc0
2052
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
cbf811cd3b |
fix(recovery): prove a replaced deployment's copyback owner gone automatically (#19913)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save their workspace after the result is accepted, under
a copyback owner receipt
> - A deploy can restart the server while that save is in progress
> - The new server runs in a new container, so it cannot see the old
process and fails closed
> - The run then waits on a board action that asks a person to verify
the previous controller
> - Recoveries must continue without a human step, and copy must not ask
anyone to inspect a server
> - This pull request adds a deployment-level rule to the shared
owner-liveness primitive and uses it in workspace finalization
> - The benefit is that a save interrupted by a deploy resumes on its
own, with no second writer
## Linked Issues or Issue Description
**What happened?**
A deploy restarted the server while three runs were saving their
accepted results. Each run kept a copyback owner receipt that names the
old Railway deployment on another container. The restarted server
reported `native_workspace_finalization_owner_unverified`, owned by the
board, with the next action "Verify the previous controller and its
workspace-copyback processes have stopped...". The cloud harness stop
receipt did not arrive, so the runs stayed in "Recovery needed" for
about an hour, until a person confirmed the stop by hand.
**Expected behavior**
The restarted server proves that the previous controller is gone,
resumes the save, and settles the run with no human step. No user-facing
text asks a person to verify a server or controller.
**Steps to reproduce**
1. Start a native run on a Railway service with a volume.
2. Let the result be accepted and the workspace save start.
3. Deploy, so that the server restarts during the save.
4. See that the run waits for a board confirmation.
**Paperclip version or commit**
Observed on
|
||
|
DottaandClaude Opus 5.5
|
99529f9d71 |
feat(observability): local speed budgets and a run-log budget check (#19863)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When someone messages an agent, the reply must start as fast as
possible, and restart-safety work must not slow it
> - The durable ownership plan makes this invariant 8 ("fast turns") and
asks Phase 1 to turn the Phase 0 baseline into p50/p95 budgets per
adapter
> - Without budgets, a regression in time to first token, agent boot or
restart recovery shows up only when someone reruns the baseline by hand
> - This pull request records the budgets and adds a per-run check that
writes a run-log row when a run exceeds its p95 budget
> - The benefit is that every later phase can show "no regression
against the budgets" from the run log, without new waits on the turn
path
## Linked Issues or Issue Description
Refs #19831 (plan, "Performance constraint", rule 4, and invariant 8).
Refs #19846 (the Phase 0 baseline these budgets come from). Stacked on
#19861.
**Problem**
The plan requires p50/p95 budgets per adapter for TTFT, agent boot and
restart recovery latency, and a check against them. Neither exists.
**Expected behavior**
Budgets are recorded in the plan and in code. A run that exceeds a p95
budget gets one run-log row that names the metric, the measurement and
the budget.
**Impact**
Phase 2–5 exit criteria can cite budget breaches from chaos waves and
staging. The check adds no wait before `turn.start`.
## What Changed
- New `server/src/services/native-runtime/native-speed-budgets.ts`: the
local budget table (codex, claude, opencode) for TTFT and agent boot
(cold and warm) and restart recovery latency. Each budget is 1.2× the
baseline, rounded up to 0.5 s. `evaluateSpeedBudget` judges one run
against p95. An unknown start kind uses the looser budget. Remote runs
and adapters without a baseline are not judged.
- `native-run-trace.ts`: the trace takes an optional `budget` subject.
It checks `task.time_to_first_token`, agent boot (end of
`heartbeat.queue` to the end of the first `runner.turn.submit` or
`runner.session.resume`) and restart recovery once each. It writes
`run.performance.budget_exceeded` through `defer`, so the write is
detached from the event path. `finish` now drains diagnostics that defer
more work.
- `native-session-executor.ts`: passes the adapter and environment to
the trace, labels cold or warm at the existing `provider_session` reuse
receipts, and checks restart recovery at a recovered run's first
provider event, measured from the startup recovery `ready` time.
- Docs: budget tables in
`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` ("Speed
budgets (Phase 1)"), and the new event in `doc/run-log-events.md`.
## Verification
- `npx vitest run
src/services/native-runtime/native-speed-budgets.test.ts
src/services/native-runtime/native-run-trace.test.ts` (in `server/`): 34
tests pass. They cover:
- every budget has p50 ≤ p95;
- the provider-to-adapter mapping;
- p95 judgement, and the looser budget when the start kind is unknown;
- one breach for a slow boot, and none for a second submit or a fast
TTFT;
- a deferred TTFT breach is written before the trace closes;
- restart recovery is checked once;
- remote runs and runs with no budget subject write nothing.
- `npx tsc --noEmit -p server`: no errors.
## Risks
- Low risk. This is a run-log change only (AGENTS.md §5.7, no extra
review). The OTLP and Telemetry paths are unchanged.
- The baseline samples are small (2–12 per cell) on a shared host.
Budgets carry 20% headroom, and the codex restart budget (8.0 s p95) is
tight. Expect some breach rows on a loaded machine. They are diagnostics
and change no behavior.
- Restart-recovery budgets use run 4 of the baseline, as merged in
#19846.
- Restart recovery is measured from the startup recovery `ready` time to
the first provider event, which matches the baseline definition.
- Previous-release story: no persisted run state changes. The new event
is additive.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
d5dd09c966 |
docs(plans): design turn-scoped ACPX MCP authority; correct the Claude baseline (#19912)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Follow-up messages must start fast. #19888 made Codex and OpenCode follow-ups 2.6–5× faster by keeping their runners warm. > - Claude (ACPX) did not gain, because ACPX relaunches the Claude process and reloads its session on every follow-up (3.2–4.2 s). > - That relaunch is a safety boundary: the ACPX MCP endpoint lives as long as the process and cannot attribute a tool call to a turn, so reaping the process is the proof against late calls. > - Removing it needs a new authority model, which needs a reviewed design before any code. > - This pull request adds that design and corrects the Claude baseline. ## Linked Issues or Issue Description Refs #19888 (warm default), #19846 (TTFT baseline, finding 4). **Docs issue** The TTFT docs did not explain why Claude follow-ups relaunch, what removing that would require, or the corrected Claude numbers. **What is wrong or missing** `doc/plans/2026-11-07-warm-follow-up-ttft.md` reported a Claude follow-up at 10.5 s, measured under heavy host load. A quieter re-measurement gives 6.3 s per-turn against 7.1 s warm. There was no design for removing the per-turn relaunch. **Proposed fix** A design document for turn-scoped ACPX MCP authority, and a corrected-baseline section in the warm follow-up doc. ## What Changed - New `doc/plans/2026-11-07-turn-scoped-acpx-mcp-authority.md`, which covers: - the measured cost breakdown; - how the current boundary works (tool bridge bearer, call-ID tombstones, `restart_idle_provider`, `run.attach`); - a threat model (late tool calls from a settled turn, replay, credential replay, cross-turn and cross-run confusion, restart replay); - three candidate designs with their safety arguments: turn-scoped credentials in a live session, a proof that no calls are in flight, and an in-process session reload with per-generation credentials, which is recommended for a spike; - restart and chaos implications, the expected TTFT gain, and next steps. - `doc/plans/2026-11-07-warm-follow-up-ttft.md`: corrected Claude baseline. Per_turn stays the default; the earlier 10.5 s was host noise. Finding 4 now points to the design and drops the idle-retirement idea, which would only bring warm level with per-turn. - Docs only. No code changes until Dotta has reviewed the design and a security review is done. ## Verification - The Claude numbers come from `ttft-baseline.ts --adapters claude --samples 6 --phases coldwarm --lifecycle <per_turn|warm>` on master `fc6e1b0346` (load average about 9). - The step breakdown comes from temporary timing in runnerd's ACPX backend. It was not committed. - `node scripts/check-maps.mjs` passes. ## Risks - Docs only. The design proposes no behavior change until it has been reviewed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
177f642da7 |
fix(recovery): continue native runs that failed before their first claim (#19911)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs go through a coordinator claim before any provider work
starts
> - A run can fail before that claim, for example while it restores the
previous harness backup
> - The failure path then records a no-replay hold, as if the run had
acted
> - Workspace recovery saves the files, but the task stays blocked and
asks a person to send a message
> - Recoveries must continue without a human step when no side effect is
possible
> - This pull request lets the existing safe-replacement worker continue
such runs once, after their files are saved
> - The benefit is that these tasks resume automatically, and runs that
may have acted keep today's reconciliation
## Linked Issues or Issue Description
**What happened?**
Native runs that failed before their first coordinator claim were never
continued automatically. They had no claim, no provider process and no
provider or tool events. Their result recorded `executionRecovery: {
kind: "bootstrap", providerWorkStarted: false }`. The failure path still
moved the coordinator to `terminal_failure` with
`native_continuation_requires_reconciliation` and recorded a resolved
`preserve_without_replay_v1` hold (replay blocked, board owner).
Workspace recovery then saved the files, but the next action stayed
"Saved work recovered. Send a message to continue."
`nativeReplacementPending` accepted only a few failure codes, and the
interrupted-preparation path covers only legacy runs.
**Expected behavior**
When a native run provably failed before its first claim, Paperclip
continues it once, automatically, after its files are saved (or when it
had no workspace to save). Runs with a claim, a process, a provider or
tool event, or recorded provider work keep the existing reconciliation.
**Steps to reproduce**
1. Start a native run with a remote workspace.
2. Make it fail before `native.coordinator.claim` (for example, abort
the harness-backup restore).
3. Let workspace recovery save the files.
4. See that the task stays blocked and asks for a message.
**Paperclip version or commit**
Observed on
|
||
|
DottaandClaude Opus 5.5
|
fc6e1b0346 |
perf(native): keep local Codex and OpenCode runners warm by default (#19888)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When someone messages an agent, the turn must start fast. Time to first token (TTFT) is a first-class metric. > - The Phase 0 baseline (#19846) found that every local turn launches a new runner and loads the provider session again, because `per_turn` is the default lifecycle. That made a follow-up message slower than a new task. > - Warm runners keep the runner and provider session alive between runs. #19874 fixed the defect that broke warm OpenCode follow-ups, and #19879 bounds an idle warm runner whose controller dies. > - This pull request makes warm the default for local Codex and OpenCode agents, and migrates existing agents that only carried the old creation default. > - The benefit: follow-up TTFT p50 drops from 12.0 s to 4.7 s (Codex) and from 21.7 s to 4.2 s (OpenCode), with no new restart-chaos failures. ## Linked Issues or Issue Description Refs #19846 (TTFT baseline), #19852 (startup sub-spans), #19874 (warm completion contract), #19879 (warm reconnect grace). **What existing behavior does this improve?** The default runner lifecycle for native agents, and how a follow-up message starts. **Current behavior** Agent creation writes `lifecycleMode: "per_turn"` into every native agent. Every follow-up launches a new runner and restores the provider session: 4.2 s (Codex) and 16.3 s (OpenCode) at p50 before the turn starts. **Proposed behavior** New native agents store no lifecycle. On a local target, an unset lifecycle resolves to `warm` with the default 5-minute idle window for Codex and OpenCode. Other harnesses, remote targets and managed providers without warm support stay `per_turn`. An explicit choice always wins. Migration 0443 clears the stored creation default from existing agents, except where an edit set "Turn by turn" explicitly. **Reason and benefit** Measured locally (`doc/plans/2026-11-07-warm-follow-up-ttft.md`). Warm follow-ups are 2.6–5× faster for Codex and OpenCode, and restart recovery latency is equal or lower. ## What Changed - `native-lifecycle.ts`: an unset `agentMode` resolves to warm (5-minute idle window) for local `codex_app_server` and `opencode_server` agents. `runtime-selection.ts` passes `localTarget`. Explicit `per_turn` or `warm` wins. An admitted session keeps its persisted policy. ACPX Claude stays `per_turn`: it relaunches its provider process on every follow-up by design, so warm did not shorten its follow-up (10.7 s warm vs 10.5 s per_turn). - Creation and configuration writes no longer store `lifecycleMode: "per_turn"` for Codex, Claude or OpenCode. This covers `normalizePaperclipRunnerAdapterConfig` (runs on every create and update), `paperclipRunnerTransitionConfig` (switching to the native runner), `agent-runner-selection.ts`, `codex-local` `build-config.ts`, and the simplified new-agent setup (`paperclipRunnerCreationLifecycle` in `@paperclipai/shared`). Grok, Pi and Cursor keep their explicit value. - Agent form: the Runner lifecycle field shows the unset state as "Automatic". "Turn by turn" and "Warm session" remain. - Migration `0443_native_warm_follow_up_default.sql`: clears `lifecycleMode = 'per_turn'` from native Codex, OpenCode and Claude agents unless `agent_config_revisions` shows an edit that set it to `per_turn`. Each migrated agent gets an `agent.updated` activity entry with the old value (`actor_id = migration:native-warm-follow-up-default`). Creation records no revision, so a "Turn by turn" pick made at creation cannot be told apart from the old default. The product owner approved migrating those agents. Claude agents follow the default, which is `per_turn` today. - Restart chaos harness and TTFT driver: `--lifecycle per_turn|warm|default` (saved in the wave plan). `default` removes the catalog's pinned lifecycle, so a wave exercises the real default path. - Docs: `doc/DEVELOPING.md` "Runner lifecycle default", and the results in `doc/plans/2026-11-07-warm-follow-up-ttft.md`, which also documents baseline finding 4 (Claude's resumed turn-start gap is ACPX's per-turn process generation, not the provider). ## Verification - `server`: `native-lifecycle.test.ts` (new default cases: local Codex and OpenCode warm, Claude, remote, unknown target, unmeasured harness, managed Claude, admitted per-turn session, explicit choices, non-native adapter), `agent-runner-selection.test.ts`, `runtime-selection.test.ts` (335 pass). - `packages/db`: `native-warm-follow-up-default-migration.test.ts` on embedded Postgres. Default agents are cleared, the explicit edit is kept, other harnesses and legacy adapters are untouched, audit rows are written once, and a rerun is a no-op. Without the explicit-edit filter the test fails. `check:migrations` passes. - `codex-local` `build-config.test.ts` (65), `ui` `config-fields.test.tsx` (40), restart-chaos unit tests (134), `tsc --noEmit` for `server`. - `adapter-utils` `paperclip-runner-lifecycle-default.test.ts`: configuration writes and adapter conversion store no lifecycle for Codex, OpenCode and Claude, keep `per_turn` for the others, and preserve explicit choices. It fails without the fix. - End-to-end default check on this branch (`ttft-baseline.ts --lifecycle default`, codex and opencode, 2 samples): the agents read back with no stored lifecycle, and every follow-up reused the runner and provider session (no `runner.process.launch`). Follow-up TTFT was 3.6–4.8 s. This check found the per-write `per_turn` injection fixed here. - Restart chaos (codex and opencode, A1/B1/B5, K1/K2/K6, seed 4242), 8/8 PASS each with no open signatures: forced warm, forced per_turn, and `--lifecycle default` on this branch (agents created without a lifecycle). - `pnpm verify` selects broader CI (harness config is a global input), so required CI supplies the broad coverage. ## Risks - Behavior change: local Codex and OpenCode agents keep a runner and provider process for up to 5 minutes after a run. Instance sleep waits for that idle window. The process is reaped by the idle timer, by graceful shutdown, or by the #19879 reconnect grace if its controller dies. - Migration: agents whose "Turn by turn" was chosen at creation become Automatic. The activity entry records the old value; choosing "Turn by turn" restores it. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): a task session keeps the lifecycle policy it was admitted with, so sessions saved before this change stay `per_turn`. New sessions get the default. No run, checkpoint or launch state changes shape. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
76994c5027 |
fix(native): bound failover backup restores by inactivity, not total time (#19906)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native per-turn session that continues an earlier conversation first restores its failover backup from durable storage. This happens before the coordinator claim, so no provider has started. > - That restore ran under a fixed 300 s total deadline (`AbortSignal.timeout(300_000)`). The deadline covers the download and the local decrypt, digest, extraction and provider-artifact steps. > - On staging, restore time for resumed sessions grew during the day. It reached 300 s, and then every resumed run of one review agent failed with the generic "The operation was aborted". Fresh sessions were not affected. > - A failure at this point also enters the saved-work reconciliation path, so the user request stopped. > - This pull request bounds each durable read by inactivity instead of total time. A slow transfer continues while bytes arrive. A read that stops sending bytes fails with a named code. > - The benefit is that resumed sessions no longer fail only because their backup is large or the store is slow. ## Linked Issues or Issue Description Refs #19894 (adds the same idle-deadline rule for workspace snapshot reads; this PR covers the separate native failover-backup reads). **What happened?** On staging, a PR-review agent had 17 runs fail in about 80 minutes. Each failure had the message "The operation was aborted" and the code `native_continuation_requires_reconciliation`. Every failing run resumed an existing session (`sessionIdBefore` set, per-turn lifecycle). Every fresh-session run in the same window succeeded. In each failing run, the error came 304–309 s after the `environment.startup` span, and no `native.coordinator.claim` span was recorded. The run result had `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. Earlier resumed runs of the same agent show the time from `environment.startup` to `native.coordinator.claim` growing from about 30 s to about 200 s before the failures started. The only fixed 300 s deadline on that path is the failover-backup restore in `restoreDurableHarnessForExecution`. **Expected behavior** A resumed session restores its backup when the store keeps sending bytes, even when the total time is more than five minutes. A read that stalls fails with a named, diagnosable code. A failed restore publishes no partial backup and keeps the saved backup. **Steps to reproduce** 1. Persist a native failover backup with provider artifacts. 2. Restore it through a storage reader that sends bytes slowly, so the full restore takes longer than the deadline. 3. Before this change, the restore fails with "The operation was aborted". A reader that stops sending bytes, with no caller deadline, waits indefinitely. ## What Changed - `native-harness-backup-storage.ts`: add `downloadIdleTimeoutMs` to `restoreDurableNativeHarnessBackup` and `verifyDurableNativeHarnessBackup`. When set, each durable `getObject` (the backup object and the provider artifacts) runs under an idle timer. Each received chunk renews the timer. The timer is active only while a read is in flight, so local decrypt, digest, extraction and copy are never timed. On expiry the whole restore aborts and throws `native_harness_storage_download_stalled`. Caller cancellation keeps its own error. - `verifyRecoverableNativeHarnessBackup` and the executor's `restoreDurableHarnessForExecution` use the five-minute idle limit (`NATIVE_HARNESS_DOWNLOAD_IDLE_TIMEOUT_MS`) instead of `AbortSignal.timeout(300_000)`. - The whole restore or verification keeps a 30-minute outer limit. It fails with `native_harness_storage_restore_deadline_exceeded`, so a hung local step stays bounded and named. - The executor passes the startup Stop signal (`providerPackSourceAbort`) into the restore, so Stop (also a Stop that came before the restore began) ends it. Only a startup that already completed does not reuse its signal. The restore checks the signal after the local copy and before it publishes `current`. - The wrapped storage is a `Proxy` that replaces only `getObject` and forwards every other member to the real service. - `doc/DEVELOPING.md`: document the idle limit and the named code. Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted shape changes. Backups, artifact rows, receipts and checkpoints saved by the previous release restore unchanged. A deploy that lands during a restore stops the process. The new process starts the restore again from the saved durable backup under the new limit. The existing hash, size, encryption and atomic publication checks are unchanged. ## Verification Run from `server/`: - `pnpm exec vitest run src/services/native-runtime/native-harness-backup-download-deadline.integration.test.ts` — 4 tests passed (embedded PostgreSQL, local-disk storage, an OpenCode backup with provider artifacts). The tests are in a focused suite, so the existing 990-line storage suite does not grow. - `pnpm exec vitest run src/services/native-runtime/native-harness-backup-storage.integration.test.ts` — 68 existing tests passed. - New: "keeps restoring a backup whose download outlasts the idle window while bytes keep arriving". With the former total-deadline policy, the same slow transfer fails with "The operation was aborted" and publishes nothing (the staging message). With the idle limit, the restore completes after more than twice the window, restores the session and the OpenCode launch artifacts, and verification also completes. - New: "names a download that stops sending bytes and publishes no partial backup". It checks restore and verify, checks that no `current` backup is published, and then restores the same saved backup successfully. - New: "stops at the outer bound during local copy and publishes nothing". - New: "lets Stop end a progressing download with its own reason, including a Stop before it starts". - RED: with the production files reverted to `origin/master`, the stall test does not finish and times out after 10 s, because no stall is detected. The other two tests pass on the old code, because the total deadline was the executor's caller policy. The first test reproduces that policy directly. - `pnpm exec vitest run` on `native-harness-online-preservation`, `native-harness-filesystem-independence` and `native-controller-migration` integration suites passed. `native-warm-checkpoint-ownership.integration.test.ts` fails locally on master as well, because the local Rust `paperclip-runnerd` binary is not built. - `node scripts/typecheck-layers.mjs` passed for all layers. Not verified: a live resumed run on staging with this build. The staging evidence shows where the time goes (before the coordinator claim, on resumed sessions only) but does not show which restore step was slowest. The fix removes the total deadline for every step on that path. ## Risks - A restore that progresses very slowly can now take longer than five minutes. Before, it failed. The run still stops on caller cancellation. - A local step that hangs (for example a stalled disk) is now cut off by the 30-minute outer limit instead of 5 minutes. A stalled local writer also pauses the read through backpressure, so it can be reported as `native_harness_storage_download_stalled`. - A failure before the coordinator claim still enters the existing saved-work reconciliation path. This PR does not change that classification. It removes the cause of these failures. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edits, read-only browser for API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
57ae2fc309 |
perf(workspaces): reduce snapshot stream scheduling overhead (#19900)
Use bounded 1 MiB filesystem buffers during workspace archive hashing, encryption, and restore. Preserve integrity, ownership, cancellation, and snapshot format guarantees. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d7f02c51b4 |
fix(skills): scope file protection to admitted selections (#19904)
Allow unused skill deletion when unstarted admission entries have no retained execution. Limit unknown legacy selections to skills that could have existed when the validated immutable run finished. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
c9959814ba |
fix(workspaces): cancel stalled snapshot reads and preserve restore refusals (#19894)
Cancel encrypted snapshot reads after five minutes without incoming bytes. Renew the deadline on progress, preserve caller cancellation, settle the physical request and private scratch, and retain saved snapshots for normal retry admission. Keep Node warnings out of server-owned helper refusal labels. Verified both complete owner suites (145 tests), the prior unchanged storage and transport coverage, server typechecks, final-head CI and Greptile 5/5. The earlier CI cancellation-signal assertion failure was reproduced locally and corrected with checks for abort state, reason, stream closure and cleanup. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
12681d1611 |
fix(sentry): stop resending retained task interruption failures (#19041)
Keep task-state Sentry reports within the recent saved-change window while retaining reports for newly failed recovery and delayed Retry preparation. Include the safe run error code without changing event grouping. Verified 402 relevant local tests, every server typecheck layer, all final-head CI checks, and Greptile 5/5 with no unresolved review threads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
f439150cb3 |
fix(recovery): limit the null-host owner fallback to single-host servers and record each use (#19883)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #19868 and #19871 made restart recovery judge a recorded pid only in the PID space that recorded it > - Rows saved before those host columns existed have no recorded host. They kept the previous local dead-pid proof, so in-flight native runs would not be stranded at upgrade > - On an ephemeral (multi-host) server that fallback is unsafe: the row may have been written on another host > - This pull request limits the fallback to single-host servers, records every use, and gives ephemeral servers a working board action > - The benefit is that single-host upgrades still recover on their own, multi-host servers never accept a cross-host proof, and the next release can remove the fallback with evidence ## Linked Issues or Issue Description **Bug description** After #19868 and #19871, a `native_run_finalizations` row with `controller_host_hash = null`, or a run with `process_host_hash = null`, still accepted "the pid is dead here" as proof on every server. That includes an ephemeral server, where the row may come from another host. Nothing recorded how often the fallback was used, so it had no safe removal path. **Expected behavior** - Single-host server: the fallback applies for one release, and each use writes a `legacy_host_unknown_dead_pid_proof` activity-log entry. - Ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`): the row is unprovable, and the board has an action to confirm the previous server stopped. - A mismatched host is always unprovable. Refs #19868, #19871, #19846 ## What Changed - `native-restart-recovery.ts` `evaluateNativeControllerTakeover`: - A coordinator row with `controllerHostHash: null` returns `controller_host_unrecorded` when `ephemeralServerEnabled()` is true. - Otherwise a pid proof from that row carries `legacyHostFallback: true`. - `operator_attested` is accepted as proof (`operator_attested_controller_stopped`). - Receipt-built owners, which have no host key, are unchanged. - New `recordLegacyHostFallbackProof` writes the activity-log entry. It is not a run-log event: run events are progress evidence for restart recovery's crash-loop credit, and the record must not change any recovery decision. It is called from restart recovery, the dead-controller result-lease release, native retirement, and remote warm-session recovery. - New `native-controller-host-recovery.ts`: - `requestNativeControllerStopConfirmation` (board-owned `native_controller_host_unverified` recovery action, idempotent, never displaces another active action). - `confirmNativeControllerStopped`. It verifies the exact recorded controller and stamps `operator_attested` with compare-and-set. It then resolves the action, wakes live-run recovery, and logs `issue.native_controller_stop_confirmed`. - Route `POST /issues/:id/recovery-actions/confirm-controller-stopped`. It is board-only and requires `runtime:confirm_owner_stop`. Generic `restored`/`false_positive` resolution of this cause is refused. A shared `confirmNativeControllerStoppedSchema` validates the body. - UI: `WorkspaceExportRecovery` reuses the "Confirm the previous server stopped" control for this cause, and the generic recovery card hides it. Adds the `issuesApi.confirmControllerStopped` client call. - `recovery/service.ts` orphan reaper: a null `process_host_hash` is judged only when not ephemeral. A successful use records the entry best-effort, after the terminal write. On an ephemeral server the run is left to its existing Stop control. - `ephemeralServerEnabled` moves to a leaf `ephemeral-server-mode.ts` (re-exported by `ephemeral-server-contract.ts`). Importing the contract module from recovery code created an import cycle in the runner child process. - The new route is registered in OpenAPI. - `doc/execution-semantics.md`: documents the host rule and the one-release sunset. ## Verification - `native-restart-recovery.test.ts`: null host on a single-host server is released with the flag. Null host on an ephemeral server is unprovable. A mismatched host is unprovable in both modes. An attested controller is accepted. Receipt owners are unchanged. - New `native-controller-host-recovery.test.ts` (embedded Postgres): - The fallback writes the activity-log entry and no run event. - Ephemeral mode creates exactly one board action. - A wrong run id is refused. - Confirmation stamps the attestation, after which takeover is allowed. The action resolves and the activity is logged. A second confirmation is refused. - `recovery-stale-issue-lock-sweep.test.ts`: on a single-host server a null host is terminalized and records one event. On an ephemeral server it is left alone with no event. The existing audit-failure case still clears the lock. - `WorkspaceExportRecovery.control.test.tsx`: the controller confirmation posts to `confirmControllerStopped` and never to the copyback endpoint. - Locally: these suites plus `remote-warm-session-recovery`, `native-controller-recovery.integration` and `native-startup-cleanup-recovery` (136 passed), and after the CI fixes `native-runner-restart-preparation` and `native-runner-restart-recovery` integration, `openapi-routes` and the suites above (256 passed); the UI recovery suites (52 passed); `node scripts/typecheck-layers.mjs` with no errors in the touched files; `pnpm check:token-gates` clean. ## Risks - Previous-release story: no schema change. Null-host rows saved by earlier releases recover as before on single-host servers, now with an event, and wait for the board on ephemeral servers. The next release removes the fallback; the event count shows whether any rows remain. - On an ephemeral server, a null-host orphaned run process is not reaped automatically. The operator uses Stop. That is the intended "unprovable" outcome. - A board confirmation is an attestation. It applies only to the exact recorded controller (boot id + pid) and only while the host is still unrecorded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
5a70999306 |
Add a versioned online workspace migration command (#19882)
## Thinking Path > - Paperclip preserves the files and recovery state produced by its agents. > - Canonical online migration can preserve selected local workspaces while unrelated work continues. > - Operators sometimes need a newer migration owner before the serving web process can take that release. > - The existing packaged maintenance entry requires stopped writers. > - This pull request adds a finite online entry from one coherent reviewed release tree. > - The entry delegates capture and publication to the existing owners and keeps tool and serving identities separate. ## Linked Issues or Issue Description Refs #19854, #19855 and #19872. The duplicate search found related owner fixes and no separate versioned online entry. **What happened?** The finite online workspace importer was available as a library in the serving image. The packaged all-phase migration entry required maintenance mode. **Expected behavior** Run the reviewed online owner from a separate complete release tree without replacing the web process, stopping unrelated work or changing its release policy. **Deployment mode** A managed instance with its original local volume, database, existing encryption key and shared S3 storage. ## What Changed - Add strict inspect, single, cohort and exact single-journal resume requests with explicit company, workspace, deadline and concurrency bounds. - Verify the private tool root and its own build stamp separately from the serving source commit and deployment. - Reuse the nonmutating full migration-manifest inspector and require the additive controller host-identity column before publication. - Hydrate existing configuration and key bindings, then use the original ESM database pool and canonical storage, source locks and migration owners. - Keep multi-member inspection with the optional canonical cohort preflight in a read-only transaction with denied storage. Releases without that owner refuse cohort operations; single import and resume remain available. - Register the optional runtime bundle entry and document the supported full-tree source mode, resource admission and reconciliation. - Add hash-bound old-writer/reader compatibility and encrypted restore tests. ## Verification Final local revision: `c9345b72f27aee1158047dccd0b1060f9a0be6df`, on master `fbe689938c3bf8c5849d2cd56b9da90029db9389`, macOS arm64 with Node 24.22.2. - Entry and runtime packaging: **34/34 PASS**. - Real PostgreSQL/filesystem compatibility: **6/6 PASS**, disposable database wrapper exit 0 and cleanup complete. The frozen old source lock owner contends with the new entry; unrelated work proceeds. The old authority and snapshot readers decrypt and restore newly published bytes. - Pending and uploaded journal cases use the unchanged old persisted format and preserve operation/snapshot IDs through resume. - Full layered server source and test typecheck: **PASS**. - Module boundaries and `git diff --check`: **PASS**. - Actual source-mode module import under Node 24 succeeds without starting the application. - A separate temporary full-old-producer fixture did not get past module setup: 90 seconds under Vitest and a 15-second direct Node control. Neither is counted as an executed old-producer test or a product failure. The smaller frozen old-writer/reader proof and old-format journal cases are the local evidence. - Hosted resume of the actual saved operation and Linux artifact qualification remain operational acceptance gates. This PR does not claim those ran. - Review regression: a real successful import returned a null operation ID (genuine RED). The repaired entry binds the exact returned snapshot to its succeeded migration journal after commit (GREEN). Committed or superseded snapshots are accepted; a missing readback requires reconciliation rather than a blind retry. - Final affected check: **40/40 PASS** across the entry, packaging and actual PostgreSQL compatibility suites; wrapper exit 0 and cleanup complete. Full layered typecheck and boundaries pass after the user-requested rebase and current Runner rebuild. - A second real postcommit database-read fault also has genuine RED/GREEN proof. The succeeded journal, saved authority and original bytes remain intact, and every readback error now preserves the reconcile-required warning. - Final-head [CI run 42404116001](https://github.com/paperclipai/paperclip/actions/runs/42404116001): **PASS**, with all 59 check contexts terminal (55 success and 4 intentional skips). Fresh Greptile review is **5/5** on `c9345b72f27aee1158047dccd0b1060f9a0be6df`, with no unresolved threads; check and thread pagination are complete. ## Risks - The tool must come from one authenticated complete release tree with its own dependencies. A copied bundle chunk can retain old absolute paths and is not a portable release. - Concurrency is not resource admission. The operator must check memory, CPU and scratch headroom and start with one import or two cohort members. - Schema must be prepared by the existing schema owner. This entry never applies DDL. - Cancellation awaits the canonical owner. An SSH interruption does not prove the process stopped; durable journals must be reconciled before retry. - The entry preserves remote obligations and source files. It does not prove enrollment readiness or zero-downtime operation. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact serving snapshot and context-window size are not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
0189fedffc |
fix(native): bound an idle warm local runner whose controller died (#19879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A warm runner (`lifecycleMode: warm`) keeps runnerd alive between runs so a follow-up turn starts fast. The follow-up default flip (next PR) makes warm the default for local Codex and OpenCode agents. > - The controller owns a warm runner's idle window. Runnerd ignores the idle timeout itself. > - A graceful controller shutdown suspends idle warm sessions. A controller that dies (SIGKILL, crash) does not. Local runners launch with no process reconnect grace, so the idle runner kept running until its 1-hour connection lease ran out, owned by nobody. > - This pull request gives warm local runners the 7-minute reconnect grace that remote runners already use. > - The benefit: idle runners are reaped by proof within a bounded time, never left orphaned, before warm becomes the default. ## Linked Issues or Issue Description Refs #19874 (warm completion contract), #19846 (TTFT baseline), #19852. **What existing behavior does this improve?** Ownership of an idle warm local runner after its controller dies. **Current behavior** `runnerProcessReconnectGraceMs` is `undefined` for local targets, so runnerd launches without `--reconnect-grace-ms`. An idle warm runner whose controller was killed stays alive until its connection lease (one hour) expires. The restart-chaos harness cannot show this: its K1 kills the controller's whole process tree, runner included. **Proposed behavior** `localRunnerProcessReconnectGraceMs(policy)` returns the remote grace (7 minutes) for a warm policy and `undefined` for per-turn. After the grace, runnerd records `transport_reconnect_grace_exceeded`, stops its provider, keeps durable state and exits. The next run resumes cold. **Reason and benefit** The restart plan requires idle runners to be reaped by proof, never orphaned. Seven minutes is far longer than a controller restart (13–20 s locally) and the long-outage fault holds (3–5 minutes), so a live turn still reattaches. ## What Changed - `native-runner-recovery-contract.ts`: `localRunnerProcessReconnectGraceMs`. - `native-session-executor.ts`: the local runnerd transport passes it as `runnerProcessReconnectGraceMs`. Remote targets keep `null` (lease-owned). - `runnerd-warm-idle-orphan.test.ts` (new): real runnerd and fake Codex. An idle warm runner, after its turn completed, whose controller detaches. - `doc/DEVELOPING.md`: the grace and its bound. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-warm-idle-orphan.test.ts`: with a process grace, the idle runner exits with `transport_reconnect_grace_exceeded` and the next transport resumes the same provider session cold. With no grace (`null`, today's local launch), the runner is still alive after the controller left. - `cd server && npx vitest run src/services/native-runtime/native-runner-recovery-contract.test.ts`: warm gets the remote grace; per-turn keeps none; the grace outlasts a controller restart and a long outage. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - Local restart measurement with warm runners (K1/K2, codex, claude, opencode, n = 3 each): every interrupted run recovered, with recovery latency equal or lower than per-turn. The chaos waves with this change are in the default-flip PR. ## Risks - A warm local runner whose controller stays down for more than 7 minutes exits. A turn in progress then resumes through `resume_dead_runner`, the same path as a controller crash that kills the runner. Remote runners already behave this way. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted state changes. A warm runner launched by the previous release keeps the launch arguments it started with (no grace) until it exits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
965ca91dd0 |
Capture legacy workspace cohorts under one source fence (#19873)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Workspace migration preserves local files as authenticated portable
snapshots.
> - Workspaces that share Git storage require one complete source fence.
> - Independent imports contend on that fence and cannot safely run as a
batch.
> - This change adds an opt-in cohort owner that captures members
concurrently and publishes them atomically.
> - Existing source, recovery, snapshot and publication checks remain
authoritative.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The administrative remote-workspace migration owner for workspaces in
one shared physical source component. Related changes: #19688, #19854
and #19855.
**Current behavior**
Each independent import acquires the complete source fence. Parallel
independent calls therefore contend instead of overlapping archive work.
A caller cannot use a surrounding transaction and assume that a
successful savepoint confirms the outer commit.
**Proposed behavior**
An explicit `migrateRemoteWorkspaceCohort` call selects one to eight
leaves from one proven component. It owns the full
source/lifecycle/copyback fence, uses the original pool for bounded
archive work, validates every member, publishes serially, and returns
success only after the outer commit.
**Reason and benefit**
This supplies a correctness-gated path for a controlled two-member
hosted qualification. It makes no wall-clock speedup claim. No running
app call site or scheduler selects cohorts automatically.
**Breaking changes**
The existing single-import API retains its guards and rejects a journal
owned by a cohort. New cohort journals require the same complete member
set on resume.
## What Changed
- Extract the existing reservation, capture, validation and publication
stages for reuse by both owners.
- Add an opt-in cohort API with default concurrency two and a maximum of
eight. Every member must share the same complete physical/Git/alias
component; selected roots must be leaves relative to each other.
- Persist exact cohort and operation membership in the existing journal.
Compare all selected fingerprints with the aggregate proof before the
first reservation under the complete fence, and recheck each member when
reserving its journal. Refuse foreign journals, incomplete or mixed
replay sets, lost fences and changed sources. Keep all authorities and
activity events unpublished if the outer commit fails.
- Share the merged nested-child verifier inside one complete fence.
Queue its prepare calls, retain its two-child bound and authenticated
immutable proof, and recheck local bytes and database bindings on every
pass.
- Document the API, exact-membership resume protocol, resource bounds
and required hosted qualification.
## Verification
- Integration head `fce1896010`: 32/32 PostgreSQL/filesystem cases
across the cohort owner, the complete merged nested-verifier suite, and
five existing parent-import cases.
- Final source-proof repair: 19/19 cohort-owner cases pass, including
real first- and last-member metadata changes before reservation,
changed-source partial-resume refusal, mixed local/imported refusal with
storage forbidden, cancelled/failed/finished admission, a late finished
timestamp, complete fences, queued child-proof reuse, cold restore,
cancellation, commit failure and exact replay. All local database checks
use the canonical owned wrapper, one Vitest worker and Node 24.22.2 on
macOS arm64/PostgreSQL 18.1.
- Final rebased head is based on
`fbe689938c3bf8c5849d2cd56b9da90029db9389`, including current
source-owner, database-schema and Runner contract changes. The 19-case
owner suite, full server layered source/tests/test-tree typechecks,
module boundaries and diff checks are rerun on that base; required CI
supplies the native Runner qualification.
- Selection-only `node scripts/local-verify.mjs --base
|
||
|
DottaandPaperclip
|
4d5265fe7e |
Preserve fully covered historical workspace sources (#19872)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace migration must preserve local files while retaining unfinished recovery obligations. > - A shared Git source needs proof for its companion workspaces before import can publish. > - Recovery backfill can add a historical-source marker to an exhausted prepared-export hold. > - The hold owner rejected that marker even when every recorded allocation already belonged to its proven prepared source. > - This pull request accepts only that fully validated form through the existing source owners and publication fences. > - Unrelated work can continue while import saves the local files and keeps the remote recovery hold intact. ## Linked Issues or Issue Description Refs #19830. No separate public issue was found. A search for the marker found no open duplicate PR. **What happened?** A companion workspace with an exhausted prepared-export hold blocked the whole shared Git cohort after canonical recovery backfill added its historical-source marker. The prepared and historical source readers both accepted the same allocation. **Expected behavior** Local file preservation can proceed when all historical references are already covered by the proven prepared source. The recovery hold and remote obligations remain unchanged. Unknown or uncovered sources stay blocked. **Steps to reproduce** 1. Create a shared Git repository with a selected leaf and a companion workspace. 2. Give the companion a settled failed export, accepted result, exhausted recovery and blocked task. 3. Run canonical historical recovery backfill for its retained allocation. 4. Import the selected leaf through the normal online migration owner. **Paperclip version or commit** Reproduced on `4185e308659abf0887930e20049ca6e7cbfc0e81`. **Deployment mode** A source installation with legacy local Git workspaces and object storage. ## What Changed - Validate the exact historical marker schema, run and unique lease references against the already proven prepared source and declared allocations. - Reuse the canonical historical-source inspector and compare the same provider allocation and remote working directory. - Reject unknown fields, malformed markers and foreign, duplicate or uncovered references. - Keep the full marker in the existing hold fingerprint and retain the existing source, task, lifecycle, finalizer and publication checks. - Add real PostgreSQL/Git tests through shared-cohort inspection and the encrypted import/restore owners. ## Verification - The unchanged owner failed the actual selected-leaf importer regression with `online_workspace_source_scope_unproven`. The canonical prepared/historical reader controls passed before that refusal. - Production head `3e678e10fa` passed 41 focused PostgreSQL/Git cases across three files in 18.93 seconds after rebasing onto the new finalizer schema migration. This includes 14 new cases; 80 unrelated cases were filtered out. The final test-only review update passed both additional multi-allocation controls plus the original selected-leaf cold restore (3/3, 11.29 seconds); TypeScript passed again. - The positive uses the actual historical backfill producer. It cold-restores an encrypted selected-leaf snapshot with the original Git tree unavailable and keeps the peer's task, finalizer, result, allocation and recovery action unchanged. - The tests retain an unrelated active object-backed run, prove the peer finalizer lock is held, refuse bad bindings and reject late marker changes without discarding the uploaded snapshot. - Server TypeScript, module-boundary checks and `git diff --check` passed. The canonical PostgreSQL wrapper exited and cleaned up. - Final-head CI passed: 56 successful checks and four intentional skips, with no failures or pending jobs ([run](https://github.com/paperclipai/paperclip/actions/runs/42397090541)). Greptile reviewed `0cbf0a0255` at 5/5 with no open findings; GitHub reports CLEAN/MERGEABLE. ## Risks - The allowance applies only to the existing exhausted prepared-export hold. Other recovery forms and unknown or uncovered references remain blockers. - A marked hold adds database reads proportional to its already declared allocations during maintenance inspection. It adds no provider calls, payload download or recurring background work. - Import saves the current local files. It does not claim that unsaved remote files were recovered, that a provider stopped or that execution may resume. Existing holds and warnings remain authoritative. - Local tests use the real encryption and restore owners with a local object backend. Hosted backfill and normal-delivery qualification remain separate. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools and code execution. The exact serving snapshot and context-window limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
738058553f |
fix(recovery): resume a continuation that never launched its runner in its own sandbox (#19850)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A chat follow-up on a remote native task runs as a continuation: it starts from its predecessor's saved provider session and, for a warm session, copies the predecessor's runner process identity onto the new run > - When the continuation fails before it asks for a runner of its own, its due same-run retry sees a recorded process identity and reattaches to "this run's" runner. That runner never existed, so reattachment always fails with `native_remote_recovery_lease_mismatch` and the task is blocked > - On staging, 34 OpenCode runs on 31 tasks failed this way in one evening. Each first attempt failed while staging its provider pack, recorded no runner start, and kept its own sandbox lease active > - This pull request proves that shape from durable state, retires the inherited identity, and resumes the run the way its failed attempt ran, in its own retained sandbox > - The benefit is that these retries recover automatically, without a second sandbox and without the unsaved-work hold, while every run that may hold its own provider work keeps verified reattachment ## Linked Issues or Issue Description Refs #19839, Refs #19399, Refs #19755, Refs #19779, Refs #19827 **What happened?** A remote OpenCode chat continuation failed during provider-pack staging. The run had no runner start event, no process identity event, no controller checkpoint, and no provider event of its own. Its saved session (no active turn, no pending request) and its process identity were inherited from the previous run. Its failure detail recorded `providerEventsExist: false`, and its own sandbox lease stayed active. The due same-run retry chose `reattach_remote_runner` because a process identity was recorded. Reattachment failed with `native_remote_recovery_lease_mismatch`, and the run ended in `native_continuation_requires_reconciliation`. **Steps to reproduce** 1. Continue a warm remote native session so the new run inherits the predecessor's runner process identity and saved session. 2. Fail the new run with a retryable error before it requests a runner (for example during provider-pack staging). 3. Let the same-run retry become due. Recovery requests `reattach_remote_runner`, and reattachment fails. **Expected behavior** The retry resumes the run the way its failed attempt ran, in the same sandbox, and continues the inherited provider session. A run with any provider work, runner start, or controller checkpoint of its own keeps verified reattachment. ## What Changed - New `server/src/services/native-runtime/native-inherited-runner-retry.ts`: - `retireInheritedRemoteRunnerIdentity` locks the issue, the coordinator, then the run (the same order as same-run resumption). It retires the recorded process identity only when all of these hold: the coordinator is a result-less, unowned `exact_checkpoint_resume` retry with `providerEventsExist: false`; the run has no runner start, process identity, adapter invocation, tool start, runner event, controller checkpoint, provider event or provider session of its own; the saved session has no active turn and no pending runtime request; and a settled predecessor of the same native session recorded the exact same process identity (PID, group, start time). - It records `nativeInheritedRunnerRetirement` on the run profile, with the run's own sandbox location when that lease is still active, and an `inherited_runner_never_launched` lifecycle event. - `readInheritedRunnerRetainedSandbox` returns that sandbox only while it still matches the run's saved remote location. - `heartbeat/recovery.ts`: a due remote retry calls the retirement first. It returns `retired`, `not_applicable` or `transient`. A retired run (including one a concurrent sweep already retired) goes to the ordinary same-run resume (`claimNativeSessionResumptions`), with every existing admission gate. A lock timeout or database error is `transient`: the sweep defers the run and checks again, and never falls back to reattachment. Any other run keeps verified reattachment. - `heartbeat/workspace-preparation.ts`: an ordinary same-run resume with a retired inherited runner acquires its retained sandbox through the existing exact-lease path (`reattachRemoteLease`), which validates and resumes that lease and never acquires a replacement. It acquires as an ordinary resume does only when the exact-lease failure proves the sandbox is unusable (`stage=environment_acquire|environment_resume`, `field=lease|lease_status|lease_expired|provider_lease_id`). Any other failure, such as a scope mismatch on a live lease, is kept and acquires nothing. - New `server/src/services/retained-sandbox-absent-error.ts`: a sandbox provider's resume reply with no provider lease and `expired: true` (the reply Daytona and other providers send when the sandbox is gone) now throws `RetainedSandboxAbsentError`. Every other mismatched reply keeps the existing error. The exact-lease reattachment in `environment-run-orchestrator.ts` maps it to `native_remote_recovery_lease_mismatch: stage=environment_resume field=lease_expired`, so the same-run resume above acquires as an ordinary resume when the provider has removed the retained sandbox. A transient resume failure is not treated as absence and never acquires another sandbox. - `native-restart-recovery.ts` exports the existing `hasNativeExecutionStartEvidence` so both paths use one definition of "execution reached a runner". - `doc/execution-semantics.md` documents the rule. ## Previous-release state - No schema change. The new profile field is additive. - A run saved by an older release without `providerEventsExist`, without a native session ID, or without a matching predecessor keeps verified reattachment. That path is unchanged. When it loses its runner, it still requests the user's Retry once (#19399). - A run that an older release left with its own runner start, controller checkpoint or provider event keeps verified reattachment, because its outcome may be ambiguous. - A profile without the new marker never changes acquisition. ## Verification - `npx vitest run server/src/services/heartbeat/recovery.test.ts -t "never launched its own runner|keeps verified reattachment"` (embedded PostgreSQL): - Staging shape (inherited session and process identity, no runner start of its own, saved remote workspace, active own lease, recorded PID). Before the fix: fails, because recovery requested reattachment. After the fix: passes. The PID fields are cleared, the marker names the predecessor and the retained lease, `executeRun` is called once with a plain `nativeLeaseOwner` claim, the run still has exactly one (active) lease, and no process is signalled. - Negatives, passing before and after: a provider turn of its own, a runner start of its own, a controller checkpoint of its own, no matching predecessor, a pending runtime request in the saved session, and a failure detail without `providerEventsExist`. Each keeps verified reattachment and gets no marker. - `npx vitest run server/src/services/heartbeat/workspace-preparation.test.ts -t "retained sandbox|retired inherited"`: - A retired inherited runner acquires its retained sandbox with `reattachRemoteLease`. Before: fails. After: passes. - When the retained sandbox is gone, it acquires as an ordinary resume. Before: fails. After: passes. - Without the marker, acquisition is unchanged (passes before and after). - A `nativeWorkspaceSync` reference also resumes its retained sandbox. A live lease's `lease_scope` failure is kept and acquires nothing. - `recovery.test.ts` also covers a second retirement call (returns `retired`, one event) and a held issue lock (defers; no reattachment, no dispatch, PID kept). - `environment-runtime.test.ts`: on exact resume, only a provider reply with no lease and `expired: true` (with or without a workspace sentinel mismatch) is `RetainedSandboxAbsentError`; a different lease or a reply without `expired` keeps the existing error. `environment-run-orchestrator.test.ts`: confirmed absence becomes `stage=environment_resume field=lease_expired` and acquires nothing. - Affected suites pass locally: `recovery.test.ts`, `workspace-preparation.test.ts`, `heartbeat-native-remote-failure-retention.test.ts`, `workspace-recovery-retry-copyback.test.ts`, `native-runner-restart-recovery.integration.test.ts`. `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - **No second executor.** The rule applies only when this run never requested a runner, so no runner of this run exists. The predecessor must be settled, and its identity is matched exactly. The resume then takes the normal same-run claim, which holds the issue lock and admission gates. - **No repeated side effects.** The run has no provider event, turn or tool start of its own. A saved session with a pending runtime request or an active turn keeps verified reattachment. The resumed provider session goes through the existing provider recovery, which refuses a pending side-effecting call with a permanent, named code. - **Bounded.** The retirement takes effect once per run, because afterwards the run has no recorded identity. A contended retirement waits for the next sweep. Further failures use the existing three-attempt provider budget. - I did not run a live Daytona sandbox, or a full preparation through a real sandbox plugin, through this path. The orchestrator's exact-lease checks are covered by its own unit tests. The evidence is recovery and preparation tests on a real database with the staging run shape. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file editing, read-only browser access to staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
4185e30865 |
Recover archived commits from original bare sources (#19855)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace migration must preserve saved files before the web server releases its local volume. > - An archived Git checkout can retain a trusted branch-deletion receipt after its private clone and generated project cache are removed. > - The original repository can still exist at its recorded file URL, but the archive owner did not check that source. > - This pull request permits a narrow, verified original bare-repository source for the exact recorded commit. > - The existing encrypted snapshot and publication owners then preserve the workspace through their normal migration path. ## Linked Issues or Issue Description No public issue exists. Searches for the archived-source error and original bare Git source found no duplicate change. **What happened?** Migration reported `archived_git_source_unavailable` for an archived checkout even though its original bare repository still contained the exact saved commit. The private clone and generated cache were absent. **Expected behavior** The canonical owner can use the recorded original local repository when its full commit identity and source ownership are proven. Uncertain or changed sources remain blocked. **Steps to reproduce** 1. Create an isolated checkout from a local bare repository. 2. Close it through the normal cleanup owner and retain the successful branch-deletion receipt. 3. Remove the private clone and generated cache, while retaining the recorded original repository and full base revision. 4. Inspect the archived workspace for migration. **Paperclip version or commit** Reproduced on `84908d176b2800f4c792c4249c610ec0baebdd28`. **Deployment mode** A source installation with existing local Git workspaces and shared object storage. ## What Changed - Consider the exact configured file-URL repository only after the canonical primary source is definitively absent. - Require a matching source record and successful cleanup receipt, the exact full base/deleted commit, a contained bare SHA-1 repository, bounded standard configuration and alias-free metadata. - Restrict this source path to local Git transport and a minimal environment. Reconstruct the tree only in private migration scratch. - Retain existing source fingerprints, journal identity, encrypted upload/readback and final publication checks. Lock and recheck the configured source and cleanup record before publication. - Add real PostgreSQL/Git coverage for encrypted cold restore, lost-reply retry and refused source or publication changes. ## Verification - The unchanged owner failed the real four-object original-source regression with an ineligible inventory. - The review regressions failed before each repair: four returned-primary retry/publication cases and two actual production metadata/reaper cases. - Final head `8925519310` passed all 40 focused PostgreSQL/Git tests in 56.38 seconds. Four positive formats use canonical/legacy metadata and ordinary/verified-tip cleanup. Each imports an encrypted snapshot, removes the original file URI from reach, then authenticates and cold-restores exact files, executable mode and Git history. - The same file covers source/receipt/writer guards, changed sources before publication, authenticated lost-reply reuse and 18 returned-primary directory/alias/unreadable cases. Refused publication retains returned edits and uploaded ciphertext. - Server TypeScript, module-boundary checks and `git diff --check` passed on this revision. Canonical test wrappers exited and cleaned up. - An earlier focused run also passed 22 existing archived-Git/restore-point controls. `pnpm verify --explain` completed without executing extra suites. Required CI supplies final broad coverage. - Final head passed all 60 CI/review contexts: 56 successful checks and four expected skips. Greptile reviewed this exact head at 5/5, with no open review threads. - One unchanged archive segment-boundary test timed out on both framework attempts in the initial server shard. The one job-only retry passed with unchanged code and timeout. The test and its resolved local import graph match a green control revision. The original failures remain retained; the cause is unresolved. ## Risks - The fallback is intentionally narrow. Original repositories with a different full tip, unsupported configuration, aliases (including an aliased instance root), external object storage or more than 10,000 metadata entries remain blocked. - The first source check and final publication check add bounded local filesystem and Git work to this maintenance path. They add no recurring worker or network Git access. - Original repository files remain intact. Importing a retained workspace does not make its configured local file URL portable for future fresh project work. That separate configuration dependency still requires preservation and qualification before volume removal. - The local regression uses the real encryption and restore owners with a local object-storage backend. It does not claim a hosted backfill or a deployed migration. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools and code execution. The exact serving snapshot and context-window limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
a9ac2e2039 |
feat(native): split local runner startup into stage, launch and session-load spans (#19852)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a person messages an agent, the agent must start its turn fast. Time to first token (TTFT) and agent boot are first-class metrics. > - The Phase 0 baseline (#19846) found that local runner startup costs 1.6–17 s at p50 and up to 50 s at p95. It is the largest part of TTFT. > - The run log records that startup as one span (`runner.session.bootstrap` or `runner.session.resume`). We cannot see if the time goes to staging, to the `paperclip-runnerd` process launch, or to the provider session load. > - This pull request splits local runner startup into three sub-spans in the run log. > - The benefit: the follow-up work to cut follow-up-turn TTFT can measure each step before and after. ## Linked Issues or Issue Description Refs #19846 (TTFT baseline, finding 1 and gap "No sub-spans exist inside local runner startup"). Refs #19838. **What existing behavior does this improve?** The run-log performance spans for local native runner startup. **Current behavior** The local path records one `runner.session.bootstrap` or `runner.session.resume` span that covers runtime staging, the runner process launch and the provider handshake together. The remote path already records `runner.runtime.stage` and `runner.process.launch`; the local path does not. **Proposed behavior** The local path records three sequential child spans of `runner.session.startup`: `runner.runtime.stage`, `runner.process.launch` and `provider.session.load`. **Reason and benefit** The TTFT work needs to know which step dominates before it changes the runner lifecycle. Without these spans, a change to warm runner reuse cannot show which cost it removes. ## What Changed - `RunnerdCodexTransport` has a new optional `onStartupStep` callback. `#start` and `#resume` time three milestones and call it synchronously. A failing callback never changes startup. A nested startup (exact adoption from `thread/start`) keeps the outer clock. - `runner.runtime.stage`: launch preparation before the runner process exists. - `runner.process.launch`: spawn until runnerd accepts the run (`run.prepare`, or the recovery attachment on resume). An adopted live runner records no launch. This keeps the existing meaning in `summarizeStartupReuse`: a launch span means a fresh runner. - `provider.session.load`: `session.open` (or the recovery probe) until the provider identity is confirmed. - `recordNativeRunnerStartupStep` (`native-run-trace.ts`) records each step as a `run.performance.span` row with `mode` and `target: "local"`. It uses `trace.defer`, so the run-log write never delays `turn.start`. - The native executor passes the callback only for a local target. Remote launchers keep their own spans, so there are no duplicates. - `redaction.ts`: allowlist `runner.process.launch` and `provider.session.load` as native span names. - `doc/run-log-events.md`: new section "Local runner startup sub-steps". Data path (AGENTS.md §5.7): run log only, plus Observability spans when an operator sets an OTLP endpoint. The attributes (`mode`, `target`, `outcome`) are already in the closed span-attribute allowlist. No Telemetry change. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-startup-steps.test.ts`: a real runnerd and the fake Codex fixture. Bootstrap and a cold resume each report stage, launch and session load in order, with `ok` outcomes and no overlap. A throwing sink does not change startup. - `cd server && npx vitest run src/services/native-runtime/native-run-trace.test.ts src/__tests__/redaction.test.ts`: the rows have the correct parent, offsets, durations and failed level. A stalled run-log sink does not block the caller. The rows settle before `task.provider_session.measured`. The names survive redaction. - The existing `runnerd-codex-transport*` suites pass (412 tests). Under host load (load average 16–27), the first full run had 7 timing failures in suites that do not set the callback. Each failing file passed when I ran it again alone. `runnerd-codex-transport-settlement` passed 3/3 for the failing case alone. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - Low risk. Without `onStartupStep`, the transport behaves as before: the clock is not created and each milestone call is an optional-chain no-op. - More run-log rows: three per local runner startup. - The step boundaries are controller-side observations. `runner.process.launch` includes the runnerd authentication and the `run.prepare` acknowledgement, not only the OS spawn. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6cfb69b4b0 |
docs(plans): Phase 0 ownership inventory and TTFT baseline (#19846)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart can land between two durable writes. It leaves an in-progress record owned by a dead process. About 69 restart-chaos PRs fixed that shape one subsystem at a time. > - The durable ownership plan (#19831) replaces those fixes with one owner stamp, one prover, write-ahead intents and one reconciler. Phase 0 must first list every in-progress record kind and measure how fast turns start today. > - Without the inventory, Phase 1 cannot choose one stamp or one proof. Without the baseline, no later phase can show that it kept turns fast (invariant 8). > - This pull request adds the Phase 0 inventory, the TTFT, boot and restart-recovery baseline, and the run-log-only driver that produced the baseline. > - The benefit is one reviewed map of 83 record kinds to plan sections A–G, and measured p50/p95 budgets per adapter. ## Linked Issues or Issue Description Refs #19831. Refs #19838 (the TTFT anchor that the baseline uses). **What is the problem?** Phase 0 of the durable ownership plan has no inventory of in-progress record kinds and no speed baseline. **What is the expected outcome?** Two reviewed documents in `doc/plans/`, plus a repeatable way to measure the baseline again in later phases. **Why does it matter?** Phase 1 cannot start without them, and every later phase's exit criteria compare against the baseline. ## What Changed - `doc/plans/2026-11-07-ownership-inventory.md`: - 83 durable in-progress record kinds (94 reader rows) across native runtime, environments, workspaces, accounting and the wake queue; - each kind with its table and column, writer, states, owner, release rule, proof, obligation, invariants and plan sections A–G; - the overlaps, the merge decisions and the zero-downtime upgrade record kinds. - `doc/plans/2026-11-07-ttft-baseline.md`: - TTFT, agent boot, pre-turn steps and restart recovery per adapter (codex, claude, opencode), local, cold and warm, p50/p95; - restart recovery from the overnight chaos evidence, the slowest steps ranked, and the gaps. - `tests/restart-chaos/ttft-baseline.ts`: an on-demand driver on the chaos supervisor. It reads the run log only, never Telemetry, and always stops its embedded Postgres. - `tests/restart-chaos/ttft-baseline-stats.ts` and its unit tests: pure percentile, boot, pre-turn ranking and recovery statistics. No runtime behavior changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/ttft-baseline-stats.test.ts` - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` - The baseline numbers come from the driver run named in the report. To reproduce them, follow the report's "Method" section. - The owner spot-checked the inventory's file:line claims at `f2d8a51027`. One example: `processLossRetryCount` has no writer, and `heartbeat/recovery.ts:2675` is its only reader. ## Risks - Low risk. This adds documentation and an opt-in test driver. Nothing runs in CI except the pure stats unit tests. - The driver spends a small amount of provider credit when someone runs it by hand. - The baseline sample is small (6 per cell). The local host was shared with other worktrees, so the p95 tails include load. The report states both. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use, code execution and five parallel read-only subagents for the inventory. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
54b83b15a3 |
Reuse authenticated nested workspace verification (#19854)
## Thinking Path > - Paperclip preserves the work that its agents produce. > - Portable workspace migration can save nested workspaces before their parent. > - The parent must prove that a committed child snapshot covers any files it omits. > - The existing owner downloads and restores the same immutable child archive before and after parent capture. > - The fresh local-file checks are required, but the repeated immutable archive check is avoidable within one live source fence. > - This pull request reuses only an authenticated archive proof with an unchanged reference and provider identity. > - It also bounds independent child branches at two workers while retaining every fresh local and Git check. ## Linked Issues or Issue Description Refs #19805 and #19688. A duplicate search found no separate fix for repeated child archive verification. **What existing behavior does this improve?** Online Git parent migration after nested workspaces have committed portable snapshots. **Subsystem affected** server/ — workspace migration and nested snapshot coverage. **Current behavior** Each parent verification pass downloads, authenticates and restores every child archive, then captures the child's current local tree. The second pass repeats the immutable download and restore. Independent child branches also run in sequence. A large parent can exhaust its existing budget before parent capture begins. **Proposed behavior** One private verifier belongs to the existing live source-owner transaction. It can reuse a completed archive proof only when the full snapshot reference, operation, child closure, ETag and byte length still match. Missing or weak ETags and non-cancellation HEAD errors keep full verification. Caller cancellation still stops work. A changed identity forces fresh authenticated readback. The owner still captures current local bytes and Git state in both passes. **Reason and benefit** Remove duplicate immutable work without treating an old local-file check as current authority. Two independent topmost branches can overlap. Recursive descendants stay serial within those branches, so the bound does not grow with depth. ## What Changed - Add a private per-fence verifier to the existing tree owner. It clears its proof map at the end of the import and refuses a different owner transaction. - Reuse only completed ciphertext, GCM, plaintext, and canonical restore verification with a matching fresh provider identity. - Retain all current database, operation, recursive coverage, local-file, Git and manifest checks on each pass. - Bound independent branches at two. Abort queued work on failure and await active peers before scratch cleanup or fence release. - Use the verifier only in the existing online Git importer. Keep the offline full-verification path and all capture deadlines unchanged. - Document resource admission, fallback behavior and the hosted acceptance limit. ## Verification - Genuine baseline RED through the real parent importer: child GET counts 2 + 2, canonical restores 4, maximum overlapping readbacks 1. The four fresh local child captures already pass. - Exact positive GREEN: child GET counts 1 + 1, canonical restores 2, maximum overlapping readbacks 2. All four fresh local child captures remain. - The fixture uses real encrypted archives and the canonical S3 provider's local HTTP HEAD parsing with actual payload-derived ETags. It uses fake fixture credentials only. - Final focused suite: **13/13 PASS**, 148.36 seconds, at `eaf17d3340`. This includes recursive max-two completion, failure/cancellation joining, identity/ref/content changes, same-fence alias reuse and stale-owner refusal. - Both affected existing migration suites: **256/256 PASS**, 201.84 seconds, at initial head `eaf17d3340`. - The HEAD-failure review repair has genuine two-case RED evidence and three passing failure/cancellation controls. Full layered source/test-tree typecheck and module boundaries pass at repaired head `763648adad`. Final exact-head controls: **21/21 PASS**, 86.63 seconds (all 16 new controls plus five existing parent cases). - The first extended recursive fixture correctly failed because it created a Git branch after taking an earlier snapshot. The fixture now creates all branches before any import; the source-change guard remains intact. - Exact-head CI: 56 success, four expected skips, and one neutral Paperclip Review. Required gates pass; Greptile is fresh 5/5. Its optional aggregate-handler cleanup recommendation is resolved with the intentional fail-closed rationale. - Local verification is macOS arm64 / Node26.4.0. The observed hosted parent still needs to complete within its unchanged budget after normal delivery. No hosted timing improvement is claimed here. ## Risks - Two simultaneous child verifications can use more I/O, memory and scratch space. Operators must count logical verifications, even in one process. - HEAD availability and provider identity are required for proof reuse. Providers without a usable ETag perform the full check. - Reuse lasts only for one live transaction. A retry, changed reference, changed provider identity, process exit or owner change requires fresh verification. - Faster verification does not waive changed child bytes or guarantee a specific hosted completion time. No provider turn, authority marker, source deletion or larger timeout is introduced. ## Model Used OpenAI GPT-6 through Codex, with repository and code-execution tools. The exact serving model revision and context-window size are not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
377bc87df1 |
feat(connectors): show and select personal AI account defaults (#19845)
Show a crown on each saved personal AI provider default. Add a provider-specific menu action to other eligible accounts. Share saved-state queries, block duplicate writes, and keep the saved crown until a write succeeds. Keep account details available for accessible grants. Show session lookup errors with Retry. Resolve list permissions once per request and index account lookups. Add UI and API regression coverage and account documentation. Co-Authored-By: Paperclip <noreply@paperclip.ing> Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> |
||
|
DottaandClaude Opus 5.5
|
e2d490c61b |
feat(observability): anchor native time to first token at the accepted wake (#19838)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When someone messages an agent, the reply must start fast. The durable ownership plan (#19831) makes time to first token (TTFT) a hard invariant. > - Phase 0 of that plan records a TTFT baseline per adapter and environment, measured from the moment the message or wake is accepted. > - Today `task.time_to_first_token` starts at the wake comment, the answered question, or the run row's creation. A wake request that was accepted earlier (deferred, coalesced or retried) is not part of the measurement, and the span does not say which start it used. > - This pull request moves the start back to the earliest accepted input and names it on the span. > - The benefit is that the baseline and later budgets measure what the user waits for, with no new work before `turn.start`. ## Linked Issues or Issue Description Refs #19831 **What is the problem?** `task.time_to_first_token` does not include the time between an accepted wake request and the creation of its run row. It also does not record which input it started from. **What is the expected outcome?** The span starts at the earliest accepted input and carries an `anchor` attribute: `comment`, `question_response`, `wake_request` or `run_created`. **Why does it matter?** Phase 0 of the durable ownership plan sets TTFT budgets from this baseline. A start that skips queue time hides the latency that deferred wakes add. ## What Changed - `resolveNativeTimeToFirstTokenStart` (pure) in `native-run-trace.ts` picks the start and the anchor. `anchor` joins the closed span-attribute allowlist. - `readNativeRunWakeRequestedAtMs` reads `agent_wakeup_requests.requested_at` for the run. It never throws. - `native-session-executor.ts` calls it only after the first deliverable text, so it adds no wait before `turn.start`. The span now ends at the text commit time, not after the trace write. - Updated `doc/run-log-events.md` and `doc/observability.md`. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-run-trace.test.ts src/services/native-runtime/native-run-wake-anchor.test.ts` (26 pass). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t first` (41 pass). The existing TTFT test now asserts the anchor for each ingress. ## Risks - Low risk. This changes only the run-log and OTLP span. It is not Telemetry (AGENTS.md §5.7), and it does not change runtime behavior. - One indexed primary-key read per native dispatch, after the first text. A read failure drops the anchor and keeps the old start. - TTFT values can grow for deferred wakes. That is the intended correction. `startOffsetMs` can be negative when the wake request comes before the trace start. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
84908d176b |
fix(recovery): name the failed check in native_remote_recovery_lease_mismatch (#19839)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote native runs execute in a sandbox, and recovery can reattach a run to its original sandbox runner > - Reattachment checks the saved lease, workspace, and runner identity in five places. Each failed check throws the same bare `native_remote_recovery_lease_mismatch` > - On staging, 34 OpenCode runs on 31 tasks failed with this bare error in one evening. The run records could not show which check failed, so the cause could not be confirmed from the run alone > - This pull request makes every producer name the failed step and field with closed values > - The benefit is that an operator and the next investigation can see which reattachment check failed, without new logging or identifiers in the error ## Linked Issues or Issue Description Refs #19399, Refs #19733, Refs #19755, Refs #19779 **What happened?** Remote same-run retries failed with `native_remote_recovery_lease_mismatch` and the task was blocked for reconciliation. The error is thrown by environment acquire, environment resume, workspace preparation, workspace sync, and remote runner verification. All of them used the same bare string. The run's `error` field and its run events did not show which check failed. **Steps to reproduce** 1. Start reattachment for a remote native run whose saved lease, workspace reference, or runner identity no longer matches. 2. Read the run's `error`. 3. It shows only `native_remote_recovery_lease_mismatch`. **Expected behavior** The error names the step and the field that differed, with no identifiers, and existing consumers of the code keep working. ## What Changed - New `server/src/services/native-runtime/remote-recovery-lease-mismatch.ts`: the code constant, a closed `stage` and `field` vocabulary, and `remoteRecoveryLeaseMismatch(stage, field)`. It builds `native_remote_recovery_lease_mismatch: stage=<stage> field=<field>`. - `environment-run-orchestrator.ts` (stages `environment_acquire` and `environment_resume`), `heartbeat/workspace-preparation.ts` (`workspace_preparation`), `native-workspace-sync.ts` (`workspace_sync`), and `verifyRemoteRunnerReattachment` in `native-session-executor.ts` (`runner_verification`) now throw the detailed form. Each check keeps the same condition and order. Only the message changes. - `restart-interrupted-turn.ts` uses the shared constant. It already matches the code exactly or as the `code: ` prefix, so automatic continuation after a reattachment loss is unchanged. No other consumer matches the code. - `doc/execution-semantics.md` documents the detail format. ## Previous-release state This change adds no persisted state and changes no saved identity. Runs that a previous release failed keep the bare `native_remote_recovery_lease_mismatch` in `error`. Runs failed by this release store the detailed form. The only consumer (`isReattachmentLoss` in `restart-interrupted-turn.ts`) accepts both forms, and `workspace-recovery-retry-copyback.test.ts` covers both. ## Verification - Exact-detail assertions in the producer tests: - `server/src/__tests__/environment-run-orchestrator.test.ts`: every scope, status, expiry, provider, lease-ID, cwd, plugin-task, and resume case asserts its stage and field. Against the previous producer, 21 of these cases fail (the message was the bare code). With this change, all 57 tests pass. - `native-session-executor.test.ts`: a replaced sandbox reports `runner_verification`/`provider_lease_id`. A missing durable runner identity reports `identity_run_id` before any sandbox read. - `native-workspace-sync.test.ts`, `heartbeat/workspace-preparation.test.ts`, `native-controller-recovery.integration.test.ts` assert their exact detail. - End to end on embedded PostgreSQL: `heartbeat-native-remote-failure-retention.test.ts` records `native_remote_recovery_lease_mismatch: stage=environment_resume field=lease` on the run when the provider no longer resumes the lease, and it still requests the user's Retry once. `workspace-recovery-retry-copyback.test.ts` adds a detailed-form row that still continues once. - The affected server test files pass locally. `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. Only the error text changes. Conditions, ordering, and fail-closed behavior are unchanged. - Logs and run errors show a longer message. The detail has closed values only, so it adds no identifiers or secrets. - A first version of this PR also changed remote retry selection. Review showed that a plain resume in a new sandbox hits the remote workspace's unsaved-work hold, and that a `destroy` disposition does not prove the old sandbox is gone. That change is removed. It needs a separate design. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file editing, read-only browser access to staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Dotta
|
86bc3db693 |
feat(runner): search and load unassigned company skills (#19676)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents use tools to read reusable company skills. > - The existing native skill tools read only skills assigned to the agent. > - An agent can stop when a task needs another skill that is already in the company library. > - This pull request adds tools to search and read accessible company skills on demand. > - Agents can use the saved instructions without a persistent assignment change. ## Linked Issues or Issue Description **Problem or motivation** An agent cannot use an unassigned company skill through the standard native skill tools. A generic API workaround needs extra discovery and can return the full library. **Proposed solution** Add `search_skills` and `load_skill` to the native tool catalog. Search returns small pages of stored metadata. Load returns text from a saved version and lists its files. Both tools use the current run identity and existing company and project access checks. **Alternatives considered** Assign each skill before a read, or use generic API discovery. Both add work to a read that the company already permits. **Roadmap alignment** This extends the existing skills discovery path. It does not add a store, permission model or installation flow. Related: #15409 adds an assigned-skill loader to a different adapter. ## What Changed - Add bounded company skill search. Search does not synchronize sources or fetch repositories. - Add versioned text reads for SKILL.md and supporting files. Page text and file lists. Reject unsafe paths and binary files. - Recheck access after storage reads. Keep the existing closed Muse, skill-test and completion-review tool scopes. - Add integration tests with isolated PostgreSQL and real local package storage. Update the native tool count assertion and skill documentation. - Limit text files to 16 MiB before body reads. Accept saved file paths up to 4,096 characters plus the repository prefix. Return the response digest only on the first page. - Add optional byte limits to the package readers. Select only the requested legacy file in PostgreSQL. Cast the size comparison to `bigint` so existing callers can omit the limit. - Cache decoded text and its digest for one run. Limit the cache to 64 MiB and 64 files. Check access on each page and clear the cache when the run closes. - Read repository source files and wrapper inputs through verified warm reads or isolated recovery. These reads do not hold the company startup lock. ## Verification - Local verification tested tree `7ba5828387ca0a9c2ee99655818a28de1235a854`, published as commit `c9d04cba601541cca506f5b72aa6f7272a25379f`. The published tree matches that tested checkout. Current head `33fa3d3d4c7988288685ce7c6499d064e0cb9079` (tree `8daaa2f3264d1cd65a9b6d85dbb9d3729ac4a29d`) removes six comment lines from three source files; its executable code is unchanged from the locally tested revision. - Four native skill and authority Vitest suites: 72 passed and 7 existing sandbox checks skipped. All 20 skill-library integration tests passed. - Skill-library and package-store suites: 65 passed. This includes the 45 package-store cases and the same 20 integration cases. Ran with umask `022` for executable-file tests. - Server source and test typecheck layers passed in low-memory mode. The affected source cycle also passed a final separate check. - `git diff --check` and `node scripts/check-module-boundaries.mjs` passed. - Red/green evidence: the new default inline-read case reproduced PostgreSQL error `22003`. Two max-size paging cases reproduced extra body reads. Warm and cold repository reads timed out while another connection held the company startup lock. All five cases pass after the fixes. - Both 16 MiB paging cases check that the second page performs no additional body read. They also check that source-access revocation blocks a cached page. Warm and cold lock cases load both the generated wrapper and a supporting file while the startup lock remains held. - The first cold-read test attempt had an incorrect helper import. Corrected the test import, reproduced the lock timeout, then verified the fix. - [CI on the previous revision](https://github.com/paperclipai/paperclip/actions/runs/42319652795) passed typecheck, build and policy. Serialized shard 2 failed in the unchanged test `schedules bounded retries for failed accepted interaction continuation wakes`. The same case passed locally: 1 passed; 416 cases were excluded by the filter. The cause of the earlier CI failure remains unknown. [CI for |
||
|
DottaandPaperclip
|
b8e02fc2b4 |
fix(native): preserve bounded larger web inventories (#19832)
## Thinking Path > - Paperclip manages AI agents and preserves their work across server changes. > - Native harness preservation stores retained web-server files in authenticated archives. > - A retained skill can contain hundreds of valid local files. > - The WEB evidence owner rejected any directory or inventory above 128 entries. > - This blocked preservation of two 462-file skills even though their payload fit the existing byte limits. > - This pull request shares finite inventory bounds between capture and archive validation. > - It preserves every file and keeps the execution, identity, writer, and publication guards. ## Linked Issues or Issue Description Refs: #19814 **What happened?** WEB evidence preservation rejected a retained source with 927 local skill files. The source had settled execution ownership. The files fit the existing payload limits. The directory enumeration failed before archive publication because its entry limit was 128. **Expected behavior** Preserve a valid finite inventory within explicit file, payload, path, and metadata limits. Keep remote recovery obligations unchanged. **Steps to reproduce** 1. Retain two 462-file skills and one three-file skill under a settled native source. 2. Request canonical WEB evidence preservation. 3. The old owner rejects directory enumeration. The added integration test reproduces this refusal with real files and PostgreSQL. ## What Changed - Share the WEB limit through the existing controller archive owner: 10,000 total entries, 4 MiB of manifest metadata, and 4,096 bytes per path. - Apply the limits to filesystem capture, published binding validation, and archive descriptor validation. - Keep the existing 16 MiB per-file and 64 MiB payload bounds. Keep authority history at 128 entries and 32 epochs. - Add real archive and cold-restore coverage for 927 files, legacy manifest compatibility, and bound refusals. Document the limits. ## Verification - Genuine baseline RED: the new 927-file integration case reaches the original `captureWebEvidence.directoryNames` limit and fails before publication. - The initial setup attempt failed before test collection because the generated plugin SDK was absent. The required SDK build fixed setup; it was not counted as a regression result. - The first affected three-suite run passed 93 cases. The large roundtrip exceeded the generic 15-second case deadline. That failure is retained. The isolated large case passed with a 60-second case limit, but full CI later exceeded that limit while the other 4,577 tests in its shard passed. Both timeout failures are retained. The final fixture keeps all 927 files and 18.5 MB, uses bounded 16-way independent comparisons, and has a 120-second whole-case limit. The canonical preservation signal remains 60 seconds. No production deadline or assertion changed. - The 927-file case then passed: encrypted archive, authenticated cold restore, all bytes/hashes/modes, unchanged remote leases, and database-only validation with source volume absent and storage reads forbidden. - The final metadata repair adds a genuine RED case: an entry list below 4 MiB previously emitted a larger complete manifest without rejection. The owner now checks the complete serialized manifest. All ten focused cases passed after that repair. The final bounded-I/O fixture passed the same ten cases in 37.26 seconds, including the full 927-file roundtrip. All 93 other affected cases passed in the preceding three-suite run. Fresh required CI supplies broad final-head coverage. - Server TypeScript, module boundaries, and whitespace checks pass. Run the focused PostgreSQL checks from `server/` with the repository wrapper: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts --testNamePattern 'preserves all 927|existing 128-entry WEB|WEB manifest beyond|oversized WEB filesystem' --hookTimeout 90000 --maxWorkers 1 --fsModuleCache ``` ## Risks The owner can hold more entries in memory. The fixed 64 MiB payload, 4 MiB metadata, path, and entry bounds constrain that cost. The archive format and old receipt bytes do not change. Older builds still have the 128-entry limit and cannot qualify a larger new receipt. This change does not settle remote recovery, grant execution authority, alter finalizer locks, or change publication CAS rules. No hosted preservation or deployment was performed for this PR. ## Model Used OpenAI `gpt-6.1-sol`, with code execution and repository tools. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
061c84d280 |
fix: preserve local copies behind settled remote recovery holds (#19830)
## Thinking Path > - Paperclip manages AI agents and the work they save. > - Online migration must save legacy local files while other remote work continues. > - A failed native run can retain a remote recovery hold after its local writer has settled. > - The current source proof recognizes only some of those recovery producers. > - This blocks stable local files and their shared Git aliases behind an interrupted session or exhausted export recovery. > - This change admits two exact producer shapes under the existing physical and task locks. > - It preserves the failed run, accepted result, remote hold, retained allocations, and recovery history. ## Linked Issues or Issue Description Refs #19642, #19688, #19810. A public search found these related changes. This is a narrow correction to their local preservation proof. **What happened?** Online workspace import rejects a stable local source with `online_workspace_source_scope_unproven` when its prepared native source has an ambiguous session interruption hold or an exhausted file recovery hold. Shared Git aliases inherit the refusal. **Expected behavior** Save the stable local copy under the existing source locks. Keep remote recovery blocked until its own recovery owner supplies the required evidence. **Steps to reproduce** 1. Create a failed native run with an unowned terminal finalizer and an exact prepared allocation. 2. Keep either its board-owned interrupted-session failure block or its exhausted system recovery and blocked file recovery with an accepted result. 3. Import the local Git source or a shared alias through `migrateRemoteWorkspace`. 4. Before this change, both actual importer tests reject the source. With this change, both save and restore its files while retaining the hold. **Paperclip version or commit** Reproduced on master at `b2af45f0f1` with local PostgreSQL 18.1 on macOS arm64 and Node 26.4.0. ## What Changed - Reuse the exact prepared-source failure proof for ambiguous native session interruptions. - Admit an exhausted export hold only with its exact run, blocked task, unowned finalizer, accepted result, original allocation, and current recovery receipts. - Bind the complete accepted result into the publication fingerprint. Keep the existing source, alias, Git, finalizer, and task locks. - Add real importer, authenticated restore, refusal, and publication-race controls. Document the separate local and remote obligations. ## Verification - Baseline: the exhausted-export importer case failed at `remote_workspace_migration_blocked:online_workspace_source_scope_unproven`. - Review found that the first interruption fixture used the artifact-failure resume mode. The corrected fixture reads `nativeProviderRecoveryEvidence` and `nativeSessionFailureDisposition`. Its canonical ambiguous interruption case failed at the source guard before the cause-specific mode repair. - Fresh recorded metadata then showed that an interrupted source can have a boolean `providerSessionEstablished: false`. The final fixture derives that exact false+ambiguous state from a checkpoint without a provider identity or provider events. It failed against the intermediate shared-true guard. The final cause-specific proof passes it. The artifact-failure case still requires true+exact mode. - Both importer cases remove the originals and restore saved files through the real encrypted snapshot owner. They keep the failed run status, native source reference, result bytes, tasks, recovery actions, retained allocations, and unrelated active remote work unchanged. The existing migration owner adds only its local source recipe. - Final affected PostgreSQL suites: **107/107 passed**, 74.44 seconds, canonical shared PostgreSQL wrapper with one worker and cleanup complete. Controls reject the wrong interruption mode, a string session flag, unreleased owners, active controllers, stale fingerprints, changed allocations, changed recovery state, and changes during upload. Command from `server`: `node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/remote-workspace-online-source.test.ts src/services/online-legacy-workspace-recovery.test.ts --fsModuleCache --maxWorkers=1 --hookTimeout=90000`. - Final server source and all test typecheck layers passed. Module boundary check and `git diff --check` passed. - A first setup attempt could not load the unbuilt plugin SDK and ran zero tests. Building the SDK corrected the setup. An unchanged-baseline run repeated source refusals after a local edit used the wrong path. The first 105-case and 106-case passes predate the final producer correction and are superseded by the 107-case pass. Earlier failures and passes are retained separately. - A read-only probe loaded the exact committed proposed inspection code against three current stored holds in one repeatable-read snapshot. All three passed the actual function. It verified the owned accepted result and both interruption fields without changing source or provider state. This is applicability proof; an import must still acquire its locks and recheck after deployment. - The first final-head CI run failed browser shard 5 at `cloud-auth.spec.ts:34` with an APIResponse disposed during route teardown. After the workflow completed, one scoped failed-job retry passed that shard and its browser aggregate on the same head. The initial failure is retained. No code, test, timeout or assertion changed for that retry. - `verify --explain` returned 2 because the documentation path requires broad CI. Required CI passed on `2050fa058cabe75e3d8f64f1f6bdb4b9b2fd0873`. Fresh Greptile review is 5/5 with zero open threads. No hosted workspace, provider, task, or recovery state was changed for this PR. ## Risks The new proof could accept an inconsistent recovery producer. Exact company, run, task, allocation, action, result, and receipt checks keep uncertain sources blocked. Changed evidence during capture prevents publication. A saved local snapshot is not proof that remote files were recovered, and this change never creates such a receipt. ## Model Used OpenAI GPT-6 (Codex), with reasoning, tool use, and code execution. The runtime does not expose a more specific model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
0f5ce5050e |
docs(plans): plan durable ownership and reconciliation for restart safety (#19831)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs must survive a control-plane restart: they must continue on their own, keep their work, and offer a Retry that works > - The 2026-11-06/07 restart chaos campaign found about 40 restart failures and merged about 69 fixes (#19469 through #19807) > - Most of these failures have one shape: a restart stops a process between two durable writes, and each subsystem then decides on its own whether the leftover record is safe to release > - Each fix closed one crash window, and about 12 fixes rebuilt the same "this owner is dead" proof > - This pull request adds a plan for one shared ownership record, write-ahead intents, one reconciler, and an on-demand crash-point suite > - The plan also makes time to first token and agent boot hard requirements, so restart safety never makes turns slow > - The benefit is that we fix the whole class of failure once, and we stop fixing one window at a time ## Linked Issues or Issue Description Refs #19696, #19713, #19717, #19724, #19741, #19747, #19784, #19807 **What existing behavior does this improve?** Recovery after a control-plane restart. Today each subsystem keeps its own in-progress records and its own release rules. **Subsystem affected** Native runtime, runner, environments and sandboxes, workspaces, accounting and budgets, and the wake queue. **Current behavior** A restart can leave a lease, a fence, a claim, a workspace operation, a receipt, or a deferred wake that belongs to a dead process. If no subsystem can prove that the owner is dead, the system fails closed. The task then needs a Retry, the company holds new work, or a run stays `running`. About 40 server files read a controller boot id, and about 25 files check process liveness in different ways. **Proposed behavior** One owner stamp and one liveness proof for every in-progress record. Intents are written before outside effects. One reconciler brings every record to a terminal state. Admission never waits on cleanup. Accepted work is delivered exactly once. Time to first token and agent boot are measured and stay within budgets. **Reason and benefit** Fewer restart failures, less duplicate code, and a fast first response after a restart. ## What Changed - Added `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`. The plan contains: - the five failure classes from the campaign, with example IDs and PR counts - eight invariants - a performance section with time-to-first-token, boot, and restart-latency metrics and rules - the design (sections A–G) - an on-demand crash-point suite - six phases - risks - the decisions that Dotta made on 2026-11-07 ## Verification - This change adds documentation only. No code changes. - Read the plan. Check that each failure class cites real merged PRs. - Check the code claims: - `git grep controller-process-identity -- server/src` shows three users. - `git grep task.time_to_first_token -- server/src` shows the existing metric. ## Risks - Low risk. The change adds one plan document and changes no code. - The plan proposes later behavior changes. The largest is the removal of the company-wide accounting gate. Each later phase will come as its own pull request with tests. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, git, GitHub CLI). The model coordinated the chaos campaign and wrote the plan from its evidence. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
2e52dfa864 |
fix(slack): retain DM thread tasks after completion (#19486)
## Thinking Path > - Paperclip manages AI agents and their work. > - Slack conversations store that work in tasks and comments. > - A native thread must keep its task when work finishes. > - The server treated all DMs as linear conversations. > - A reply in a completed DM thread therefore started a new task without its earlier comments. > - This pull request uses the native thread identity for task continuity. > - People can continue the same work with its saved context. ## Linked Issues or Issue Description **What happened?** A reply in a Slack DM thread created a new task after the preceding task was Done or Cancelled. It also created a new task when the saved conversation was completed. The new task lacked earlier comments. **Expected behavior** A native Slack thread keeps one task in channels and DMs. Later replies retain earlier user and agent comments. Top-level DMs keep their existing task generation rules. **Steps to reproduce** 1. Start work in a Slack DM thread. 2. Mark its task Done. 3. Reply in the same thread. 4. Before this fix, the reply creates another task. **Paperclip version or commit** Reproduced from master at |
||
|
DottaandPaperclip
|
c740cb9ded |
fix(ui): dismiss skipped confirmations when sending a message (#19496)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tasks and Agent Chat show saved requests above the message composer. > - A person can send a separate message while a confirmation is open. > - The old code closed only question forms, after the message save finished. > - A confirmation could stay open or open again after a refresh. > - This change closes the form when sending starts and keeps the request in the feed. > - People can open the request later and make the original decision. ## Linked Issues or Issue Description **What happened?** A confirmation stayed open when a person sent a separate message. **Expected behavior** The form closes immediately. The request stays in the feed. A person can open it later. Sending a message does not record an approval. **Steps to reproduce** 1. Open a task with a pending confirmation. 2. Send a separate message below the form. 3. Observe the form while the message save is pending. 4. Reload the task and open the request from the feed. **Paperclip version or commit** Reproduced at |
||
|
DottaandClaude Opus 5.5
|
a4050187ec |
docs(rules): make upgrade safety part of run and recovery design (#19827)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs save session, checkpoint, lease, and launch state, and recovery resumes that state later > - Deploys and restarts often land between the save and the resume, so one release resumes state that another release saved > - On 2026-11-06/07, staging deployed four times. Most of 182 failed runs were saved by one release and resumed by another (#19462, #19659, #19755) > - These refusals showed as the generic retryable `native_session_interrupted`. Retries repeated them, and recovery replaced a thread (now prevented by #19733). Remote-only code also shipped without a test that applied Daytona confinement (#19743) > - This pull request makes upgrade safety a required part of how we design, fix, and review run and recovery code > - The benefit is that authors, agents, and Greptile check the release cutover before a change ships, not after a deploy breaks saved runs ## Linked Issues or Issue Description Refs #19462, Refs #19516, Refs #19622, Refs #19659, Refs #19733, Refs #19743, Refs #19755 **Issue type** Missing contributor guidance for run and recovery code. **Where is the issue?** `.agents/skills/fix-failed-paperclip-run/`, `rules/`, `AGENTS.md`, and `doc/execution-semantics.md`. **What's wrong?** No rule or skill step asks how state saved by the previous release is handled. Fixes added one error code at a time. Saved-identity refusals used a generic retryable code. **Suggested fix** Add a scoped repository rule and additive skill and contract text that require a previous-release story, permanent named refusal codes, and cutover tests. ## What Changed - Add `rules/upgrade-safe-run-state.mdc`. It covers native runtime, recovery, session preparation, runner, ACPX engine, and sandbox image paths, and the `packages/db` schema for native run, checkpoint, session, and run identity state. It requires authors to state how previous-release state is handled. It requires a permanent, named code for unknown or changed saved identity, a class-level fix, an old-saved/new-resumed regression, a deploy-lands-mid-run case where relevant, and a remote-confinement test for sandbox paths. - Register the rule in `.greptile/files.json` with the same scope as its `globs`. - Link the rule from the "Scoped repository rules" section of `AGENTS.md`. - `fix-failed-paperclip-run` skill: step 1 asks if a deploy or restart separated save from resume, by comparing served revisions. Step 2 asks this early. Step 4 adds a "Persisted run identity across releases" safety row and an upgrade-safety paragraph. - `references/failure-patterns.md`: add "Saved run state meets a different release". - `doc/execution-semantics.md`: add §9.4 "Release compatibility of saved run state". ## Verification - Docs, skills, and rules only. No product code changes. - Confirm that each rule glob matches real paths, for example `server/src/services/native-runtime/`, `server/src/services/recovery/`, `server/src/services/heartbeat/session-preparation.ts`, and `packages/adapter-utils/src/acpx-engine/`. - Confirm that `.greptile/files.json` parses and that its new `scope` array equals the rule's `globs`. - `rules/README.md` names no lint check for rules. I did not edit maps, so `node scripts/check-maps.mjs` does not apply. ## Risks - Low risk. The change adds guidance only. - The rule scope includes all of `packages/paperclip-runner/**`. Greptile can apply the rule to runner docs or UI files. The rule's exception section says that changes with no persisted shape or identity need no cutover test. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2e7c66aac9 |
fix(ci): stop the PR merge review notifier from flooding the Actions queue (#19630)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Its GitHub Actions CI shares the `ubuntu-latest` runner pool across all workflows > - #19480 added the `PR merge review notification` workflow. It triggers on `workflow_run`, on every completed `check_run`, and on every commit `status` > - One PR push completes about 60 checks, so it queues about 60 notifier jobs. Each job takes a runner > - The Actions queue now holds about 340 queued runs. About 260 of them are this notifier. Other `ubuntu-latest` jobs wait more than one hour (for example, the `authorize` job of Runner Full-Stack E2E image builds) > - This pull request skips events that cannot complete a green PR state before a runner is allocated, and it collapses the remaining events to the newest one per PR head > - The benefit is that this unjams the Actions queue and keeps the advisory review notification ## Linked Issues or Issue Description Refs #19480 **What happened?** The `PR merge review notification` workflow from #19480 schedules one runner job for each `workflow_run`, `check_run`, and `status` event. One PR push makes about 60+ jobs. The repository Actions queue has about 340 queued runs (about 260 from this notifier at the time of writing). Other `ubuntu-latest` jobs wait more than one hour. **What did you expect?** The notifier requests one advisory merge review when PR verification is complete. It must not use a runner for each intermediate check. **Steps to reproduce** 1. Push to any PR. 2. Run `gh run list --workflow pr-merge-review.yml -L 200 --json event,status`. 3. See tens of queued `check_run`, `status`, and `workflow_run` runs for the same push. ## What Changed - `.github/workflows/pr-merge-review.yml`: the job `if:` now schedules the job only for events that can complete a green PR state: - `workflow_run` from `pull_request` or `pull_request_target` with conclusion `success`, `skipped`, or `neutral`. - `check_run` from a non-Actions app (app id is not `15368`) with conclusion `success`, `skipped`, or `neutral`. GitHub Actions check runs are already covered by the `workflow_run` list. - `status` with state `success`. - GitHub cannot filter `check_run` or `status` at the trigger level. A skipped job makes a run record but does not take a runner. - The job has a concurrency group `pr-merge-review-<PR head SHA>` with `cancel-in-progress: true`. The notifier reads the current PR state, so the newest event subsumes older ones. The group is on the job, not the workflow, so a skipped event cannot cancel a scheduled notifier. - The environment, permissions, trusted default-branch checkout, and secret handling do not change. - `.github/scripts/tests/notify-pr-merge-review.test.mjs`: a new test pins the trigger kinds, the job condition, and the job-level concurrency group. - `doc/PR-MERGE-REVIEW-WEBHOOK.md`: describes the event filter and the concurrency group. **Expected runner jobs per PR push** (typical PR: `PR`, `Storybook Visual`, `commitperclip PR Review` workflows; Greptile, Paperclip Review, Socket (2), Superagent (2) external checks; one Snyk status): | | Before | After | | --- | --- | --- | | Eligible events that take a runner | about 60–70 (every check run + 2 statuses + 3 workflow runs) | at most about 10 (3 workflow runs + 6 external checks + 1 status, only when green) | | Jobs that run to completion | about 60–70 | usually 1–4: events that arrive while a job is pending or running cancel it, so a burst collapses to the newest event. In a backed-up queue, all events for one head collapse to one job | ## Verification - `node --test .github/scripts/tests/notify-pr-merge-review.test.mjs`: 22 tests pass. - The new workflow test fails on the old workflow (red) and passes on the new workflow (green). - `actionlint .github/workflows/pr-merge-review.yml`: no findings. - I confirmed with `gh run list` that `workflow_run.head_sha` for the `pull_request_target` workflow (`commitperclip PR Review`) is the PR head SHA. So one SHA key groups all event kinds for one PR head. - I confirmed that all workflows that make PR checks are in the `workflow_run` list (`CI routing` and `Trusted PR CI` are reusable workflows called by `PR`). - After merge: check that `gh run list --workflow pr-merge-review.yml` shows mostly `skipped` runs for each push, and a small number of runs that execute. ## Risks - A check provider that reports a final green state through a GitHub Actions check run from a workflow that is not in the `workflow_run` list does not trigger the notifier. All current PR workflows are in the list. The doc already tells authors to add new PR workflows to the list. - A failed or cancelled check that later succeeds on re-run still triggers the notifier, because the re-run completes with `success`. - `cancel-in-progress` can cancel a notifier during delivery. The newer job evaluates the same or a more complete state, and the Paperclip idempotency key deduplicates deliveries for the same PR, head, and base. - Each event still makes a workflow run record. Skipped runs do not take a runner. - Low risk otherwise: the notifier is advisory and `continue-on-error`. It does not affect PR verification results. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
45c2bd94c6 | fix(native): name the interrupted attempt when an exact resume is refused (#19709) | ||
|
DottaandClaude Opus 5.5
|
8373ae905a |
fix(accounting): settle a never-started run whose recorder bound an empty fence map (#19826)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budgets hold new work while a native run's final usage receipt is outstanding > - A run that never started a provider has nothing to account. #19700 added a rule that settles such a run at once. > - In restart chaos waves, Retry runs that failed before any provider start still held company admission for up to 10 minutes > - This pull request fixes the condition that stopped the rule from matching > - The benefit: a failed run that provably never started a provider settles at once and never holds other work ## Linked Issues or Issue Description Refs #19700, #19724. **What happened?** An OpenCode Retry run failed at once with `native_session_cleanup_quarantined`. It had no model, no usage receipts, no process start and no provider events, yet it held company admission for the full 600-second closure bound. Two more waves had the same shape, with holds of 309 and 324 seconds. **Root cause** `nativeProviderWorkNeverStarted` (`run-cost-accounting.ts`) required `usage_json.accountingReceiptSourceFences` to be absent. `createRunUsageRecorder` binds every run with that key, as an empty map `{}`, even when no earlier receipt source exists. So the rule never matched a run that had a usage recorder. The run then waited for the closure bound as a run whose receipt was still closing. The restart-evidence check for the unpriced-usage exemption (`budget-exposure.ts`, from #19724) had the same flaw: it treated the presence of the key as restart evidence. **Expected behavior** A failed native run with no receipts, no replaced receipt source and a process that never launched (or was verified stopped) settles at once with no charge. A run whose provider might still be spending keeps holding. ## What Changed - `run-cost-accounting.ts`: the never-started rule treats an empty fence map as "no replaced source". Only a non-empty map shows that provider work began. - `budget-exposure.ts`: restart evidence also requires a non-empty fence map. - `doc/SPEC-implementation.md`: documents the rule. ## Verification - Red then green, `closed-native-receipt-accounting.test.ts`: - "settles a quarantined Retry whose recorder bound an empty fence map at once". This is the exact chaos row: failed, `native_session_cleanup_quarantined`, terminal_failure finalization, receipt sequence 0, empty fence map, no receipts, no process events. - "does not count an empty fence map as restart evidence for a timed-out closure". - Existing tests still pass. A run with a recorded but unverified process still waits for the bound. The restart-derived exemptions still apply to runs with a non-empty fence map or a recovery transition. - Local: `closed-native-receipt-accounting` (51), the budget suites, `heartbeat-cost-accounting`, `adapter-cost-failure-paths`, `native-stop-accounting` and `native-session-executor` all pass. Server `tsc --noEmit` is clean. ## Risks - Low. The change only narrows what counts as a replaced receipt source. A run whose source was really replaced has a non-empty map and behaves as before. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Dotta
|
3112e4c955 |
Verify original Dot OAuth authority for signed Cloud requests (#19819)
## Thinking Path > - Paperclip manages AI agents and their work. > - Dot uses a user OAuth grant to act in one company. > - Cloud must use that original grant when it resolves a selected user's identity. > - A Cloud signature alone must not create user or board authority. > - The tenant's OAuth owner must check the token and its current restrictions. > - This pull request adds a signed private HTTP boundary for that check. > - The result contains only the checked identity fields. It grants no task or board access. ## Linked Issues or Issue Description **Subsystem affected** Server external-agent identity, public MCP OAuth, Cloud control verification, and HTTP logging. **Problem or motivation** Cloud needs a private way to resolve the original Dot token for the selected user, client, company and resource. The existing generic Cloud control route creates an operator actor. That route is not the right owner for this identity check. **Proposed solution** Add `POST /mcp/runner/oauth/cloud-principal`. Require the existing Cloud signature and replay verifier. Bind the whole request to a new, closed `mcp-dot:principal` action with a maximum 60-second lifetime. Authenticate the original token once through the normal Dot OAuth owner. Check current namespace authority before and after that call. **Alternatives considered** A Cloud-created grant or a board actor would change the authority owner. Neither is used. The private route reuses the existing OAuth and Cloud identity owners. **Roadmap alignment** This is a narrow boundary for the existing MCP and Cloud areas in `ROADMAP.md`. It adds no work queue, public hosted profile, or provider enablement. Related public work: Refs #14846, #19427, #19750 and #19776. Searches for the exact private path and action found no duplicate PR. The matching private Cloud caller has merged. The current App head passed its required checks and review. ## What Changed - Add an original-token principal reader. Keep the original client, selected user, company, resource, purpose, scope, membership and feature restrictions. - Add a 16 KiB raw JSON private route. Reject compressed, invalid UTF-8, malformed and unknown-field bodies. Bind an ordered whole-body digest to the signature and existing nonce fence. - Compare the current stack, claim, instance, origin and connector identity before and after OAuth. An absent owner or namespace fails closed. - Mount the route in normal `createApp` after hostname, runtime-claim and standby checks, before session/actor work. Do not add it to generic board-control routing. - Keep private request and response logs content-free, including raw buffers, headers, query data and reflected errors. - Add controlled HTTP/privacy tests and a real normal-App OAuth/Cloud-signer integration case. Document the machine-only contract and correct the existing `assistant-connections/client-use` access entry. No new capability ID or agent API entry is added. No schema, lockfile, workflow, Runner, Muse or useful-task fixture changes belong to this PR. ## Verification - Current `70c03593`: normal module-boundary policy and all seven App compiler programs passed. The public module factory has an explicit portable Express return contract. Whole HTTP/log controls passed again: 30/30. The existing real paired OAuth/Cloud-signer/private HTTP case then passed again on a fresh PostgreSQL fixture: 1/1, zero skipped. All known child processes and five database connections closed; the final independent SQL audit was empty. - The original remote policy failure rejected an App import of a module-internal path; dependent E2E/verify gates stopped on that failure. The fix delegates through the public module API. A first local declaration run reported TS2883 for its inferred return type; the explicit public type fixed it. Original logs remain preserved. No checker, compiler option, test body or authority check was weakened. - The normal full App graph passed on the integrated source. The later fixture-only repairs each passed the complete affected owning program:405 entries, including 403 source roots. The remaining programs and own dependency outputs carried only where their source was exact. - Whole private HTTP/log files passed:30 tests, zero failures or skips. The HTTP controls use a controlled OAuth owner with real signatures. The log controls use the actual HTTP logger. - The paired Cloud source passed its normal strict build and whole 26 signer/client/namespace and credential-error controls. These are controlled caller tests, not provider tests. - One normal-App paired case passed with a dedicated PostgreSQL database. It used real DCR 201, PKCE, consent, token exchange and a persisted unpaired grant; actual signed metadata publication and a committed Cloud receipt; the real Cloud signer/client and actual App HTTP; exactly-once authentication; refresh; current namespace restrictions; and revocation. The Cloud registry/provider were the existing controlled in-memory/Fake fixtures. Recorded inbound Hosts matched, with zero metadata overflow. No grant, readiness or mirror ACK was fabricated. - The original principal helper's 21 controlled tests and two real-OAuth tests passed before the HTTP slice. Those three files remained unchanged through integration. They are separate source-scoped evidence, not an extra combined run. - Preserved failures: a helper import and test typing error were corrected before the passed checks. An earlier suite failure did not serialize its hook error and remains unexplained. Later runs captured a missing normal managed-config initialization and HTTP 404 from the loopback mapper. The fixture now calls the normal initialization owners and uses the existing Host-preserving canonical HTTP adapter. All original authority assertions and deadlines remain. - Reproduce the local files with `pnpm --filter @paperclipai/server exec vitest run src/modules/external-agents/cloud-principal-http.test.ts src/modules/external-agents/cloud-principal-log.test.ts --maxWorkers=1 --no-file-parallelism`. The real integration file is `server/src/__tests__/cloud-principal-normal-app.postgres.test.ts`; it requires its explicit dedicated test DB and an own compiled matching Cloud checkout. - The normal `node scripts/check-maps.mjs` check passed for 35 areas and 192 capability IDs. This validates map structure, not business acceptance. The later edits are documentation/map only; all executed business source and own outputs from the paired case remained exact. - Current-head remote CI passes: 59 successful checks and three intentional skips. Greptile reviewed this exact head at 5/5 with no findings or review threads. The prior Ponytail review was Ship 5/5 at the earlier head; no current-head Ponytail turn is present, which is nonblocking under the authorized review policy. The three full-repository commands were not repeated locally; the focused normal App graph and affected tests above passed, and required remote CI supplies the broad gates. No deployed, provider-capacity, cold-wake or full hosted lifecycle result is claimed. ## Risks - This is a security boundary. Signature, current namespace and original OAuth restrictions must continue to fail closed. The response gives identity only; callers still need normal operation authority. - The request carries the original token. The reserved route uses a closed log projection on both accepted and refused requests. Privacy regressions remain a material risk. - Both repositories must agree on the action and digest contract. Older code rejects the unsupported action; no fallback creates authority. - This is a prerequisite private boundary. The remaining hosted discovery, wake, capability, operation, event, retry/reactivation and revocation lifecycle work, plus Muse, remains open. No profile or capability is enabled here. - No migration is needed. Rollback removes the private boundary and paired caller; normal OAuth and Personal MCP remain with their existing owners. - The engineering documentation describes this private machine boundary and its access restriction. No operator-facing feature is enabled. Documentation for full hosted enablement remains with that later work. ## Model Used OpenAI `gpt-6.1-sol` (Sol6.1), through Codex, produced the source and tests with tool use and local compiler/test execution. A GPT-6-based Codex coordinator reviewed the work. Its exact model ID is unavailable. Context window sizes are unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
b2af45f0f1 |
fix(runner): let OpenAI API-key Codex agents generate images with their existing key (#19654)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - New users often connect a Codex agent with an OpenAI API key during
onboarding.
> - A user asked such an agent for a logo. The agent said that the
built-in image generator was not available and that its fallback needed
a separate API key. It then drew an SVG by hand.
> - #19516 removed `--disable image_generation`. That change is correct,
but it does not reach this path. The pinned Codex 0.160.0 shows its
`image_gen` tool only when Codex signs in with ChatGPT. The check is
`image_generation_available` in `codex-rs/core/src/tools/spec_plan.rs`.
An API key never gets the tool.
> - The runner keeps the provider key out of the agent's shell on
purpose, so the skill's `scripts/image_gen.py` fallback cannot run.
> - This pull request adds a Paperclip `generate_image` tool. The server
makes the image with the agent's existing OpenAI key. The key never goes
to the agent process or the shell.
> - The benefit is that API-key Codex agents can make images with no
second key. The spend goes to the agent's budget and the budget can stop
it.
## Linked Issues or Issue Description
Refs #19516
**What happened?**
An agent connected through onboarding with an OpenAI API key (Codex
harness, `gpt-5.6-sol`) could not generate images. The agent said: "The
image skill's built-in generator isn't available in this run, and its
fallback would require a separate API key." It then drew an SVG instead.
**Steps to reproduce**
1. Create a company with a Codex agent that uses an OpenAI API key.
2. Ask the agent for a raster image, for example a logo PNG.
3. The agent reports that image generation is not available. It does not
make the image.
**Expected behavior**
The agent makes the image with the OpenAI key that already powers it.
**Root cause**
The agent ran on the native runner's Codex app-server path, with
`auth.json = {"OPENAI_API_KEY": ...}`. Codex 0.160.0 removes
`image_gen.imagegen` from the model's tools unless
`current_auth_uses_codex_backend()` is true. That function is true only
for a ChatGPT sign-in.
I proved this with the real pinned binary. I gave it the runner's
isolated launch arguments and the API-key auth file, and pointed it at a
local Responses endpoint. The first model request offers only `exec`,
`wait`, `request_user_input` and the collaboration tools. #19516's test
checked only `codex features list`, so it could not see this gate.
## What Changed
- **New `server/src/services/native-runtime/generate-image-tool.ts`:**
- The tool schema and definition.
- `imageGenerationBindingForRun`. It applies only to native Codex runs
on a direct OpenAI `api_key` connection with no router selection.
- The OpenAI Images API call (`gpt-image-2`, `b64_json`, PNG check, 180
s timeout). The reply is read as a stream with a byte cap before it is
parsed. A decoded image larger than 20 MiB is refused.
- A reviewed rate card for `gpt-image-2`: text input $5, image input $8
and image output $30 per 1M tokens, from
developers.openai.com/api/docs/pricing, retrieved 2026-11-06.
- A create-only, symlink-confined local write.
- **`paperclip-runner-tool-authority.ts`:**
- Adds the `imageGeneration` binding.
- Advertises the tool only for standard mode with a workspace. A remote
target must also have an installer. Dot, Muse, Grok and native review do
not get the tool.
- Adds the `#generateImage` handler. Before any spend, it checks again:
the run and task, budget authority, `issue:mutate`, case handoff, and
the connection.
- Stores a claim-before-spend per-call receipt in
`resultJson.imageToolReceipts`. A retry or recovery replays the receipt
and does not pay again.
- Saves a `charged` receipt (usage, price, request ID and charge time)
as soon as OpenAI returns. A failed save, failed accounting write or
crash can then still settle the spend. A retry or recovery records the
cost once and never buys again.
- Records one `cost_events` row through `createCostEventInTransaction`.
The row is `metered_api` with `costStatus: "estimated"`, and the budget
is evaluated. The row is recorded even if saving fails.
- Checks the task's project budget, as well as the company and agent
budgets, before spending.
- Writes content-free `runner.image_generate` activity entries: started,
completed, failed and unconfirmed.
- **`ai-connections.ts`:** adds `selectImageGeneration`. It checks again
that the connection is an active, healthy OpenAI `api_key` connection
with no custom routing. It also checks the responsible user's
membership, the grant audience and the agent or company installation.
These are the same rules that runtime selection uses.
- **`native-session-executor.ts`:** gets the binding from the run's
admitted `contextSnapshot.aiConnection`. The value comes only from the
server, never from tool input.
- **`native-session-resume.ts`:** the native tool contract changes from
`v18` to `v19` and now lists `generate_image`. A persisted provider
thread cannot add tools when it resumes, so older threads must rotate.
- **`doc/runner-api-tools.md`:** a new section about image generation
for OpenAI API-key Codex agents.
- **#19516: kept, not reverted.** Its changes have no effect on the
API-key path. They are still correct for Codex agents that sign in with
ChatGPT, because `image_gen` works there and saves to
`CODEX_HOME/generated_images`. Those agents need the read grant that
#19516 added. On the API-key path, the new tool replaces the built-in
tool. The tool's description tells the agent to use it instead of
`image_gen` or `scripts/image_gen.py`.
## Verification
**Red on `master` (
|
||
|
DottaandPaperclip
|
ae1f781cac |
fix(native): preserve web files without settling remote recovery (#19814)
## Thinking Path > - Paperclip manages work that AI agents perform for users. > - Native sessions retain files on the web server and state in a remote provider. > - Replacing the web server must preserve its files without changing remote recovery authority. > - The existing maintenance owner requires destroyed-provider receipts even when all web writers have settled. > - This can keep the old web volume required only because a remote recovery obligation remains unresolved. > - This pull request adds a closed web-byte evidence mode to the existing preservation owner. > - The benefit is durable web-file coverage with unchanged remote references, failures and continuation policy. ## Linked Issues or Issue Description Refs #19748, #19806. A search for native harness filesystem evidence and open web/harness changes found no duplicate implementation. **What happened?** A settled successful native source can retain a raw controller, a separate warm supervisor and provider-home files while its historical external lease has no termination receipt. The existing filesystem-independence owner refuses before it can preserve these web files. One retained skill file can also remain physically present after its version row was deleted. **Expected behavior** Preserve every nonexcluded web file as authenticated evidence. Keep missing files absent and deleted skill versions deleted. Leave VM references, recovery actions, failures and execution authority unchanged. Validate the published web-byte receipt on an empty replacement using database bindings only. **Steps to reproduce** Use the included successful-source fixture with a settled web owner, an unresolved external lease, real controller/supervisor files and a retained skill whose version row is deleted. The baseline rejects at its destroyed-provider guard. The new explicit maintenance mode preserves the files and validates on a replacement. A separate fixture uses a prior successful source, a failed retained last run, a genuinely absent controller, a warm supervisor and 267 durable checkpoint parts. **Paperclip version/commit** Baseline `384fa2b7be`; final branch rebased onto `585c885168`. Current head: `990210d7e9`. **Deployment mode** Original-volume maintenance followed by an ephemeral replacement. No hosted operation or provider request is part of this PR. ## What Changed - Add explicit `web_evidence` maintenance mode and a separate `paperclip.native-harness-web-evidence.v1` receipt. Keep the existing terminated-provider mode and old manifest shapes. - Archive the closed canonical session-root inventory, the separate warm supervisor and exact admitted skill manifests through the existing encrypted `original_volume_source_evidence` owner. Preserve bytes and modes. Keep canonical credential exclusions. - Preserve local skill edits and deleted-version files as forensic evidence. Create no skill version, executable checkpoint, provider backup or recovery authority. - Require original-volume capability, settled web writers, physical/finalizer locks, exact task/source/cohort/input bindings and final byte/mode/membership comparisons. Keep genuinely absent raw controllers absent. - Bind allocation and stamp identity while allowing later external status and cleanup observations. Validate replacements from durable database and archive bindings without the original anchor, filesystem or storage reads. ## Verification - Node 24.22.2, PostgreSQL 18.1, macOS arm64. The unchanged baseline fails at the real destroyed-provider guard. The final revision passes all 85 owner, original-volume capability and unchanged continuation-policy cases in 21.14 seconds. - Tests perform real encryption, persistence through the S3 storage interface backed by the owned local fixture, authenticated restore and exact byte/mode comparisons. The 281-byte file remains present while its version row remains absent. The evidence archive stays separate from the provider-session backup scope. - The supervisor-only case preserves 267 database checkpoint parts, keeps the raw controller absent and validates without original-volume identity. Existing strict-mode and old receipt controls pass. - Negative controls cover active web writers, expired but owned finalizers, physical owners, foreign controller/supervisor identities, missing manifests, symlinks, changed bytes/modes/membership/source during upload, changed durable bindings, incomplete receipt coverage and invalid late publication. Concurrent attempts retain one valid winner and one audit event. - Server TypeScript, module boundaries, diff checks and private-data scans pass. The verification planner's first invocation used a wrong script path; its canonical invocation refused a plan because fixture inputs changed during selection. Neither attempt counts as a pass. Required CI will provide broader application coverage. - Earlier fixture and treatment failures remain recorded. They exposed a missing helper, JSONB property-order instability, missing separate-supervisor mode coverage and fixture timing/source/count errors. The final same owner suite passes without weakened assertions. - A canonical cleanup audit found two additional observation fields: the cleanup controller and manual-attempt ID. Their real publication updates reproduced false receipt invalidation on the first PR head. The narrow correction keeps WEB coverage valid through those observations while changed allocation identity still fails. The exact same 85-case suite passes afterward. The final `990210d7e9` head has fresh Greptile 5/5 with no comments or unresolved threads. Required CI run [42370607882](https://github.com/paperclipai/paperclip/actions/runs/42370607882) passes: 55 successful checks and four intended skips, plus successful Snyk. The old head's CI was superseded by this correction and is not counted as a pass. From `server/`: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts src/services/native-runtime/native-harness-original-volume.test.ts src/services/native-runtime/native-harness-filesystem-independence-continuation.integration.test.ts --hookTimeout 90000 --maxWorkers 1 --fsModuleCache ``` ## Risks This changes a maintenance trust boundary. It proves web-byte preservation, not provider recovery or termination. External recovery obligations and existing resume failures remain. Unknown layouts, missing manifests, active local owners and changed authority fail closed. Invalid receipts are not automatically replaced. The inventory has independent limits of 128 entries, 16 MiB per file and 64 MiB total. Larger valid data stays on the original volume and blocks this mode. Canonical credential exclusions remain excluded. Evidence uses the existing encrypted ledger and retention roots; it does not become an executable backup. Old receipts remain byte-compatible. Startup adds bounded per-retained-scope database reads. Measure the affected cohort's actual serving startup cost before enrollment. Validation adds no filesystem or object read, worker or recurring poll. Live source qualification and publication remain separate after review, merge and normal delivery. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and PostgreSQL/filesystem test execution. The runtime does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
585c885168 |
fix: preserve prepared sources held by provider artifact failures (#19810)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace migration archives local files before it changes their storage authority. > - A failed provider session can leave an active board-owned recovery hold over a prepared local source. > - The existing migration owner accepts finalized sources and workspace-export failures. > - It rejects a settled provider-artifact failure even when the prepared source and recovery allocation have complete bindings. > - This PR permits that exact preservation case under the existing source and finalizer locks. > - The remote recovery hold remains active. The archive preserves local bytes and the existing recovery recipe. ## Linked Issues or Issue Description Refs #19642 and #19805 for the existing online migration and source-closure owners. **What happened?** Online workspace migration rejects a prepared source held by a terminal `runner_remote_provider_artifact_incompatible` recovery. This also blocks another local workspace that shares the Git source. The provider session had executed work. A pre-execution exemption would be unsafe. **Expected behavior** Preserve the local source through the existing immutable archive owner when the board recovery, task version, terminal run, finalizer, and declared allocation all match. Keep the remote recovery action and authority unchanged. **Steps to reproduce** 1. Create a prepared native source and a failed run with provider session and tool events. 2. Set its finalizer to an unowned terminal artifact failure. Add the matching active board recovery and task status version. 3. Remove the local descriptor and inspect or migrate a shared local Git alias. 4. Before this change, the source proof is rejected. The regression fixture reproduces this failure. **Paperclip version or commit** Reproduced on master `1674902b4f`. The final branch is based on `5a92d53f7d`. ## What Changed - Accept only the matching active board-owned artifact failure with a current task block, no live execution authority, an exact producer fingerprint, and a declared prepared-source allocation. - Include the task version and complete existing source/recovery evidence in publication revalidation. - Lock the proven recovery tasks in sorted company scope before publication checks. A held descendant task produces the existing busy error. - Add a real PostgreSQL regression with 124 tool executions, a missing descriptor, shared Git aliases, immutable download, and cold restore of the local archive. - Add stale, foreign, owned, controlled, malformed, and publication-race rejection cases. - Document that this archive preserves local bytes and the recovery recipe. It does not prove remote export or release the remote hold. ## Verification - Genuine baseline RED: the positive migration regression rejects the source before the production fix. - Focused treatment: 12/12 PostgreSQL cases pass. - Review-race RED: both a selected source and a Git companion published while a descendant recovery task was locked. The corrected tests reject both cases. - Two affected owner suites after the lock repair: 91/91 pass in 23.18 seconds. - Strict server TypeScript check passes after the lock repair. - Module boundaries and `git diff --check` pass. - [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/42367152081) passes: 56 successful checks, four conditional skips, and one neutral review check. No checks remain pending or failed. - Greptile and Paperclip reviews are both 5/5 on `0a6e6c3ece`, with no open findings. - The first descendant fixture attempt hit the immutable task/privacy guard before it reached the race. The corrected fixture creates that task binding at admission and produces the two genuine REDs described above. ## Risks - A broad classification could let a migration move local authority without a complete recovery recipe. The new branch requires the exact failure, producer, task version, source allocation, and released execution owners. - Evidence can change during upload. Existing locks and full publication revalidation reject those races. - The remote allocation may contain changes absent from the local source. This PR preserves only local bytes and the durable recovery route. It does not mark remote work exported, finalized, or safe to discard. - No schema, provider lifecycle, admission, or polling behavior changes. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, tool use, and local code execution. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d2efe3392b |
fix(native): preserve retired authority evidence (#19806)
## Thinking Path > - Paperclip manages work that AI agents perform for users. > - Native sessions retain control-plane files when execution authority moves to a later run. > - Volume migration must preserve these historical bytes before a server can stop depending on its old volume. > - The existing preservation owner archives the current controller and supervisor, but refuses a nonempty authority-epoch directory. > - A settled four-run session can therefore remain blocked after its current controller archive is complete. > - This pull request adds a closed historical inventory to the existing encrypted evidence archive. > - The benefit is preserved source evidence and database-only replacement validation without granting execution or recovery authority. ## Linked Issues or Issue Description Refs #19748. A search of pull requests for filesystem independence and issues for authority-epoch history found no duplicate change. This fixes its uncovered historical-file case. **What happened?** A settled native session retained three prior controller files in authority-epoch directories. Its three historical database checkpoint headers were absent. Maintenance preserved the current controller, then refused filesystem independence because those historical files remained uncovered. **Expected behavior** Preserve the exact historical bytes and modes through the existing encrypted evidence owner. Keep original files, prior archive rows, run outcomes, warnings, stamps, session pointers, and recovery policy unchanged. Validate the resulting receipt on a replacement without the source volume. **Steps to reproduce** Use the included four-run fixture with a current-controller-only archive, three prior authority epochs, destroyed provider receipts, released ownership, and absent historical checkpoint headers. Call the canonical filesystem-independence owner. The base rejects the nonempty directory; the treatment preserves the complete evidence and validates it after removing source-volume identity from the test process. **Paperclip version/commit** Baseline `ef83d566cacf1d5f740d701560597dad2bfa4619`; final branch rebased onto `27170dca49`. **Deployment mode** Original-volume maintenance followed by an ephemeral replacement. This PR performs no hosted migration or provider operation. ## What Changed - Add optional closed authority-epoch entries to the existing `original_volume_source_evidence` archive. Preserve nested file bytes and modes through the existing encryption, readback, ledger, audit, and retention owners. - Bind each epoch to its retained run, canonical checkpoint identity when present, or explicit header absence plus the exact admitted authority binding. Keep current-source checkpoint and artifact checks strict. Historical evidence never becomes an executable checkpoint. - Recheck the complete historical inventory at archive and receipt publication fences. Keep physical locks, all-run ownership and termination checks, source capability checks, and final CAS. - Bind the history inventory and its database facts into the authenticated archive manifest. Replacement validation uses database bindings only. Later header presence, removal, retirement, or changed identity invalidates the receipt. - Leave history-free manifest bytes and receipt shapes unchanged. Keep an existing current-controller-only archive when a later complete archive is added. Unknown children and other nonempty history directories remain blockers. ## Verification - Node 24.22.2, PostgreSQL 18.1, macOS arm64. The final four-run fixture fails on the baseline owner at its nonempty-history guard. The same final fixture and treatment pass all 51 focused PostgreSQL/filesystem cases. - The positive case performs actual encryption, object-store persistence, authenticated restoration, and exact byte/mode comparisons. It proves replacement database-only validation with absent source paths and storage reads refused. - Controls cover foreign identities, unknown children, owned prior runs, changed bytes or new files after upload, missing or newly present historical headers, malformed and retired headers, an unknown authority override, receipt-identity tampering, and a late concurrent winner. Existing current-controller evidence and history-free manifest bytes remain intact. - The baseline/treatment driver restored only its two owned source files in `finally` and verified their exact hashes. Earlier setup and treatment failures remain recorded: missing bundled PostgreSQL library links, an overly narrow raw-schema check, and an incorrect synthetic transient-workspace binding. The final baseline and treatment use the corrected same fixture; those earlier attempts are not counted as passes. - Review follow-up `21183b75f4` closes the missing-history-field gap. Deleting history from an actual published receipt reproduced an invalid true result before the fix. Unconditional manifest reconstruction now rejects it; all 46 owner cases pass. Genuine history-free manifests retain their exact bytes. - Review follow-up `f3d305dd36` adds suspended historical Runner-state bytes and mode checks through authenticated restore, active/foreign state refusals, exact 16 MiB and 64 MiB boundaries, and retention of an oversized file without publication. All 51 cases and server TypeScript pass; production behavior is unchanged. - CI run `42362784772` on the previous head had one unrelated 30-second timeout in `manual-private-cell-report-assets.test.ts`, with 6,864 support tests passing. The failure remains recorded. The prior head `21183b75f4b223cd3ec28124cdc194f24b4f79ba` passes [required CI run 42363408227](https://github.com/paperclipai/paperclip/actions/runs/42363408227), including Runner support: 55 successful checks and four intended skips, plus successful Snyk. The final test/documentation head `f3d305dd3690a7db38b57c4a77389c17206cb76b` passes [required CI run 42364958386](https://github.com/paperclipai/paperclip/actions/runs/42364958386): 55 successful checks, four intended skips, one neutral review check, and successful Snyk. Fresh Greptile is 5/5; all four review threads are resolved. No timeout or assertion was weakened. - The count warning is resolved as independent bounds: the actual descriptor accepts 32 control-plane-only epochs (97 entries) and 32 epochs with 31 Runner states (128 entries), and rejects 129 entries. The 32-epoch and 128-entry maxima both apply; the capture budget is unchanged. - Server TypeScript and module-boundary checks pass. The local verification planner selects ten broad application lanes; required CI supplies that wider coverage instead of duplicating it locally. - Reproduce the focused owner suite from `server`: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts --hookTimeout 90000 --maxWorkers 1 --fsModuleCache ``` ## Risks This changes a maintenance trust boundary. A source capture is limited to 128 entries, 16 MiB per file, and 64 MiB total. These limits are intentionally below the Runner's 256 MiB state maximum. Larger valid epochs remain on the original volume and continue blocking filesystem independence. It uses bounded directory reads and file descriptors, rejects aliases and unknown paths, and rechecks bytes before publication. Storage failures retain original files and ledger evidence; they do not settle recovery or authorize a retry. Startup adds bounded database reads for retained historical header bindings or their absence. It adds no source filesystem or storage reads. The selected cohort's serving startup cost still needs measurement before enrollment. Other historical layouts require their own canonical source qualification. No missing provider history is declared recovered, and no automatic corruption repair is added. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source editing, shell tools, and PostgreSQL/filesystem test execution. The runtime does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
51ab7ebdc6 |
fix(recovery): admit recovered scheduled preparation retries (#19802)
Allow explicit Retry after a recovered scheduled setup failure. Recognize the exact host scheduled-retry promotion only when its saved parent, attempt, deadline and reason match the source run and context. Keep changed, extra or foreign evidence held. Preserve the existing provider-absence, stopped-source, recovered-file and authorization gates. Keep refusal tests receipt-only until they change saved proof, then invoke the real retry handler. Positive tests retain real HTTP delivery. Verification: 217 unique tests passed across five recovery suites. The 12-test affected suite and its typecheck layer passed again after the test-race fix. All applicable CI passed; exact-head Apex review was 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
bf12b2d8ad |
fix: retry reviewed chat setup after temporary contention (#19787)
Retry reviewed chat preparation with the original accepted input. Bound retries for authorization proof waits that expire before provider work. Preserve the original answer receipt and queue later wakes separately. Keep plan confirmation semantics by assigning the interaction marker only to question answer delivery and legacy question payloads. Verification: 433 tests across four backend suites, all 17 project-chat browser tests, and all server source/test typecheck layers passed. The plan approval regression was reproduced on the previous head before the fix. All applicable CI passed and exact-head Greptile Apex scored 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
27170dca49 |
fix(workspaces): preserve the read-only capture phase budget (#19803)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspace saves must preserve the files produced by a completed turn. > - When a sandbox disk is full, the existing fallback transfers a stable source and seals it on the controller. > - The fallback gives metadata checks and the portable codec the same two-minute timer. > - Slow but successful metadata checks can leave the codec only milliseconds, although normal sandbox capture allows five minutes. > - This change keeps the metadata budget and gives the local codec the existing finite capture budget. > - All source checks, archive checks, ownership checks, and failure cleanup remain required. ## Linked Issues or Issue Description Refs #19465. Related: #19364 and #19619. The open diagnostic change in #19619 records fallback causes; it does not separate these phase budgets. No duplicate budget fix was found. **What happened?** A retained workspace failed five read-only exports at `stage=archive` with `transport_timeout`; four warnings also named `ABORT_ERR`. The fallback had already transferred the source and passed the received-tree and source inventory checks. Its controller codec received the same two-minute signal used by those checks. **Expected behavior** Metadata checks share a finite two-minute budget. Sealing uses the same finite five-minute policy as normal sandbox capture. The final source check uses only the unused metadata budget. A changed source, expired budget, or cancelled codec must not produce a preservation receipt. **Steps to reproduce** 1. Create a standalone Git fixture with staged changes, an untracked result, and a confined symlink. 2. Make the initial sandbox helper report the exact disk-full error. 3. Transfer real files and run the actual inventory scripts. Delay a successful second inventory reply until just before the native two-minute deadline. 4. Observe that the real portable codec starts with a live signal, then aborts when that shared timer expires. ## What Changed - Name and reuse the existing 300-second command policy for controller capture. - Retain the aggregate 120-second metadata budget. Preserve its unused remainder for the final source recheck. - Keep the exact disk-full error, source comparisons, file/Git checks, and cleanup behavior. - Add six real-filesystem and real-codec controls. Document the separate phase limits and the current absence of a caller abort signal on the save interface. - Repair the existing onboarding test's fake-environment mapping with the same real-agent PATCH mapping used by its sibling. Verify the persisted agent through a real GET. Production UI code is unchanged. ## Verification - Native-timer RED on base `35d41f11bb`: a successful source reply arrived at 119,981.62 ms. Its command took 119,854.47 ms within its 119,873 ms cap. The actual codec started at 119,982.70 ms with a live signal and aborted at 120,001.84 ms. Its nested cause matched the retained warning: `stage=archive reason=operation_failed transport=transport_timeout code=ABORT_ERR`. The source inventory was unchanged and private staging was removed. - The real-archive control sealed 10,166,395 bytes and 533 entries in 555.55 ms. It restored HEAD, staged data, untracked data, and the symlink, and passed Git fsck. The recorded delay is synthetic; it is not a measurement of the failed hosted workspace or provider throughput. - The native RED leaves about 18 ms for the final check too. The repaired code must still refuse that exact exhausted-metadata case; it is not claimed as a native-clock GREEN. - The committed fast regressions control phase time and delegate to the actual inventory commands and portable codec. They cover successful restore after the old metadata signal expires during sealing, pre-capture metadata expiry, codec deadline/cancellation, and final metadata expiry after a real completed archive. - From `server/`: `node ../node_modules/vitest/vitest.mjs run src/services/remote-workspace-readonly-capture-budget.test.ts src/services/remote-workspace-transport.test.ts src/services/remote-workspace-readonly-inventory.test.ts src/services/remote-workspace-transfer-timeout.test.ts --fsModuleCache` — 156 tests passed in 42.30 seconds before removal of a redundant test duration assertion. The final-head six-case regression also passes after rebasing onto `fff70fe49e`; only unrelated Runner and E2E files changed in that rebase. - Review follow-up `ad565dd984` makes the phase clock fully controlled. It checks the requested final 2,000 ms budget separately from the real subprocess's 30-second test limit. All six controls pass in 11.94 seconds, and server TypeScript passes again. The earlier real-time-plus-offset clock could falsely fail on a loaded machine; that review finding is fixed. - Server TypeScript and module boundary checks pass. The first TypeScript attempt could not resolve the unbuilt plugin SDK. Building that existing workspace prerequisite resolved the setup errors; no source workaround was added. - The first preflight attempt used the host pnpm shim and was stopped. The second lacked sandbox network access and was stopped. The canonical cached preflight then completed in 22.1 seconds with the Runner build restored from cache. A later test launch used the wrong working directory for a test rewrite; its wrapper was stopped before waiting through a second native-timer test. None of these interrupted attempts is a pass. - Final-head CI first failed one unchanged Runner-support privacy test at 30 seconds: [job failure](https://github.com/paperclipai/paperclip/actions/runs/42359560195/job/128365192635). The other 6,864 tests passed and four were skipped. The failing test and its report owners match base `fff70fe49e`; the exact case passes locally unchanged in 2.97 seconds (one pass, 40 unselected). Job cleanup reported an orphan `unzip`; that does not establish the timeout cause. GitHub rejected the scoped retry while the parent workflow was still running, so no retry has started. No old-head retry was started because the separately proven onboarding fixture repair requires a new head and a fresh full pipeline. The original Runner failure remains recorded. - The same CI run failed onboarding case 160 at 60 seconds; its retained page showed `Environment not found`. The fixture injected an environment only in browser responses, then sent that fake ID to the real agent update. The original case passed locally in 37.44 seconds despite rejected HTTP 422 updates. Adding a real update-success guard produced a local RED in 26.83 seconds (expected 200, got 422). Reusing the sibling fixture's fake-ID-to-local-default PATCH mapping produced GREEN in 22.52 seconds. A real GET verifies the same persisted agent ID, name, and default environment. Existing sign-in, cancellation, and no-hire assertions are unchanged. CI screenshot/error context and the local persistence RED remain retained. - All other product checks on `ad565dd984` passed, including build, TypeScript, all general and serialized server shards, and native Runner checks. Final head `58bb4ee2411dfefca462f0e9b5a701d84ce1326e` passes the fresh pipeline: [CI run](https://github.com/paperclipai/paperclip/actions/runs/42360927780). Runner support passes in 3m36s and onboarding shard 5/8 passes in 5m46s, with no Runner-support source change. The original failures remain part of this verification history. - Environment: macOS arm64, Node 26.4.0, real Node/Git/filesystem operations. No provider or recovery calls, replayed turn, source mutation, or hosted recovery acceptance was performed. Final-head required CI and review pass; the exact gate state is recorded below. - Exact-head final gate: 57 successful checks, three skipped, two neutral, and no failures. Both required gates (`ci / verify` and `ci / e2e`) pass. Greptile reviewed this exact head at 5/5, and zero review threads remain open. The non-required companion `review` job is still queued with no steps; it has not passed, and GitHub therefore reports `UNSTABLE` while the PR remains mergeable. ## Risks - Controller work after transfer can now take up to two minutes for metadata plus five minutes for sealing. Each phase remains bounded. Existing entry, repository, configuration, and transfer limits stay in force. - The save interface has no caller abort signal. This change adds no Stop guarantee and does not replace the recovery retry window with an operation deadline. Existing physical ownership and publication fences remain in force. - This proves the timer mechanism with a synthetic slow inventory. It does not measure the actual dirty workspace size or show that its archive finishes within five minutes. A guarded file-only recovery and independent durable readback remain required after normal delivery. - The onboarding fixture maps only its exact synthetic environment ID. It still calls the real agent update and read routes. It does not loosen production environment validation. - No schema, provider transport, task status, recovery policy, or source cleanup authority changes. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, repository tools, code execution, and regression tests. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (both required gates and all product checks pass; non-required companion review remains queued as stated above) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
2468b3f907 |
fix(native): tell a restart continuation that earlier processes may be gone (#19804)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a control-plane restart loses a turn, the native runtime sends a same-conversation continuation (#19501, #19699, #19771) > - A restart can also kill shell or background processes that earlier turns started > - In three restart storms, the continuation agent could not inspect its killed `bench.sh` and set the task to blocked > - The continuation prompt told the agent not to "blindly rerun an action whose outcome is unknown", so the agent did not re-run the command > - This pull request tells the agent that earlier processes may be gone, and to check the workspace outputs and re-run a command whose expected output is missing > - The benefit is that a task can finish after a restart instead of blocking on a process that no longer exists ## Linked Issues or Issue Description Refs #19501, #19699, #19771 **What happened?** Restart chaos workload B5 (Codex and Claude) failed this way three times: storm 6b t14, s7111 t16 and storm 10 t12. The agent started `bench.sh` in the background. A restart killed it. The restart continuation agent tried to find its tool session and got "process inspection is denied". It then blocked the task. The comment read: "Blocked by the Paperclip runtime restart: the benchmark session is unavailable…". It added that it "cannot verify whether the original run is still active or recover its output, so I have not rerun it". **Expected behavior** The agent knows it is in a resumed session after a restart. It checks the workspace for the command's outputs. If the expected output is missing, it re-runs the command and finishes the task. **Root cause** `packages/paperclip-runner/src/native-session-runtime.ts`, `restartContinuation`: the prompt did not say that processes from earlier turns may be gone. It told the agent to inspect each unfinished command's "current state" and not to "blindly rerun an action whose outcome is unknown". With the process gone, the agent could not inspect it, so it blocked. ## What Changed - The restart continuation prompt now says: - This is a resumed session after a Paperclip control-plane restart. - Shell commands and background processes from earlier turns may no longer be running, and their tool sessions may be gone. Do not block only because one cannot be found or inspected. - Check the workspace for the outputs or artifacts the command should have produced. If the expected output is present, use it. If the expected output is missing, re-run the command. - Do not repeat completed work. Before repeating an action with external effects (a push, a message, a deployment), check whether it already took effect. If you cannot tell, report the blocker. - The prompt does not say that re-running is always safe. Process teardown (#19727) has not merged. - `doc/execution-semantics.md`: one additive sentence. ## Verification - `native-session-runtime.test.ts`: the three restart continuation tests (Codex, ACPX, OpenCode) now check the new guidance. They were red before the prompt change and are green after it. - Guard: "uses the admitted session cwd in the task envelope" now checks that fresh turns and ordinary resumed turns do not contain the restart guidance. - `npx vitest run src/native-session-runtime.test.ts src/drivers/codex/ src/live/runnerd-restart-continuation.test.ts` in `packages/paperclip-runner`: 603 passed. - `tsc -p tsconfig.json` and `tsc -p tsconfig.test.json` for `packages/paperclip-runner` are clean. - Not run: a live restart-chaos replay with a real model. ## Risks - Low. This changes only the prompt text of the restart continuation. Ordinary turns do not change. - An agent may now re-run a command whose earlier run had partial side effects. The prompt asks it to check external effects first, and to re-run only when the expected output is missing. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ef83d566ca |
fix(native): ask for the result once after a restart continuation ends without one (#19771)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs continue a turn that a control-plane restart lost, in the same conversation (#19501, #19699) > - In restart storms 5, 6b and 8, ACPX Claude runs crossed restarts, ran the continuation, and then failed with `native_session_retry_exhausted` > - The continuation completed, but Claude ended it with "I'll wait for the notification when it completes" for a background command, and did not call `paperclip_finish` > - The next same-run attempt then sent no turn at all and failed with "session returned no semantic result" > - This pull request makes the continuation ask for the result, and asks once more in the same attempt if the continuation still ends without one > - The benefit is that a restart-interrupted task settles without a Retry, and the restarts that the run survived cannot block the request ## Linked Issues or Issue Description Refs #19501, #19699, #19720 **What happened?** Storm 8, run 84c532f7 and its Retry 0faead66 (ACPX Claude, task CHA-5). The steps are the same in both runs: 1. `resume_dead_runner`, then `session.resumed`, then `turn.failed` with `provider_turn_lost_on_restore` (recoverable). 2. The restart continuation runs. Claude starts the bench in the background and ends the turn with "I'll wait for the notification when it completes". The turn completes with no `paperclip_finish`. 3. Lifecycle: "provider turn completed without a semantic result; same-run disposition recovery persisted" (attempt 2). 4. 30 s later, attempt 3 restores the session. It sends no turn. After about 2 s it fails with `native_finalization_missing: session returned no semantic result`, and the attempt count makes it `native_session_retry_exhausted`. **Expected behavior** The continuation tells the agent to finish and report its result. If the continuation still completes without a result, Paperclip asks for the disposition one time. Restarts that the run already survived do not block that request. An agent that never reports a result still fails closed. **Root cause** - `packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts` `startTurn` (lines 224 and 248): the restart continuation uses the driver's one-shot disposition marker (`dispositionOnlyRecoveryAvailable`). #19699 uses the marker on purpose, so that a crash can adopt the accepted continuation turn. The marker then stays consumed with the continuation turn id. - `packages/paperclip-runner/src/drivers/codex/codex-session-terminal.ts` (line 226): only a *lost* continuation gives the marker back. A *completed* continuation with no result does not. - `packages/paperclip-runner/src/native-session-runtime.ts` (`dispositionRecoveryWasSubmitted`, about line 2761): on the next attempt, the consumed marker looks like a disposition request that already ran. Recovery replays the continuation's terminal and starts no turn. So the disposition request never happens. - `server/src/services/native-runtime/native-session-executor.ts` (lines 9755-9761 and `nativeSessionFailureDisposition` at 7768): the `semantic_result_missing` same-run retry charges a provider attempt on the same counter as restart resumes. In this run, attempt 3 was the last allowed attempt. - The continuation prompt (`restartContinuation`) did not tell the agent to report a result, or not to stop and wait for a background notification. ## What Changed - Codex-protocol driver (used by Codex and by ACPX through runnerd): when a restart continuation completes without a semantic result, the driver gives back the one-shot disposition marker. The disposition request is not a continuation, so it consumes the marker for good. - Native session runtime: after that, the same attempt sends one disposition request (`Paperclip semantic-result recovery after a restart continuation ... invoke exactly one paperclip_finish or paperclip_block`). It does this only when the driver shows that the continuation borrowed the marker and gave it back. Other drivers keep the current behavior. - Restart continuation prompt: it now says to finish in this turn, not to end the turn to wait for a background command or a notification, and to report one `paperclip_finish` or `paperclip_block`. - `doc/execution-semantics.md`: one additive paragraph. - No server change. The disposition request runs in the same attempt, so it uses no provider attempt. This avoids `native-restart-recovery.ts` (#19747). ## Verification Red before the fix, green after: - `codex-app-server-driver.recovery.test.ts` "keeps one disposition request after a restart continuation completes without a result". Red: marker stays consumed on `turn-2`. Green: the marker is given back, one disposition turn starts, and a resultless disposition turn does not re-arm. - `native-session-runtime.test.ts` "settles a restart continuation that ended without a result through one disposition turn (Codex driver)". Real Codex-protocol driver with a fake transport. Red: only one `turn/start`, then `native_finalization_missing`. Green: the disposition turn `turn-3` reports the result and the run completes. - `native-session-runtime.test.ts` "asks once for the disposition when a restart continuation ends without a result" and "fails closed when the driver keeps the continuation's disposition marker". Also run: - `npx vitest run src/native-session-runtime.test.ts src/drivers/codex/` in `packages/paperclip-runner`: 593 passed. - `src/live/runnerd-restart-continuation.test.ts` (real runnerd and fake Codex, killed runner): 5 passed. - `tsc --noEmit` for `packages/paperclip-runner`. - `pnpm verify` exits 2 because the doc change is a global CI input. Required CI supplies the broad coverage. Not run: a live replay of storm 8 t05 with Claude. The chaos harness needs provider keys from the operator's shell. ## Risks - Low to medium. A run whose continuation ends without a result now gets one more provider turn in the same attempt. The bound is one disposition request per continuation. Each continuation still needs an admitted restart, and the provider-attempt budget bounds those. - If the controller crashes after the continuation completes but before its checkpoint, the checkpoint keeps the stricter consumed marker. That case fails closed as before. - The prompt text changes for restart continuations. It does not change a protocol schema or a profile digest. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6adc8b1049 |
fix(recovery): keep restarted Retry attempts as task directories and settle failed re-preparation (#19759)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task recovery lets a user click Retry after a run fails, and a native run restarts its own preparation after a control-plane restart > - A Retry for a task without a project runs in a fresh task directory under `isolated-workspace-attempts` > - When a restart hit that Retry during bootstrap, preparing it again treated the task directory as the base of a new git worktree and failed > - The failed run then kept its native recovery claim open, so the task showed Retry but every Retry click created no run > - This pull request keeps the restarted attempt in its task directory and settles a failed re-preparation > - The benefit is that a Retry finishes after a restart, and a failed one can be retried again ## Linked Issues or Issue Description Refs #19666 **What happened?** A task without a project failed after a restart (`native_session_interrupted`). The user clicked Retry. The Retry run started in a fresh task directory. A restart hit it during bootstrap. Restart recovery prepared the same run again and failed with `workspace_validation_failed`: "requested isolated_workspace with git_worktree, but base workspace `.../isolated-workspace-attempts/<company>/<issue>/<run>` is not a git checkout". The task rested blocked with Retry shown. A later Retry click created no run: the receipt stayed deferred with "New work must wait while a native run recovers its unfinished accounting." A second storm wave showed the same failure on a Claude C1 task. **Expected behavior** Preparing a run again after a restart reuses the task directory that run already bound. A git worktree base is only the project's checkout or the task's own worktree. A run whose re-preparation fails leaves a working Retry. **Paperclip version or commit** `master` at `51eb4ff705` and `131b566ca4` (both include #19666). Restart-chaos waves on 2026-11-07. **Steps to reproduce** 1. Use a task with no project and isolated workspaces enabled. Let a native run fail after a restart. 2. Click Retry. The run starts in `isolated-workspace-attempts/<company>/<issue>/<run>`. 3. Restart the control plane while the run bootstraps. 4. The re-preparation fails with `git_worktree_base_not_git_checkout`. Click Retry again: no run starts. Root cause: - `server/src/services/fresh-workspace-continuation.ts` `readFreshWorkspaceReadiness` returns `null` for a native run that already has admitted input and a bound workspace (from #19693). This is correct: the restart must resume the admitted workspace. - `server/src/services/heartbeat/environment-selection.ts` (`boundTaskDirectory`) excluded every native recovery binding from the task-directory path. With readiness no longer fresh, `useIsolatedTaskDirectory` was false. The default `isolated_workspace` strategy (`git_worktree`) applied, with the task directory as its base. - The restart recovery claim set the native coordinator to `observed` with its lease owner. `failRunSetup` failed the run, but nothing released that claim. `budgets.ts` reads an open coordinator as unfinished accounting, and Retry admission reads it as `controller_settling`. ## What Changed - Environment selection treats a native recovery binding as the task directory when it is the task's current binding and a bound isolated task directory. The directory strategy is kept; it never becomes a git worktree base. A recovery binding the issue no longer points to keeps its existing path. - New `settleFailedNativeRecoveryPreparation`. When a `bootstrap_incomplete` or `resume_dead_runner` restart claim fails its re-preparation before any provider dispatch, it settles the claimed coordinator as `terminal_failure` and wakes deferred work. It only changes a coordinator that is still `observed`, owned by that claim, without a result, for a failed run. Reattach claims (live runner) are not changed. - `doc/task-recovery.md` describes both rules. ## Verification - `environment-selection.test.ts`, RC-58 cases: a restarted fresh attempt bound to the task re-prepares as its task directory (`useIsolatedTaskDirectory` true). It failed before the fix and passes now. A binding the issue moved away from keeps the old behavior. - `native-runtime/native-recovery-preparation-failure.test.ts` (embedded Postgres): before settlement, Retry admission reports `controller_settling` (the dead Retry). After settlement the coordinator is `terminal_failure` with the run's error code, and admission no longer waits on the controller. Settlement is idempotent. Another owner, a reattach claim, a run that is not failed, and a coordinator finalizing elsewhere are left unchanged. - Local regression: `environment-selection`, `workspace-preparation`, `retry-workspace-continuity`, `native-recovery-preparation-failure`, `explicit-native-continuation` (588 tests passed). - Restart-chaos: local restart-chaos runs on `3a7abef4e5` passed, but neither one reproduced the exact failure (a restart during the Retry's bootstrap). The race depends on timing, so the unit tests above are the red/green proof for both defects. - `--replay w-20261107T011010-s7403/t02` (codex C1 K4 with the original fault sequence): PASS, R1-R6 and T. The replay's Retry did not cross a bootstrap restart. - Wave `w-20261107T064432-s7210` (codex+claude, C1+B1, K1/K4/K6, `--triggers retry`, concurrency 2): 8/8 PASS. It had 4 Retries, including C1 tasks with a fault fired right after the Retry click. No `git_worktree_base_not_git_checkout` and no dead Retry. The server log has no "incomplete native runner bootstrap" restart in this wave either. - `ce7e3f9b19` registers execution work before the settlement writes (Greptile finding). ## Risks - A native recovery now re-prepares a bound task directory through the task-directory path. That path still validates the binding, owner, status and directory identity before use. - Settling a failed re-preparation ends that run's recovery claim. It applies only when the previous runner is verified gone (`bootstrap_incomplete`, `resume_dead_runner`) and no provider dispatch started in this attempt. A Retry still passes its normal stop, ownership and admission checks. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
af0fe8fe81 |
Keep personal MCP connections functional while idle tenants sleep (#19427)
## Thinking Path > - Paperclip manages work for AI agents. > - Personal MCP connections should remain enabled while an idle tenant sleeps. > - Unfinished OAuth, transfer and delivery work must retain its owner and deadline. > - Cloud can receive bounded metadata and incoming work while the tenant remains the company-scoped authorizer. > - Current identity, revision and delivery acknowledgments establish when ownership has transferred. > - This change preserves useful wake, retries and revocation while making settled ownership inspectable without writes. ## Linked Issues or Issue Description Refs: #19377 and #19750. Hosted companion: [paperclip-cloud#1217](https://github.com/paperclipai/paperclip-cloud/pull/1217). **What happened?** Enabled personal MCP grants and subscriptions could keep an otherwise idle tenant awake. Ignoring those records would lose discovery, authorization, delivery and revocation obligations. **Expected behavior** Unconsumed authorization codes, device requests, file handoffs and delivery work retain their original deadlines. Settled connections remain enabled and become quiet after the current external owner acknowledges responsibility. Incoming useful work resumes normal authorization and delivery; revocation still takes effect for the current company. **Steps to reproduce** Complete normal PKCE/device consent and company selection, synchronize client/device and revocation state, retain the connection while idle, then exercise useful tools/events and revocation. Sleep eligibility requires current incoming/deadline ownership throughout. ## What Changed - Add bounded signed client/device metadata and revocation synchronization with exact revision, identity and ACK checks, startup recovery and finite legacy-device cutover. - Preserve company selection, authorization and operation identity. Register grant work before its INSERT in the same transaction; retain rollback, expiry and retry behavior. - Require current managed identity/resource before hosted publication. Receipt-cleared re-enable must obtain a fresh source ACK before device creation. - Reuse read-only settings and inspection budgets; connect MCP owners to normal startup, drain and shutdown. - Bound metadata pages and the actual signed request body while preserving the default limits of other operations. - Generate migration 0440 after the merged Legacy migration. Preserve all 438 existing executable migrations and the journal prefix; retain the normal five newest snapshots. - Preserve the merged transaction-owner forwarding and regression coverage from #19750 through normal ancestry integration. ## Verification - Current own Shared/SDK, database, eval-kernel and Runner builds passed, together with all seven normal application TypeScript programs. Missing generated package exports/workspace links were restored through normal setup; the original setup failures were retained. No source, compiler option, test assertion or timeout was weakened. - Normal migration generation changed only the five intended MCP tables; its SQL is byte-identical to the reviewed MCP delta. Fresh reference databases and both retained master/Muse histories upgraded through 0440, preserving historical hashes, row fields, schema and idempotence. Direct assertions cover the Legacy null default and MCP null/zero defaults. Two selected history cases passed; the other 21 database cases were unselected. - All six current MCP lifecycle cases passed with zero skips: normal PKCE approval intent, ACK recovery after outage, mutation during publication, disable/re-enable with original expiry, company-deletion rollback/commit and key rotation. - All 81 checks in five current whole files passed with zero skips: OpenAPI 34, read-only settings 2, diagnostic catalog 25, PostgreSQL diagnostics 7 and dependency scheduling 13. PostgreSQL fixtures ran normal migrations before test imports. - Earlier canonical proof at App `cd1c8c6` / Cloud `9f67531` includes real 60-second rate and three 15-second quiet windows, useful work and three revokes. Later paired proof at App `43af514` / Cloud `914eb0f` plus its reviewed candidate includes normal DCR, PostgreSQL metadata, a 144,017-byte signed publication, actual ACK, cap/default-413 controls and a real 15-second quiet window. The relevant producer/signing/wire inputs remain byte-identical in this integration; shared changes are qualified above. These remain source-scoped observations, not a fresh full canonical run on the integrated tree. - All known test owners, process groups and OS sockets closed; an independent final PostgreSQL audit found no remaining test sessions. One dependency-test trace recorded 46 opens and 45 close events. Its cause is unknown; this is not claimed as proof of graceful closure. - Fresh exact-head remote checks passed: 59 successful and 4 skipped contexts, with a complete current 5/5 Greptile review and all seven review threads resolved. Targeted local checks do not constitute a new whole-repository test run. ## Risks App and Cloud must agree on identity, revision, namespace, deadlines and revocation. Partial rollout, unavailable authority or missing ACK must retain awake debt. Shared startup/shutdown and settings changed since the earlier canonical profile, so historical observations carry only within their stated source scope. No deployed Railway/Neon suspension, Cloud alarm, savings or complete hosted Dot/Muse acceptance is claimed. ## Model Used OpenAI `gpt-6.1-sol` produced code and verification, with a Codex coordinator reviewing and integrating the changes. The coordinator's exact runtime model ID and context-window sizes were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandPaperclip
|
909f4d1a33 |
fix(native): recover exact historical provider pack code (#19779)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native sessions retain the provider pack that admitted their saved
state.
> - An image update can replace that pack's executable, proxy, or
contract.
> - Older sessions can predate the durable three-file launch artifact
receipt.
> - The current image's archive cannot satisfy a different pinned
manifest.
> - This pull request obtains the exact public code from its recorded
release image.
> - Full source, archive, and installed-file checks still control
provider admission.
## Linked Issues or Issue Description
Refs #19558, #19622, #19755.
A saved OpenCode session can fail with
`runner_remote_provider_archive_invalid` after an image update. Its
original provider state and checkpoint remain valid, but its pinned pack
differs from the controller's pack. The old manifest can name a release
that still has the exact code in GHCR. Matching the OpenCode version
does not prove matching proxy or contract bytes.
## What Changed
- Resolve a pinned historical pack from the recorded source-SHA image in
the public Paperclip GHCR repository. Verify the platform manifest, raw
config, source identity, dedicated code layer, selected pack manifest,
archive contents, paths, and hashes.
- Keep a private temporary cache of verified public code. Rebuild it on
an empty host. Bound each source to 4 GiB and three minutes, with two
active imports or cache verifications, 32 distinct queued sources, and
disk-headroom checks. Share warm-cache verification by manifest while
each caller keeps its own cancellation.
- Select this verified archive through the existing uploader. Preserve
installed-pack and preinstalled-pack checks. Refuse unavailable
historical transport before clearing the target. Keep the current-source
identity guard from #19755. Refuse an oversized layer argument before
any target effect.
- Attach the source wait to the actual per-turn startup. Stop, lease
loss, and startup completion cancel it. Keep existing Runner admission
and turn limits.
- Document the registry retention requirement and the distinction
between public code recovery and encrypted provider-state backup.
## Verification
- Genuine baseline RED: the real uploader rejects the selected old pack
before any remote effect. Treatment installs the exact old code through
that uploader.
- Current head: all 19 source/integrity/cache/deadline controls, all 45
archive controls, and 29 actual-caller and manifest/startup controls
pass (93 total). Four review regressions fail against the prior head and
pass with the fixes. The server and colocated test TypeScript graph
passes. Module-boundary and whitespace checks pass.
- Actual public GHCR cold reconstruction: 1,054,587,008 compressed code
bytes, 115.4 s, about 248 MiB maximum owner RSS. All 114 file layers and
the overlay passed source validation.
- Linux Node 24.22.2 fixture: the real historical pack passes the normal
uploader, full installed-file verifier, and pinned OpenCode version
check. It used one transfer, no application startup, no network, and no
provider calls. Revision12b0d2ba passed in 220.2 s with about 282 MiB
owner RSS. The earlier prototype passed in 108.5 s; both measurements
are retained. The three-minute source limit and the fixture's
four-minute limit were unchanged. The later review fixes add shared
cache verification, hash cancellation, and an early size guard. They do
not change extraction or the accepted Linux fixture input.
- Earlier fixture failures are retained: unsafe hardlink classification
was corrected with confined-link controls; a cached image lacked the
required zstd CLI; Docker Desktop bind storage caused GNU tar
directory-identity errors. The passing fixture uses the official Debian
zstd binary and an owned Linux tmpfs target. Production extraction code
is unchanged.
- Fresh Greptile review is 5/5 on
|
||
|
DottaandClaude Opus 5.5
|
7475577cdb |
fix(daytona): keep a runner sandbox alive through the controller reconnect grace (#19754)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote agent run executes in a Daytona sandbox and keeps working while the Paperclip server restarts > - The server reattaches that runner within a 7-minute reconnect grace > - Daytona stops an idle sandbox after its `autoStopInterval`, and it counts only SDK and lifecycle calls as activity > - With the server down, nothing touches the sandbox, so a 5-minute auto-stop kills the runner before the server returns > - This pull request makes the host tell the provider its reconnect grace, so Daytona never auto-stops a runner sandbox inside that grace > - The benefit is that a controller outage of up to 7 minutes no longer turns a healthy remote run into a blocked task ## Linked Issues or Issue Description Refs #19199 (the same floor for the warm runner idle window). Refs #19741 (follow-up delivery through restarts; no overlap). **What happened?** A restart-chaos wave on Daytona killed the server for 4m51s while a per-turn remote run was active. The environment used `autoStopInterval: 5`. The last sandbox activity was at 06:09:16. The server came back at 06:14:19 and then crashed again. At 06:15:21 recovery called the sandbox. The plugin found it not started and started it again. The runner identity marker was still on disk, but its process was gone: `native.remote_runner_recovery_failed {stage: process_verification, reason: process_not_running}`, then `runner_remote_process_identity_unavailable`. The task went from done to blocked. An earlier wave showed the same failure after a 4m45s outage. **Expected behavior** A sandbox that holds a live remote runner survives any controller outage up to the reconnect grace. The server reattaches the runner and the run continues. **Steps to reproduce** 1. Create a Daytona environment with `autoStopInterval: 5` and a `paperclip_runner` agent. 2. Start a remote run and kill the server for more than 5 minutes (less than 7). 3. Restart the server. Recovery reports `process_not_running` and the task becomes blocked. ## What Changed - `server`: add `resolveRunnerReconnectGraceMsForRun`. It returns `REMOTE_RUNNER_RECONNECT_GRACE_MS` for `paperclip_runner` on a sandbox environment. - `server`: forward `runnerReconnectGraceMs` to the provider worker on lease acquisition and on the recovery reattach resume. It uses the same provider-only path as `warmRunnerIdleTimeoutMs`. It is not lease identity, lease metadata, or saved environment config. - Daytona plugin: the effective auto-stop is at least the largest host floor (warm idle window or reconnect grace) plus 5 minutes. With the current grace this is 12 minutes. The plugin applies it at creation and raises an older sandbox's shorter interval on resume (bounded, best effort). - Daytona plugin: keep the forwarded grace out of the normalized config. Update the `autoStopInterval` manifest description. - `doc/execution-semantics.md`: document the reconnect-grace floor next to the warm floor. ## Verification - Red/green Daytona plugin tests: `cd packages/plugins/sandbox-providers/daytona && npx vitest run --config vitest.config.ts`. The new cases failed before the fix (for example `expected 120000 to be >= 720000`, and no `setAutostopInterval` call on resume). All 605 tests pass after the fix. - Server tests: `cd server && npx vitest run src/__tests__/environment-run-orchestrator.test.ts src/__tests__/environment-runtime.test.ts`. 247 pass. They prove acquisition forwards the grace only for `paperclip_runner` on a sandbox, that the reattach resume forwards it, and that it does not reach lease metadata or saved config. - `pnpm typecheck` for the plugin and `tsc --noEmit` for the server are clean. - `pnpm verify` asks for broad CI coverage because a doc changed. Required CI supplies it. - Evidence from the kept wave database: lease metadata shows `autoStopInterval: 5`, `reuseLease: false`, `runnerLifecycleMode: per_turn`. The run had no events between 06:09:16 and 06:14:19. ## Risks - Cost: a runner sandbox that leaks (no cleanup) now auto-stops after 12 minutes instead of a shorter configured value. Cleanup still stops or destroys released sandboxes. Explicit `0` and longer values are unchanged. Auto-archive and auto-delete are unchanged. - I did not inspect the sandbox state in Daytona directly. Cleanup destroyed it before this investigation. The proof is the timing, the lease config, and the `process_not_running` result after a restart that kept the disk. - No live Daytona replay ran. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edit, code execution) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
e2c7d81067 |
fix(server): keep reply recovery off web startup critical path (#19769)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server restores retained native replies during startup. > - Fair reply recovery can admit more due work while an earlier group remains active. > - Waiting for that whole pass can hold web readiness on unrelated retained replies. > - This pull request tracks reply recovery with the existing background startup owner. > - The server can become ready while idle and shutdown still account for that work. ## Linked Issues or Issue Description Refs: #19726. **What happened?** The startup path awaited the native response worker's first pass. That pass can remain active while fair dispatch admits later replies. A controlled `startServer` regression stays in recovery until the held reply pass finishes. A deployed backlog supported investigating this boundary, but no captured runtime stack proves it caused the observed startup delay. **Expected behavior** Retained reply recovery must keep its authority checks, retry deadlines, and fair dispatch. Its completion must not be a condition for unrelated web readiness. Active recovery must still block idle and be joined at shutdown. **Steps to reproduce** Run the startup regression with a held native response worker promise. Before this change, `startServer` does not complete. After this change, startup reports ready, idle work remains active, and shutdown waits until the promise settles. ## What Changed - Track the initial native response pass with the existing heartbeat scheduler work owner. - Extend the actual startup wiring test to cover readiness, idle accounting, worker stop, and shutdown join. - Document the response recovery boundary for idle and readiness. ## Verification - Mandatory agent preflight passed on macOS arm64 with Node 26.4.0. - True RED: the new startup case failed because `startServer` had not completed while reply work was held. The same case passed after the change. - Final head `279e97587b` passed the full startup test file: 123/123 tests. - The unchanged real PostgreSQL later-page case passed. It holds 100 old publication rows, publishes the independent 101st reply once while the first pass remains pending, then verifies completion and no recurring idle SQL. - Final-head CI passed: 60 successful checks and 3 skipped checks, including `ci / verify`. Fresh Greptile is 5/5 with no open threads. - Server TypeScript check passed: `node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit`. - Initial test-launch and missing generated SDK setup failures were resolved before the regression ran. They are not RED or pass evidence. - No deployment or live recovery action was performed for this PR. Focused commands, from `server/`: ```sh node ../node_modules/vitest/vitest.mjs run src/__tests__/server-startup-feedback-export.test.ts --maxWorkers=1 --no-file-parallelism node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-response-publication-retry.test.ts -t 'publishes a later-page reply while older publication groups remain contended' --maxWorkers=1 --no-file-parallelism --hookTimeout 90000 ``` ## Risks Web readiness can complete before retained reply recovery finishes. The existing scheduler still owns errors, retries, scope checks, and delivery authority. The existing idle tracker and shutdown join retain in-flight work. Other startup recovery gates stay in place. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, and code execution. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
beb3e55072 |
Continue a released Runner after confirmed graceful eviction (#19760)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core can release a completed Runner while its host retains the session. > - A host can later retire that Runner under its retention policy. > - Graceful retirement saves a suspended session before physical shutdown. > - A cached Core owner still expects the old physical task and rejects a new allocation. > - This pull request retires that dead local controller after verified stop. > - The benefit is ordinary session continuation for Codex and qualified Claude. ## Linked Issues or Issue Description Refs #19611, #19683, #19196, #19733 and #19157. After a host gracefully evicts a released task, the next run can receive a new allocation. A cached controller rejects it with `native_session_released_owner_scope_mismatch`. The detached-controller path can already recover the suspended session. ## What Changed - Require the exact source, old task and current account/workspace scope before releasing a cached controller. - Use the existing physical-stop receipt checks. Detach only the local controller, then use the same suspended-state admission as the detached path. - Preserve the current run's instruction reservation. Retain a failed detach for cleanup. - Add actual Core, PostgreSQL and native Runner tests for Codex and qualified Claude through ACPX. Route them once through the native CI lane. ## Verification - The original real-entry Codex case failed at the cached-task scope check after native retirement, confirmed process exit and a recorded stop receipt. - The current head's native CI lane passed all 16 eviction cases for Codex and qualified Claude. These check cached and detached owners, current instruction preparation, reused-allocation inspection, missing stop/state, foreign identity and non-suspended state. They verify the same provider session, two ordinary turns, one accepted source result and no replacement submit on refusal. - Local checks passed the composition matrix, 12 targeted executor controls, 43 CI lane-selection checks and server TypeScript. All eight refusal cases now assert the expected error and whether the checkpoint request or host read was reached. This addresses the first standard review's test finding. - Commands: `pnpm exec vitest run src/services/native-runtime/plugin-runner-eviction.integration.test.ts`, focused `native-session-executor.test.ts` controls, `node ../node_modules/typescript/bin/tsc --noEmit -p tsconfig.json`, and `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/42349579107) passed on head `9a25709ef43365b1f1849413b935785838eca19b`. [Standard review](https://github.com/paperclipai/paperclip/pull/19760#issuecomment-6703629255) is 5/5 on this head, with no unresolved findings. Whitespace and added-line secret/PII checks passed. ## Risks - Controlled provider processes and an authenticated local host fixture model the Launcher edge. This does not qualify live vendor transcript storage, mounts or a deployment. - This covers graceful eviction before allocation or before stopped-task inspection during `describe`. Loss after that inspection or an ambiguous claim remains held for reconciliation. No unavailable reply proves physical stop. - Missing or invalid suspended state still refuses continuation. The deterministic refusal rule from #19733 remains unchanged. Abrupt death without a suspended checkpoint stays with the separate #19157 work. - No new capability, mode, schema, dependency, deployment or activation is added. The existing draft #19157 and dependent branches remain unchanged. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with repository tools and code execution. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
3895195f4b |
fix(native): report a retained provider pack the controller does not ship as incompatible (#19755)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote OpenCode runs stage a controller-owned provider pack into the sandbox before the provider starts > - Since #19558, an ordinary OpenCode continuation retains the pack that an earlier release admitted for its saved provider session > - After a controller release changes the pack, the controller no longer ships those bytes, and the continuation must stay blocked > - The configured-pack upload path did not detect that case: it read the build-time archive receipt for the old digest and failed with `runner_remote_provider_archive_invalid`, a generic, retryable error that suggests corrupt image files > - This pull request compares the configured pack's identity with the selected identity before any upload and fails as `runner_remote_provider_artifact_incompatible` > - It also puts a fixed reason in each real archive integrity error > - The benefit is a correct, permanent classification that matches the bundled-image path, no wasted retries, and a diagnosable message for real archive corruption ## Linked Issues or Issue Description Refs #19558, #19743 **What happened?** On a hosted deployment, two remote OpenCode runs of a review agent failed before the provider process started. The error was `runner_remote_provider_archive_invalid` (classified as a generic `native_session_interrupted`). Both runs continued an issue whose provider session was admitted by an earlier release. In both runs, the run log shows `provider_pack.compatibility`, `provider_pack.verify`, and `provider_pack.verify_preinstalled` fail, then `provider_pack.archive.verify` fails after 4–6 ms. Fresh runs of the same agent on the same controller passed pack preparation with the current pack. The cause is not a cache race. The archives are created at image build time (`Dockerfile` `runner-provider-pack` stage) and the server never rewrites them. The retained authority selects the old pack digest. `readProviderPackLayers` then compares the receipt's `providerPackDigest` (current release) with that old digest and throws `runner_remote_provider_archive_invalid`. Directory staging (no archive) instead clears the staged remote pack, uploads the different pack, and fails only at the final verification. **Expected behavior** A continuation pinned to a pack that the controller does not ship fails as `runner_remote_provider_artifact_incompatible`, as the bundled-image path already does ("the image provider pack did not match the bundled release identity"). It does not clear or upload anything. Real archive corruption keeps `runner_remote_provider_archive_invalid` with a fixed reason that the run report shows. **Steps to reproduce** 1. Build a provider pack and its archives for digest A. 2. Call `uploadVerifiedProviderPack` with that pack and a selected (retained) digest B. 3. Before this change: the call fails with `runner_remote_provider_archive_invalid` (v1 and v2 archives), or resolves after it clears and re-uploads the remote pack (directory source). ## What Changed - `uploadVerifiedProviderPack` takes the configured pack's `sourceDigest`. If it is not the selected `expectedDigest`, the call fails with `runner_remote_provider_artifact_incompatible: selected provider pack <12 hex> is not the controller's pack <12 hex>; ...`. The check runs before the remote clear, the archive read, the directory hash and any upload. - The executor passes the configured pack's identity (`readRemoteProviderPackIdentity(root, false)`, the same reader that selects the current pack). - Archive integrity errors keep the `runner_remote_provider_archive_invalid` prefix and add a fixed reason: `receipt_unreadable`, `receipt_identity_mismatch`, `receipt_invalid`, `layer_invalid`, `layers_unreadable`, `archive_unreadable`, `archive_not_file`, `size_mismatch` or `sha256_mismatch`. The run report keeps only the outer message, so the reason must be in the message. Filesystem causes stay on `cause` and do not put private paths in the message. - The v1 reader now uses the shared `verifyArchive` helper. - `doc/execution-semantics.md` documents the rule. ## Verification - Red before the fix (production files unchanged, new tests only): `npx vitest run src/services/native-runtime/provider-pack-archive.test.ts` gave 5 failed / 38 passed. The v1 and v2 cases failed with exactly `runner_remote_provider_archive_invalid`, which is the hosted error. The directory case resolved after it cleared and uploaded the remote pack. The reason tests got a bare `runner_remote_provider_archive_invalid`. - Green after the fix: the same file gave 43 passed. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1245 passed. This includes the #19558 continuation cases. - `remote-provider-pack`, `remote-provider-pack-authority` and `remote-provider-pack-verification` tests: 69 passed. - `pnpm --filter @paperclipai/server typecheck`: all layers passed. - Review follow-up (`c8156841a9`): an unreadable numbered layer now reports `archive_unreadable`. The layered-corruption test deletes `0.tar.zst` and checks for that reason before any transfer. The archive file gave 43 passed. - Hosted evidence came from read-only run, event and issue-run reads only. No hosted retries, recovery or deploys. ## Risks - This change does not unblock an affected session. It makes the failure honest and permanent: `runner_remote_provider_artifact_incompatible` is already a permanent, pre-session failure code, so the run is not retried with the same result. To recover, the session needs its original artifacts (the captured provider-artifact path, which #19743 repairs for new admissions) or an authorized continuity break. - Low risk for other paths. Fresh launches select the configured pack's own identity, so the digests are equal. Restored provider artifacts and compatible installed packs return before this upload path. - Callers that match `runner_remote_provider_archive_invalid` with a substring still match. No code compares the exact message. - Not tested end to end against a hosted sandbox. The executor-level wiring of `sourceDigest` has typecheck coverage but no dedicated executor test. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic tool use with shell, code editing, test execution and read-only authenticated browser diagnostics. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |