PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-12 14:19:00 +01:00

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	6fd46c0025	perf(budgets): judge Paperclip's share of restart recovery; report the provider's session load (#20183 ) ## Thinking Path - Speed budgets protect the time to first visible agent activity, and restart recovery is one of the judged metrics (`native-speed-budgets.ts`, wave grader). - In the B1 concurrency-6 chaos wave (`w-20261109T045320-s9807`), Codex `restart_recovery` was p50 8.6 s and p95 11.2 s, against a budget of 6.5 s and 8.0 s. At concurrency 4 (K13) it was 3.8 s and 5.0 s. - The preparation spans on a copy of the kept database break it down: - Paperclip's share (ready → claim → prepare → runtime stage → runner launch) is a steady 1.2–2.0 s. - `provider.session.load`, the Codex app-server boot plus `thread/resume` of long saved sessions, takes 3.0–6.3 s, 65–75% of each recovery. - Resumes run concurrently, and the worktree lock and the startup reconciler are not on the path. - Judging the full latency therefore mixes our code with provider and session-size effects. Budget the part we control, so a regression in our code shows up regardless of session size or concurrency, and report the rest. ## Linked Issues or Issue Description What happened? `restart_recovery` breached its budget whenever many long provider sessions resumed at once, although Paperclip's own recovery work stayed within about 2 s. The breach did not point at anything Paperclip could fix, and a real regression in our code could hide inside the provider-load variance. Steps to reproduce 1. Run a local chaos wave at concurrency 6 with long-running Codex tasks and K1/K2/K4 faults (for example B1, seed 9807). 2. Compare `restart_recovery` in `speed.json` with `provider.session.load` in the recovered runs' run logs. Expected behavior The budget judges Paperclip's share of a recovery. The provider's session load and the full latency are reported but not judged. ## What Changed - `server/src/services/native-runtime/native-speed-budgets.ts`: - New budgeted metric `restart_recovery_control_plane`: from `ready` to the start of `provider.session.load`, or to the next provider event when nothing reloads (for example a reattached runner). - Its budget is 2.5 s / 4.0 s for claude, codex and opencode. Measured: 1.2–2.0 s per run at concurrency 6. - The full `restart_recovery` stays in the table for reporting. - Breach payloads carry `restartTotalMs` and `providerLoadMs`. - `server/src/services/native-runtime/native-run-trace.ts`: - The per-run check records when `provider.session.load` starts, keeps the recovery measurement, and judges it when the run finishes. - It judges after the deferred diagnostics drain, so a provider span committed after the provider event still counts. - It emits `restart_recovery_control_plane` breaches only. - `tests/restart-chaos/ttft-baseline-stats.ts`: each recovery sample gains `controlPlaneMs` and `providerLoadMs`. The provider-load start is the span event's write time minus its duration. - `tests/restart-chaos/wave-speed.ts`: - Three restart rows per adapter: `restart_recovery` (report-only), `restart_recovery_control_plane` (budgeted) and `restart_provider_load` (report-only). - All three have outage time removed. - `tests/restart-chaos/ttft-baseline.ts`: the recovery table adds a "Paperclip share" column. - Docs: `doc/run-log-events.md` (metric and payload) and the speed-budget table in `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`. ## Verification - `server`: `npx vitest run src/services/native-runtime/native-speed-budgets.test.ts`, 18 passed. Two new tests: - a recovery dominated by provider load does not breach; - a slow Paperclip share breaches with `restartTotalMs` and `providerLoadMs`, even when the provider span is committed after the provider event. The existing recovery tests moved to the new metric. All six were red before the trace change. - `tests/restart-chaos`: `npx vitest run --config tests/restart-chaos/vitest.config.ts wave-speed.test.ts ttft-baseline-stats.test.ts`, 29 passed. New tests cover the sample split and the grader rows (a judged share, a report-only provider load, zero breaches). - Server typecheck: no errors in the touched files. ## Risks - Lower sensitivity to provider-side slowdowns. A provider regression in session load no longer breaches. It stays visible in the report-only rows and in breach payloads. - Runs with no `provider.session.load` span (reattach, or an adapter that doesn't emit it) are judged on the full latency against the control-plane budget. That's conservative. - Run-log consumers that filter on `metric: "restart_recovery"` see `restart_recovery_control_plane` from now on. The doc notes the change date. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 01:22:28 -06:00
DottaandClaude Opus 5.5	95d476c57b	docs(plans): record the measured seed experiment for per-turn uploads (#20139 ) ## Thinking Path - #20100 proposed sending only the changed files on per-turn follow-ups, with the repository seed held inside Daytona in a snapshot or a volume. It asked for one measured experiment before building. - Snapshot and volume operations were blocked by our Daytona key's permissions. A restricted key with those permissions now exists, so the experiment has run. - This PR records the numbers and the decision in the design doc. ## Linked Issues or Issue Description Refs #20100 What happened? The #20100 design's seed-location choice was unmeasured: snapshot build time and volume read speed were both open questions. Steps to reproduce 1. Run the SDK timing of direct upload, volume extract, and snapshot build and create on the paperclip seed (58.7 MB gzip) in Daytona `us`, 2 vCPU. Expected behavior The design doc states the measured winner, the per-turn saving, the one-time cost, and the work needed to build it. ## What Changed - `doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md`: - The status now reads: measured; snapshot chosen; not built. - New "Experiment results" table: - direct upload: 6.4 s per turn; - volume: 6.8 s per turn, which rules it out; - snapshot: about 2–3.5 s per turn, after a 177 s build per seed generation. - Findings: fork is unsupported, compression is exhausted, and the snapshot builder runs as `daytona` and does not extract archives. - New "Work to build the snapshot seed" list: lifecycle, sandbox creation, verification and fallback, expiry and cleanup, and rebuild policy. ## Verification Docs only. Every Daytona resource the experiment created was deleted, and an account-wide listing confirmed none remained. ## Risks None at runtime. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 18:21:08 -06:00
DottaandClaude Opus 5.5	2a16be36ab	docs(plans): per-turn follow-ups without re-uploading the repository (#20100 ) ## Thinking Path - Per-turn remote sandboxes stay the default while warm reuse (#20048) is on hold. - On the paperclip clone, a per-turn follow-up spends about 13 s in `workspace_sync.transfer`, though its changes pack in 0.5 s: every fresh sandbox receives the whole 70 MB (gzip) seed again, at about 5 MB/s from the controller. - The coordinator asked for a design that uploads only changed files into a fresh sandbox, with a full-upload fallback, measured against the spans. It also asked to build it only if the implementation is bounded and safe. ## Linked Issues or Issue Description Refs #20048 What happened? Per-turn follow-ups on large repositories wait about 13 s re-uploading the full workspace seed even when almost nothing changed (pack 0.54 s, transfer 13.3 s, time to first token about 31 s). Steps to reproduce 1. Run `tests/restart-chaos/ttft-baseline.ts --env daytona --workspace-repo <paperclip clone>` with per-turn sandboxes. 2. Compare `workspace_sync.pack` and `workspace_sync.transfer` on the follow-up turn. Expected behavior A follow-up uploads only its overlay into a sandbox that already holds the seed, falling back to the full upload when unsure. ## What Changed - `doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md`. It contains: - the measured span table and the size analysis (220 MB archive, 70 MB gzip, about 5 MB/s); - the expected saving: about 12–13 s per follow-up, time to first token about 31 s → about 18 s; - a design keyed by seed generation with a content-hashed manifest, using a Daytona snapshot (preferred) or a Daytona volume to hold the seed; - the overlay-only turn flow, with a verified fallback; - one bounded experiment to choose the seed location before building. - Not built yet: both seed locations need provider capabilities (snapshot build time, volume read speed) that must be measured first, so the implementation isn't bounded until the experiment picks one. ## Verification Docs only. ## Risks None at runtime. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 14:21:24 -06:00
DottaandClaude Opus 5.5	c02fd8d427	docs(plans): salvage-then-retire design for kept restore-source and done-task warm sandboxes (#20087 ) ## Thinking Path - Paperclip keeps some remote sandboxes after their run: warm ones for reuse, and failed runs' stopped sandboxes as workspace-restore sources. - In the warm Daytona chaos waves for #20048, the real leaks were restore sources that nothing retires once a user Retry closes their recovery actions. Separately, a done task's suspended warm sandbox is only destroyed when its environment is deleted. - Retiring them early needs a provider-I/O salvage step: compare against the durable save, export anything unsaved to the issue, then destroy. That spans recovery, storage, attachments and teardown. The coordinator asked for a design note so it can be its own reviewed task instead of a rushed patch. ## Linked Issues or Issue Description Refs #20048, #20085, #20086 What happened? Closed restore sources and done-task warm sandboxes stay suspended until their environment is deleted, costing storage. No path compares them with the last durable save or exports their unsaved files before retirement. Steps to reproduce 1. A run fails; its sandbox is kept as a `retain_on_failure` restore source. 2. The user clicks Retry; a fresh run succeeds and the task is done. 3. The restore-source sandbox is never retired until the environment is deleted. Expected behavior After a clean compare, or a confirmed export of anything unsaved to the issue, the sandbox is destroyed. If salvage fails, it stays suspended and is retried. ## What Changed - `doc/plans/2026-11-08-restore-source-salvage-and-warm-retirement.md`: a design for one owned kind (`environment_kept_sandbox_retirement`) on the shared reconciler. It covers listing rules for both cases (24 h default retention for done-task warm sandboxes), a three-stage salvage state machine (compare, export, retire) with a provider-I/O step and receipt table, failure handling that never destroys without proof, and the test plan. ## Verification Docs only; no code or tests changed. ## Risks None at runtime. Additive documentation. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 14:13:05 -06:00
DottaandClaude Opus 5.5	b32ef9f111	perf(budgets): judge time to first visible agent activity; keep time to first text as a reported metric (#20077 ) ## Thinking Path - The native speed budgets (plan invariant 8) judge how fast an agent responds. They judged `task.time_to_first_token`: time to the first deliverable text. - In the clean local restart storm (w-20261108T153612-s9401), Claude breached its TTFT budget (cold p50/p95 17.4/34.7 s against 16.5/30). The per-run spans showed this was not a control-plane wait: in all 49 cold runs (and 76% of warm), Claude started a tool call before writing any text. Its first visible activity came at 7.4 s p50 from run creation; first text came at 19.6 s. Codex never calls a tool first. - So first text charges model tool work as latency for tool-first adapters. What the user experiences as "the agent responded" is the first visible activity: a text delta, a tool-call start, or a thinking indicator. - The coordinator decided: budgets judge time to first visible agent activity from run creation, with the budget numbers unchanged; first text stays a reported metric with no budget for tool-first adapters. ## Linked Issues or Issue Description What happened? Local storm s9401: the Claude TTFT budget breached on every aggregate, while the control-plane steps (queue 0.05 s, prepare 0.3 s, runner startup ~4 s, first provider event 1.3–1.8 s) were well within budget. The rest of the first-text wait was Claude streaming tool input (about 5.5 s per call) and running tools before it wrote text. Steps to reproduce 1. Run a Claude task whose first action is a tool call (most real tasks). 2. Read the run log: `provider.time_to_first_agent_event` ends at the tool call; `task.time_to_first_token` ends many seconds later at the first text. 3. The speed budget judges the later one and reports a breach. Expected behavior The budget judges the first visible agent activity from run creation (`task.time_to_first_agent_activity`). Time to first text is still recorded and reported. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: when the first meaningful agent event commits (the existing `firstMeaningfulAgentEventKind`: text delta, tool-call start, or reasoning), record `task.time_to_first_agent_activity`. It uses the same detached anchor resolution as TTFT (accepted comment, answered question, durable wake request, or run creation), so it adds nothing to the turn path. - `server/src/services/native-runtime/native-speed-budgets.ts`: the metric is now named `time_to_first_activity` (was `ttft`), with identical budget numbers. The type comment documents the definition and why first text is unbudgeted. - `server/src/services/native-runtime/native-run-trace.ts`: the budget check judges `task.time_to_first_agent_activity` from run creation, keeping the existing input-wait handling. `task.time_to_first_token` is no longer budgeted. - `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`: report first activity from run creation (the budgeted value) next to TTFT. - Docs: `doc/observability.md` (new span) and `doc/plans/2026-11-07-ttft-baseline.md` (metric table). - Tests: - A tool-first turn records first activity at the tool call (5 s) while TTFT records first text (16 s). - A tool-first trace with slow first text but fast first activity is within budget. - The existing budget tests now judge the activity span. Observability path (AGENTS.md section 5.7): the new run-log span uses only the existing `eventKind` and `anchor` attributes. Its name survives run-log redaction intact; checked with `redactEventPayload`. The operator OTLP gate is unchanged. It is not Telemetry. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts`: 44/44 pass. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "first visible activity\|first assistant text once\|first-token timing"`: 7/7 pass. - `npx vitest run --root tests/restart-chaos`: 159/159 pass; `tsc` on `tests/restart-chaos` is clean. ## Risks - A metric rename. Consumers of `run.performance.budget_exceeded` see `metric: "time_to_first_activity"` instead of `"ttft"`. In-repo there are none besides the tests; Phase 3's chaos speed grade switches to the new span. - A looser budget for slow first text. Intentional: first text is reported, not judged. A regression in control-plane latency still shows up in first activity. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 12:32:42 -06:00
DottaandClaude Opus 5.5	e8a3f1f6f2	feat(recovery): prove a replaced container's records gone through a fenced controller epoch (#20054 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, the control plane settles in-progress records (leases, claims, intents) only once it proves their owner process is gone > - The proof compares host and PID space, so a stamp from a replaced container or a rebooted machine stays unprovable forever, and its records never settle > - Self-hosted users on Docker, Kubernetes and Fly hit this on every redeploy; a Railway deployment with an exclusive volume is covered by `deployment_replaced` (#20026) > - This pull request adds a fenced controller epoch: each boot holds a database lease, and an expired boot is fenced so its records can be proven gone > - The benefit is that work stranded by a replaced container is recovered, without keeping an idle database awake ## Linked Issues or Issue Description Follow-up to #20026 and the Phase 2 durable-ownership data. Related: #19965 (owner stamps), #19883 (process space). No duplicate PR was found. What happened? A record stamped by a server process in another container reports `different_process_space`. A dead PID elsewhere is never proof, so the record stays `unprovable` and is never settled. Every redeploy of a self-hosted container leaves such records behind. Expected behavior A record owned by a server boot that was replaced (stopped cleanly, or lost its lease and was fenced) is proven gone and settled exactly once. A live boot's records stay with it. An idle server issues no recurring queries. Steps to reproduce 1. Run Paperclip in a container and start work, so an environment lease is stamped by that boot. 2. Replace the container. The new container has a new PID namespace. 3. The ownership reconciler reports `different_process_space` (unprovable) for the old lease, forever. Paperclip version or commit `master` before this PR. ## What Changed - New `controller_instances` table (migration 0449): boot id, epoch (bigserial), process space, pid, a lease on the database clock, `fenced_at` / `fenced_by_epoch`, `stopped_at`. - `controller-instance-lease.ts`, registered with the shared delivery-work coordinator (`controller-instance-lease` queue, runs during drain): - Holds work → renews; idle → silent. One upsert creates the row on first use and extends the lease only if the boot holds work: a stamp within the last 90 s, or a stamp whose ownership write may still be pending (held until the server's tracked work settles, capped at 10 minutes; a renewal that sees some other held record does not clear it, so a slow ownership write cannot be outlived), or an in-progress record held by its boot id. Those records are active environment leases, in-flight cleanup claims, sandbox creation intents, workspace finalize attempts, agent lifecycle steps, native finalizer and cleanup leases (`lease_holder` bound to the token), open native coordinator claims, and running legacy runs. Renewal runs every 30 s while the boot holds work. When it finds none, the lease lapses and the worker sleeps until the next stamp (`onControllerStamp`). - No idle trailing statements. When the server's last tracked work settles, the worker re-checks holdings at once, at most every 5 s (the rate limit also absorbs the worker's own run). So the last check happens with the work, not 30 s later. - Quiesce, don't crash. Each renewal has a 10 s bound. A failed or timed-out renewal applies an internal task drain, so no new work is dispatched. A recovered renewal resumes. The boot exits only after 60 s of failure (counted from the first failed attempt's start) while it still holds work, or at once when it finds itself fenced. - Fence on demand, by a newer epoch. For a stamp from another host or PID space, the prover reads that boot's row: - `stopped` or `fenced`: gone. - live lease: alive. - otherwise, the boot's last sign of life is the later of its lease expiry and the record's last write plus the TTL. Within 30 s of that, the owner stays `alive` (`controller_lease_grace`), so the settle watch keeps rechecking. After that, the prover fences the boot, but only with a lease row of its own (created on demand) and only an older epoch. - A graceful shutdown records `stopped_at`. - A reconciliation pass that settles nothing (`record: false` or `settle: false`) never fences and never creates the boot's row. - `owner-proof.ts`: these proofs apply only to server-held records. A stamp with a recorded process group stays unprovable, because agent processes can outlive their server. The reconciler passes each record's `updatedAt` to the proof. - `task-admission.ts`: - the quiesce drain never replaces an operator's drain, and has no idle owner, so HTTP stays open; - a `subscribeIdleWorkSettled` hook fires when the last tracked work finishes. - Off on a Railway deployment with an exclusive volume (it has its own replacement proof). `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=on\|off` overrides this. A stateless Railway service keeps the lease. A server without its own lease row never fences. - `maintenance-server-routes.test.ts`: in the post-run quiet window, the lease may check once more that the run's records were released. The test now excludes those statements, then asserts that the lease issues nothing in a further window. - Docs: `doc/execution-semantics.md`; the Phase 2 data doc records the two known gaps. ## Verification - `controller-instance-lease.test.ts` (15 cases): - a replaced container with all Railway variables empty is fenced and settled exactly once over two passes; - a live lease is alive; - an owner in grace, and work written after an idle lapse, stay alive and watched; - an older epoch never fences a newer one; - no fencing without its own lease row (off or unregistered), and an idle prover creates its row on demand; - a stopped boot is gone; - process-group stamps stay unprovable; - idle issues zero statements, and one upsert renews only while records are held; - a 45 s outage: quiesced, no exit, resumes; - a renewal that never answers: bounded, and the boot exits on time; - a 70 s outage: exits at 60–65 s, and the successor recovers the work exactly once; - fenced: stops; - the settle re-check; - graceful stop; off on a volume-exclusive Railway deployment, on for a stateless one, and the on/off override; - a running legacy run keeps the lease live. - Red, with the proof's instance check removed: the replaced-container, live, stopped and successor cases fail. - Wider: task-drain, idle-sleep, idle-local-work, owner-proof and ownership suites, plus the maintenance quiet-window tests. Server low-memory typecheck is clean. ## Risks - Clock skew. The proof compares a record's `updatedAt` (often the writer's clock) with the lease (database clock). The 30 s grace absorbs ordinary skew. - A new exit path exists. A server whose database is unreachable for more than 60 s while it holds work now exits, after first quiescing. This is the intended safety backstop. - Bounded trailing activity. A lease that holds work issues one statement every 30 s, plus at most one check when its work settles. An idle server issues none. - Boots from before this release have no row, and their records stay unprovable. The board "mark owner gone" action for those follows in a separate PR. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 11:06:32 -06:00
DottaandClaude Opus 5.5	c966334ec3	fix(recovery): settle a run-less lease once its stamped holder is proven gone (#20026 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane recovers in-progress records after a server restart through a shared ownership path > - Run-less sandbox leases (login, probe, harness verification) were released only by a five-minute idle sweep > - That sweep releases by age: it leaves a dead holder's sandbox for minutes and takes a live login session's sandbox > - This pull request settles a run-less lease only when its stamped holder is proven gone > - The benefit is no stuck sandbox after a restart, and no live session loses its sandbox ## Linked Issues or Issue Description Related: #19965 (named the owner of login/probe leases), #19964 (the run-lease settle this mirrors). No duplicate PR was found. What happened? The final Daytona chaos wave `w-20261108T083036-s9401` recorded I2 `environment_login_probe_lease` (`pid_dead`) on 2 records. These were the codex agent's harness-verification leases. A restart killed their holder. The leases stayed `active` until the five-minute idle sweep released them. The same sweep releases any run-less lease after five idle minutes, also when its holder is alive. Expected behavior A run-less lease is released as soon as its stamped holder is proven gone. A lease whose holder is alive is never released, however long it is idle. Steps to reproduce 1. Start a Daytona wave: `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image <image>` (K1/K4 faults). 2. Let a fault kill the server while an agent's harness verification holds its sandbox lease. 3. See the reconciler report I2 `environment_login_probe_lease` until the idle sweep releases it. Paperclip version or commit `master` at `8475826d2d`. ## What Changed - New `settleEnvironmentLoginProbeLease`, registered on `environment_login_probe_lease`. It runs only for a stamped holder that is proven gone, and compares and swaps on that exact stamp. - It moves the lease to `pending_cleanup` with the recovered-orphan reason, so the cleanup sweep destroys the sandbox in its next tick. - It leaves the lease alone when a legacy shutdown fence applies, or when another live lease holds the same provider resource. - The idle sweep for run-less leases now covers only legacy rows saved before holders were stamped (no `leaseController` key). Stamped rows are never released by age. - `holderUnchanged` is now exported from the run-lease settle so both settles share it. - The controller stamp (`leaseController`, and every other `ControllerProcess` stamp) now records the platform deployment when the platform names one (Railway: project, environment, service, deployment, volume). `controllerStamp` passes it to `proveOwnerGone`. A container redeploy changes the PID space, and the old stamp would read `different_process_space` (unprovable) forever. Now the existing `deployment_replaced` proof shows the old holder gone. This also covers `environment_run_lease_active` and the other kinds built on `controllerStamp`. - `currentDeploymentIdentity` in `owner-proof.ts` now delegates to `currentControllerDeployment`, so there is one parser. - Data doc: added the final Daytona wave, the six Daytona-only crash points (4 pass, 2 not reached, no violations), and the cause of the roughly 450 s codex TTFT. Runs waited behind the agent's harness verification, which restarted after each fault; boot itself took about 32 s. - Restart-chaos README: corrected which Daytona crash points a plain task reaches. ## Verification - Red, before the change: `environment-login-probe-lease-ownership.test.ts` failed 3 of 6. No settle happened, and the idle sweep released 3 live holders' leases. - Green: `pnpm exec vitest run` passes for `environment-login-probe-lease-ownership`, `environment-run-lease-ownership` and `heartbeat-orphaned-active-lease-sweep` (45/45), and for `environment-lease-owner-stamps`, `heartbeat-observed-workspace-preservation`, `ownership-reconciliation-report`, `heartbeat/recovery` and `workspace-recovery-warm-handoff`. - Redeploy: a stamp from a replaced Railway deployment settles with `deployment_replaced`. A stamp from the same deployment in another PID space stays `active`. This was red when `controllerStamp` dropped the deployment. - Live-owner counter-cases: the current process, another live PID, another PID space, and a lease re-stamped after the proof all stay `active`. A read-only pass never settles. - Not re-run live: the Daytona wave and crash points ran before this fix. The next Daytona wave should show no `environment_login_probe_lease` I2. ## Risks - A stamped run-less lease whose live holder never releases it, because of a bug in that process, is no longer released after five idle minutes. It is released once that process exits. This is the intended trade: never take a live holder's sandbox. - On an ephemeral server without a platform deployment identity (not Railway), a stamp from a replaced container is still unprovable. Such a run-less lease stays `active` until the provider expires the sandbox, and the reconciler reports it as unprovable. Before this PR, the five-minute sweep released it by age. This is the same open gap as unstamped legacy leases on an ephemeral server: there is no operator action yet. It is listed as a Phase 2 follow-up. - Legacy unstamped run-less leases keep the age fallback. These are rows written before #19965, and no current code writes them. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 06:43:45 -06:00
DottaandClaude Opus 5.5	68d5fb5a6e	feat(chaos): check I6 and I8 per crash point; ready the Daytona points (#19984 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The on-demand crash-point suite (#19946) kills the control plane at named durable write boundaries and checks the plan's invariants > - Its first full run measured I1–I4 and I7, but I5, I6 and I8 were reported as not measured > - #19960 (generation stamping) and #19863 (speed budgets) now record what I6 and I8 need in the run log > - The six Daytona-only points have not run yet, and three of them cannot be reached by a plain task > - This pull request checks I6 and I8 per point, names the I5 gap, and readies the Daytona points for the coming Daytona wave > - The benefit is that every invariant is either asserted or a named gap, and a Daytona run explains any point it cannot reach ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5). Follows #19946 (the suite). Uses #19960 (event generations) and #19863 (speed budgets). Problem - The suite reported I5, I6 and I8 as "not measured". - A Daytona run would report three points as `not_reached` with no reason. - A Daytona run without its key or image failed only after a 3-minute boot. Expected behavior - I6 and I8 are checked at each point. - I5 is a named gap with its reason. - Daytona points say what they need, and a Daytona run checks its inputs before it boots. ## What Changed - `tests/restart-chaos/invariants.ts`: `runLogViolations` checks two invariants from the run log, scoped to the wave's company: - I6: a run that failed with an error that names a stale or earlier-generation event; - I8: each `run.performance.budget_exceeded` event, with the budget evidence (metric, measured, p95, adapter, start kind). Both are keyed by run or event id, so the per-point baseline diff charges each one to the point that produced it. I5 is reported as `notMeasured` with a reason (`I5_GAP`). - `packages/shared/src/crash-point-registry.ts`: an optional `prerequisite` on the three Daytona points a plain task cannot reach. These are the teardown claim (needs a failed release), the reusable destroy (needs a `reuse_by_environment` lease), and stop-and-retain. - `tests/restart-chaos/crash-points.ts`: - `--list` prints each point's `needs:`, and a `not_reached` result names it too; - `--env daytona` refuses to start without `DAYTONA_API_KEY` and an image. - Docs: the README and the plan describe the I5/I6/I8 coverage and the Daytona points. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/crash-points.test.ts`: 12 passed. - `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and `npx tsc --noEmit -p packages/shared` both pass. - I ran `runLogViolations` against a throwaway Postgres 18 with seeded rows, then stopped and deleted it. It returned: - I8 for the budget breach, with its evidence; - I6 for the run of this company that failed on `stale_generation_event`. It excluded the other company's stale run, the `process_lost` run and a non-budget event. - `node scripts/crash-points.mjs --list --env daytona` lists 19 points, with `needs:` on the three. `--env daytona` with no key or image fails at once with the missing names. - No harness run: the harness slot is held for other work. The next suite run exercises the new checks. ## Risks - Test harness only; no product code. - I6 is a proxy: it matches the run's error text and code. A failure that names a generation for another reason would be reported, with its evidence, for a person to judge. - I8 can flag breaches on a loaded host. Each breach carries its measured and budget values, so a load-caused breach is visible as one. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 00:06:01 -06:00
DottaandClaude Opus 5.5	46bf913fb9	docs(plans): record the Pi steer duplicate after a restart and its follow-up (#19982 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can steer a running Codex or Pi turn, and since #19970 a queued follow-up steers automatically > - Restart recovery (#19962) delivers a steered message that the model did not read before the restart, exactly once for Codex > - Pi gives the run log no read evidence for a steer, so after a restart a Pi steer that the model already read can arrive again > - The fix needs changes to the closure-pinned Pi distribution, which waits for the next Pi re-qualification > - This pull request records the limit and the follow-up plan so they are not lost > - The benefit is that the next Pi re-qualification can include the definitive fix ## Linked Issues or Issue Description What is the documentation problem? The Pi steering limit after a restart, and its planned fix, were only in pull request text and session notes. Where? `doc/plans/2026-11-07-pi-steer-restart-limit.md` (new). What should it say? Codex delivers an unread steered message exactly once after a restart. Pi can duplicate a message it already read, and never loses one. The note also gives the reasons, the follow-up plan (option A), its costs, and the rejected options. Refs #19962, Refs #19970 ## What Changed - New `doc/plans/2026-11-07-pi-steer-restart-limit.md`. It covers the limit, why Pi has no read evidence, the follow-up (forward Pi's native user `message_start` through pi-acp, acpx, the sidecar and Rust into a `userMessage` item), its re-qualification costs, and the rejected options. ## Verification - Documentation only. Read the note. The facts match `packages/paperclip-runner/src/steering-delivery.ts`, `patches/pi-acp@0.0.33.patch`, `pi-closure-pins.ts` and `rules/acpx-profile-identity.mdc`. ## Risks - None. Documentation only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 23:51:57 -06:00
DottaandClaude Opus 5.5	051834bdc8	feat(chaos): add the on-demand crash-point suite and its manual CI workflow (#19946 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart between two durable writes leaves an in-progress record owned by a dead process > - Random restart-chaos waves found these 20 ms–1 s windows only by luck > - #19937 added `crashPoint(name)`, and #19938 put it at 20 durable write boundaries > - Nothing yet runs a task, kills the server at a chosen point, restarts it, and checks the plan's invariants > - This pull request adds that on-demand suite, as a local command and as a manual CI workflow (not a required check, per Decision 2) > - The benefit is a repeatable work list: which boundary breaks which invariant, for Phases 2–4 ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5 and Decision 2). Refs #19861 (report-only reconciler used for I1–I7). Stacked on #19938 (call sites), which is stacked on #19937 (helper). Problem The named crash points cannot be exercised end to end. No command arms one point, runs real agent tasks across it, restarts, and grades the result. Expected behavior `node scripts/crash-points.mjs --points <glob> --kinds <id> --adapters <list>` runs one wave per selected point and reports each point as pass, fail (with failing R rules and violated invariants), not reached, or harness error. A `workflow_dispatch` workflow runs the same thing on demand. ## What Changed - Runtime arming. `PAPERCLIP_CRASH_POINT_CONTROL=<file>`, in the shared helper and the runner mirror. Each hit reads `<token> <name>[:<nth>]`, and a new token resets the count. It is test mode only: `NODE_ENV=production` ignores it, and the disarmed path is still one comparison. - `tests/restart-chaos/main.ts --crash-point <a>:<nth>,<b>:<nth>,...` (config, types, ledger): - one booted server and no Vite UI; - per point: take an invariant baseline, launch the point's batch, arm the point, then on rest disarm it and check invariants as violations that are new since the baseline; - every task is graded R1–R6 and T; - `crash-point.json` (v2) holds one entry per point. - `tests/restart-chaos/supervisor.ts`: - every spawn gets the control file; - a SIGKILL whose receipt names the child's pid is restarted like K1: the control file is emptied first, the owned tree is killed, and a replacement starts. A failed tree cleanup fails the fault; - the isolated instance pins `PAPERCLIP_IN_WORKTREE=false` (a `.paperclip/.env` above the checkout otherwise suppressed all scheduling); - `RESTART_CHAOS_HEALTH_TIMEOUT_MS`. - `tests/restart-chaos/invariants.ts`: - the Phase 1 report-only pass with `record: false`: two real-clock passes for I1 and I2, and one pass 15 minutes ahead for I3, I4 and I7; - spool kinds are read from the wave's instance; - an unfinished pass is a harness error; - each violation carries its wake and run state as evidence. - `tests/restart-chaos/crash-points.ts` and `scripts/crash-points.mjs`: - select points by glob, kind and environment, and run them in one wave; - a per-point outcome: pass, fail, not_reached or harness_error; - `SUMMARY.md` with a violation evidence column. - `.github/workflows/crash-points.yml`, manual only, under the paid-workflow contract: - a credential-free `authorize` job; - a `suite` job in `runner-e2e-paid` whose first step reauthorizes; - a frozen install with `--ignore-scripts` and no shared cache; - only the selected adapters' keys; - the workflow is added to both security-test allowlists. - Docs: "Crash-point suite" in `tests/restart-chaos/README.md`, and "Built (Phase 5)" plus the first work list in the plan. ## Verification - Focused tests: - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/crash-points.test.ts`; - `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15 tests, including runtime arming in a real child process; - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 72 tests; - `tests/runner-e2e/workflow-security.test.ts`: 19 tests. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Loop line: question "how long is one crash-point loop?" \| loop: `node scripts/crash-points.mjs --points <one>` (codex A1) \| elapsed 2.7 min end to end: boot 9 s, crash 9 s after arming, replacement ready in 6 s, rest about 2 min \| result: one point alone takes about 3 min. Before this change it took about 15 min (one cold boot per point, Vite UI, host at load 77). - Full local run, 14 points, codex A1, 11.8 min in all: - every point was reached (the server killed itself at each one, and the replacement came back); - 11 pass; - `workspace.finalize.after_start_before_copyback` and `workspace.finalize.after_copyback_before_done` fail I2 and R6: a `workspace_finalize` operation stays running after the run succeeded; - `wake.claim.after_run_running_before_wakeup_claimed` reports I4. Its evidence shows the wake failed with "…retrying once", and the retry delivered the work, so this is the known refusal-regex false positive. - The 6 Daytona-only points were not run. ## Risks - Test harness and an on-demand workflow only. No product code changes. - `.github/**` needs a CODEOWNER review. The workflow uses provider keys, so it refuses non-default branches and actors that are not on the allowlist, like `runner-full-stack-e2e.yml`. - `PAPERCLIP_IN_WORKTREE=false` is pinned in every chaos wave, not only crash-point waves. A chaos instance is never a worktree dev instance, so scheduling must not be suppressed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 23:41:15 -06:00
DottaandClaude Opus 5.5	b4842df1e4	docs(plans): Phase 2 Daytona wave results (#19947 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Phase 2 of the durable-ownership plan moves in-progress record kinds to one shared ownership path, guided by chaos data. > - The first data doc (#19927) covered local waves only. Remote runners, leases and sandboxes need a Daytona wave. > - This pull request adds the results of one Daytona wave to the data doc. > - The benefit is that Phase 2 and Phase 3 prioritize remote kinds from measured data. ## Linked Issues or Issue Description Refs #19927 (data doc), #19934 (claimed-wake settle), #19863 (speed budgets). What is the problem? The Phase 2 data doc had no remote data. Where is it documented? `doc/plans/2026-11-07-phase2-reconciler-data.md`. What should change? Add the Daytona wave's per-kind violations, settle latency, failures and speed numbers. ## What Changed - Added a "Daytona wave" section: 6 tasks with K1, K2 and two K4 outages of about 575 s. 5 of 6 passed, and R6 was clean. - Recorded the t03 failure: a Retry was cancelled by the company-wide accounting gate (Phase 3 scope). - Recorded the claimed-wake leak again (#19934 fixes it) and remote settle times of 76–107 s. - Corrected the earlier note about the image content-ID mismatch. ## Verification - Wave `w-20261108T012041-s9301`; the numbers come from a copy of the kept database and the read-only ownership sampler. - This is a docs-only change. ## Risks - Low. Docs only. The samples are small and the doc says so. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:37:38 -06:00
DottaandClaude Opus 5.5	f35eb1e5ef	fix(observability): judge the TTFT budget from run creation; report the wait behind a live turn (#19941 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Invariant 8 of the durable-ownership plan says turns must start fast. #19863 added per-run speed budgets that write `run.performance.budget_exceeded` to the run log. > - The Phase 2 chaos data (#19927) showed follow-up TTFT about 3x over the claude budget. > - The cause is measurement scope. A follow-up comment posted during a live turn waits for that turn to end, and TTFT is anchored at the comment. The budgets came from idle follow-ups. > - This pull request judges the TTFT budget from run creation and reports the wait separately. > - The benefit is that budget breaches point at real start-up slowness, not at time spent behind a turn the user is already watching. ## Linked Issues or Issue Description Refs #19863 (speed budgets), #19927 (Phase 2 data), #19852 (startup sub-spans), #19888 (warm default). What happened? In two local storm waves, C1 follow-up runs breached the claude warm TTFT budget (p50 27.6–29.9 s against 9.5 s). Their spans show `comment.to_run_created` at 6–62 s (waiting for the live turn), then session resume at 1.4–3.8 s and submit-to-first-event at 2–5 s. Codex without the wait was about 4.8 s, the same as the warm-follow-up baseline. What did you expect? The budget should judge start-up latency, which is what its baseline measured. The wait behind a live turn should be visible but not judged. Steps to reproduce 1. Run a local chaos storm with C1 tasks. 2. Read `run.performance.budget_exceeded` rows with metric `ttft` for `issue_commented` runs, and compare them with each run's `comment.to_run_created` span. ## What Changed - `native-run-trace.ts`: the TTFT budget check measures from run creation (start of `heartbeat.queue`, or end of the ingress span). The TTFT span itself does not change. - `native-speed-budgets.ts`: breaches can carry `inputWaitMs` (TTFT only). - `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`: the wait behind a live turn is its own measured step (`inputWait`), next to the full TTFT and the TTFT from run creation. The TTFT table gains a "Wait behind live turn" column. The full comment-to-first-token time stays the user-visible metric. - `doc/run-log-events.md` and the plan's speed-budget section describe the rule. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts`: 43 pass, 2 new. `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/ttft-baseline-stats.test.ts`: 12 pass, 1 new. They cover a follow-up under budget after a 60 s wait (no breach), a follow-up over budget (breach with `inputWaitMs`), and queueing after run creation still judged. - Server typecheck passes. ## Risks - Low. This is a run-log-only change (AGENTS.md §5.7): no Telemetry and no span attributes change. - A slow wait before run creation is no longer a TTFT budget breach. The TTFT span still records it, and the breach payload names it. Admission holds after run creation are still judged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:27:41 -06:00
DottaandClaude Opus 5.5	17b9e10252	docs(plans): Phase 2 reconciler data; let chaos K4 holds reach 600 s (#19927 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Control-plane restarts leave in-progress records that each subsystem releases in its own way. The durable-ownership plan replaces those paths with one shared path. > - Phase 1 added a report-only reconciler. Its light local waves showed zero violations, and that proves little. > - Phase 2 moves kinds to the shared path. It needs data that shows which kinds really leak, and the speed numbers to protect. > - This pull request records that data from harsher waves. It also fixes the harness so that K4 outages past the reconnect grace qualify. > - The benefit is that Phase 2 moves kinds in order of measured risk, against a known speed baseline. ## Linked Issues or Issue Description Refs #19831 (durable-ownership plan), #19861 (report-only reconciler), #19863 (speed budgets), #19797 (K4 hold cap). What is the problem? Phase 2 needs per-kind invariant data and speed data from harsh restart waves. Only light local waves existed. In addition, a K4 hold over about 405 s always failed its fault receipt, so the 600 s cap from #19797 could not be used. Where is it documented? `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` (Phase 2 rows) and `tests/restart-chaos/README.md`. What should change? Record the data in a dated plan doc. Let chaos K4 holds run up to 600 s and still qualify. ## What Changed - Added `doc/plans/2026-11-07-phase2-reconciler-data.md`. It covers 5 unmasked local waves (two storms, a crash loop, two K4 runs). It gives per-kind violation counts with example rows, owner verdicts, TTFT, boot and restart recovery against the budgets, Phase 3 gate notes, and what is not covered (Daytona). - Added a short pointer to that doc in the durable-ownership plan. - `tests/restart-chaos`: new flag `--k4-hold-max-seconds` (120–600). It raises the upper end of the seeded K4 hold. The default range (120–300 s) does not change. - `tests/runner-e2e/controller-outage.ts`: the receipt's ready bound is now a parameter (`CONTROLLER_READY_BY_MS`, default 420 s). Runner suites keep 420 s. The chaos supervisor passes hold + 420 s, so a hold past the Runner's reconnect grace qualifies. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 12 files, 135 tests pass, including the new flag test. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/controller-outage.test.ts`: 36 pass, including the new long-hold receipt test. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and `tests/runner-e2e`: pass. - Live: the K4 wave at seed 9203 failed 3 of 6 tasks with the old bound (fault error "did not produce a qualifying replacement receipt"). It passed 6 of 6 with this fix, with holds of 294–579 s. ## Risks - Low risk. The changes are test harness code and docs only. The runner suites keep the 420 s ready bound by default. - The data is local only. The doc says which kinds the waves do not reach. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). The model ran the chaos waves and wrote the analysis and code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 19:55:02 -06:00
DottaandClaude Opus 5.5	99529f9d71	feat(observability): local speed budgets and a run-log budget check (#19863 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When someone messages an agent, the reply must start as fast as possible, and restart-safety work must not slow it > - The durable ownership plan makes this invariant 8 ("fast turns") and asks Phase 1 to turn the Phase 0 baseline into p50/p95 budgets per adapter > - Without budgets, a regression in time to first token, agent boot or restart recovery shows up only when someone reruns the baseline by hand > - This pull request records the budgets and adds a per-run check that writes a run-log row when a run exceeds its p95 budget > - The benefit is that every later phase can show "no regression against the budgets" from the run log, without new waits on the turn path ## Linked Issues or Issue Description Refs #19831 (plan, "Performance constraint", rule 4, and invariant 8). Refs #19846 (the Phase 0 baseline these budgets come from). Stacked on #19861. Problem The plan requires p50/p95 budgets per adapter for TTFT, agent boot and restart recovery latency, and a check against them. Neither exists. Expected behavior Budgets are recorded in the plan and in code. A run that exceeds a p95 budget gets one run-log row that names the metric, the measurement and the budget. Impact Phase 2–5 exit criteria can cite budget breaches from chaos waves and staging. The check adds no wait before `turn.start`. ## What Changed - New `server/src/services/native-runtime/native-speed-budgets.ts`: the local budget table (codex, claude, opencode) for TTFT and agent boot (cold and warm) and restart recovery latency. Each budget is 1.2× the baseline, rounded up to 0.5 s. `evaluateSpeedBudget` judges one run against p95. An unknown start kind uses the looser budget. Remote runs and adapters without a baseline are not judged. - `native-run-trace.ts`: the trace takes an optional `budget` subject. It checks `task.time_to_first_token`, agent boot (end of `heartbeat.queue` to the end of the first `runner.turn.submit` or `runner.session.resume`) and restart recovery once each. It writes `run.performance.budget_exceeded` through `defer`, so the write is detached from the event path. `finish` now drains diagnostics that defer more work. - `native-session-executor.ts`: passes the adapter and environment to the trace, labels cold or warm at the existing `provider_session` reuse receipts, and checks restart recovery at a recovered run's first provider event, measured from the startup recovery `ready` time. - Docs: budget tables in `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` ("Speed budgets (Phase 1)"), and the new event in `doc/run-log-events.md`. ## Verification - `npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts` (in `server/`): 34 tests pass. They cover: - every budget has p50 ≤ p95; - the provider-to-adapter mapping; - p95 judgement, and the looser budget when the start kind is unknown; - one breach for a slow boot, and none for a second submit or a fast TTFT; - a deferred TTFT breach is written before the trace closes; - restart recovery is checked once; - remote runs and runs with no budget subject write nothing. - `npx tsc --noEmit -p server`: no errors. ## Risks - Low risk. This is a run-log change only (AGENTS.md §5.7, no extra review). The OTLP and Telemetry paths are unchanged. - The baseline samples are small (2–12 per cell) on a shared host. Budgets carry 20% headroom, and the codex restart budget (8.0 s p95) is tight. Expect some breach rows on a loaded machine. They are diagnostics and change no behavior. - Restart-recovery budgets use run 4 of the baseline, as merged in #19846. - Restart recovery is measured from the startup recovery `ready` time to the first provider event, which matches the baseline definition. - Previous-release story: no persisted run state changes. The new event is additive. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 15:27:42 -06:00
DottaandClaude Opus 5.5	d5dd09c966	docs(plans): design turn-scoped ACPX MCP authority; correct the Claude baseline (#19912 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Follow-up messages must start fast. #19888 made Codex and OpenCode follow-ups 2.6–5× faster by keeping their runners warm. > - Claude (ACPX) did not gain, because ACPX relaunches the Claude process and reloads its session on every follow-up (3.2–4.2 s). > - That relaunch is a safety boundary: the ACPX MCP endpoint lives as long as the process and cannot attribute a tool call to a turn, so reaping the process is the proof against late calls. > - Removing it needs a new authority model, which needs a reviewed design before any code. > - This pull request adds that design and corrects the Claude baseline. ## Linked Issues or Issue Description Refs #19888 (warm default), #19846 (TTFT baseline, finding 4). Docs issue The TTFT docs did not explain why Claude follow-ups relaunch, what removing that would require, or the corrected Claude numbers. What is wrong or missing `doc/plans/2026-11-07-warm-follow-up-ttft.md` reported a Claude follow-up at 10.5 s, measured under heavy host load. A quieter re-measurement gives 6.3 s per-turn against 7.1 s warm. There was no design for removing the per-turn relaunch. Proposed fix A design document for turn-scoped ACPX MCP authority, and a corrected-baseline section in the warm follow-up doc. ## What Changed - New `doc/plans/2026-11-07-turn-scoped-acpx-mcp-authority.md`, which covers: - the measured cost breakdown; - how the current boundary works (tool bridge bearer, call-ID tombstones, `restart_idle_provider`, `run.attach`); - a threat model (late tool calls from a settled turn, replay, credential replay, cross-turn and cross-run confusion, restart replay); - three candidate designs with their safety arguments: turn-scoped credentials in a live session, a proof that no calls are in flight, and an in-process session reload with per-generation credentials, which is recommended for a spike; - restart and chaos implications, the expected TTFT gain, and next steps. - `doc/plans/2026-11-07-warm-follow-up-ttft.md`: corrected Claude baseline. Per_turn stays the default; the earlier 10.5 s was host noise. Finding 4 now points to the design and drops the idle-retirement idea, which would only bring warm level with per-turn. - Docs only. No code changes until Dotta has reviewed the design and a security review is done. ## Verification - The Claude numbers come from `ttft-baseline.ts --adapters claude --samples 6 --phases coldwarm --lifecycle <per_turn\|warm>` on master `fc6e1b0346` (load average about 9). - The step breakdown comes from temporary timing in runnerd's ACPX backend. It was not committed. - `node scripts/check-maps.mjs` passes. ## Risks - Docs only. The design proposes no behavior change until it has been reviewed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 15:24:12 -06:00
DottaandClaude Opus 5.5	fc6e1b0346	perf(native): keep local Codex and OpenCode runners warm by default (#19888 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When someone messages an agent, the turn must start fast. Time to first token (TTFT) is a first-class metric. > - The Phase 0 baseline (#19846) found that every local turn launches a new runner and loads the provider session again, because `per_turn` is the default lifecycle. That made a follow-up message slower than a new task. > - Warm runners keep the runner and provider session alive between runs. #19874 fixed the defect that broke warm OpenCode follow-ups, and #19879 bounds an idle warm runner whose controller dies. > - This pull request makes warm the default for local Codex and OpenCode agents, and migrates existing agents that only carried the old creation default. > - The benefit: follow-up TTFT p50 drops from 12.0 s to 4.7 s (Codex) and from 21.7 s to 4.2 s (OpenCode), with no new restart-chaos failures. ## Linked Issues or Issue Description Refs #19846 (TTFT baseline), #19852 (startup sub-spans), #19874 (warm completion contract), #19879 (warm reconnect grace). What existing behavior does this improve? The default runner lifecycle for native agents, and how a follow-up message starts. Current behavior Agent creation writes `lifecycleMode: "per_turn"` into every native agent. Every follow-up launches a new runner and restores the provider session: 4.2 s (Codex) and 16.3 s (OpenCode) at p50 before the turn starts. Proposed behavior New native agents store no lifecycle. On a local target, an unset lifecycle resolves to `warm` with the default 5-minute idle window for Codex and OpenCode. Other harnesses, remote targets and managed providers without warm support stay `per_turn`. An explicit choice always wins. Migration 0443 clears the stored creation default from existing agents, except where an edit set "Turn by turn" explicitly. Reason and benefit Measured locally (`doc/plans/2026-11-07-warm-follow-up-ttft.md`). Warm follow-ups are 2.6–5× faster for Codex and OpenCode, and restart recovery latency is equal or lower. ## What Changed - `native-lifecycle.ts`: an unset `agentMode` resolves to warm (5-minute idle window) for local `codex_app_server` and `opencode_server` agents. `runtime-selection.ts` passes `localTarget`. Explicit `per_turn` or `warm` wins. An admitted session keeps its persisted policy. ACPX Claude stays `per_turn`: it relaunches its provider process on every follow-up by design, so warm did not shorten its follow-up (10.7 s warm vs 10.5 s per_turn). - Creation and configuration writes no longer store `lifecycleMode: "per_turn"` for Codex, Claude or OpenCode. This covers `normalizePaperclipRunnerAdapterConfig` (runs on every create and update), `paperclipRunnerTransitionConfig` (switching to the native runner), `agent-runner-selection.ts`, `codex-local` `build-config.ts`, and the simplified new-agent setup (`paperclipRunnerCreationLifecycle` in `@paperclipai/shared`). Grok, Pi and Cursor keep their explicit value. - Agent form: the Runner lifecycle field shows the unset state as "Automatic". "Turn by turn" and "Warm session" remain. - Migration `0443_native_warm_follow_up_default.sql`: clears `lifecycleMode = 'per_turn'` from native Codex, OpenCode and Claude agents unless `agent_config_revisions` shows an edit that set it to `per_turn`. Each migrated agent gets an `agent.updated` activity entry with the old value (`actor_id = migration:native-warm-follow-up-default`). Creation records no revision, so a "Turn by turn" pick made at creation cannot be told apart from the old default. The product owner approved migrating those agents. Claude agents follow the default, which is `per_turn` today. - Restart chaos harness and TTFT driver: `--lifecycle per_turn\|warm\|default` (saved in the wave plan). `default` removes the catalog's pinned lifecycle, so a wave exercises the real default path. - Docs: `doc/DEVELOPING.md` "Runner lifecycle default", and the results in `doc/plans/2026-11-07-warm-follow-up-ttft.md`, which also documents baseline finding 4 (Claude's resumed turn-start gap is ACPX's per-turn process generation, not the provider). ## Verification - `server`: `native-lifecycle.test.ts` (new default cases: local Codex and OpenCode warm, Claude, remote, unknown target, unmeasured harness, managed Claude, admitted per-turn session, explicit choices, non-native adapter), `agent-runner-selection.test.ts`, `runtime-selection.test.ts` (335 pass). - `packages/db`: `native-warm-follow-up-default-migration.test.ts` on embedded Postgres. Default agents are cleared, the explicit edit is kept, other harnesses and legacy adapters are untouched, audit rows are written once, and a rerun is a no-op. Without the explicit-edit filter the test fails. `check:migrations` passes. - `codex-local` `build-config.test.ts` (65), `ui` `config-fields.test.tsx` (40), restart-chaos unit tests (134), `tsc --noEmit` for `server`. - `adapter-utils` `paperclip-runner-lifecycle-default.test.ts`: configuration writes and adapter conversion store no lifecycle for Codex, OpenCode and Claude, keep `per_turn` for the others, and preserve explicit choices. It fails without the fix. - End-to-end default check on this branch (`ttft-baseline.ts --lifecycle default`, codex and opencode, 2 samples): the agents read back with no stored lifecycle, and every follow-up reused the runner and provider session (no `runner.process.launch`). Follow-up TTFT was 3.6–4.8 s. This check found the per-write `per_turn` injection fixed here. - Restart chaos (codex and opencode, A1/B1/B5, K1/K2/K6, seed 4242), 8/8 PASS each with no open signatures: forced warm, forced per_turn, and `--lifecycle default` on this branch (agents created without a lifecycle). - `pnpm verify` selects broader CI (harness config is a global input), so required CI supplies the broad coverage. ## Risks - Behavior change: local Codex and OpenCode agents keep a runner and provider process for up to 5 minutes after a run. Instance sleep waits for that idle window. The process is reaped by the idle timer, by graceful shutdown, or by the #19879 reconnect grace if its controller dies. - Migration: agents whose "Turn by turn" was chosen at creation become Automatic. The activity entry records the old value; choosing "Turn by turn" restores it. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): a task session keeps the lifecycle policy it was admitted with, so sessions saved before this change stay `per_turn`. New sessions get the default. No run, checkpoint or launch state changes shape. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 14:37:36 -06:00
DottaandPaperclip	5a70999306	Add a versioned online workspace migration command (#19882 ) ## Thinking Path > - Paperclip preserves the files and recovery state produced by its agents. > - Canonical online migration can preserve selected local workspaces while unrelated work continues. > - Operators sometimes need a newer migration owner before the serving web process can take that release. > - The existing packaged maintenance entry requires stopped writers. > - This pull request adds a finite online entry from one coherent reviewed release tree. > - The entry delegates capture and publication to the existing owners and keeps tool and serving identities separate. ## Linked Issues or Issue Description Refs #19854, #19855 and #19872. The duplicate search found related owner fixes and no separate versioned online entry. What happened? The finite online workspace importer was available as a library in the serving image. The packaged all-phase migration entry required maintenance mode. Expected behavior Run the reviewed online owner from a separate complete release tree without replacing the web process, stopping unrelated work or changing its release policy. Deployment mode A managed instance with its original local volume, database, existing encryption key and shared S3 storage. ## What Changed - Add strict inspect, single, cohort and exact single-journal resume requests with explicit company, workspace, deadline and concurrency bounds. - Verify the private tool root and its own build stamp separately from the serving source commit and deployment. - Reuse the nonmutating full migration-manifest inspector and require the additive controller host-identity column before publication. - Hydrate existing configuration and key bindings, then use the original ESM database pool and canonical storage, source locks and migration owners. - Keep multi-member inspection with the optional canonical cohort preflight in a read-only transaction with denied storage. Releases without that owner refuse cohort operations; single import and resume remain available. - Register the optional runtime bundle entry and document the supported full-tree source mode, resource admission and reconciliation. - Add hash-bound old-writer/reader compatibility and encrypted restore tests. ## Verification Final local revision: `c9345b72f27aee1158047dccd0b1060f9a0be6df`, on master `fbe689938c3bf8c5849d2cd56b9da90029db9389`, macOS arm64 with Node 24.22.2. - Entry and runtime packaging: 34/34 PASS. - Real PostgreSQL/filesystem compatibility: 6/6 PASS, disposable database wrapper exit 0 and cleanup complete. The frozen old source lock owner contends with the new entry; unrelated work proceeds. The old authority and snapshot readers decrypt and restore newly published bytes. - Pending and uploaded journal cases use the unchanged old persisted format and preserve operation/snapshot IDs through resume. - Full layered server source and test typecheck: PASS. - Module boundaries and `git diff --check`: PASS. - Actual source-mode module import under Node 24 succeeds without starting the application. - A separate temporary full-old-producer fixture did not get past module setup: 90 seconds under Vitest and a 15-second direct Node control. Neither is counted as an executed old-producer test or a product failure. The smaller frozen old-writer/reader proof and old-format journal cases are the local evidence. - Hosted resume of the actual saved operation and Linux artifact qualification remain operational acceptance gates. This PR does not claim those ran. - Review regression: a real successful import returned a null operation ID (genuine RED). The repaired entry binds the exact returned snapshot to its succeeded migration journal after commit (GREEN). Committed or superseded snapshots are accepted; a missing readback requires reconciliation rather than a blind retry. - Final affected check: 40/40 PASS across the entry, packaging and actual PostgreSQL compatibility suites; wrapper exit 0 and cleanup complete. Full layered typecheck and boundaries pass after the user-requested rebase and current Runner rebuild. - A second real postcommit database-read fault also has genuine RED/GREEN proof. The succeeded journal, saved authority and original bytes remain intact, and every readback error now preserves the reconcile-required warning. - Final-head [CI run 42404116001](https://github.com/paperclipai/paperclip/actions/runs/42404116001): PASS, with all 59 check contexts terminal (55 success and 4 intentional skips). Fresh Greptile review is 5/5 on `c9345b72f27aee1158047dccd0b1060f9a0be6df`, with no unresolved threads; check and thread pagination are complete. ## Risks - The tool must come from one authenticated complete release tree with its own dependencies. A copied bundle chunk can retain old absolute paths and is not a portable release. - Concurrency is not resource admission. The operator must check memory, CPU and scratch headroom and start with one import or two cohort members. - Schema must be prepared by the existing schema owner. This entry never applies DDL. - Cancellation awaits the canonical owner. An SSH interruption does not prove the process stopped; durable journals must be reconciled before retry. - The entry preserves remote obligations and source files. It does not prove enrollment readiness or zero-downtime operation. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact serving snapshot and context-window size are not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 13:20:35 -06:00
DottaandPaperclip	965ca91dd0	Capture legacy workspace cohorts under one source fence (#19873 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace migration preserves local files as authenticated portable snapshots. > - Workspaces that share Git storage require one complete source fence. > - Independent imports contend on that fence and cannot safely run as a batch. > - This change adds an opt-in cohort owner that captures members concurrently and publishes them atomically. > - Existing source, recovery, snapshot and publication checks remain authoritative. ## Linked Issues or Issue Description What existing behavior does this improve? The administrative remote-workspace migration owner for workspaces in one shared physical source component. Related changes: #19688, #19854 and #19855. Current behavior Each independent import acquires the complete source fence. Parallel independent calls therefore contend instead of overlapping archive work. A caller cannot use a surrounding transaction and assume that a successful savepoint confirms the outer commit. Proposed behavior An explicit `migrateRemoteWorkspaceCohort` call selects one to eight leaves from one proven component. It owns the full source/lifecycle/copyback fence, uses the original pool for bounded archive work, validates every member, publishes serially, and returns success only after the outer commit. Reason and benefit This supplies a correctness-gated path for a controlled two-member hosted qualification. It makes no wall-clock speedup claim. No running app call site or scheduler selects cohorts automatically. Breaking changes The existing single-import API retains its guards and rejects a journal owned by a cohort. New cohort journals require the same complete member set on resume. ## What Changed - Extract the existing reservation, capture, validation and publication stages for reuse by both owners. - Add an opt-in cohort API with default concurrency two and a maximum of eight. Every member must share the same complete physical/Git/alias component; selected roots must be leaves relative to each other. - Persist exact cohort and operation membership in the existing journal. Compare all selected fingerprints with the aggregate proof before the first reservation under the complete fence, and recheck each member when reserving its journal. Refuse foreign journals, incomplete or mixed replay sets, lost fences and changed sources. Keep all authorities and activity events unpublished if the outer commit fails. - Share the merged nested-child verifier inside one complete fence. Queue its prepare calls, retain its two-child bound and authenticated immutable proof, and recheck local bytes and database bindings on every pass. - Document the API, exact-membership resume protocol, resource bounds and required hosted qualification. ## Verification - Integration head `fce1896010`: 32/32 PostgreSQL/filesystem cases across the cohort owner, the complete merged nested-verifier suite, and five existing parent-import cases. - Final source-proof repair: 19/19 cohort-owner cases pass, including real first- and last-member metadata changes before reservation, changed-source partial-resume refusal, mixed local/imported refusal with storage forbidden, cancelled/failed/finished admission, a late finished timestamp, complete fences, queued child-proof reuse, cold restore, cancellation, commit failure and exact replay. All local database checks use the canonical owned wrapper, one Vitest worker and Node 24.22.2 on macOS arm64/PostgreSQL 18.1. - Final rebased head is based on `fbe689938c3bf8c5849d2cd56b9da90029db9389`, including current source-owner, database-schema and Runner contract changes. The 19-case owner suite, full server layered source/tests/test-tree typechecks, module boundaries and diff checks are rerun on that base; required CI supplies the native Runner qualification. - Selection-only `node scripts/local-verify.mjs --base `68db7ed9e3` --explain` exceeded its 60-second observation bound before producing a plan. It supplies no test pass; required CI must supply broad coverage. - Genuine replay RED: a committed two-member cohort accepted a one-member replay. The repaired owner requires its full succeeded-journal/snapshot membership. - Genuine terminal-journal RED: read-only preflight accepted a cancelled, finished saved member, and apply created two pending snapshots and the missing peer journal before refusing later. The repair rejects that state before preflight eligibility or snapshot work and rechecks unfinished state before publication. - Genuine source-proof race REDs: early workspace metadata drift could strand a partial journal. A first repair checked each member only when reserving it; an additional last-member regression showed that an earlier journal could still be written before noticing the changed member. The final owner compares the complete preimage in the existing fenced preflight before the first reservation and retains per-reservation checks. Both first- and last-member cases now leave zero journals/snapshots and permit a fresh attempt. Changed sources after a genuine partial reservation still refuse. - Coverage includes parallel uploads, cold restore with originals removed, complete source/copyback fences, exact partial-reservation resume, actual read-only preflight with storage forbidden, foreign/changed journals, peer failure and cancellation, fence connection loss, and a real deferred outer-commit failure. - Prior head `70bc2740d0` passed all CI after one explicitly bounded same-head retry of server shard 4. Its first attempt observed an unrelated heartbeat/plugin test still running at a five-second wait; automatic test retry then hit the fixed-name environment uniqueness constraint. The test and 1,100 relevant owner files matched the nearest successful base gate. The one job retry passed the target test in 3,491ms and all 3,045 shard tests. Original failures remain recorded; their initial cause is unknown. This does not supply CI acceptance for the later repair head. - Earlier prototype evidence remains historical: 286 PostgreSQL controls and twelve cold restores passed on its prior base. Six macOS timing runs were inconclusive under contention, including slower cohort runs. They are not speedup or hosted acceptance evidence. - Hosted acceptance remains a two-member same-component leaf canary with no imported children, followed by same-host timing, peak-memory and scratch-headroom observations before considering a larger batch. Run the focused owner file from `server/` with the canonical temporary PostgreSQL owner: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/remote-workspace-cohort-migration.test.ts --fsModuleCache --maxWorkers 1 --hookTimeout 90000 ``` Final head `112ffeb20c272c6ef5e1e0956be889c85511f0d0` passed [required CI](https://github.com/paperclipai/paperclip/actions/runs/42402942287) with 56 successful checks and four skips; the aggregate is green. [Greptile reviewed this exact head at 5/5](https://github.com/paperclipai/paperclip/pull/19873#issuecomment-6710780943), and a fresh paginated review-thread read has zero unresolved findings. ## Risks - Concurrent archive work increases CPU, memory, scratch space, database and storage load. Admission must count logical member workers and up to two nested-child workers, not Node processes. Eight-member use remains unqualified. - The complete cohort fence lasts until all started work joins. Slow callbacks can delay cancellation and associated source writers. - The exact journal exception is internal and binds company, cohort, workspace and operation IDs. Wrong or changed membership fails closed; operators must inspect unknown outcomes before resuming the same full set. - Existing remote recovery holds, provider obligations, source files and history remain preserved. A successful import does not settle those independent obligations. ## Model Used OpenAI GPT-6 (Codex), with reasoning, code execution and repository tools. This runtime does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 13:02:54 -06:00
DottaandClaude Opus 5.5	6cfb69b4b0	docs(plans): Phase 0 ownership inventory and TTFT baseline (#19846 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart can land between two durable writes. It leaves an in-progress record owned by a dead process. About 69 restart-chaos PRs fixed that shape one subsystem at a time. > - The durable ownership plan (#19831) replaces those fixes with one owner stamp, one prover, write-ahead intents and one reconciler. Phase 0 must first list every in-progress record kind and measure how fast turns start today. > - Without the inventory, Phase 1 cannot choose one stamp or one proof. Without the baseline, no later phase can show that it kept turns fast (invariant 8). > - This pull request adds the Phase 0 inventory, the TTFT, boot and restart-recovery baseline, and the run-log-only driver that produced the baseline. > - The benefit is one reviewed map of 83 record kinds to plan sections A–G, and measured p50/p95 budgets per adapter. ## Linked Issues or Issue Description Refs #19831. Refs #19838 (the TTFT anchor that the baseline uses). What is the problem? Phase 0 of the durable ownership plan has no inventory of in-progress record kinds and no speed baseline. What is the expected outcome? Two reviewed documents in `doc/plans/`, plus a repeatable way to measure the baseline again in later phases. Why does it matter? Phase 1 cannot start without them, and every later phase's exit criteria compare against the baseline. ## What Changed - `doc/plans/2026-11-07-ownership-inventory.md`: - 83 durable in-progress record kinds (94 reader rows) across native runtime, environments, workspaces, accounting and the wake queue; - each kind with its table and column, writer, states, owner, release rule, proof, obligation, invariants and plan sections A–G; - the overlaps, the merge decisions and the zero-downtime upgrade record kinds. - `doc/plans/2026-11-07-ttft-baseline.md`: - TTFT, agent boot, pre-turn steps and restart recovery per adapter (codex, claude, opencode), local, cold and warm, p50/p95; - restart recovery from the overnight chaos evidence, the slowest steps ranked, and the gaps. - `tests/restart-chaos/ttft-baseline.ts`: an on-demand driver on the chaos supervisor. It reads the run log only, never Telemetry, and always stops its embedded Postgres. - `tests/restart-chaos/ttft-baseline-stats.ts` and its unit tests: pure percentile, boot, pre-turn ranking and recovery statistics. No runtime behavior changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/ttft-baseline-stats.test.ts` - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` - The baseline numbers come from the driver run named in the report. To reproduce them, follow the report's "Method" section. - The owner spot-checked the inventory's file:line claims at `f2d8a51027`. One example: `processLossRetryCount` has no writer, and `heartbeat/recovery.ts:2675` is its only reader. ## Risks - Low risk. This adds documentation and an opt-in test driver. Nothing runs in CI except the pure stats unit tests. - The driver spends a small amount of provider credit when someone runs it by hand. - The baseline sample is small (6 per cell). The local host was shared with other worktrees, so the p95 tails include load. The report states both. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use, code execution and five parallel read-only subagents for the inventory. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 10:47:31 -06:00
DottaandPaperclip	54b83b15a3	Reuse authenticated nested workspace verification (#19854 ) ## Thinking Path > - Paperclip preserves the work that its agents produce. > - Portable workspace migration can save nested workspaces before their parent. > - The parent must prove that a committed child snapshot covers any files it omits. > - The existing owner downloads and restores the same immutable child archive before and after parent capture. > - The fresh local-file checks are required, but the repeated immutable archive check is avoidable within one live source fence. > - This pull request reuses only an authenticated archive proof with an unchanged reference and provider identity. > - It also bounds independent child branches at two workers while retaining every fresh local and Git check. ## Linked Issues or Issue Description Refs #19805 and #19688. A duplicate search found no separate fix for repeated child archive verification. What existing behavior does this improve? Online Git parent migration after nested workspaces have committed portable snapshots. Subsystem affected server/ — workspace migration and nested snapshot coverage. Current behavior Each parent verification pass downloads, authenticates and restores every child archive, then captures the child's current local tree. The second pass repeats the immutable download and restore. Independent child branches also run in sequence. A large parent can exhaust its existing budget before parent capture begins. Proposed behavior One private verifier belongs to the existing live source-owner transaction. It can reuse a completed archive proof only when the full snapshot reference, operation, child closure, ETag and byte length still match. Missing or weak ETags and non-cancellation HEAD errors keep full verification. Caller cancellation still stops work. A changed identity forces fresh authenticated readback. The owner still captures current local bytes and Git state in both passes. Reason and benefit Remove duplicate immutable work without treating an old local-file check as current authority. Two independent topmost branches can overlap. Recursive descendants stay serial within those branches, so the bound does not grow with depth. ## What Changed - Add a private per-fence verifier to the existing tree owner. It clears its proof map at the end of the import and refuses a different owner transaction. - Reuse only completed ciphertext, GCM, plaintext, and canonical restore verification with a matching fresh provider identity. - Retain all current database, operation, recursive coverage, local-file, Git and manifest checks on each pass. - Bound independent branches at two. Abort queued work on failure and await active peers before scratch cleanup or fence release. - Use the verifier only in the existing online Git importer. Keep the offline full-verification path and all capture deadlines unchanged. - Document resource admission, fallback behavior and the hosted acceptance limit. ## Verification - Genuine baseline RED through the real parent importer: child GET counts 2 + 2, canonical restores 4, maximum overlapping readbacks 1. The four fresh local child captures already pass. - Exact positive GREEN: child GET counts 1 + 1, canonical restores 2, maximum overlapping readbacks 2. All four fresh local child captures remain. - The fixture uses real encrypted archives and the canonical S3 provider's local HTTP HEAD parsing with actual payload-derived ETags. It uses fake fixture credentials only. - Final focused suite: 13/13 PASS, 148.36 seconds, at `eaf17d3340`. This includes recursive max-two completion, failure/cancellation joining, identity/ref/content changes, same-fence alias reuse and stale-owner refusal. - Both affected existing migration suites: 256/256 PASS, 201.84 seconds, at initial head `eaf17d3340`. - The HEAD-failure review repair has genuine two-case RED evidence and three passing failure/cancellation controls. Full layered source/test-tree typecheck and module boundaries pass at repaired head `763648adad`. Final exact-head controls: 21/21 PASS, 86.63 seconds (all 16 new controls plus five existing parent cases). - The first extended recursive fixture correctly failed because it created a Git branch after taking an earlier snapshot. The fixture now creates all branches before any import; the source-change guard remains intact. - Exact-head CI: 56 success, four expected skips, and one neutral Paperclip Review. Required gates pass; Greptile is fresh 5/5. Its optional aggregate-handler cleanup recommendation is resolved with the intentional fail-closed rationale. - Local verification is macOS arm64 / Node26.4.0. The observed hosted parent still needs to complete within its unchanged budget after normal delivery. No hosted timing improvement is claimed here. ## Risks - Two simultaneous child verifications can use more I/O, memory and scratch space. Operators must count logical verifications, even in one process. - HEAD availability and provider identity are required for proof reuse. Providers without a usable ETag perform the full check. - Reuse lasts only for one live transaction. A retry, changed reference, changed provider identity, process exit or owner change requires fresh verification. - Faster verification does not waive changed child bytes or guarantee a specific hosted completion time. No provider turn, authority marker, source deletion or larger timeout is introduced. ## Model Used OpenAI GPT-6 through Codex, with repository and code-execution tools. The exact serving model revision and context-window size are not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 10:45:17 -06:00
DottaandClaude Opus 5.5	0f5ce5050e	docs(plans): plan durable ownership and reconciliation for restart safety (#19831 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs must survive a control-plane restart: they must continue on their own, keep their work, and offer a Retry that works > - The 2026-11-06/07 restart chaos campaign found about 40 restart failures and merged about 69 fixes (#19469 through #19807) > - Most of these failures have one shape: a restart stops a process between two durable writes, and each subsystem then decides on its own whether the leftover record is safe to release > - Each fix closed one crash window, and about 12 fixes rebuilt the same "this owner is dead" proof > - This pull request adds a plan for one shared ownership record, write-ahead intents, one reconciler, and an on-demand crash-point suite > - The plan also makes time to first token and agent boot hard requirements, so restart safety never makes turns slow > - The benefit is that we fix the whole class of failure once, and we stop fixing one window at a time ## Linked Issues or Issue Description Refs #19696, #19713, #19717, #19724, #19741, #19747, #19784, #19807 What existing behavior does this improve? Recovery after a control-plane restart. Today each subsystem keeps its own in-progress records and its own release rules. Subsystem affected Native runtime, runner, environments and sandboxes, workspaces, accounting and budgets, and the wake queue. Current behavior A restart can leave a lease, a fence, a claim, a workspace operation, a receipt, or a deferred wake that belongs to a dead process. If no subsystem can prove that the owner is dead, the system fails closed. The task then needs a Retry, the company holds new work, or a run stays `running`. About 40 server files read a controller boot id, and about 25 files check process liveness in different ways. Proposed behavior One owner stamp and one liveness proof for every in-progress record. Intents are written before outside effects. One reconciler brings every record to a terminal state. Admission never waits on cleanup. Accepted work is delivered exactly once. Time to first token and agent boot are measured and stay within budgets. Reason and benefit Fewer restart failures, less duplicate code, and a fast first response after a restart. ## What Changed - Added `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`. The plan contains: - the five failure classes from the campaign, with example IDs and PR counts - eight invariants - a performance section with time-to-first-token, boot, and restart-latency metrics and rules - the design (sections A–G) - an on-demand crash-point suite - six phases - risks - the decisions that Dotta made on 2026-11-07 ## Verification - This change adds documentation only. No code changes. - Read the plan. Check that each failure class cites real merged PRs. - Check the code claims: - `git grep controller-process-identity -- server/src` shows three users. - `git grep task.time_to_first_token -- server/src` shows the existing metric. ## Risks - Low risk. The change adds one plan document and changes no code. - The plan proposes later behavior changes. The largest is the removal of the company-wide accounting gate. Each later phase will come as its own pull request with tests. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, git, GitHub CLI). The model coordinated the chaos campaign and wrote the plan from its evidence. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 08:36:05 -06:00
Dotta Paperclip Codex	af0fe8fe81	Keep personal MCP connections functional while idle tenants sleep (#19427 ) ## Thinking Path > - Paperclip manages work for AI agents. > - Personal MCP connections should remain enabled while an idle tenant sleeps. > - Unfinished OAuth, transfer and delivery work must retain its owner and deadline. > - Cloud can receive bounded metadata and incoming work while the tenant remains the company-scoped authorizer. > - Current identity, revision and delivery acknowledgments establish when ownership has transferred. > - This change preserves useful wake, retries and revocation while making settled ownership inspectable without writes. ## Linked Issues or Issue Description Refs: #19377 and #19750. Hosted companion: [paperclip-cloud#1217](https://github.com/paperclipai/paperclip-cloud/pull/1217). What happened? Enabled personal MCP grants and subscriptions could keep an otherwise idle tenant awake. Ignoring those records would lose discovery, authorization, delivery and revocation obligations. Expected behavior Unconsumed authorization codes, device requests, file handoffs and delivery work retain their original deadlines. Settled connections remain enabled and become quiet after the current external owner acknowledges responsibility. Incoming useful work resumes normal authorization and delivery; revocation still takes effect for the current company. Steps to reproduce Complete normal PKCE/device consent and company selection, synchronize client/device and revocation state, retain the connection while idle, then exercise useful tools/events and revocation. Sleep eligibility requires current incoming/deadline ownership throughout. ## What Changed - Add bounded signed client/device metadata and revocation synchronization with exact revision, identity and ACK checks, startup recovery and finite legacy-device cutover. - Preserve company selection, authorization and operation identity. Register grant work before its INSERT in the same transaction; retain rollback, expiry and retry behavior. - Require current managed identity/resource before hosted publication. Receipt-cleared re-enable must obtain a fresh source ACK before device creation. - Reuse read-only settings and inspection budgets; connect MCP owners to normal startup, drain and shutdown. - Bound metadata pages and the actual signed request body while preserving the default limits of other operations. - Generate migration 0440 after the merged Legacy migration. Preserve all 438 existing executable migrations and the journal prefix; retain the normal five newest snapshots. - Preserve the merged transaction-owner forwarding and regression coverage from #19750 through normal ancestry integration. ## Verification - Current own Shared/SDK, database, eval-kernel and Runner builds passed, together with all seven normal application TypeScript programs. Missing generated package exports/workspace links were restored through normal setup; the original setup failures were retained. No source, compiler option, test assertion or timeout was weakened. - Normal migration generation changed only the five intended MCP tables; its SQL is byte-identical to the reviewed MCP delta. Fresh reference databases and both retained master/Muse histories upgraded through 0440, preserving historical hashes, row fields, schema and idempotence. Direct assertions cover the Legacy null default and MCP null/zero defaults. Two selected history cases passed; the other 21 database cases were unselected. - All six current MCP lifecycle cases passed with zero skips: normal PKCE approval intent, ACK recovery after outage, mutation during publication, disable/re-enable with original expiry, company-deletion rollback/commit and key rotation. - All 81 checks in five current whole files passed with zero skips: OpenAPI 34, read-only settings 2, diagnostic catalog 25, PostgreSQL diagnostics 7 and dependency scheduling 13. PostgreSQL fixtures ran normal migrations before test imports. - Earlier canonical proof at App `cd1c8c6` / Cloud `9f67531` includes real 60-second rate and three 15-second quiet windows, useful work and three revokes. Later paired proof at App `43af514` / Cloud `914eb0f` plus its reviewed candidate includes normal DCR, PostgreSQL metadata, a 144,017-byte signed publication, actual ACK, cap/default-413 controls and a real 15-second quiet window. The relevant producer/signing/wire inputs remain byte-identical in this integration; shared changes are qualified above. These remain source-scoped observations, not a fresh full canonical run on the integrated tree. - All known test owners, process groups and OS sockets closed; an independent final PostgreSQL audit found no remaining test sessions. One dependency-test trace recorded 46 opens and 45 close events. Its cause is unknown; this is not claimed as proof of graceful closure. - Fresh exact-head remote checks passed: 59 successful and 4 skipped contexts, with a complete current 5/5 Greptile review and all seven review threads resolved. Targeted local checks do not constitute a new whole-repository test run. ## Risks App and Cloud must agree on identity, revision, namespace, deadlines and revocation. Partial rollout, unavailable authority or missing ACK must retain awake debt. Shared startup/shutdown and settings changed since the earlier canonical profile, so historical observations carry only within their stated source scope. No deployed Railway/Neon suspension, Cloud alarm, savings or complete hosted Dot/Muse acceptance is claimed. ## Model Used OpenAI `gpt-6.1-sol` produced code and verification, with a Codex coordinator reviewing and integrating the changes. The coordinator's exact runtime model ID and context-window sizes were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-11-07 02:12:48 -06:00
scotttong	f5fb8ada3b	Restore chat contacts, agent pins, and scrollable recent navigation (#19685 )	2026-11-06 19:17:25 -08:00
DottaandPaperclip	d1306f338b	fix(runtime): preserve settled failed harness backups online (#19715 ) ## Thinking Path > - Paperclip coordinates agents, tasks, and their saved execution state. > - A replacement server needs native provider backups in shared storage. > - The online pre-copy owner accepts successful runs with a committed result. > - A sealed backup from a failed or cancelled run can still contain user history. > - The existing owner defers those files even after their source ownership is settled. > - This change adds a separate preservation owner with exact source and publication checks. > - The copy retains the failed outcome and original files while making the backup readable from a fresh home. ## Linked Issues or Issue Description Refs #19137 and #19108 for the existing pre-copy and encrypted backup owners. What happened? Online pre-copy reports `source_not_successful` for a sealed legacy backup from a settled failed or cancelled run. The provider files remain tied to the old disk. Expected behavior Copy the exact recorded backup when the source has no remaining writer ownership. Preserve the run status, warnings, result, allocation, selected session, and local files. Steps to reproduce 1. Create a lease-stamped suspended native backup. 2. Settle its source as a failed run with a terminal-failure coordinator and no owner. 3. Keep an unrelated execution active. 4. Run canonical online pre-copy. The previous owner defers the source. ## What Changed - Add a narrow preservation owner for terminal source files. It checks the physical finalization lock, task, coordinator, run, selected session, source lease, and source fingerprints. - Recheck that owner before the encrypted upload receipt and current-backup publication. - Reuse the finite workspace keepalive during long source checks and defer source-local invalid archives. - Add optional exact-root stamp verification. A valid previous backup cannot authorize a changed current payload. Existing callers keep their fallback behavior. - Bind the archive input to the checked owner and record an activity event after publication. - Keep pending object receipts after an uncertain or failed publication. - Add focused PostgreSQL and filesystem controls. Update the existing migration runbook. ## Verification - Real PostgreSQL baseline on unchanged production: the new positive case fails with `source_not_successful` instead of publishing the backup. - Same case passes after the change and decrypts/restores the exact provider history. - Final head `4b6cf093be`: three focused suites pass, 81 tests in 62.26 seconds, including existing storage and stamp consumers. The canonical PostgreSQL wrapper exited zero and completed cleanup. - Controls cover cancelled and failed sources, real committed results, expired nonnull owners, physical owner/lock contention, late source/owner changes, previous-tree fallback, and archive root/scope binding. - Both review regressions fail with the committed pre-fix owners: a real 30-second database idle timeout expires during a 32-second exact-source check, and a stamp-valid invalid archive stops the pass. The fixed owner keeps its transaction live and copies the next valid source. Source hashes and automatic file restoration were recorded for each leg. - Final-head strict server TypeScript and module-boundary checks pass after the master rebase and required Runner/shared declaration rebuild. - No source migration, provider mutation, or archive apply was executed. ## Risks This path does not reconstruct missing backups or prove that missing provider state is recoverable. Such sources remain blockers. It does not take expired owners or report failed work as successful. Source hashing adds bounded disk work while source row locks are held; unrelated executions remain active in the PostgreSQL fixture. Existing archive size limits, authenticated encryption, object readback, and publication fencing remain in place. No schema, API route, scheduler, or rollout-policy change is included. ## Model Used OpenAI Codex (GPT-6 family; exact runtime model ID and context capacity are not exposed in this session), with reasoning, repository editing, code execution, and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 21:08:27 -06:00
DottaandPaperclip	55c9c44e41	fix: preserve database-only failed controller evidence (#19714 ) ## Thinking Path > - Paperclip preserves failed work as well as successful work. > - Ephemeral enrollment must retain each selected controller source. > - A failed source can already exist entirely in an encrypted database checkpoint. > - Its original inline snapshot does not create a local filesystem dependency. > - The migration owner must prove that both legacy files are absent before accepting that case. > - This change preserves the failed evidence without creating execution authority. ## Linked Issues or Issue Description Refs: #19678. What happened? Startup still reports `native_controllers_require_migration` for a failed retained source with a valid database checkpoint, its original inline snapshot, and no local controller or supervisor. The online archive owner reports that the source is already absent. The existing database-only preservation case accepts only a missing inline snapshot. Expected behavior Accept the exact database-only failed evidence after full checkpoint and identity validation. Keep a present local payload blocked. Retain the failed run, original snapshot, coordinator, task pointer and queued work. Steps to reproduce 1. Store a valid encrypted checkpoint for a failed native run. 2. Keep its matching inline session snapshot and terminal-failure coordinator. 3. Remove both legacy controller files. 4. Check local controller obligations. 5. The old owner reports an obligation even though the evidence is already in the database. Paperclip version or commit The fix builds on current master `d5c4f7dd02`. Deployment mode Hosted native Runner with durable controller checkpoints. The local regression uses the existing real PostgreSQL and encrypted checkpoint fixture. ## What Changed - Add a narrow database-only failed-source preservation case to the existing migration owner. - Require the decoded checkpoint, protocol header, admitted execution, selected run and inline snapshot to match their exact scope. - Require both the controller and supervisor paths to be absent. A file or link keeps the local obligation blocked. - Retain the earlier null-snapshot predicate and all original failed execution evidence. Create no recovery metadata or executable authority. - Exercise the actual startup and maintenance consumers. Document the preservation boundary. ## Verification - Final revision `1b1da543dafd576e3c420657e3c2c6d0c882172d`: all checks are terminal, with 56 successful and four skipped, zero pending and zero failures. The [required CI aggregate](https://github.com/paperclipai/paperclip/actions/runs/42329501690/job/128275664414) passed. Build, types, server/Runner/serialized suites, end-to-end aggregate and Snyk passed. [Fresh exact-head review](https://github.com/paperclipai/paperclip/pull/19714#issuecomment-6701018015) is 5/5 with no open threads. - Review found that the first non-recovery assertion stopped at the queued fixture's missing current binding. The fixture now binds the exact native issue/session before calling the real resolver and asserts `unsupported:metadata_format` from the selected failed source. That corrected case passes in 4.58 seconds, with wrapper exit 0 and cleanup complete. The production predicate is unchanged. The original startup RED remains valid. The final source/test type graph, module boundaries and diff checks pass. Fresh revision CI and review remain required. - The actual startup regression failed before the source change: one case, 16.62 seconds, expected the absent-files obligation to be false but received true. The wrapper exited 1 and completed cleanup. This failure is retained. - The final focused owner run passed nine cases in 10.79 seconds. It covers the new DB-only case, controller/supervisor presence, every inline scope field, foreign checkpoint Runner/session identities, the unchanged null-snapshot case, and existing online/offline archival controls. The wrapper exited 0 and completed cleanup. - The new case verifies unchanged run, coordinator, checkpoint and task-session rows, retained queued work, no upload, and no invented recoverable session. - The full qualified server/test source type graph passes with db/shared/Runner source aliases. Module boundaries and diff checks pass. Required broad package builds and regression suites remain in CI. - Read-only source diagnosis validated one exact existing checkpoint with the canonical decoder and matching header, run, Runner and session bindings. No source profile or secret values were emitted. No hosted mutation, admission hold, cancellation, status change, or writers-stopped claim was performed. ## Risks - A false database-only classification could hide local bytes. Both exact legacy paths must be absent, and every protocol and database binding must match. - Corrupt or mismatched checkpoints still fail closed. Existing failed evidence does not become successful or executable. - The original null-snapshot case is unchanged. Present local files still require the existing preservation path. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 20:45:29 -06:00
DottaandPaperclip	d5c4f7dd02	fix: preserve shared Git sources during online workspace migration (#19688 ) ## Thinking Path > - Paperclip preserves the work produced by its agents. > - Remote workspace migration must preserve legacy Git history and local edits. > - Linked checkouts can share administration and objects with other tasks. > - A task lock alone cannot exclude writers of a shared or borrowed store. > - The online source owner must derive and retain the full physical ownership closure. > - This change reuses task, project, lifecycle and finalizer locks for that closure. > - Idle legacy Git work can then migrate while unrelated remote turns continue. ## Linked Issues or Issue Description Refs: #19642. What happened? Online migration keeps legacy Git sources blocked because several task checkouts can share a project base or a private Git cache. A read-only source observation found 160 pending Git records, 120 distinct paths and nine common stores. Its largest project cohort contained 126 records and 110 task owners. An ordinary-directory fence cannot preserve these sources by itself. Expected behavior Derive the physical source owners from stable Git metadata. Hold their existing source and lifecycle keys through publication. Retain all history and recovery obligations. Reject foreign aliases, unknown stores and changed sources. Let unrelated object-backed work continue. Steps to reproduce 1. Create a managed project checkout and linked task worktrees. 2. Keep a settled prepared recovery source on a sibling task. 3. Keep unrelated remote work running. 4. Request an online import of an idle leaf checkout. 5. The existing owner keeps the shared source blocked. Paperclip version or commit This branch builds on merged #19642 (`7ac99718a4`) and fresh master `fb3cf1a072`. It adds the Git closure. It does not add a writers-stopped shortcut. Deployment mode Hosted native Runner with remote workspace storage. Local qualification uses owned synthetic Git repositories, disposable PostgreSQL databases and the actual archive and storage owners. ## What Changed - Read bounded stable Git pointers, common-directory metadata and local alternates. Derive the complete physical source ownership cohort. Preserve all object stores. - Retain common and borrowed store owners in preparation, lifecycle, runtime, expiration and actual reopen paths. Keep company boundaries and the existing 128-key bound. - Hold exact sibling native recovery proofs under their existing physical finalizer keys. Recheck descriptors, allocations, recovery actions and source identities before capture and publication. Preserve original states and warnings. - Require leaf-first nested imports. Reuse the existing immutable child download, restore and source comparison before parent exclusion. - Bind the self-contained archive owner to the held Git namespaces. Keep its three walks, history normalization, fsck, bytes and digest format. Reject bare, detached, network, unknown or changed Git storage. - Check both foreign path fields. Keep foreign metadata private. Document the proof, bounds, source observation and remaining eligibility limits. ## Verification - Final revision `2934e41c923b93e16508b978c849b747c86be645`: all checks are terminal, with 56 successful and four skipped, zero failures and zero pending. The [required CI aggregate](https://github.com/paperclipai/paperclip/actions/runs/42326464408/job/128266279210) passed. Build, types, all server/Runner/serialized suites, end-to-end aggregate and Snyk passed. [Fresh exact-head Greptile review](https://github.com/paperclipai/paperclip/pull/19688#issuecomment-6699760104) is 5/5 with no open threads. GitHub reports MERGEABLE and CLEAN. - Review then found two correctness gaps: an ordinary managed directory inside an unrelated host repository was blocked, and a Git parent could miss a completed nested operation after its child snapshot became missing or deleted. Three actual-owner regressions reproduced both gaps in 9.70 seconds. The corrected owner passes 12 focused cases in 17.08 seconds. It imports and restores only the ordinary root bytes, preserves the outer repository, rejects both missing/deleted child receipts before a parent journal, and retains valid/changed child coverage and private-cache contention. Its wrapper exited 0 and cleanup completed. The first bootstrap attempt failed before tests ran because the host exhausted its 32 shared-memory segment limit; no shared services were stopped and no pass is claimed for that attempt. The final qualified source/test type graph, module boundaries and diff checks pass. - Receipt scans keep canonical path resolution. A lexical SQL prefix can omit children reached through symlinked ancestors. The observed full metadata closure completed in 1.17 seconds; no measured bottleneck justifies weakening that alias proof. Writer admission still rejects uncertain Git ownership before mutation. Expiration returns its typed skip, and reopen returns its structured failure. Runtime and lifecycle do not bypass an unknown source. No broad error-contract change was added. - Full CI at `720104ea975e` exposed five existing boundary cases across three server shards. Four exact cases reproduced locally as failures in 25.98 seconds. Reopen now preserves its structured failure when source inspection rejects the path, without reconstructing it. Runtime recognizes only the exact regular ignore-only `.git/info/exclude` fragment; migration still rejects that fragment. The original arbitrary-path and file-URL import/delete-host/resume fixtures keep both supported input forms and use the genuine offline setup path after their sole writer finishes. The production mode gate is unchanged. A new real-owner control proves the offline flag still rejects a running legacy database writer. - The final focused run passed all ten cases across four suites in 22.75 seconds, including all five CI failures, invalid-pointer admission, fragment rejection, offline writer exclusion and actual nested reopen lock ordering. Its wrapper exited 0 and cleanup completed. The known invalid-pointer case remains fail-closed; the separate unexpected-I/O diagnostic retains safe error metadata without command text or credentials. Existing Git stderr redaction coverage remains intact. The final metadata reader suite passed 20 cases in 2.68 seconds. The qualified server/test source type graph, module boundaries and diff checks pass. No production assertions or timeout defaults were weakened. Fresh CI and review are required for the repair revision. - A later review found that the ordinary early return still used only one foreign path field. The actual ordinary providerRef-only regression failed as incorrectly eligible. One source-path owner now enumerates both local locators for company/foreign ordinary and Git scans, layout checks, early admission, lifecycle/reopen/runtime and expiration/move/preparation retention. Capture and display keep their original preferred path. The actual owner canary passed 49 cases across two suites in 109.15 seconds. It covers cwd-only/ref-only/different/symlink locators, same-company mismatches, foreign owners, lifecycle ordering, Git/recovery imports and expiration/reopen. Three selected-locator fallback cases also failed before their guard; the final eight-case control passed in 31.05 seconds, including the configured alias and valid import/private-child controls. All wrappers exited 0 and completed cleanup. The metadata/locator suite passed 19 cases. The final source/test type graph, module boundaries and diff checks pass. - Review found two P1 gaps. Both configured-directory symlink cases failed before canonicalization. The actual parent import lost an already-imported child's unique private cache and failed its archive fence instead of retaining that owner. The corrected real-owner canary passed four cases in 20.78 seconds: configured symlink runtime provisioning, private-cache contention before the parent journal, successful parent publication after release, and unchanged/changed child coverage. The metadata suite passed 15 cases in 4.14 seconds. Both wrappers exited 0 and cleanup completed. - A fresh source read-only observation found 160 pending Git records and nine common stores. An additional exact receipt query found 1,243 completed migration receipts and 557 nested source paths under pending Git roots. All 126 records in the largest project were present in the captured Git metadata set. For all 110 present candidate workspace records, the derived closure fits the unchanged 128-key cap. The worst case needs 110 issue keys and one project key, with 17 keys of headroom. Its 173 imported children introduce no physical issue owner outside those 110 logical task owners. This checks the observed key bound only. It does not claim migration eligibility or a coherent snapshot. - The initial stacked CI did not run four Runner test lanes: trusted master passed new --package-lane/--native-server options to the old base lane script. The branch is now rebased onto actual merged master and retargeted before synchronization. It includes the canonical current lane parser without an unrelated Runner patch. Fresh new-head CI is required. - The actual importer/source owner run passed 36 cases across three suites in 187.75 seconds. It covers sibling prepared recovery, unrelated running remote work, common-cache contention, changed coordinator/allocation/path state during upload, exact nested coverage, dirty unpushed work, and existing expiration/reopen owners. Wrapper exit 0 and cleanup completed. - A separate previously hanging lifecycle-worker restore passed in 17.33 seconds with the unchanged 15-second test-body limit. An earlier run timed out in three expiration/reopen cases and blocked cleanup; its wrapper later exited 1. These failures remain recorded. They are not counted as passes. - Two boundary controls first failed: a foreign providerRef-only alias was incorrectly eligible, and actual reopen did not admit its distinct cache owner at the initial source boundary. The narrow guards now inspect both foreign locators and derive physical Git keys before the raw lifecycle transaction. The foreign alias and existing real nested reopen contention passed. The private-cache control first retained contradictory recorded source provenance and stayed fail-closed after release. The corrected present legacy-worktree fixture passed in 12.61 seconds. These failed attempts remain recorded; they are not passes. - The metadata reader and serialized/local archive controls use actual Git repositories. They compare archive bytes and digests, remove every original source, restore staged and unstaged work, and run fsck. The metadata reader passed 13 cases in 2.85 seconds. The local/serialized archive controls passed nine cases in 9.27 seconds. All processes exited 0. An additional actual importer control changed a relative commondir pointer to the absolute name of the same store during upload. Publication was rejected, the uploaded journal remained available, and original files stayed intact (one case, 5.86 seconds; wrapper exit 0 and cleanup complete). - The final rebased full local server/test type graph passes with db/shared/Runner source aliases because reused dependency declarations predate new master exports. The first rebased attempt reported those stale declaration fields/tables, and is not counted as a pass. Module boundaries and diff checks pass. Standard package builds, types and broad regression checks remain required in CI. - No hosted import, admission hold, cancellation, provider mutation, source filesystem write or shared-cache Git command was performed for this change. The source observation is read-only and is not a locked eligibility proof. ## Risks - A missing writer owner could make physical source exclusion incomplete. Existing mutation owners retain the same company/task/project keys, and uncertain metadata fails closed. - A project can contain a large cohort. Limits remain finite. A bound failure keeps the source blocked. - Bare, detached, unowned or network Git stores remain blocked online. Missing archived sources retain their existing authenticated reconstruction path. - Exact recovery proofs preserve authority and obligations. They do not infer provider termination or mark a prepared descriptor finalized. - This change does not claim all observed Git records are immediately eligible. Existing preparation, provider, recovery, runtime and source-change blockers remain. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 20:08:19 -06:00
DottaandPaperclip	7ac99718a4	fix(workspaces): fence legacy sources for online migration (#19642 ) ## Thinking Path > - Paperclip coordinates AI agents and preserves their saved work. > - Remote workspace storage replaces legacy host directories with durable snapshots. > - Several legacy workspace records can name one physical project directory. > - A task lock alone cannot exclude a new task that selects that directory. > - Terminal recovery records can retain valid sources after physical writers settle. > - This change fences the source through its real preparation and lifecycle owners. > - Idle ordinary sources can then migrate while unrelated remote work continues. ## Linked Issues or Issue Description What happened? Online migration rejects ordinary legacy directories when several workspace records name the same path. A source inspection found 45 aliases across three project roots with no root Git marker or previously imported nested child. The instance also had unrelated running remote work. Requiring the whole instance to drain prevented useful migration of these idle sources. Terminal native sources and superseded finalization attempts also remain blocked after their exact physical and durable owners settle. Their files, provider allocations and recovery records must remain available through the change of file authority. Expected behavior Migrate an idle ordinary source under a derived physical source fence. Let unrelated object-backed turns continue. Preserve provider and recovery obligations. Keep unknown writers, shared Git stores and uncertain source state blocked. Steps to reproduce 1. Create several local workspace records for one managed project directory. 2. Give the aliases different source tasks. Leave their local writers idle. 3. Keep a valid remote workspace turn running on another source. 4. Inspect or apply migration to an alias. The old owner returns `overlapping_workspace_requires_maintenance`. 5. Retain a terminal native prepared source with a null coordinator owner, or an old finalize attempt with an exact later committed receipt. The old online path keeps these sources blocked. Paperclip version or commit Current tested head: `7478c817f09f7eed396922c0f7831af07ae6ff14`, based on master `8621bcf53ee2e4bbefb7d193add42dd544ca06e8`. Related recovery work: #19131. Deployment mode Hosted native Runner with remote workspace storage. Qualification uses disposable local PostgreSQL databases, actual archive/storage owners and synthetic source trees. The source observations were read-only. ## What Changed - Retain the existing company/task source keys and add one physical managed-project key. Heartbeat source selection, materialization, reopen and lifecycle owners share it. Configured aliases of canonical task directories retain their original task key. - Fence runtime provision and each configured service CWD through both startup owners. Check all unsettled service CWDs for physical source overlap, even when workspace IDs differ. Runs with no services keep their filesystem-free path. - Derive a bounded ordinary alias closure. Lock every alias task and lifecycle key. Reject foreign companies, changed membership, unproven descendants, included Git metadata and other unresolved writer obligations. - Commit remote authority on the connection that holds source exclusion. Refresh a stale selected local workspace after admission if migration published remote authority. Publish activity after commit. - Admit exact terminal native recovery sources under the existing physical finalizer keys. Bind descriptors, allocations and owner checkpoints to journaled fingerprints. Recheck them before capture and publication. - Add a separate company-scoped online entry for exact superseded finalization receipts. Keep the offline entry's deployment prerequisites and keep inventory read-only. - Document the proof and the remaining Git/common-store extension. The original files stay in place. No new scheduler, deployment action or database migration is added. ## Verification - Review found that the real reopen path could hold lifecycle exclusion while waiting for a project source. The new actual nested-reopen regression failed before the repair: lifecycle lock count was 1 while another connection held the project fence. The repair acquires declared and physical project sources before reconstruction and lifecycle. The contention case, ordinary-fetch restore and runtime nested restore passed. The lifecycle-worker restore first completed in 16.64 seconds but exceeded its unchanged 15-second body limit; a bounded warm rerun passed at that same limit. Both wrappers exited and completed cleanup. The source type graph, module-boundary check and diff check pass. Final 7478 CI passed: 58 successful jobs, three skipped, one neutral and a successful Snyk context. Fresh Greptile is 5/5 for exact7478, with no unresolved threads. [Final CI](https://github.com/paperclipai/paperclip/actions/runs/42318150314/job/128240631573). - Full CI on `e4283b3452` passed the other server, build and type checks. One exposure owner case failed: two execution workspaces received one preview pair after source admission serialized their starts. The first local seven-case run reproduced that failure; adding the owner argument alone did not change the hash payload. The completed repair binds run-scoped reuse to its execution-workspace owner. All seven existing real-process/fake-broker cases then passed in 13.81 seconds, with wrapper exit 0 and cleanup complete. Public scope IDs and explicit project/agent sharing are unchanged. The final7478 head passed all required CI and fresh review gates. - The service/source/restore owner canary passed 25 cases across four suites before the final no-service fast-path guard (41.33 seconds). This includes actual project service processes with a different project ID and no execution-workspace ID. Their startup waits for source ownership, migration stays blocked while they run, and the source becomes eligible after their verified stop. The final three-case runtime canary passed at the final source inputs (18.70 seconds). It covers that fast path and both service-CWD variants. Both wrappers exited 0 and completed cleanup. - The boundary-repair canary passed 9 cases across four suites at `7c18f1c170`. It covers the real rehome/conversion gateway, project command provisioning, future task selection, stale imported authority, foreign-company binding, and existing standalone filesystem callers. The shared PostgreSQL wrapper exited 0 and completed cleanup. Duration: 94.15 seconds. - The P1 diagnostic passed 46 cases across five suites at the pre-rebase P1 revision. It covers the actual importer with an export warning created by `recordNativeFinalizationFailure`, strict prepared-source/action bindings, finalizer locks, source/lifecycle races, unrelated remote execution, and bare Git with external alternates. It uses the repository shared PostgreSQL wrapper, `--fsModuleCache`, and a local-only 90-second hook allowance. Duration: 245.44 seconds. No product timeout changed. - The canonical retained-directory compatibility check passed 22 cases across two suites. Missing archived task directories still import from authenticated restore points; they do not need a present-directory source proof. - Both initial P1 regressions were observed before the fix. The real finalizer warning made the importer ineligible. The old archive owner accepted a nested bare repository borrowing external objects. The new cases pass and preserve the original warning, prepared descriptor, allocations and files. - Prior attempts remain non-passes: the first 23-case canary passed on an earlier head; its exact-head rerun hit 30-second database setup/cleanup limits. A read of the owned PostgreSQL server found a template clone waiting on `DataFileExtend` for 58 seconds. A broader owner run was intentionally interrupted during slow existing heartbeat setup. One diagnostic invocation used an invalid CLI option and executed no tests. A later review canary passed 21 cases and failed the new real restore contention case: cwd-only admission missed the declared project of an external worktree. Its fixture also omitted the required file-repository authorization callback and produced an unhandled rejection after the failed assertion. The repaired run passed 22 cases, exited 0 and completed cleanup. These earlier failures are not reported as passes. - The recovery owner passed 37 PostgreSQL cases, including 14 new and 23 existing finalized-recovery cases. The operation reconciliation owner passed 40 cases. The initial archive run passed 103 cases and found a missing closed diagnostic entry; the corrected diagnostic and expanded Git rejection cases pass in the P1 run. - The full server type graph passes with the Runner source exports mapped locally because this checkout has no built Runner declarations. The latest module-boundary check and `git diff --check` pass. Standard package build/typecheck and broad final-head regression coverage remain required in CI. - The initial CI exposed missing admission scopes for standalone filesystem APIs and a changed foreign-company error. The latest repair keeps those APIs independent of a database and holds physical fences at their actual database-backed mutation owners. The branch also includes the existing master fix for unchanged Runner default-model assertions. New review findings and new-head CI are being checked. - The local canary command and bounded source selection are documented in `doc/plans/2026-11-06-online-ordinary-source-fence.md`. Source observations only read data. A fresh bounded metadata probe found 254 pending legacy records and zero unsettled local-process services. No live import, hold, cancellation, provider action or source filesystem write was performed for this change. ## Risks - Run-scoped reuse now includes execution-workspace ownership in its private key. Explicit project/agent sharing keeps its previous identity. - A missing writer owner could make a source proof unsafe. Admission derives physical aliases, includes future project selection and retains all existing source/lifecycle/finalizer locks. Unknown and changed state fail closed. - Cross-company physical alias checks use bounded minimal-path pages. Lexical-prefix filtering cannot safely replace them because a symlink can hide a foreign alias. Unknown paths or more than 10,240 foreign records leave the source explicitly unproven. More than 1,000 same-company local records also fail closed. - The project key can serialize local preparation within one project. A first project waits; additional project acquisition fails on contention to avoid a lock cycle. Remote authority keeps its remote preparation path. - Recovery preservation does not prove provider termination or successful file recovery. Existing provider allocations, checkpoints, task intent and recovery actions remain authoritative. - This slice does not finish shared Git migration. Git administrative/common-object/alternates closure and exact nested imported-child coverage remain the next owned extension. No global writers-stopped shortcut is used online. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 18:42:45 -06:00
DottaandPaperclip	72f7b42fe3	Preserve settled failed controller state during online migration (#19678 ) ## Thinking Path > - Paperclip controls agent work across web-server releases. > - Volume-free servers need retained controller state in shared storage. > - Successful settled sources already support online migration. > - Some retained failed sources have no accepted result and must stay available for recovery inspection. > - This change copies those sources under task, coordinator and publication fences while unrelated work continues. > - The original files, failed status and recovery warnings remain unchanged. ## Linked Issues or Issue Description Refs #19038. Refs #19631. What are you trying to improve? Preserve retained failed controller files without waiting for global idle. What problem does this solve? The online producer skips terminal failed controllers. Their files can still be needed for recovery inspection even when no accepted provider artifact proves executable continuation. What would the improvement look like? A settled exact source can enter the existing encrypted archive store. An accepted committed failed source can use the existing checkpoint import. Busy, changed, unowned or foreign sources remain blocked. ## What Changed - Add a source owner bound to the selected task session, run, coordinator generation, admitted input and file digests. - Check the task, physical finalizer lock, released coordinator, canonical resume source and active related turns before upload and before publication. - Preserve failed sources through the existing encrypted archive format. Do not treat archived bytes as execution authority or old-writer stop proof. - Allow the existing accepted committed checkpoint import for a finished failed run without changing its status. - Validate admitted input bindings before looking up local controller files. - Document the online preservation path and the final static writer fence. ## Verification - Real PostgreSQL: 125 cases passed across migration, controller recovery and durable backup storage. One added assertion inspected an unadmitted queued fixture; it was corrected while retaining the queued-turn preservation and stale-publication assertions. The final migration rerun passes 51/51; the other three affected suites pass 75/75. - Genuine prepatch RED: online failed-source preservation and accepted committed failed-source import both fail on the previous producer and recovery owners. - New cases cover independent restore after deleting the original source, an expired but non-null coordinator owner, source mutation after upload, a new related queued turn during upload, foreign input binding and an accepted committed failed result. - The full rebased server source and test-tree typecheck passes after restoring its missing ignored workspace SDK link and building that existing dependency. - Module boundaries and whitespace checks pass. - Rebased onto current master; both provider-artifact imports and the new archive owner are preserved. Rebased type and module-boundary checks pass. All six archive cases pass on the exact rebased head (39.68 seconds). All CI gates pass on exact head `81fc974cd9`. Fresh Greptile review is 5/5 with no unresolved comments. No serving migration, deletion or deployment is performed by this PR. - Review memory fix: 64 KiB streamed hashes avoid retaining another controller string at each publication fence. A 256 MiB source is hashed three times while the original string remains live under a 512 MiB heap. The old full-string comparison reproduces an OOM; the fixed two-case suite passes. Full source/test-tree typecheck and module boundaries pass on the new code. All six real-PostgreSQL archive cases pass again with streaming enabled (17.31 seconds); ownership, queued-turn and post-upload mutation guards remain covered. ## Risks - This is a copy-only preservation operation. It does not prove the original process has stopped. Final enrollment must arm static ephemeral mode on the original volume, observe the old deployment stopped and repeat data eligibility. - A source change during upload leaves an unpublished pending object. The normal retention owner handles obsolete attempts; the source remains required until a verified current archive exists. - An archive is forensic evidence. It does not make an unaccepted failed provider result resumable. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The host does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 18:38:50 -06:00
DottaandPaperclip	6be71028ad	Prepare online ephemeral enrollment through signed Cloud control (#19660 ) ## Thinking Path > - Paperclip manages agent work through a control-plane web server. > - A replacement web server needs shared durable data and the original signing identity. > - Existing enrollment exports that identity only after a global drain and offline migration. > - Online enrollment needs a separate preparation operation while agent work continues. > - This change adds a purpose-bound Cloud control operation with existing data and key owners. > - The result is an audited preparation receipt, with no permission to stop agents or detach storage. ## Linked Issues or Issue Description Refs #19038. Refs #19024. What are you trying to improve? Allow Cloud to prepare an already migrated tenant for volume removal without the offline migrator or a global-idle gate. What problem does this solve? Cloud must preserve the original decision key before starting a replacement with static ephemeral mode. A tenant session must never gain key-export authority. What would the improvement look like? A fresh exact-action Cloud assertion obtains a private, deployment-bound preparation receipt. Unfinished durable-data migration and invalid replacement configuration block export. ## What Changed - Add the signed `ephemeral-enrollment:prepare` control action and a strict UUID-only POST route. - Check existing durable-data eligibility and ephemeral configuration before reading the original key. Never create a missing key. - Audit only public preparation metadata. Keep HTTP diagnostics secret-sensitive and the response non-cacheable. - Bind the receipt to the attempt, stack, build commit, Railway deployment, time, static mode and key digest. - Always return `enrollmentReady: false`. Cloud still needs to arm a replacement, verify old writers have stopped, repeat eligibility and qualify native handoff before detaching the retained volume. ## Verification - PASS: 80 focused tests across the preparation owner, signed route, existing Cloud control, HTTP privacy and OpenAPI route coverage. - PASS: 2 real PostgreSQL cases. Concurrent copies of one signed request export once; the durable receipt still rejects reuse after clearing the process fence. Normal run admission before skill pins is allowed; missing durable skill installation remains blocked. - Genuine prepatch RED: the same two PostgreSQL cases exported twice and rejected ordinary admission. The fixed owner was restored automatically after the comparison. - PASS: server TypeScript check with workspace dependencies built. - Fixed the first CI route-coverage failure by declaring this private control endpoint in the existing explicit OpenAPI exclusion registry. - Initial local HTTP execution lacked sandbox socket permission; the reviewed run passed. Initial PostgreSQL fixtures enabled durable files too late and reused an issue prefix; those fixture defects were corrected before the red/green comparison. - All required CI gates pass on `5346721847`. Fresh Greptile review is 5/5 and all review threads are resolved. No deployment, volume detachment, admission change or agent stop is performed by this PR. ## Risks - The private response contains the original signing key. Only a fresh stack-bound assertion for this exact action can obtain it. The Cloud consumer must pass it directly to its existing vault owner and must not expose it through admin responses or registry metadata. - Eligibility is an observation. It cannot replace static writer restrictions or qualified ownership handoff. This PR deliberately supplies no detach or readiness authorization. - Cloud transition integration is still required. This PR alone does not enroll an instance. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution and browser tools. The host does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 18:00:28 -06:00
DottaandPaperclip	349b8b3f7b	Migrate settled controller checkpoints while unrelated work runs (#19631 ) ## Thinking Path > - Paperclip manages work performed by AI agents. > - Server replacement must preserve retained Runner sessions. > - Older sessions can retain controller checkpoints on the server volume. > - The existing importer requires all execution writers to stop, including unrelated runs. > - This change adds a separate online path for successful, settled sources. > - It locks the source task and finalizer while it reads and publishes that source. > - Unrelated work continues. The offline importer keeps its existing authority. ## Linked Issues or Issue Description What happened? A retained controller checkpoint cannot be converted by the existing maintenance importer while another task runs. Requiring a busy server to become globally idle prevents online migration. Expected behavior An operator can convert a successful retained source after its finalizer releases ownership, while unrelated tasks continue. An expired owner remains an owner. Related active work and uncertain evidence remain blocked. Steps to reproduce 1. Retain a valid legacy controller for a successful run with an accepted result and a released committed finalizer. 2. Start a different task. 3. Run controller preparation. The old importer refuses because execution writers are active. Related preservation contracts: #19038 and #19024. This change does not replace workspace migration or authorize volume removal. ## What Changed - Add `prestageNativeControllers` with task, finalizer, and source-run row locks. Recheck related runs and exact checkpoint identity under those locks. - Allow legacy checkpoint material through normal settled-owner import checks. Keep offline authority separate. - Compare captured local controller and supervisor bytes before commit. A changed source rolls back publication. - Add `--controllers-only` to the existing finite native preparation entry. This avoids re-reading provider archives during controller conversion. - Add PostgreSQL cases for unrelated work, expired owners, related active work, source-lock contention, and source changes. - Document the finite command and its limits. ## Verification - Real PostgreSQL affected suites: 56 passed across controller migration and recovery. A first per-server run was interrupted because it was slow; the completed run used the repository's shared PostgreSQL test wrapper. - Initial online cases: 3 passed, covering unrelated work, an expired owner, and related active work. - Source lock and changed-file cases: 2 passed. Publication rolled back on a source change. - Final source-evidence regression: 2 passed. A first unproven source deferred and a later valid source imported; changed-file rollback remained intact. - Server TypeScript check passed with the SDK and Runner declarations built. - Server package build passed, including the Runner vendor checks and compiled command. - Runtime bundling suite: 10 passed. - Full repository CI and exact-head automated review remain required before merge. This draft is not staging acceptance. - PASS: 7 CLI cases after adding the controller-only dependency fixture and explicit-mode coverage. The first CI run exposed missing mock/export coverage; the fixture repair passed after rebase. - The unrelated model-default failures were fixed on master; this branch now includes that fix. ## Risks - A source task can wait briefly on its own migration lock. A one-second lock acquisition budget defers a contended source. No global admission hold is acquired. - Only successful sources with a released committed finalizer are eligible. Failed or unsettled sources stay deferred and require their preservation owners. - The operator must run the command on the host that owns the original files. The command does not certify enrollment readiness or detach any volume. - The existing offline path remains unchanged and still requires stopped writers. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 17:19:23 -06:00
DottaandPaperclip	124b2cd0ad	fix(server): certify background deadlines for idle tenants (#19001 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Background services own future checks, retries, and recovery work. > - An idle tenant can stop only when a durable wake covers its remaining duties. > - The current alarm catalog covers several domains but leaves these owners blocked. > - A saved future timestamp alone cannot prove that reconstruction or admitted work has settled. > - This pull request binds durable domain snapshots to the actual registered worker revisions. > - The tenant can hand off supported future duties and retain all checks for uncertain or active work. ## Linked Issues or Issue Description No matching public issue was found. Related work: [domain deadline handoff #18656](https://github.com/paperclipai/paperclip/pull/18656), [owned blocker diagnostics #18632](https://github.com/paperclipai/paperclip/pull/18632), and [completed history proof #18646](https://github.com/paperclipai/paperclip/pull/18646). The separate [native idle-session change #18988](https://github.com/paperclipai/paperclip/pull/18988) covers runtime retirement. [Far-future retry re-probes #13534](https://github.com/paperclipai/paperclip/pull/13534) changes recovery responsiveness after an upstream success; this change preserves existing retry decisions. What happened? Future status-card, external-object, skill-sync, watchdog, retry, and quota duties lacked complete owner-bound coverage in the idle deadline proof. A completed quota task could also retain its monitor policy and timestamp and remain blocked after the scheduler had disarmed it. Expected behavior A settled registered owner can hand off its exact supported future deadline. Active work, dirty reconstruction, transaction uncertainty, changed revisions, claims, unknown state, and unsupported duties keep the tenant awake. Completed history retains its data and stops recurring work when the owner has no remaining duty. Steps to reproduce 1. Produce a supported future duty through its normal service. 2. Let the registered worker finish reconstruction and admitted work. 3. Request an owned idle report with the negotiated domain deadline catalog. 4. Check the saved earliest deadline and generation before durable alarm registration. 5. Stop and reconstruct the worker. Check that ordinary admission and claims resume useful work once due. 6. Complete a quota task. Keep its saved policy and timestamp. Check that linked unfinished work still blocks sleep and that settled history does not. Paperclip version or commit The owned patch was qualified at `699c8569fa34338dd75cdc5731381ec1e2efdb45`. Publication head `491fac368ee72eaac8ae105d53e6bf6b305d8a6a` normally merges actual master `039e0dad2c51c80251e1302a19c5566e039c357a`. All 43 owned files are byte-identical. All 29 new upstream paths are preserved. The later normal current-master integration is `8c912e1ba4fc66397a7341ade59373af8075068f`, described separately below. Deployment mode Source checkout with real PostgreSQL 17 and a controlled local Node process. No deployed alarm, tenant suspension, database suspension, or fleet result is claimed. ## What Changed - Add registered-worker reconciliation revisions and invalidate them on stop. - Keep status-card, external-object, and skill deadline interpretation with their owners. - Reuse the watchdog classifier against the supplied database snapshot. - Validate the exact saved retry run and paired queued wake. - Validate legacy quota source bindings and the current assignee or reviewer policy. - Preserve common issue scope, dependency, sibling, and state checks for covered waits. - Compose normalized owner snapshots within the existing bounded deadline proof and recheck revisions after inspection. - Bind actual service and worker instances from server startup. Missing bindings preserve existing blockers. - Retain completed quota policy and timestamps while independently rejecting linked unfinished runs or wakes. - Block unconsumed budget evaluation inputs through the owning queue predicate. Accounting policy and settlement stay unchanged. - Bind completed quota history through fixed-size UUID-array parameters, preserve exact company/task/source associations, and guard external references before the run-ID join. - Batch watchdog subtrees and shared classifier facts without changing the 64-query or two-second proof budget; preserve exact company pairing for first-run evidence. - Extend production-path regression controls and document supported coverage and limits. ## Verification Current integration head: `688abc6cac6016b9fb50e5832f81e2975ed5d49f`. Fresh exact-head checks completed with 58 successes, three intentional skips and one neutral review, including aggregate `ci / verify`. The complete current Greptile review is 5/5. The complete current Ponytail verdict is Ship (4/5), with only optional cleanup of an unused MCP proof input; both prior correctness findings are resolved. All review threads and nested pages were inspected. The normal master merges preserve the registered retry deadline owner and current startup completion log after the same readiness await. The earlier focused evidence below remains tied to its tested source revisions. Current-master integration `8c912e1ba4fc66397a7341ade59373af8075068f`: - Normally merge actual master `46fb933b0155d183ce64a8bdc4950d1b58c926f6`. Every nonoverlapping tree blob retains its owning parent across 14,044 paths. All 42 owner-only files and 15 critical binding/proof/readiness files remain byte-identical. Three automatic overlaps preserve the existing watchdog binding, conservative unavailable-reader fallback and optional actual owner arguments. The two scoped-deadline conflicts retain the entire upstream resource-reservation/mid-pass admission/abort algorithm inside the existing reconciliation revision/running/failure fence, with both parent test sets. - Actual TypeScript 7.0.2 server noEmit passed in 9.434 seconds, normal source declaration layers in 12.102 seconds, and four selected test/helper/config layers extending the actual CI configuration in 3.167 seconds. No compiler option weakening, heap override, donor declarations or SDK replacement stub. - Focused PG0 controls passed: 102 scheduler/proof/owner controls in 16.451 seconds, including all 21 scoped-deadline cases; 121 startup controls; and all 107 unchanged instance-settings route cases in 6.456 seconds. The latter first hit a local prewire-guard listener-registration race. Only the ignored guard was corrected to register the same actual OS-bound server port immediately after native listen returns. Closed, unknown, foreign, offhost and PG destinations remain blocked. Original failure evidence is retained; product code and assertions were unchanged. - The exact current UI policy command now reports zero findings (0.362 seconds); its seven scanner controls pass. The earlier dependent PR used an old scanner/source baseline under a newly enforcing trusted workflow and reported 397 findings. Carrying the genuine current master resolves that mismatch without editing unrelated UI sites or relaxing the gate. - Every meaningful worker checked its executing PID/thread/preload hash and matching receipt. PG0 controls opened no PostgreSQL sockets; all owned processes exited. Original dependency-cache/DNS, ignored-harness syntax/type and guard failures remain separate from product results. - This is fresh semantic and focused runtime qualification, not a local whole-repository or current-head PostgreSQL acceptance rerun. The upstream workspace-busy parked-retry/terminal-holder notification path remains unchanged by this merge and needs its separate current-graph producer check. Historical six-case/scale results below stay tied to their measured sources. A matching controlled default-cadence follow-up is in #19191; no fresh current-head two-cycle or deployed suspension claim follows from this merge. - All 59 exact-head check contexts now pass (55 success, one neutral, three intentional skips), plus the legacy security status. The complete edited Greptile review is 5/5 for this exact commit, with all root/nested pages inspected and no unresolved threads. Ponytail stopped without completing its new turn; it supplied no new verdict or finding. Prior actual findings remain addressed. Earlier scores/checks below belong to their explicit earlier commits. On frozen clean merged source `699c8569fa`: - Real PostgreSQL acceptance: 6/6 passed,83.483 seconds wall. Actual retry producer creates the exact saved run/wake; stopped and reconstructed owner resumes through current admission, one durable start, real local process output and normal task completion. Includes cancellation, drain arrival, retained quota history with queued-run/wake negatives, watchdog review settlement and budget-input empty/pending/consumed control using the actual owner. - Affected coordinator, deadline, scoped-deadline, recovery and composed-proof controls: 78/78 in five files passed,32.114 seconds wall, worker 1/no file parallelism. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: passed,17.467 seconds. Native TypeScript 7.0.2 and Node 24.22.2. - PostgreSQL tests use one dedicated test database, matching fixture/admin URLs, pool limit four, one worker, and no embedded fallback. Every run recorded unchanged source hashes and a clean checkout. - The eight-second real-clock retry is a domain execution check. It is below the Cloud minimum wake lead and is not a positive accepted hosted sleep cycle. - The completed quota-history product failure and original fixture/protocol failures were preserved; the owning-reader fix passed a real PostgreSQL red/green regression before the fresh merged six-case pass. - After runtime completion, the dedicated database had zero other clients and no owned runtime processes. All three qualification process PIDs exited. The coordinator independently audits and drops the database. - Comment-only cleanup has equal strict TypeScript parser ASTs across 15 files. The final lexical audit finds no added comments relative to actual master. `git diff --check` passed. - Earlier retry fixtures omitted responsible-user membership, advanced JavaScript time past the real PostgreSQL lease clock, or produced stdout without normal task completion. Corrected fixtures use real membership, an eight-second real deadline, and ordinary task comment/completion. The last case exposed legitimate disposition repair rather than duplicate retry promotion. - The quota and watchdog controls first ran before admitted work or the existing 30-second reconstruction duty settled. The stronger tests require an interim blocked report and then exact settled-owner proof. Production deadlines were not changed. The separate completed-quota-history defect was fixed in its reader and verified with linked unfinished-run and wake controls. - The status/object and skill owners reviewed the actual final startup, route and post-backup bindings. Their domain source evidence carries where token-equivalent. The master coordinator coalescing change was reviewed separately. Historical domain PostgreSQL results are not presented as fresh full-suite results on this head. The later master merge adds retry predecessor cleanup checks under the existing issue claim lock. The source audit confirms unchanged saved deadlines and run/wake identities. The three actual retry lifecycle/cancellation/drain cases passed on the batched candidate with this parent (31.817 seconds). Those local dependency controls preceded the current exact-head remote run. Native backup preservation and TestOps changes from master are retained. No pass is inferred for changed dependency behavior solely from unchanged owned files. Review repair published as `643f1e7400`: - The original reader returned `unknown` after seven actual watchdog reviews settled across seven companies. The unchanged composed-report regression passed after batching; the final reader passed in 61.841 seconds, including active queued-work and stopped-owner rejection. - A separate two-company red/green control rejects terminal-run context from another company as first-run grace evidence. Shared facts are serialized once, with per-root IDs and review summaries. - The affected recovery, proof and diagnostic unit controls passed 47/47 (14.575 seconds). Startup owner-contract fixtures passed 121/121 (10.589 seconds). - Diagnostic PostgreSQL controls passed7/7 (12.406 seconds), including the normal budget-input producer and its exact blocker reason. Initial empty-state setup uses the same cleanup as per-case teardown when a dedicated test database is reused. - Actual repository `pnpm exec tsc --noEmit -p server/tsconfig.json` passed 12.710 seconds with no heap override. Final source has no added code comments; whitespace checks pass. - The original query-budget failure, two-company discriminator, initial cross-suite fixture failure and an interrupted incorrect local-driver selector are retained separately. The interrupted run is not a pass. Independent cleanup found zero owned clients or runtime processes. Quota-scale repair published as `6696a37f8a`: - The exact 10,000-row completed-history case reproduced `MAX_PARAMETERS_EXCEEDED`:90,002 bound parameters exceeded the driver's 65,534 limit, and the composed report returned `unknown` despite settled owners. Original failure and actual driver diagnostics are retained. - Fixed pending-work queries use three typed UUID-array parameters plus the result limit, keeping exact company/issue/source tuple bindings. The primary run-ID column stays uncast; a guarded CASE validates the raw external reference before conversion. The 10,000-row cap and 64-query/two-second proof budget are unchanged. - The same 10,000-row composed-report case passed 18.860 seconds, with unfinished run/wake rejection, malformed-reference safety and conservative 10,001-row overflow. Its fixture derives retained history from the actual quota producer and normal task completion, expands valid persisted issue/source pairs, then starts the actual registered owners. - The existing actual quota producer case passed 14.187 seconds, including exact future deadline, retained completion history, unfinished work, real foreign-company source references and noncanonical/malformed references. - Affected owner/proof controls passed 22/22 (26.444 seconds). Production server noEmit passed 16.039 seconds. - CI on 643 identified TS7034/TS7005 in the settled-watchdog fixture's initially untyped collection. It is now explicitly typed. A separate affected-entry semantic check extending the repository's actual `server/src/__tests__/tsconfig.json` passed 15.093 seconds: six affected tests, two owner files, ambient declarations and current imported source, with unchanged strict/module/lib/path settings. The production server config excludes that test directory and is not presented as test-layer proof. No custom suppression or replacement SDK stub was added. - All runs were sequential with pool max 4, worker 1, source/config/command hashes and cleared inherited provider/LIVE/other-database variables. Cleanup found zero owned database clients or local runtime processes. At prior head `6696a37f8a5718cd0f79054969a0084462d41d47`, all required CI gates pass. The exact-head inventory has 55 successes, one neutral review and three intentional skips. Greptile is 5/5. The complete latest Ponytail verdict confirms the quota finding is fixed; its only remaining note is optional unused MCP proof state, not a behavior defect. All three review threads are resolved. The earlier 643 result remains historical. The repository-wide typecheck, tests, build and browser gates passed in the prior `6696a37f8a5718cd0f79054969a0084462d41d47` CI run. The new `8c912e1ba4fc66397a7341ade59373af8075068f` full CI also passes. They were not run as a full local workspace command in the focused PostgreSQL lease. Local evidence above uses bounded external databases and does not claim embedded-cluster, live-provider or deployed-alarm qualification. ## Risks - An unsupported or uncertain owner state must remain blocked. An optimistic exemption could lose work while compute is stopped. - Changes during reconstruction or inspection must invalidate the proof. A memory-only timer is not durable alarm coverage. - Current authorization, budgets, cancellation, accounting, and resource ownership still apply on dispatch. The proof does not grant execution authority. - The supported writer model uses in-process transaction signals and startup/resume reconstruction. Independent SQL writers and other replicas need an explicit wake contract. - Native, review, coalesced, and taskless retry variants remain conservative until their owner cases qualify. Reviewer quota and hosted alarm qualification remain open. - The shared catalog remains opt-in. Legacy callers and missing owner bindings retain their old blockers. ## Model Used OpenAI `gpt-6.1-sol` was the configured model for the delegated service implementations and reviews. OpenAI Codex, based on GPT-6, performed managing review and integration. Both used repository tools and code execution. The exact served backend snapshots and context-window limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass on the final merged head - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — exact current-head run passes - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — complete current-head review and all paginated threads inspected - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 13:48:55 -06:00
DottaandClaude Opus 5.5	871a9b3180	test(restart-chaos): add Phase 1 control-plane restart chaos harness (#19477 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane owns task execution, recovery and Retry for every agent run > - Server restarts were the largest remaining source of stalled tasks in the staging soak campaign > - The soak could only observe restarts that staging happened to perform, so a failure could not be repeated > - This pull request adds a script harness that causes the restarts on purpose and grades each task on fixed rules > - The benefit is a repeatable restart failure with a replay command and an evidence bundle for each fixer ## Linked Issues or Issue Description What existing behavior does this improve? The restart test coverage. Today the runner-e2e restart cells test one fixed restart point per cell. They cannot run many tasks at once, cannot choose a random fault time, and cannot replay a failure. Subsystem affected Test tooling (`tests/restart-chaos/`). It reuses the runner-e2e process code in `tests/runner-e2e/`. Product code does not change. Current behavior Restart robustness is checked through a small set of fixed cells. A random restart during a real task has no defined pass rule. Proposed behavior A harness starts an isolated control plane and seeds a "Chaos Lab" company. It launches seeded workloads, injects K1, K2 or K4 faults at seeded times, and grades each task on rules R1 to R6 plus a deterministic task oracle. `--replay <wave>/<task>` repeats one task. Reason and benefit Fixers get a replay command and an evidence bundle for each failure signature. The first smoke wave already found three product signatures (dead Retry after a restart, a false over-budget stop after restarts, and an OpenCode turn killed mid-turn that ends as a provider failure). Breaking changes None. The change adds new test files, one entry in the test-project typecheck list and a gitignore entry. ## What Changed - Add `tests/restart-chaos/`: - `supervisor.ts` owns one isolated server. It reuses `replaceController`, the process-tree stopper, the environment isolation helpers, the config seed, the port reservation, the wrapper-owned Postgres and `server-entry.ts` from `tests/runner-e2e/`. It does not copy them. Faults: K1 (SIGKILL, immediate restart), K2 (SIGTERM, restart) and K4 (SIGKILL, 2 to 5 minute hold). Postgres and the controller runtime cache stay across a fault. - `seed.ts` creates the company, encrypted provider secrets, native Codex, ACPX Claude and OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project. The project is a git repository with a Python CLI, one failing test and a four-minute `bench.sh`. - `scheduler.ts` makes a seeded plan of workload, adapter, fault, fire time, hold and nonce for each task. It saves the plan so that `--replay` can run one task again. - `observer.ts` polls each issue and its runs. When a task stops and offers Retry, it sends `POST /api/issues/:id/retry` one time, with the same body as `useTaskRetry.ts`. - `grader.ts` grades R1 to R6 as pure functions. `workloads.ts` defines workloads A1, B1, B5 and C1 and their oracles. - `evidence.ts` and `ledger.ts` write an evidence bundle for each failure, a JSONL ledger and `SUMMARY.md`. All results go to the gitignored `tests/restart-chaos/results/`. Known provider key values are redacted. - Unit tests: `scheduler.test.ts` (determinism, replay and CLI parsing) and `grader.test.ts` (R1 to R6 and the oracles on fixture timelines). These tests need no credentials. - Add `tests/restart-chaos` to the test-project typecheck list (`scripts/typecheck-test-projects.mjs`), with a `tsconfig.layers.json` like `tests/runner-e2e`. The harness runs through direct script invocations, not `package.json` scripts. - After the first Greptile review: - A failed fault or an unexpected server exit now gets one bounded recovery. If the recovery fails, the wave fails and does not hang. - Incomplete cleanup is now a harness error, and the harness keeps the instance. - After the C1 follow-up comment, the observer needs a fresh rest window before it grades the task. - `--replay` now fires the recorded faults that crossed the task, at the same offsets, including the faults that other tasks requested. - A failed workspace-operation read is now a sweep error. It cannot give a clean rest. - R2 now checks that workspace files from before the first fault still exist at rest. - B5 now needs the completion record that `bench.sh` writes for each task. - After the second round of reviews: - An idle `todo` or `in_progress` task with no live run and no named wait is no longer treated as resting. The observer keeps watching it until the task resumes or passes the R5 stall limit. - A failed injected fault now fails the wave with exit 2. - T-oracle commands run agent-editable code. They now run under a macOS Seatbelt sandbox with a scratch `HOME` and no network, and they cannot read the operator's home directory. On other platforms these commands are refused unless the operator explicitly allows them. - R4 now counts every active run. A wake comment may be served again only after a run that did not succeed. - Evidence redaction now covers every environment variable name that ends in `KEY`, `TOKEN`, `SECRET` or `PASSWORD`. - If the pre-fault snapshot fails, only R2 for that task is left unchecked. The wave continues. - B5 also needs bench progress that the harness observed while it polled. - Add `doc/plans/2026-11-06-control-plane-restart-chaos.md` and its campaign ledger. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 tests pass. The tests include observer tests with a fake API, a real `bench.sh` run, and a sandbox test that cannot read a canary secret. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - `server/scripts/typecheck-layers.test.mjs`: 45 tests pass, including "covers every project under tests/". - A supervisor check with no credentials passed. K1 was ready again in about 6 seconds. K2 with a 3-second hold was ready again in about 9 seconds. Company data stayed readable. - One paid smoke wave ran: `--env local --adapters codex,claude,opencode --workloads A1,B1,C1 --faults K1 --tasks 3 --concurrency 2 --seed 7`. The harness worked from start to end, and all three tasks crossed a restart. The 0/3 result comes from product failures, which the campaign ledger records. A scan found no provider key in the results. - To run a wave: ```sh pnpm --filter @paperclipai/paperclip-runner build:runner-binaries set -a; source ~/.secrets; set +a node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude,opencode --concurrency 4 --tasks 8 --faults K1,K2,K4 --seed 42 node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay <wave>/<task> ``` ## Risks - Low product risk. The change adds test tooling only and does not change product code. - A wave spends provider money. The company budget hard cap (`--budget-cents`, default 3000) is the only spend limit. The plan's per-wave and per-night refusal is not implemented yet. - Known gaps: - `--env daytona` is reserved. It fails with a TODO in `seed.ts`. The Daytona sandbox sweep for R6 is not implemented yet. - The project requests isolated git-worktree workspaces. A B1 task still ran in the shared fixture repository. Concurrent B1 and B5 tasks can share files. The oracles use marker files for each task. - Only K1, K2 and K4 at seeded random times are available. Boundary triggers (B1 to B16), other faults and the UI probe come in Phase 2. - The paid smoke did not exercise B5 or K4. - On macOS, B1 oracle commands need `python3` outside the operator's home directory, for example `/usr/bin` or Homebrew. On Linux they are refused until a sandbox is added. - The B5 proof is best-effort against a deliberately adversarial agent, because that agent can write to the workspace. - Startup phase timings come from polling `/api/ready`. The server records its own timings only as OpenTelemetry spans. - `pnpm verify` and the CI Vitest jobs do not select these unit tests. The root Vitest projects cover workspace packages only. CI typechecks the files, but the tests run only through the vitest command above. - After the smoke, rest detection and the review fixes changed. A recovery action that the board owns now counts as a rest, and R6 reports it. A paid wave did not run again after these changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code as an agent with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 10:45:52 -06:00
DottaandPaperclip	c0214826b3	fix(workspaces): preserve pending repository requests during import (#19439 ) ## Thinking Path > - Paperclip manages agent work and its saved files. > - An ephemeral web server needs remote workspace authority before the old volume can detach. > - A task can have a repository request that has never prepared a checkout. > - The offline importer currently treats that pending request as unfinished file publication. > - This blocks migration even when the target does not exist and there is no saved commit, branch or failure. > - This pull request preserves that request in the database and imports the existing files separately. > - The next authorized sandbox preparation can acquire the repository through its existing owner. ## Linked Issues or Issue Description What happened? On Core `253bf071f0a496554b61fff897ad3f11afaf8766`, offline migration of a volume-backed workspace fails with `repository_preparation_unresolved` for an untouched pending repository request. The requested directory has never existed. The request has no pinned commit, branch or failure. Expected behavior Preserve the pending request and all existing files. Do not label the absent checkout as part of a saved archive. Keep partial publication and active preparation blocked. Steps to reproduce Create a local task workspace. Add an unmaterialized pending repository request. Run the canonical offline importer after stopping writers. The unchanged importer rejects the request before preserving the workspace. Related migration work: #19429. A GitHub search found no duplicate pending-repository import fix or matching public issue. ## What Changed - Admit an untouched pending request only at the existing offline writer fence and after proving its target is absent. - Keep the database request unchanged and exclude it from materialized snapshot descriptors. - Recheck file absence and source identity before publication. Reject a request that becomes ready during capture. - Cover cold filesystem loss, real sandbox-side Git acquisition, and eleven blocked states. Document the preservation contract. ## Verification - Red: the new real PostgreSQL/filesystem regression fails on the unchanged importer with `repository_preparation_unresolved`. - Green: both affected migration/lifecycle suites pass, 236 tests. The cold test deletes the predecessor directory, restores saved bytes, and acquires the pending repository through the full admission owner, checking ready state and the pinned commit. Real Git runs against a local HTTP origin through an exact-URL redirect at the sandbox transport boundary. It does not claim live Daytona or GitHub qualification. - Full workspace typecheck and build pass at the final head. Both affected suites pass 236 tests, including the full admission owner and eleven blocked states. All final-head CI checks pass: 54 success, 2 skipped, 1 neutral. Greptile is 5/5, with no unresolved review threads. - The full local test command failed in `recovers lost inline and check responses without duplicate GitHub publications` in the chat integration suite. That exact case passes in a fresh isolated invocation. Its cause remains unclassified; this is not a full-local-suite pass claim. The original failure is retained alongside the isolated result. - No candidate code has been applied to the serving instance. ## Risks - The exception applies only to offline migration and a pending request with no target, commit, branch or failure. Unreadable paths, symlinks, partial directories and active/repair states retain their blockers. - Requests retain normal repository authorization and acquisition behavior. This change neither fetches repositories during migration nor marks requests ready. - A changed materialized source aborts publication and leaves original files intact. ## Model Used OpenAI GPT-6 through Codex, with reasoning and tool use. The exact deployment variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 05:49:16 -06:00
DottaandPaperclip	253bf071f0	fix(workspaces): preserve resolved pending merges in snapshots (#19429 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task workspaces must survive replacement of the web server. > - Workspace snapshots therefore preserve both files and portable Git state. > - A resolved merge can still await its merge commit. > - The archive rejected this state, so an existing workspace could not migrate. > - This pull request saves and checks the complete supported merge state. > - The benefit is that migration preserves the user's choice to continue or abort the merge. ## Linked Issues or Issue Description What happened? A legacy workspace contained a resolved merge with `MERGE_HEAD`, `MERGE_MODE`, `MERGE_MSG`, `ORIG_HEAD`, and `AUTO_MERGE`. Its index had no unresolved entries. The snapshot owner refused the operation. This kept the original safe but blocked migration. Expected behavior Save this supported state without changing the source. Restore it without the original repository, borrowed object store, or linked worktree. Retain the merge parents, selected index, later edits, and Git's normal safety checks. Steps to reproduce 1. Merge two branches that conflict. 2. Resolve and stage the conflict. Leave the merge uncommitted. 3. Capture the workspace. The base implementation rejects the merge marker. Related prior work: #19310 preserves an inactive rebase bookmark. No duplicate open merge-preservation PR was found. This is a workspace preservation fix, not a new roadmap feature. ## What Changed - Admit only resolved pending merges with valid, bounded metadata and available objects. - Retain and verify the merge mode, message, parents, original head, automatic merge tree, and index. - Verify copied metadata and object closure before publication. Keep the before/after source checks. - Keep unresolved conflicts, autostash, rerere, mixed operations, malformed state, and unsafe links blocked. - Test real ordinary, linked, and borrowed repositories through local and serialized capture. Delete all original sources before continuing or aborting the restored merge. - Document the supported state and the remaining refusal cases. ## Verification - Six new continuation cases fail against the unchanged base at the pending-operation guard. - The first candidate restored and continued the merges, but six abort assertions failed. The fixture rewrote the worktree without updating Git's index cache. It now first verifies that abort protects later edits, uses normal `git restore` to discard those fixture edits, then aborts. No archive assertion was removed. - The final archive and environment suites pass all 98 tests at `3c87d13d1b`, including linked-metadata protections and pending merges in ordinary, linked, borrowed, shallow, partial, and SHA-256 repositories. All 12 local/serialized continuation cases pass after source removal. - Full repository build and typecheck pass at `1544cc4a90`. The build first found a nullability error; an explicit rejection return fixed it. - The first full local test attempt stopped before tests ran because omitted package postinstall steps left PostgreSQL library links absent. Direct `initdb --version` reproduced the missing-library failure. Restoring the package links made that probe pass; the full test rerun is active. - The initial typecheck also found missing excluded-plugin SDK links after the same install. Running the repository postinstall link helper corrected this, and the full typecheck now passes. - All current-head CI gates pass at `3c87d13d1b`: 54 success, 2 skipped, 1 neutral. Fresh Greptile is 5/5 and no review threads remain open. - Review requested shallow and SHA-256 coverage. The current commit adds both plus partial-clone coverage; all 98 archive tests and the full server typecheck pass. - All six local chat shards pass. The monolithic local general-suite run reports one viewer-permission test failure and remains active. That exact test passes alone, and its full 505-test file also passes alone. The original failure cause is unresolved; this is not a full local-suite PASS. The complete current-head CI suite is green. ## Risks - This widens capture eligibility only for the checked resolved-merge state. No migration commits or aborts the source operation. - A missing object or changed metadata still blocks publication and retains the source. - Unresolved merges and other operation formats remain unsupported. Their preservation guard is unchanged. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, and code execution. The runtime does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with the available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task id - [x] I have run the affected tests locally and they pass; full-suite limitations are recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented the risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 04:20:32 -06:00
DottaandPaperclip	c1d567ff6a	fix(workspaces): isolate archived nested Git reconstruction (#19409 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace import must preserve archived task files and Git history. > - A saved nested checkout can be a linked worktree whose branch belongs to its old absolute path. > - Archived import reconstructs that checkout in a new scratch directory. > - Ordinary linked recovery rejects the old branch at that new path. It can also write into a cache used by other import workers. > - This pull request reconstructs archived nested checkouts as private standalone repositories. > - Each import preserves saved files and extra history without changing shared worktree registrations or branches. ## Linked Issues or Issue Description What happened? Archived workspace import fails when a saved nested linked checkout has a branch owned by the old path. The new scratch path fails the linked-worktree ownership check. The existing cache lock is process-local, so separate import processes also share mutable Git state without a common lock. Expected behavior Archived import must create an independent portable copy of the saved checkout. It must retain the saved commit, branch name, staged edits, working files, ignored recovery files, settings, and extra saved commits. It must keep the source cache unchanged and enforce source authorization. Steps to reproduce 1. Create a managed nested linked checkout with an unpushed commit and staged and unstaged edits. 2. Save an encrypted restore point, including ignored files and an extra history tip. 3. Restore its saved contents under a different absolute workspace path. 4. Before this repair, the new path cannot adopt the branch owned by the old path. Paperclip version or commit The real Git regression fails before the repair at `fc1f5f3052`. The final branch includes the merged interrupted-upload and nested-source authorization fix from #19396. Deployment mode Archived remote workspace migration with real Git repositories, disposable PostgreSQL, and encrypted object storage. Local integration tests use the filesystem storage provider. Builds on merged #19402 and #19396. Related #19374 concerns interrupted clone staging; it does not implement private archived reconstruction. No duplicate open repair was found. ## What Changed - Enable private repository reconstruction only for archived migration. Ordinary linked recovery keeps its path ownership guard. - Add an object-only source to the managed checkout owner. A verified cache supplies only the saved commit and authenticated bundle prerequisites through scoped fetches. The private copy includes no unrelated task refs or objects, hard links, object alternates, or current cache files. - Fetch missing authenticated bundle prerequisites before importing a saved bundle in both private reconstruction and ordinary linked recovery. A pruned cache can use the approved remote; unavailable history still blocks restore. - Keep every authenticated restore point's extra commit under a private ref in root and nested private copies so portable archive capture and garbage collection retain it. Ordinary recovery keeps its existing history and expiration rules. - Share the verified, bounded ignore-file copy between root and nested reconstruction. Sources remain unchanged. - Retain source authorization before and after reconstruction and the final source-row locks from #19396. - Test concurrent private copies, source rejection, foreign task refs and objects, ordinary linked prerequisite recovery, a real migration, and cold restore after removing all original repositories. Test hard links, oversized excludes, unsafe info paths, and replacement during a descriptor read. ## Verification - Final affected run: 292 tests pass across ten suites. These cover real Git repositories, encrypted restore points, disposable PostgreSQL migration, portable archives, the restore-point ledger, source rejection, and concurrent copies. - Full repository typecheck and build pass after the scoped-object repair. Final server typecheck and full build also pass after restricting preserved refs to archive mode. - RED then GREEN: a linked checkout at a different path; a pruned bundle prerequisite still available from the approved remote; foreign task refs and objects entering the archive; root extra history disappearing after cold restore; and archive-only refs blocking ordinary standalone expiration. - Two concurrent private copies retain the saved head, branch, staged and unstaged edits, ignored recovery bytes, settings, and modes. Source refs, index, worktree registrations, and working files remain unchanged. - Delete the original checkout, cache, remote, and private copies after portable capture. Cold restore retains the saved files and extra history after garbage collection. No alternates or unrelated task objects remain. - Ordinary linked recovery retains its path ownership check and recovers pruned prerequisites. New and existing ordinary standalone destinations have no archive-only refs and remain eligible for expiration after normal object cleanup. - Unsafe exclude tests reject hard links, oversized files, nonregular paths, info-directory substitution, and replacement during the descriptor read. They preserve the target and replacement source bytes. - Preserve the failed attempts: an initial migration fixture changed its home after constructing the source path. Later expiration fixture attempts used a filtered current-head tip, omitted the local-source callback, or used cleanup that generated Git server-info files or left packed unreachable objects. Correct those setup assumptions before the actual regressions fail. Final affected suites pass. - Two earlier local full-suite attempts were interrupted before editing valid review findings. Neither is reported as a full-suite PASS. Final-head full CI passes; review is 5/5 with no open findings. ## Risks Archived imports use additional private disk space for independent Git objects. Missing, changed, or unsafe sources still block import. The repair does not weaken the default linked-worktree ownership check, change stored formats, or update live provider state. The shared checkout addition is optional and has no effect for existing callers that do not supply an object source. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 00:56:06 -06:00
Dotta	92452524f4	fix(server): recover authenticated empty retention checkpoints (#19417 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Ephemeral server enrollment must preserve existing task files. > - Archived task directories can depend on encrypted retention checkpoints. > - A missing object blocks migration even when its authenticated contents were exactly empty. > - Metadata counters alone cannot prove that no saved files existed. > - This change verifies the exact candidate against the original encrypted key envelope before normal migration publication. > - The benefit is lossless migration of this bounded case without resetting task history or weakening ordinary restore. ## Linked Issues or Issue Description What happened? Migration stops at an archived empty directory when its retained encrypted object is missing. The original encrypted key envelope still authenticates the payload hash, byte length, company, workspace, checkpoint ID, and generation. Expected behavior Administrative migration can reconstruct those exact original bytes when storage confirms absence and every authenticated check passes. It must keep missing nonempty, expired, corrupt, or ambiguous sources blocked. Steps to reproduce 1. Capture and encrypt an empty task directory. 2. Retain its reference and encrypted key envelope. 3. Remove the old directory and storage object. 4. Run the normal remote workspace migration. Paperclip version or commit The regression fails before this change at `205ffcb606`. Related preservation work is in #19396, #19402, and #19409. No duplicate open fix was found. Deployment mode Administrative migration to external workspace storage. Tests use real encryption, real filesystem storage, and disposable PostgreSQL. ## What Changed - Let the encryption owner verify a candidate against the existing scoped encrypted key envelope. Normal ciphertext validation keeps its previous order and checks. - Share the task-directory encoder with an exact empty payload constructor. - Add a default-off loader option for confirmed missing empty checkpoints. Only the administrative migration caller enables it. - Keep the original checkpoint reference and object unchanged. Publish the recovered bytes through the existing snapshot and source-fence owners. - Reject forged counters, wrong identities, missing nonempty data, present corruption, unreadable present objects, unknown absence, and expiry. ## Verification - Final affected run: all 238 tests pass across five crypto, snapshot, retention, and migration suites. Full repository typecheck and build pass; server typecheck and full build also pass after the cancellation repair. - RED then GREEN: exact missing-empty recovery and cancellation of a stalled HEAD through the real storage-service boundary. Cancellation retains the caller's original abort reason. - Disposable PostgreSQL migration publishes a cold-restorable empty archive. Archived status, task status, task binding, original reference, and the absent original source stay unchanged. Replay keeps one canonical snapshot. - Negative cases reject forged empty counters, an authenticated directory or file, every foreign identity field, expiry, present corruption, unreadable present objects, unknown absence, and missing nonempty data. Ordinary restore keeps its previous failure behavior. - Initial verification retained one new fixture-path failure after 145 passes, and a typecheck rejected the fixture's widened schema literal. Correct the fixture to use the returned staging path and its explicit reference type. The final complete affected run passes. - The new HEAD request runs only after a failed read with the administrative option enabled. It uses the existing cancellation contract and adds no polling or background work. - Final-head full CI passes; review is 5/5 with no open findings. This PR does not claim deployed enrollment or a live migration result. ## Risks Recovery requires the original valid encrypted key envelope and an exact format match. A format change or unavailable key continues to block migration. Existing ciphertext hash, size, authentication, and ordinary restore behavior remain enforced. This change has no schema, provider-session, API, or UI contract changes. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-06 00:50:28 -06:00
DottaandClaude Opus 5.5	958f9aa9f3	feat(issues): limit agent-created task trees to parent and child (#19312 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents split work into tasks. A task can have a parent task. > - Today agents can nest tasks to any depth: parent, child, grandchild, and deeper. > - Deep agent-made trees are hard to follow, and an agent that works on a child task makes grandchildren by default. > - This pull request limits agent-created trees to two levels: a top-level task and its children. > - A request for a grandchild does not fail. The server puts the new task beside the requester and tells the agent. > - Board users keep full control. They can still nest to any depth. Existing deep trees stay valid. > - The pull request also records which task created each new task, so the audit trail stays complete when the parent changes. ## Linked Issues or Issue Description No public issue exists. This is the feature description. Related: #16853 (budgets that include descendant work) also reasons about task trees; this change does not alter its traversal. Subsystem affected Issues (tasks): create, child create and move in the issue service and routes. Agent tools, chat suggestions and creation provenance use these paths. Problem or motivation Agents create subtasks as deep as they want. An agent that runs on a child task gets that child as the default parent, so its follow-ups become grandchildren. The trees become deep and hard to read. The parent link is also the only record of which task created a follow-up, so that record is lost when the parent changes. Proposed solution Limit agent-created trees to a parent and its children. When an agent asks for a grandchild, put the new task under the requester's parent and return `parentAdjusted`. Do not reject the request. Keep board users unrestricted. Record the originating task in a new server-set `origin_issue_id` column. Keep `origin_run_id` as the originating run. Alternatives considered Reject grandchild requests with an error. We did not do this, because agents then fail mid-task. Add a database constraint. We did not do this, because existing deep trees must stay valid and nesting can return later. ## What Changed - Add `MAX_AGENT_TASK_HIERARCHY_DEPTH` (2) and the `IssueParentAdjustment` type in `packages/shared`. - Add `resolveNewTaskParent` and `planIssueParentChange` in `server/src/services/issues.ts`. `create`, `createChild` and `update` use the agent limit by default (`parentDepthPolicy: "flatten"`). - The issue create, `/children` and PATCH routes pass `allow_nested` for board users. For agents, the routes check authorization against the parent that the server actually writes. - The requester keeps its behavior when the server moves a task beside it: `blockParentUntilDone` blocks the requester, and the new task inherits the requester's workspace and privacy. - The server never moves a task above a low-trust boundary root that contains the requester. - These subsystems keep any depth, because they find their work by parent: watchdog reviews, monitor recovery, email-channel placement, and routines that a person configured. - An agent move of a task that has children skips only the parent change and returns `reason: "has_children"`. - Add `issues.origin_issue_id` (nullable, `on delete set null`, indexed) in migration `0434`. Only the server sets it. The `createdFrom` detail field, the `createdFromIssueId` filter and task-creation provenance read it first. - Chat prompts now let a chat agent create one task, a task with children, or separate tasks. The conversation is never a parent. - The runner `create_task` tool accepts an optional `parentTaskId`. A chat agent can create a top-level task and then put subtasks under it. The run must be able to read and change the parent. A conversation is never a parent. The depth limit still applies. Project tools map the field to `parentId` through the issue route. The protocol manifest, semantic contracts and eval fixture are regenerated. - Watchdog runs that have an active watchdog keep their requested parent on all create paths, because their access is limited to the watched subtree. - Fix a bug that the test drive found: the chat (MCP) `create_task` ignored the requested `status`, so a `backlog` task became `todo` and started to run. It now uses the same rule as the native tool. - Update `skills/paperclip`, `doc/SPEC-implementation.md`, and add the plan `doc/plans/2026-11-05-two-level-task-hierarchy.md`. ## Verification - `cd server && npx vitest run src/__tests__/agent-task-hierarchy-depth.test.ts src/__tests__/agent-task-hierarchy-depth-routes.test.ts`: 19 new tests for service and route behavior, all pass. - Related suites (64 files: issues service, thread interactions, privacy, watchdogs, provenance, runner tool authority, decisions, routines, project tools): 1208 tests pass. Two tests changed because of the intended behavior and pass now. Two `email-channels.integration` tests fail locally because the test cannot find a bundled skill file from this worktree path. This change does not touch that code. - `packages/db`: migration safety, snapshot drift and journal tests pass. - Typecheck passes for `server`, `packages/shared`, `packages/db`, `cli` and `ui` (local `ui` errors come only from generated Storybook catalog files that are absent). - Local test drive on an isolated instance with real Claude agents: 29 of 29 use cases pass, including an agent on a child task (follow-ups placed beside it, with the notice) and a chat that creates a top-level task with three subtasks. - I did not run the full `pnpm test:run` or `pnpm build` locally. CI runs them. ## Risks - Behavior change for agents: some follow-ups that were grandchildren are now siblings. The parent of the requester is woken when all its children finish, which now includes these tasks. - The runner `create_task` input contract adds one optional field, so the protocol manifest and the pack contract change. The tool output does not change: it shows the actual `parentId`, and the activity log records `parentAdjusted`. - Migration `0434` adds a nullable column, a foreign key and an index. It does not backfill rows. - `IssueCreationSource.run` is now nullable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 21:22:33 -06:00
TonioandClaude Opus 5.5	9e9a628a57	feat(ui): turn Artifacts V1 on for every instance and hide its toggle (#19164 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Artifacts show the output of agent work on the Artifacts page, in the panel Artifacts tab, and in the task thread > - #17228 added the Artifacts V1 presentation behind an experimental setting that is off by default > - The presentation is now ready for all users, so an opt-in toggle is no longer necessary > - A new default alone does not reach all instances, because each experimental write persists every flag > - This pull request removes the toggle and turns V1 on for every instance > - The benefit is one artifact presentation for all users, with an off switch for hosting operators ## Linked Issues or Issue Description Refs #17228 Refs #17263 Is your feature request related to a problem? Please describe. Artifacts V1 shows only on instances where an admin turned on an experimental setting. Other users see the previous cards. Describe the solution you would like Show Artifacts V1 on all instances. Remove the toggle from Experimental settings. Describe alternatives you have considered A new default without a change to the hook. This does not work, because many instances have a stored false that nobody chose, and no toggle would remain to change it. ## What Changed - Removed the Artifacts V1 card from Instance Settings, Experimental. - Changed `enableArtifactsV1` to default on in the schema, the feature catalog, and the settings service. - Changed `resolveArtifactsV1Enabled` to ignore a stored false. It honours false only when the value comes from managed config. - Kept the flag and the previous cards, so a hosting operator can hold V1 off. - Kept the key in the Cloud staging list, so a managed false does not hold V1 off on staging. - Updated unit tests, the Artifacts V1 plan, and the directions-lab Today reference. - Moved the attachment and upload e2e specs and the runner download helper to the V1 rows. - Added V1 tests for a Markdown review: expansion, deep link, and open, raw and download actions. ## Verification - `cd ui && npx vitest run`: one test file failed before its fix. That file tests the previous cards, and it now mocks the hook to off. I reran it with the other changed UI tests, and they pass. I did not repeat the full UI run after the fix. - `npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/text-attachment-tabs.spec.ts tests/e2e/artifact-uploads.spec.ts tests/e2e/artifact-tab-arrival.spec.ts`: 10 tests pass on V1. - `tests/e2e/cloud-auth.spec.ts` failed once in CI on a task heading. It does not touch artifacts, and 10 local runs pass. - `cd server && npx vitest run src/__tests__/instance-settings`: 183 tests pass. - `cd packages/shared && npx vitest run src/feature-catalog.test.ts src/validators/instance.test.ts`: 35 tests pass. - `pnpm run check:token-gates` and the `ui` and `server` typechecks pass. I did not run the full repository check. - In the artifact directions lab, with no stored settings, the Today right panel renders the V1 rows. - I did not test this on a live instance. After merge, confirm on staging that the Experimental page has no Artifacts V1 card and that all three surfaces show V1. ## Risks - All users see V1 after the next release, not only staging users. To revert, revert this pull request. - Users cannot turn V1 off. Only a hosting operator can, through managed config, and not on Cloud staging. - An instance with a managed `enableArtifactsV1: false` stays on the previous cards. The cloud harness source has no reference to this key, but I cannot read the live fleet configuration. - V1 surfaces render before settings load. An operator-held instance can briefly show V1 before the previous cards. - Admins who use the API can still write the flag. The write has no effect on unmanaged instances. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 08:14:49 +00:00
DottaandPaperclip	f7a4c5678f	feat(runtime): pre-copy completed harness history before enrollment (#19137 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An ephemeral web server needs provider history in shared object storage. > - Offline enrollment must preserve old backups before it can detach a volume. > - A large copied dataset still exceeded the maintenance deadline with parallel transfers. > - Most of its bytes belong to successful turns with released runtime authority. > - This pull request pre-copies that history while unrelated executions continue. > - The final offline migration still verifies every source and grants enrollment authority. ## Linked Issues or Issue Description Refs #19108, #19038, #18943. What happened? A copied instance has 1,953 local harness backups and 54.8 GB of declared source bytes. A resumed parallel migration still reached its 25-minute deadline. It cannot safely enroll by moving all history during maintenance. Expected behavior Copy eligible completed history while the server remains online. Defer active, failed or changed sources. Keep all existing data and final enrollment checks. Steps to reproduce 1. Retain many local native harness backups on a volume-backed instance. 2. Attempt a full offline conversion within the maintenance deadline. 3. Observe that most time moves historical bytes rather than current execution state. ## What Changed - Add finite `native-harness-prestage.mjs` preparation from a normal runtime build. Require explicit opt-in and shared object storage. Process one source at a time with a 90-minute deadline. - Use the recorded released runtime owner of each successful, finished turn. Do not assert that all writers stopped. - Recheck checkout, related queued/running work, successful result ownership, session, Runner, exact lease stamp and unchanged backup at database publication fences. - Defer source changes, malformed source JSON and unsafe backup roots. Log each such source and its reason. Keep database failures and cancellation fatal. Keep completed objects valid for retry. Never return an enrollment-ready receipt. - Reuse the bounded migration source walker. Keep offline admission and final verification unchanged. - Keep private lifecycle state, task ownership and provider policies unchanged. Document the operator path and its limits. ## Verification - A zero-source scan now checks the existing database obligation query. A known mismatch blocks with a source-missing receipt; empty fresh/already-durable scans and nonempty deferred scans remain valid advisory results. The prior command fails the mismatch regression. All 16 command and runtime-bundle tests pass for this follow-up. - At head `7ba3dbc7aa`, 87 focused command, database-timeout, storage, archive, publication and runtime-bundle tests passed with real PostgreSQL and filesystem restoration. - The current follow-up passes all 68 backup and archive tests. A source renamed after preparation fails the new regression on the previous head, then defers and permits the next backup with this fix. An archive output ENOENT remains fatal. Existing database-error and cancellation tests pass. - New race tests fail against the previous storage owner: a changed backup, queued turn or revoked lease otherwise publishes. - Tests prove unrelated executions stay running, same-task work defers, failed history stays deferred, retry verifies existing objects, and private bytes restore after deleting the old home. - Command review fixes require the configured existing vault key, retain it for the process, set database lock/statement/idle bounds, close clients on cancellation, and cap shutdown at ten seconds. Abort immediately sets failure status and prints one blocked receipt. A real child with real shortened timers reproduces the old missing-receipt exit and verifies the fix. Tests also use a real PostgreSQL lock. - Malformed manifest or runner JSON and symlink replacement fail the new preparation and publication-fence regressions on the previous code. The fixed command defers each damaged source and copies the next eligible backup. Separate tests keep database SyntaxError and cancellation fatal. Offline migration still rejects damaged sources. - The bundle test runs the emitted preparation entry with plain Node without starting the app. - A prior copied-data experiment completed with 1,738 new publications, nine verified current objects and 206 deferred sources. Later checks found different database and filesystem capture times: 43 source runs were absent from the older database snapshot, and one required historical manifest was absent from the retained filesystem archive. This experiment does not prove a consistent restore or enrollment readiness. - Full current-head CI at `ce63517d0c` passed: 54 successful checks, two skipped and one neutral, with no failed or pending statuses. Typecheck, tests and build pass. CI: https://github.com/paperclipai/paperclip/actions/runs/41970370960. The earlier superseded run lost a spot runner after a shutdown signal; that infrastructure cancellation is retained and is not counted as a passing run. - Greptile and Ponytail both rate this exact head 5/5; all actionable review threads are resolved. Comment-only cleanup removes newly added prose comments under CONTRIBUTING.md. TypeScript output with removeComments:true is byte-identical for all seven changed TypeScript files; 20 existing command, runtime and bundle tests pass after cleanup. - Serving enrollment, full normal-image migration, empty-home restoration and active-agent release handoffs remain separate acceptance gates. ## Risks - Preparation still consumes CPU, disk and storage requests. It uses one transfer and at most two database connections. Check memory headroom for a separate operator process. - A task can gain work during upload. The final fence rejects that publication and retains the previous current object. - Deferred sources remain migration obligations. Preparation never permits detaching the volume. - Ordinary backup callers do not opt into the new preparation constraint and keep their existing behavior. ## Model Used OpenAI GPT-6.1 Codex, exact model ID `gpt-6.1-sol`. Used for reasoning, code editing, shell tools and test execution. The runtime does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 23:15:46 -06:00
DottaandPaperclip	f5f5184f23	fix(workspaces): preserve retained branches and private commit sources (#19147 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Ephemeral web servers must preserve existing task files before volume removal. > - An old private checkout can be absent while its recorded branch still exists. > - A deleted branch can also retain its exact commit in the original private object source. > - The importer currently rejects these surviving sources. > - This pull request saves the verified committed tree and records its exact source proof. > - It keeps task state, branch ownership and recovery obligations unchanged. ## Linked Issues or Issue Description Refs #19138, #19144, #18826. What happened? Existing Git workspace records can have missing checkout folders and retained branch objects. The importer returns `local_source_unsafe_or_missing`. A successful deletion receipt can also have a valid private object source that the importer overlooks in favor of a missing or changed project cache. Expected behavior Save the surviving committed tree from the exact recorded branch or deletion receipt. Reject competing owners, live worktree registrations, changed branches and invalid source identities. Do not use the current default branch as a substitute. Steps to reproduce 1. Retain an isolated workspace branch in its managed project cache. 2. Remove its clean checkout but keep the branch and original workspace record. 3. Attempt workspace migration. 4. Alternatively, retain a deleted commit in its recorded private source while its shared source is unavailable. ## What Changed - Pin a retained branch in its verified company and project Git cache. - Require the recorded private path, no competing workspace owner and no surviving Git worktree registration or index authority. - For a user-owned branch, require its exact successful unforced removal receipt after archival. Preserve the branch ownership metadata. - Prefer a verified recorded private object source for successful deletion receipts when that source still exists. - Recheck branch and receipt proof before publication. Preserve compatibility with existing migration journals. - Allow missing active records only through the existing offline writer fence. Keep their status unchanged. - Add byte-restoration and rejected-authority tests. Document the source limits. - Revalidate and honor interrupted journals' original Git source path. Reject detached registrations and retained orphan index data. Reuse the cleanup producer command formatter for paths containing spaces. ## Verification - 172 focused migration, concurrency and isolated-source tests pass with real PostgreSQL and Git repositories. - The previous importer fails the retained-branch and offline-active restoration cases. The previous source selection also fails the private-cache and clean user-owned receipt cases. - Tests remove the old home and restore the pinned commit and task file bytes. New shared default-branch work is excluded. - Tests reject competing owners, registered worktrees, missing branch ownership, deleted branches, changed branch tips, forced removal and revoked receipts. - Read-only re-inventory of 79 actual failed source records makes 78 eligible with committed owner overlays. One older local test repository remains blocked. This is source-overlay qualification, not full released-image acceptance. - A separate bounded archive replay completed six imports before its 25-minute deadline. That timing failure remains open; it does not establish complete migration. - Local server typecheck has no changed-file diagnostics. Stale linked declarations prevent a clean local whole-server result. Full clean CI remains required. - Serving enrollment and empty-home recovery remain separate acceptance gates. - Latest rebased head `9f42901683` passes 172 focused migration/archive/concurrency tests. Four review regressions fail on the previous source; the revoked journal source already rejected correctly. The real cleanup fixture uses an unforced removal under a home path containing spaces. The orphan-index fixture is locked to keep Git maintenance from deleting its synthetic retained state before inspection. - The prerebase layered server typecheck passed. Current-head full CI remains required after integrating master. The local environment reset removed earlier temporary logs; the current red/green logs and source now live in persistent worktree folders. - Current-head full CI passes on run 41965901512 after one bounded failed-shard retry. The first run failed an unrelated status-card fake-clock deadline assertion by 150 ms; the same source passed the retry. Its cause remains unestablished, and the first failure log remains in Actions and persistent local evidence. No migration-related test failed. ## Risks - A retained branch saves committed data that still exists. It cannot certify recovery of files that were already absent. - User-owned branches remain user-owned. A clean removal receipt grants preservation authority, not deletion authority. - Any retained registration, conflicting record or changed proof keeps migration blocked. - This PR does not resolve the remaining source, recovery or migration timing gates. ## Model Used OpenAI GPT-6.1 Codex, exact model ID `gpt-6.1-sol`. Used for reasoning, code editing, shell tools and test execution. The runtime does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 22:35:55 -06:00
DottaandPaperclip	c8013d06ad	fix(workspaces): preserve task-only warnings during offline migration (#19144 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Ephemeral web servers must save existing workspaces before volume removal. > - Offline migration checks file ownership and recovery authority. > - The same gate also rejects task assignment and configuration warnings. > - Those warnings do not require the old filesystem. > - This pull request preserves known task-only warnings while it saves files. > - File recovery and execution obligations keep their existing blockers. ## Linked Issues or Issue Description Refs #18943, #19038, #19138. What happened? Offline workspace migration refuses tasks with assignment, configuration, disposition or deliberate-wait warnings. It returns `workspace_recovery_unresolved` even when the warning has no file or execution recovery authority. Expected behavior Save local workspace bytes while retaining the task warning unchanged. Continue to block unknown recovery kinds, file sources, export failures and execution recovery work. Steps to reproduce 1. Retain a local workspace on a task with an active assignment or configuration warning. 2. Stop all app writers through the offline maintenance entry. 3. Attempt workspace migration. ## What Changed - Admit four known task-only warning kinds during offline file migration. - Keep file recovery markers, system recovery, native failure markers, export-failure causes and unknown kinds blocked unless their exact legacy source is already preserved. - Keep the existing live-migration recovery exclusion. - Recheck warning evidence before publication. Do not resolve warnings or change their owners. - Add restoration, rejection and publication-race tests. Update the migration plan. ## Verification - 112 focused migration and concurrency tests pass with real PostgreSQL and filesystem archives. - The four task-only restoration cases fail against the previous importer with `workspace_recovery_unresolved`. - Tests delete the old home, restore its saved bytes and compare each warning row unchanged. - File and execution markers, unknown kinds and export failures remain blocked. - A task warning that gains file recovery during upload prevents publication. - Local server typecheck has no changed-file diagnostics. Stale linked dependency declarations prevent a clean local whole-server result. Clean full CI remains required before merge. - The PR was rebased onto the merged Git preservation fix after resolving a plan-document conflict. The rebased head passed full PR CI (41943153422). Current head `ba18efc7b5` adds the reviewed deliberate-wait case and reuses the canonical export-cause set; fresh full CI is required. The earlier unmergeable head did not have a PR workflow run. - Serving enrollment and full copied-data acceptance remain separate gates. ## Risks - The allowlist must remain narrow. New warning kinds are blocked by default. - A known task-only warning with file or execution recovery evidence still blocks migration. - This change depends on the existing offline provider fence. It does not establish that old app writers stopped. ## Model Used OpenAI GPT-6.1 Codex, exact model ID `gpt-6.1-sol`. Used for reasoning, code editing, shell tools and test execution. The runtime does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 22:00:41 -06:00
DottaandPaperclip	9cdbcbeddc	fix(workspaces): restore archived commits from verified project objects (#19138 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An ephemeral web server must preserve historical task files outside its local disk. > - Archived isolated Git workspaces can have a successful branch-deletion receipt. > - Their private checkout can be gone while the project cache still owns the exact commit. > - The importer rejects that cache because it applies private-checkout validation. > - This pull request validates the managed object source for receipt-based restoration. > - The importer restores the recorded commit without using the current shared working tree. ## Linked Issues or Issue Description Refs #18826, #18943, #19038. What happened? An archived isolated workspace has a valid successful cleanup receipt and a retained commit. Offline migration rejects it with `archived_git_source_unavailable,local_source_unsafe_or_missing`. The private checkout was removed. The company and project cache still contains the exact commit. Expected behavior Restore the immutable commit from the verified project object cache. Keep strict validation for saved private edit recipes. Reject missing receipts, a foreign cache, a different origin and revoked cleanup authority. Steps to reproduce 1. Create an isolated Git workspace from a managed project cache. 2. Archive it and retain its successful branch-deletion receipt. 3. Remove its private checkout while the verified project cache retains that commit. 4. Attempt offline workspace migration. ## What Changed - Extract the existing managed Git object-source validator. - Use that validator only for exact successful deletion receipts that refer to a canonical company and project cache. - Keep private restore-point source validation unchanged. - Fetch immutable objects at the recorded commit into private scratch. Recheck receipt authority before publication. - Add real Git and database tests for restoration and rejected authority changes. Document the preservation path. ## Verification - 129 focused migration, concurrency and isolated-source tests pass across three files. - The new restoration case fails against the previous importer with the reported source-unavailable error. - Tests remove the old home and restore the retained commit. A newer shared working tree and unrelated files are excluded. - Tests reject a foreign company cache, wrong origin, missing receipt and receipt revocation during upload. - Local server typecheck has no diagnostics in changed files. Stale linked dependency declarations prevent a clean local whole-server result. Current-head full CI passes (54 passing checks, three skipped). Greptile is 5/5 with no unresolved review threads. - A replay of the 32 affected records is running in an isolated staging copy. Two committed owner overlays are under test. This is not released-image or serving-instance acceptance. ## Risks - A deletion receipt must identify an exact commit that still exists. Missing objects continue to block migration. - Shared working-tree contents must never substitute for archived private files. Only the recorded commit is fetched through this path. - The change does not recreate private edit authority, retire recovery records or alter serving configuration. ## Model Used OpenAI GPT-6.1 Codex, exact model ID `gpt-6.1-sol`. Used for reasoning, code editing, shell tools and test execution. The runtime does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 18:25:08 -06:00
DottaandPaperclip	7d46532c6d	fix(runtime): bound parallel offline harness backup transfers (#19108 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An ephemeral web server must retain provider history outside its local disk. > - Offline enrollment moves verified local harness backups to object storage. > - A cold rehearsal reached 699 current backup rows while processing 1,953 sources before its 25-minute deadline. > - The serial transfer loop prevents this existing dataset from enrolling. > - This pull request overlaps at most four transfers within the existing total scratch limit. > - The benefit is faster preservation with the same ownership and publication checks. ## Linked Issues or Issue Description Refs #19038, #18943, #18932. What happened? Offline enrollment processed 1,953 local provider backups one at a time. The normal Cloud image reached 699 current backup rows (including nine pre-existing rows) before maintenance expired. The serving instance remained unchanged during the isolated test. Expected behavior Enrollment must preserve every source within bounded time and disk use. It must stop all in-flight transfers before it reports failure. Steps to reproduce 1. Prepare a settled instance with multiple verified native harness backups. 2. Run offline enrollment against shared object storage. 3. Observe that the serial owner waits for each archive, upload, readback and publication before starting the next source. ## What Changed - Process groups of up to four verified offline harness backups. - Reserve space for the archive and encrypted copy before each group starts. Keep the total scratch limit equal to one maximum-size serial transfer. - Derive a smaller archive limit from a closed manifest directory set. Preserve unlisted files with an exclusive maximum-size reservation. - Enforce the reservation while streaming. Keep the existing maximum for ordinary runtime backups. - Cancel and await peer transfers after failure. Keep completed publications valid for retry. - Preserve the original archive-stream failure when the tar writer also fails after destruction. - Document the migration bound and add integration regressions for overlap, restoration, cancellation, retry, extra files and archive limits. ## Verification - Serial baseline: the new overlap regression fails with one active transfer when four are required. - Candidate: 59 targeted tests pass with real PostgreSQL, S3, filesystem archives and restored provider bytes. The bundled-entry test is skipped locally because this source worktree has no runtime bundle. The full normal image rehearsal remains required. - Follow-up: 48 archive/storage checks pass after correcting the PAX-header allowance. The real encoder produces a 972,801,024-byte full-entry projection, above the old 805,306,368-byte allowance and below the new 1,024,001,024-byte bound. The existing global object limit remains in force. The old formula fails the new regression. Exact-size restoration and one-byte-over-limit rejection pass. - The previous revision passed full CI after one setup-route timeout retry. That failure remains recorded; the corrected current revision `f7c2be1289` passed fresh full CI (41937700826) and Greptile 5/5. Its review thread is resolved. - Local server typecheck found four test-only Promise library errors; those are fixed. The remaining 26 diagnostics come from stale linked dependency declarations outside the changed files. Clean full CI is required before merge. - The resumed isolated staging-copy experiment also exceeded its 25-minute deadline. It reached 1,389 current backups. This improves transfer throughput but does not by itself qualify enrollment; online preparation is tracked in #19137. - Full fresh normal-image migration, empty-home restoration and hosted release handoffs remain deployment acceptance gates. ## Risks - Parallel work increases concurrent object-storage and database requests to at most four transfers. The scratch reservation still bounds archive and ciphertext files together. - A malformed or changed source can fail its streaming limit. The migration retains its source and stops before export. - Partial completed publications survive failure. Retry verifies them under the existing hash and ownership rules. - No serving configuration, deployment pin, database contract or provider lifecycle policy changes in this pull request. ## Model Used OpenAI GPT-6.1 Codex, exact model ID `gpt-6.1-sol`. Used for reasoning, code editing, shell tools and test execution. The runtime does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 18:16:36 -06:00
DottaandPaperclip	61b2455319	fix(runtime): preserve failed sources and durable checkpoint authority (#19038 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A volume-free web server must preserve retained native session data. > - An old failed follow-up can keep its own snapshot while its controller file still belongs to the preceding successful turn. > - The Runner correctly refuses to import that file as the failed turn's execution authority. > - That refusal also blocks migration when no warm supervisor is present. > - This pull request preserves the file after it proves the older run's lineage, and uses existing typed database authority when present. > - Failed history and files survive, while normal execution authority stays unchanged. ## Linked Issues or Issue Description Refs #19024, which preserves the related older-supervisor case. The duplicate search found no PR for this older-controller case. What happened? Offline native-controller migration stops with `native_runner_checkpoint_binding_mismatch` for a failed retained follow-up. The local controller still contains the preceding successful run. A per-turn source can have no supervisor file, so the older-supervisor archive path does not apply. Read-only checks matched the database snapshots and controller-file hashes between the original source and the isolated rehearsal copy. Expected behavior Preserve the available file as evidence through the existing encrypted archive. Require a valid older controller, a completed successful prior run, and the same company, agent, task, native session, Runner and provider thread. Keep successful sources and active owners outside this archival fallback. When a valid typed database checkpoint already exists, derive only its missing recovery metadata and ignore obsolete local projections. Steps to reproduce Create a successful native turn and a later failed follow-up in the same session. Retain the successful controller file. Keep the failed snapshot and admitted input in the database. Leave the per-turn source without a supervisor. Run offline migration. The new positive regression fails with the binding mismatch before this change. ## What Changed - Share the scoped older-successful-run proof with the existing supervisor preservation path. - Validate the older controller through the Runner parser and artifact checks before archive publication. - Archive this failed legacy source without changing its snapshot, status, ownership, queued work or session pointer. - Preserve the normal refusal for foreign, changed, active, successful, invalid and already-durable cases. - Permit evidence archival for a stale observed coordinator only with exact host-owned stop receipts for every remote lease, no current task ownership, no result and no retry deadline. Recheck the proof before publication. - Use execution start for lifecycle ordering. A queued follow-up can predate its prior turn completion, and a remote stop can predate a later recovery timeout. - Resolve destruction receipts through the existing reusable-lease handoff verifier. Reject stopped tips, broken chains and competing tips. - Prefer a valid typed database checkpoint over a stale or absent local warm supervisor. Add only missing recovery metadata after current admitted-input and checkpoint binding checks. - Document the evidence and execution-authority boundary. ## Verification - Positive regression: red before the owner change and green afterward. - `pnpm exec vitest run server/src/services/native-runtime/native-controller-migration.integration.test.ts server/src/services/native-runtime/native-controller-recovery.integration.test.ts server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: 81 checks passed across migration, recovery and durable storage. - The checks cover storage failure, repeated migration, exact file restoration after old-home deletion, unchanged history and ownership, and refusal to treat the archive as executable recovery authority. - A second regression reproduced the archive fence refusal for an observed failed source with a valid stopped-remote receipt. Negative controls cover missing, foreign, old and revoked receipts, live leases, pending results, retry deadlines and task ownership. Revoking the receipt during upload prevents publication. - Two additional regressions cover a follow-up queued before the prior turn finished and a remote stop before a later logical failure. Both fail with the earlier timestamp comparisons and pass with execution-start ordering. Genuine execution overlap and old receipts stay blocked. - Two historical source snapshots and controller-file hashes matched in a read-only live-versus-copy check. The first candidate reached the archive fence and exposed the observed-coordinator case. That failure is retained. A fresh candidate replay is required for the expanded proof. It does not qualify the full packaged migration or hosted Railway handoff. - Existing typed-authority checks cover stale and missing supervisors, repeat migration, byte-identical checkpoint rows, unchanged snapshots/owners/history, old-home deletion and ordinary queued-turn recovery. Foreign Runner, session and snapshot controls remain blocked. - Handoff checks cover an exact destroyed successor, a stopped successor, a broken proof and a competing tip. - Isolated replay of the two retained typed-checkpoint sources imported only derived recovery metadata. Checks confirmed unchanged checkpoint rows (12 and 9 parts), history, snapshots, finalization owners and task-session pointers. The replay used three source-module overlays with a blocked provider network; it does not qualify the released bundle. - Final-head CI attempt 1 passed typecheck, build and the migration suites but failed one unrelated budget-concurrency assertion in `decision-models.test.ts` (one provider call observed instead of two). The original log is retained. The isolated local comparison cannot classify the timing failure because this checkout has stale linked budget schema exports. The one retry of that failed CI job passed, and final-head CI attempt 2 is green. The original failure remains retained; its cause is not established. - Full local typecheck, repository tests and build are not claimed. The server typecheck reports pre-existing linked declaration mismatches outside the changed files; the full clean CI typecheck has passed. This checkout uses linked generated declarations; clean CI is required before merge. Prior head `d71dc913` had passing full CI and a 5/5 Greptile review. Head `8ff3a22109` adds the handoff and typed-authority fixes. Its fresh full CI is green and Greptile is 5/5, with no unresolved review threads. Earlier head7729 hit unrelated 5-second CLI ZIP and 15-second setup-token test timeouts; those failures remain retained. ## Risks This path preserves evidence for a source that has already failed. It does not make that source resumable. The Runner must accept the older controller under its own identity and artifact binding. Scoped successful history must match its session and provider thread. The archive owner still requires stopped writers, a finished failed run and no live lease. An observed coordinator also needs exact remote-stop receipts after execution started, no accepted result or retry deadline, and no current task ownership. The coordinator stays unchanged. Existing durable authority stays authoritative. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. The host does not expose context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run the relevant tests locally; verification limits are listed above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 15:37:54 -06:00
DottaandPaperclip	3a283fbc9b	fix(runtime): preserve failed sources with older warm supervisors (#19024 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A volume-free web server needs durable recovery data for retained sessions. > - An old failed follow-up can retain its snapshot while the warm supervisor still holds the preceding successful turn. > - The migration refuses to treat those different snapshots as the same execution authority. > - That refusal also blocks the tenant even when the failed source can be preserved as an archive. > - This pull request archives that narrow case after it verifies the older successful snapshot and all scope bindings. > - The failed run keeps its history and files while valid controller recovery keeps its existing proof requirements. ## Linked Issues or Issue Description Refs #18959, which added the existing failed-controller archive path. No duplicate PR for this case was found. What happened? Offline controller migration stops with `stale_supervisor_snapshot` for an old failed warm follow-up. Its local supervisor still contains the prior successful turn. Read-only checks found the same database snapshot and supervisor bytes in the live source and the isolated copy. The mismatch predates the rehearsal. Expected behavior Preserve the failed run's database state and available local files in the existing durable archive. Only archive when the older supervisor matches a completed successful run in the same company, agent, task, native session and Runner instance. Keep current execution and controller authority checks intact. Steps to reproduce Create a successful native turn and a later failed warm follow-up. Keep the successful snapshot in the supervisor file and the failed snapshot in the database. Give the failed source local controller bytes but no durable controller checkpoint. Run the offline migration. The new regression fails with `stale_supervisor_snapshot` before this change. ## What Changed - Verify the older supervisor against scoped successful history and the same provider thread. - Use the existing encrypted archive owner for that failed legacy source. - Preserve snapshots, run status, ownership, queued work and the session's last-run pointer. - Keep foreign, changed, active, successful and already-durable sources outside this path. - Document the preservation and recovery boundary. ## Verification - New positive regression: red on the original owner, green after the change. - Native migration and controller recovery: 27 checks passed. Final migration checks also cover refusal to replace a durable checkpoint. - Storage failure blocks publication. Repeated migration is safe. The archive restores the exact controller and supervisor bytes after the old home is deleted. - The archived failed source does not gain executable recovery authority. - Local server typecheck is blocked by reused workspace declarations for accounting, bootstrap and ingress contracts. No diagnostic names either modified owner or test file. Full local typecheck, repo-wide tests and build are not claimed; clean CI is required before merge. The exact-head CI has now passed typecheck, build, policy, Runner checks, browser E2E, and all server test shards. - Rehearsal verification against the copied historical source remains required. Full tenant migration and hosted deployment qualification remain separate gates. ## Risks The archive preserves evidence for an already failed source. It does not certify that source as resumable. A prior snapshot must match exactly, and the normal archive owner checks the terminal failed binding before it publishes. Existing durable checkpoints remain authoritative. A missing or changed proof still blocks migration. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. The host does not expose context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run the relevant tests locally; retained failures and limits are listed above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 13:40:53 -06:00
DottaandPaperclip	0eaabe5c0b	refactor(accounting): remove company serialization from paid settlement (#18917 ) Settle paid operations through operation-scoped transactions. Preserve exact ledger evidence and replay while moving aggregate budget decisions to durable deferred work. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-04 07:29:28 -06:00
Dotta	55d7df2bad	fix(runtime): bound checkpoint data and preserve durable failed sessions (#18959 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native Runner checkpoints preserve controller state across server replacements. > - Enrollment imports legacy controller state into those durable checkpoints. > - The importer passed extra source fields into each encrypted record, duplicating the whole source and exhausting memory. > - Some failed runs already had durable checkpoints but no session snapshot. Re-importing them incorrectly required resumable state. > - This PR stores only each record's binding and contents, and preserves matching failed checkpoints without changing their recovery status. ## Linked Issues or Issue Description Refs #18957 and #18943. What happened? A 538-record migrated checkpoint stored 1,108,851,510 bytes of encrypted material. Every record included the full 1.4 MB legacy source. Large writes disconnected and later reads exhausted memory. Expected behavior Each encrypted record contains only its company, issue, run, part and contents. Steps to reproduce Run offline controller migration with a retained controller that has many commands. Decrypt a saved envelope and inspect its fields. The old writer also includes the entire offline migration input. Paperclip version or commit Reproduced on `a1f3637c96dfadaf3a15ae9c83aa8227f4fed20e`. ## What Changed - Serialize only company, issue, run, part and contents in encrypted checkpoint envelopes. - Preserve an already-durable failed checkpoint when its run, runner and session match and the legacy session snapshot is absent. Keep its failure, recovery blockers, owner and pointers unchanged. - Report the number of failed checkpoints preserved during maintenance. - Test exact encrypted contents, idempotent preservation after removing the old home, and rejection of successful or foreign-session sources. - Document the migration findings and verification boundaries. ## Verification - The oversized-envelope regression failed before the serializer fix. - The already-durable failed-source regression failed before the preservation fix. - Four focused test files: 44 tests passed. The migration suite passed all 5 tests again after adding the foreign-session control. - Server typecheck passed. Greptile reviewed `c4f02a064c` at 5/5 with no findings. The exact-head PR workflow passed, including typecheck, build and all required tests. - Live staging repair preserved all 16 affected checkpoint sets: 2,982 records, 6,400,144,676 bytes reduced to 15,419,528 bytes. Each record matched its legacy source and passed re-encryption verification. Original encrypted rows and source files remain on the retained volume. - Full hosted enrollment and active-agent deployment proof remain pending. Deployment will follow the normal staging release pipeline. ## Risks The serializer affects normal and migrated checkpoint writes. The decoder already accepts the same five fields and continues to accept older envelopes. Preservation applies only to a failed run with an exact, validated durable checkpoint and no session snapshot; it grants no new recovery authority. This change does not rewrite existing database records automatically. Existing failures and missing provider files remain failures until their owning recovery path resolves them. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-04 07:14:58 -06:00
DottaandPaperclip	d12de135a7	fix(runtime): preserve legacy identities and bound checkpoint imports (#18957 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Ephemeral web servers must preserve native Runner sessions before removing local storage. > - Legacy session imports validate the workspace and saved provider authority. > - A protocol lease can retain a logical workspace ID while copy-back uses a separate physical sandbox lease ID. > - Comparing these different IDs blocks valid legacy sessions during enrollment. > - This pull request checks the workspace binding, retains the full saved protocol identity check, and bounds checkpoint inserts inside the owner transaction. > - Existing sessions can migrate without changing their identity or discarding files. ## Linked Issues or Issue Description Refs #18943 and #18841. What happened? The first full staging enrollment stopped with `native_controller_migration_artifact_binding`. A successful native session had valid controller and provider-pack identities, but its physical sandbox lease had a different ID. Expected behavior Import the original session when its workspace and saved provider authority match. Reject foreign workspaces and protocol identities. Steps to reproduce Create a retained session whose sync record points to the execution workspace and a separate physical lease. Run the offline controller migration. The prior comparison rejects this valid record. Paperclip version or commit Reproduced on `84aa989bf9d08fc005ac273c37f9e1d860a4db25`. Deployment mode Docker with external PostgreSQL and shared object storage. ## What Changed - Compare the sync record's workspace ID with the frozen execution workspace. - Keep the full saved provider-pack check for company, agent, workspace, session, Runner, protocol lease, remote path, and provider digest. - Use distinct physical and protocol IDs in the database-backed migration fixture. Reject a foreign workspace and a changed protocol lease before any checkpoint import. - Limit migration inserts to 64 records and about 512 KiB of serialized rows per statement. Send a larger individual record alone. Keep all statements in the same transaction. - Verify that a conflict in a later batch rolls back earlier batches, then import and decrypt the complete checkpoint. - Record both live failures and the comparison evidence in the preservation plan. ## Verification - The updated fixture failed on the original comparison and passes with this fix. - Targeted migration, controller recovery, and provider authority tests: 39 passed. - Full repository typecheck and the latest server typecheck passed locally. The complete PR workflow passed on `a1f3637c96` (run 41827558125), including build and test gates. The Docker Compose setup download failed once before the test step; its infrastructure retry passed. A separate full local test/build run is still in progress. - A read-only staging preflight inspected 807 selected legacy sessions. It found zero mismatches under the corrected workspace and full saved-provider checks. - A single 538-record, 2.2 MB checkpoint insert repeatedly closed the live database connection on pooled and direct endpoints. A forced-rollback comparison imported the identical checkpoint with batches of 64. An unbatched control failed again after cache warm-up. The exact server-side disconnect cause remains unknown; all failed receipts are retained. - The original failed enrollment and volume are retained. The corrected importer committed 32 additional sessions and 2,982 checkpoint records. Full import and deploy handoff remain pending while memory use is investigated. ## Risks The import checks durable session authority. A weaker identity check could import the wrong session. This change retains the saved provider authority validation and adds an explicit foreign-protocol rejection test. Batching adds statements for large imports, but a late failure still rolls back the complete owner transaction. It does not rewrite session identity, change provider lifecycle, or detach any volume. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 06:36:31 -06:00
DottaandPaperclip	84aa989bf9	Preserve native sessions through legacy workspace enrollment (#18943 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native session retains its workspace and provider session across turns. > - An ephemeral web server must restore those files without its predecessor's disk. > - Legacy enrollment changes the workspace authority and can change both workspace addresses. > - The retained provider backup checks those addresses as part of its identity. > - This pull request preserves the exact addresses in a verified migration receipt. > - The benefit is that the next turn can use the original session after enrollment. ## Linked Issues or Issue Description Refs #18932, merged as `612d9931a6`. This pull request builds on that all-Runner durability change. Related: #18918 fixes concurrent workspace preservation locks. This change covers identity preservation during offline enrollment. ## What Changed - The native workspace migration owner records the selected session bindings in the successful workspace migration operation. - Runtime selection retains the original logical address only for the exact agent, task and session. - Remote workspace preparation restores the original provider address. It refuses conflicting targets, forged receipts and occupied unowned roots. - The environment owner includes the migration operation in the reusable lease fingerprint. Warm policy stays warm and receives a fresh allocation through normal lease cleanup. - Exact migration evidence permits the existing verified provider backup to restore into that fresh warm allocation. - Preserve verified provider allocation files through a normalized, lease-bound digest receipt. Unknown files and altered markers remain refused; interrupted publication is retryable. - Retain legacy managed-instruction session fingerprints using the exact imported content hash and recorded paths, preserving all other configuration and credential checks. - Migrate existing local skill archive/manifest objects through upload claims, readback verification and receipt compare-and-swap; retain original objects and IDs. - Add explicit two-turn browser qualification for every released native Daytona profile, with warm and per-turn variants. The production maintenance entry runs offline. The old home is deleted only after a verified export and successor readiness. The oracle checks provider identity, native session, addresses, reference bytes, question and final document. ## Verification - Full repository typecheck passed at `b12ecd823e`. The production session fix passed 198 session/credential tests. Full repository build passed at `94bb917947`. The final Rust-only provider fix passed 57 ACPX provider tests and the exact Linux release build. - Workspace and transport checks passed: 117 transport/workspace tests, six final real PostgreSQL migration boundary cases, 178 environment tests, 539 Daytona tests (eight skips), and the native warm-backup admission control. - Local skill migration passed eight PostgreSQL/S3 tests (one existing opt-in skip). Failed readback retains the source. Successful readback still works after deletion of the old home. - Latest Product E2E typecheck and 38 focused harness tests pass. Earlier catalog/admission/report checks passed. Twelve unrelated support-test failures reproduce on unchanged base `1a39dea9aa`; these are not reported as passing. - Live Codex per-turn at `2d5bb9a6d1` retained the native and provider sessions, both workspace paths and exact reference bytes after deletion of the old home. It completed the task and delivered its saved document and original reference. Its original failure was an output-count grading defect. The corrected grader passes all 21 checks against retained, hashed evidence; the original result remains unchanged. - Live OpenCode per-turn and Codex warm at harness `94bb917947` pass all 21 product checks. Their launcher cleanup failed because a read-only diagnostic observer kept its disposable database open. The observer now closes before ordinary database cleanup; a no-provider reproduction confirms the fix. All owned sandboxes were independently verified absent, child keys revoked and buckets removed. Original failed campaign dispositions remain unchanged. - Claude at `94bb917947` kept its sessions and reference bytes but omitted the final clickable document link. That user-delivery failure remains recorded. The fixture now explicitly asks for the link and records migration evidence before checking navigation. The fresh Claude attempt at `b12ecd823e` passed both strict migration checks: exact provider/native session, both paths, original file bytes and independently downloaded attachment. Its full browser workflow failed again: it requested unnecessary confirmation, used the answer sentence as the reference, and omitted the clickable document link. This is retained as a failed user-workflow result, not a full pass. - Earlier live failures exposed and reproduced the skill-object, fresh-allocation marker and managed-instruction fingerprint bugs. The fixes retain original identity checks. No provider session reset or accounting bypass was used. - The real Grok continuation at `b12ecd823e` failed before provider execution because its immutable-instruction attach policy rejected relocated registered file copies. A no-provider regression reproduced the exact rejection. `b05ba3728a` selects the existing strict authenticated asset-path policy for Grok; content digests, unknown context fields, session/workspace identity and same-run mutations remain enforced. The regression and all 57 ACPX provider controls pass. The Grok browser migration now passes all 21 checks with the b05 Linux Runner, including provider identity, exact reference bytes and final user output; the launcher also reports successful cleanup. - `24fabf1d81` applies the same policy to released ACPX Codex after a second no-provider reproduction. All 58 ACPX provider controls and the final Linux release build pass. Its live browser migration passes all 21 checks on the final binary, including exact original session/file preservation and completed user output. - The full native-session executor suite passes all 1,121 tests after updating its existing lease database fixture for the migration-receipt join. The prior CI failure was this unsupported test-double method; the production guard still rejects ordinary unproven recovery. CI and Greptile 5/5 passed at `ca1e0d5fe6`. The first CI attempt lost several workers and had one rotated-recovery transport failure; its seven descriptor cases passed locally, and the single unchanged retry passed. The original failure remains recorded. Staging enrollment has not started. Initial offline enrollment and subsequent rolling availability are separate checks. - The final review follow-up adds real runtime-selection PostgreSQL cases for exact, different-session and absent receipts, plus executor-level warm restoration that copies actual backup files and verifies provider identity and the new lease stamp. Both complete affected suites pass (1,172 tests), and full repository typecheck passes. Removing either production migration connection independently makes its new positive case fail; both source files were restored byte-for-byte after these negative controls. - Final commit `0971d373a3` passes all 63 checks (57 successes, five expected skips, one neutral); Greptile is 5/5 and every review thread is resolved. ## Risks - An identity conflict blocks enrollment and keeps the original files. It does not silently reset the session. - A new migration generation retires a reusable lease through its existing owner. The original sandbox must finish normal cleanup before reuse proceeds. - Initial enrollment has an offline step. The separate rolling suites prove uninterrupted deploy behavior after enrollment. - This adds no schema migration and changes no provider credentials or accounting gate. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this session. The agent used reasoning, terminal tools, code edits, PostgreSQL/filesystem tests and isolated live browser qualification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 04:43:56 -06:00
Dotta	82dd3bb702	Preserve unstarted queued turns during offline enrollment (#18887 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Existing-volume enrollment moves server files to durable object storage. > - Enrollment stops application deployments before it changes file authority. > - A task drain keeps unstarted turns queued in the database. > - The migration rejected every queued row, so a busy instance could never finish enrollment without discarding requests. > - This change preserves unstarted queue intent and continues to reject running or previously started executions. > - Users keep their pending turns and saved files through the conversion. ## Linked Issues or Issue Description What happened? A preserving enrollment found queued turns after task admission stopped. The workspace, instruction-file, and historical-log guards treated these rows as live writers. The drain also prevented the rows from starting, so waiting could not satisfy the migration guard. Expected behavior After every old deployment has stopped, maintenance can move files while unstarted queued turns remain unchanged. The replacement claims those turns through normal admission. Actual executions and retained execution ownership continue to block migration. Steps to reproduce 1. Queue a turn behind another task. 2. Start a task drain and let the current task finish. 3. Stop the application deployments and run offline enrollment. 4. Observe `active_execution_requires_drain` for the unstarted queued turn. Paperclip version or commit Reproduced on the migration contract through #18864. This branch starts at `ab2d2098a2764c511978d1c3e7a3fc5dfcef07f4`. Related: #18826, #18835, #18841, #18864. Searches for queued enrollment and queued migration found no duplicate fix. ## What Changed - Share one offline execution predicate across workspace, agent-file, and historical-log migration. - Reuse the existing pre-dispatch evidence predicate; queued turns with execution markers, saved checkpoints, source events, coordinators, or environment leases remain blockers. - Keep task bindings and normal live-migration restrictions unchanged. - Exercise normal replacement admission after deleting the old filesystem: the same queued run reads restored workspace bytes and instructions, completes its original wake, and cannot execute twice. - Document the offline boundary and queued-turn behavior. ## Verification - 111 tests passed across the final 110-case migration suite and assembled queued-turn heartbeat proof, with real PostgreSQL and local S3. This includes workspace/log migration, queued ownership blockers, ordinary agent-file controls, and the actual bundled maintenance process. - The maintenance process preserves the queued row while archiving historical output. The new assembled heartbeat test creates a normal wake under drain, migrates without changing either row, deletes the original filesystem, and resumes through normal admission. It reads saved workspace bytes and the restored instruction bundle, persists one successful native result, completes the same wake, and verifies a second sweep does not duplicate execution. Only sandbox transport/provider execution are scripted; no live model or Daytona calls. - An earlier optional run-evidence worker integration invocation was skipped without its dedicated environment; it is outside the changed offline path. - Server typecheck, server build, and runtime bundle passed. - The first maintenance test exposed the separate agent-file guard; that guard now uses the same offline predicate and the test passes. - Initial local typecheck attempts lacked CLI dependency links in this isolated checkout. Restoring dependencies fixed setup without source changes. - Full repository checks run in CI. The first head lost spot runners in Build and serialized tests; one unchanged idle-deadline fixture crossed UTC midnight and failed its timestamp assertion. Final-head CI passed on run 41744678994 after retrying a spot-runner shutdown. Greptile reviewed the exact final head at 5/5; all inline findings are resolved. The separate Paperclip Review app remains queued. Hosted enrollment remains pending release qualification and compatibility verification for the existing provider configuration. ## Risks The offline predicate relies on Cloud stopping every old application deployment. It does not make online migration safe. Running executions always block. Requeued executions with retained ownership also block. No queued request, task binding, or recovery state is removed or rewritten. No database schema changes. ## Model Used OpenAI Codex, GPT-6 based, with reasoning and tool-assisted code execution. The exact deployed model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-03 18:33:41 -06:00

1 2 3 4 5 ...