mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 22:56:58 +01:00
b2a7d322ed0bafda04a0b78ce14e4f25c8d5d170
417
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
bf818280f2 |
perf(workspaces): reserve cohort members concurrently (#20401)
## Thinking Path > - Paperclip controls agent work and preserves its workspaces. > - The online importer moves up to eight shared-source leaves under one source fence. > - Each member must have a durable reservation before capture starts. > - Serial reservations repeat the same validation cost before any archive work. > - This pull request reserves members in parallel through their existing independent transactions. > - Every reservation finishes before an error can release the fence or start capture. ## Linked Issues or Issue Description Refs: #19873, #20387, #20394. **Current behavior** A measured six-member cohort spent about 110 seconds in serial reservations. Each reservation keeps its existing source, task, recovery and journal checks. **Proposed behavior** Start the bounded member reservations together. Join all outcomes, retain member order, and refuse capture if any member fails. Keep the exact partial-journal resume contract. **Reason and benefit** This removes serial metadata work before capture. Hosted speedup has not been measured for this change. ## What Changed - Replace only the cohort reservation caller loop with `Promise.allSettled`. - Preserve the existing per-member transaction, preassigned operation ID, source fingerprint and all source fences. - Extend the real PostgreSQL failure fixture to require overlapping reservations and complete peer closure before fence release. - Document the unchanged partial-journal and atomic publication contract. ## Verification - Canonical cached worktree preflight and plugin SDK prerequisites passed on Node 24.22.2. - The unchanged production baseline failed the new reservation overlap assertion: one reader entered instead of two. The canonical PostgreSQL wrapper cleaned up. The other 21 tests were intentionally not selected. - The first baseline command failed before PostgreSQL because root-level `tsx` resolution was absent. The corrected command uses the checkout's explicit `cli/node_modules/tsx/dist/loader.mjs`. - The four focused PostgreSQL controls passed before the final rebase: overlap/join failure, exact partial-journal resume, parallel capture with both cold restores, and atomic outer-commit rollback. Eighteen tests were intentionally not selected. Server TypeScript passed. - The final branch is based on merged #20394 at `302ada90dfda0e8fbcd8a514d506d310c9843bcb`. The two directly affected PostgreSQL reservation/resume controls passed again on final head `06d436e45705da3b616a2b91725960f6e0ea52c0` (2 passed, 20 intentionally not selected; 100.56 seconds). Server TypeScript passed again. The canonical wrapper completed cleanup. - The separate Ponytail review remains 4/5. Its nonblocking fan-out warning was resolved as the documented phase contract; no source change was required. Optional multi-error aggregation is outside this change. - Fresh Greptile review is 5/5 on `06d436e45705da3b616a2b91725960f6e0ea52c0`, with no actionable findings or unresolved threads: https://github.com/paperclipai/paperclip/pull/20401#issuecomment-6754025480. - Exact-head CI is complete: 56 success, 4 skipped and 1 neutral, with no failed or pending checks. Aggregate `ci / verify` passed: https://github.com/paperclipai/paperclip/actions/runs/42718439355. - Reproduce the final focused cases on Node 24: `node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts -t "joins concurrent reservation validation readers|resumes the exact membership after reservation stops halfway"`. Run `pnpm --filter @paperclipai/server exec tsc --noEmit` for TypeScript. ## Risks - API phase contract: `concurrency` limits capture workers. Preflight, reservation and post-finalizer validation fan out over the exact selected member set, with the canonical maximum of eight. The original DB pool bounds connections. `concurrency: 1` does not promise one metadata query or transaction at a time. - Up to eight reservations can use independent pool connections at once. Members that pass can leave durable journals when a peer fails. The existing exact-member resume path retains those journals; this change does not retry automatically. - Capture and publication remain blocked until every reservation settles. Publication stays serial in one transaction. Saved journal schemas, singleton behavior and recovery proof checks do not change. - This does not change concurrency inside the publication transaction. One PostgreSQL connection serializes that SQL, and changing its row-lock order has no measured benefit here. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9c634e42e4 |
fix(workspace): bound queued source-lock admission for imports (#20370)
## Thinking Path > - Paperclip manages AI agents and their work. > - Existing local workspaces need durable imports before a volume-free deployment. > - Imports must acquire the same source locks as workspace writers. > - Try-only admission can repeatedly lose to background maintenance, even when each writer finishes. > - This pull request adds an optional bounded wait to the canonical migration APIs. > - Operators can join the source-lock queue while all existing preservation checks remain active. ## Linked Issues or Issue Description **What happened?** An online import can repeatedly return `workspace_source_busy` while background workspace maintenance reacquires its project source lock. The import does not join the PostgreSQL lock queue. **Expected behavior** An administrative import can wait for a finite writer to finish. One wait budget covers the complete source scope. Timeout or cancellation must release acquired locks before the import reserves a journal. **Steps to reproduce** 1. Hold the source lock for an eligible local workspace. 2. Start an online import while the writer holds that lock. 3. Release the writer. Before this change, the import has already refused admission. With `sourceLockWaitMs`, the import acquires the source scope and commits normally. **Paperclip version or commit** Reproduced on master `0e3fdebea8257bdff11573081c76b306f12db230`. **Deployment mode** External PostgreSQL with an online workspace import. Related preservation work: https://github.com/paperclipai/paperclip/pull/20311. ## What Changed - Add optional `sourceLockWaitMs` to singleton and cohort migration APIs. Omitted options keep current admission behavior. - Queue the sorted source keys with one decreasing wait budget. Abort cancels only the dedicated backend identified by PID and backend start time. - Use an independent source transaction for queued singleton imports. Keep lifecycle checks, fresh source checks, journal ownership, and publication after commit. - Document the canonical API option. Add real PostgreSQL importer regressions for release, timeout, abort, and partial-key cleanup. - Correct the inherited task-case lookup summary so the existing search-discovery regression passes without changing its assertion or search algorithm. ## Verification - Baseline RED: the new singleton regression fails on the unchanged product because the import returns busy without queueing. - `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'queued|bounds the whole source queue'`: 20 passed, including five new cases. - `pnpm exec vitest run server/src/services/isolated-remote-workspace-source-lock.test.ts`: 4 passed. - Server layer typecheck, `node scripts/check-module-boundaries.mjs`, and `git diff --check` passed. - Tests used Node 24.22.2 and the existing embedded PostgreSQL fixture. Hosted rollout verification remains pending. - CI build, typecheck, browser suites, all serialized suites, and the other general shards passed. The final server shard failed only the inherited `case-history-api-discovery.test.ts:14` search-ranking assertion. Its unchanged local regression reproduces on the base and passes 3/3 with a separate one-line route-summary correction. The shared one-line summary correction is included in follow-up commit `8163ffa9d4728134676b84c8c242a5bf62c802b5`; the unchanged case-discovery suite passes 3/3 locally. The original CI attempt remains failed: https://github.com/paperclipai/paperclip/actions/runs/42704666536/job/129458605783. ## Risks Queued admission can hold earlier source keys while it waits for a later key. The caller controls the finite budget. The abort path opens a short-lived dedicated cancellation connection. If cancellation cannot connect, PostgreSQL's remaining lock timeout still bounds the wait. This option must be called at the root of administrative work; an external scope that already holds the same source locks can cause self-contention. The existing pre-held journal-resume path must keep its omitted-option behavior. ## Model Used OpenAI Codex, based on GPT-6, with tool use and code execution. The exact deployment identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
7f55323abb |
fix(recovery): end a remote workspace epoch whose writer died during preparation (#20346)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task with a remote workspace prepares that workspace before the
run selects its runtime, and preparation marks the workspace epoch dirty
with the run as its writer
> - If the server is killed during preparation, the run never selected a
runtime and never started provider work, but the dirty epoch remains
> - Failed-workspace preservation, the path that resolves a dead
writer's dirty epoch, applies to native runs only, so nothing owned this
epoch
> - The dirty epoch kept the dead run's sandbox lease as a pending
export and held the workspace against the task's retry, so the task
never recovered
> - This pull request lets the ownership reconciler end such an epoch at
the last committed save
> - The benefit is that a restart during workspace preparation no longer
wedges the task
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave on Daytona (a remote workspace that clones a
real repository), the controller was killed while each run's first
workspace save was in flight. That save is now settled by #20326. Each
run ended `process_lost` with a bounded retry, and both runs were still
in `execution_stage: preparing` with `runtime_mode_resolved_at: null`.
The workspace authority stayed `dirty: true`, `snapshotId: null`, with
the dead run as writer. The dead run's environment lease was proven
owner-gone but was refused by its settle as `workspace_export_pending`,
and the reconciler reported `I2_settled` violations. The retries stayed
`queued` until the test harness deleted the environment 21 minutes
later. Then they failed `remote_workspace_execution_required`, because
the environment was gone.
**Expected behavior**
When a writer dies before runtime selection, its unsaved files are only
the restored source and preparation output, because no agent turn ran.
Recovery must end that epoch at the last committed save, so the lease
settles and the retry restores the workspace into a new directory. A
writer that reached runtime selection must keep its existing recovery.
**Steps to reproduce**
1. Run a task whose agent uses Paperclip Runner with a remote sandbox
workspace.
2. Kill the server while the run is still preparing the workspace (for
example during its first save).
3. After the restart, the run ends `process_lost` and its retry is
queued, but the workspace stays dirty and owned by the dead run, the
lease stays active, and the retry never starts.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
c67909cb0f |
fix: keep tool approvals valid through OAuth refresh and delayed review (#20340)
## Thinking Path > - Paperclip lets people manage agent work and approve app actions. > - An approval must bind the action that the person reviewed. > - OAuth access tokens can expire while that person is away. > - The old target snapshot treated each token refresh as a new authorization. It also imposed a one-hour review deadline. > - This pull request separates token renewal from authorization identity and keeps the human decision durable. > - Current access, the reviewed arguments, meaningful target changes, task closure, and execution receipts still control dispatch. > - People can approve later without repeating a decision after routine connection maintenance. ## Linked Issues or Issue Description **What happened?** A pending app action returned `approved_tool_target_changed` after its OAuth tokens refreshed. The tool and connection settings were unchanged. The same card also expired after one hour. A sign-in problem could discard an accepted decision before provider dispatch. **Expected behavior** The original card remains reviewable after routine OAuth refresh and a server restart. A later approval runs the exact reviewed action once. A connection repair preserves the decision while Paperclip verifies the same authorization. Meaningful changes require a new review. **Steps to reproduce** 1. Connect an OAuth MCP app and configure an action to require approval. 2. Request the action, then refresh its access and refresh tokens before approving. 3. Approve the original card. Before this change, it returns HTTP 409 even when the authorization is unchanged. 4. Wait more than one hour. Before this change, the card expires without a task-specific deadline. **Paperclip version or commit** Reproduced from master at `76254ead2229`. The final branch is rebased onto current master. ## What Changed - Add versioned approval target comparison. New snapshots bind credential identity epochs, consent, permissions, ownership, connection configuration, tool contracts, and credential references. Authenticated OAuth refresh preserves identity. Manual credential replacement does not. - Remove the implicit one-hour deadline from ordinary app action reviews. Keep explicit deadlines and terminal-task expiry. Synchronize the nullable interaction deadline in shared contracts and UI. - Keep accepted, undispatched actions when sign-in is required. Show a saved-approval repair state. Resume on committed connection changes after target and access revalidation, without an idle retry loop. - Preserve the existing execution claim, result receipt, and uncertain-outcome no-replay rules. - Explain meaningful target changes without inviting the person to retry the stale card. - Add a production task-thread Storybook example and document compatibility and recovery behavior. ## Verification - Red: production-refresher invalid-grant handling and formal-approval withdrawal during a sign-in wait also reproduced as failing regressions before their fixes. - Red: before implementation, the local OAuth HTTP fixture failed for both public and confidential clients with `approved_tool_target_changed` after token refresh. - Green: the extended fixture passes for public and confidential clients with personal and shared grants. It covers actual local HTTP token exchanges, multiple refreshes, a rebuilt gateway, an ended originating run, approval after 25 hours, duplicate approval, preserved sharing arguments, sign-in waiting, and event-driven repair. The gateways use the production OAuth refresher; an invalid refresh preserves the accepted decision before provider dispatch. Omitted-scope responses cover equivalent scope-reporting changes. The provider is an isolated local fixture, not a deployed Notebook service. - Negative controls refuse changed consent, scopes, tool definition, destination, and manual credential replacement before provider execution. - Existing gateway service, connected-tool acceptance, secret owner, secret service, unsigned-review queue, and connection-health suites passed. Explicit-deadline fixtures still prove expiry behavior, including deadlines on accepted sign-in waits. A withdrawn formal approval cancels a waiting action before connection recovery. - Approval card, task-chat, error-copy and target-comparison tests passed. Server and UI typechecks passed. Token gates and Storybook catalog checks passed. - Inspected the new saved-approval repair card in local Storybook using the production task thread. It shows a reconnect link and no second approval button or false running state. - `pnpm verify --explain` returned exit 2 and requested broad CI coverage. [Full CI](https://github.com/paperclipai/paperclip/actions/runs/42677855507) passed on `eb4f1c7caa9ee6e802c314ffc5eadec475a5de0c`, including build, typecheck/release registry, general and serialized tests, and all eight browser shards. - Greptile reviewed the same final head at 5/5. Both findings from the first review are fixed and their threads resolved; no open follow-ups remain. ## Risks - No database migration is needed. The new comparison reuses the vault identity epoch. Only authenticated OAuth refresh code can preserve that epoch; HTTP credential-write schemas cannot select that behavior. - Earlier signed requests keep their original exact-version checks and saved deadlines. Already-expired cards are not revived. A historical card whose credentials already changed needs a fresh review. - A new consent, account, permission set, destination, or manually replaced credential must not inherit the old authorization. Reconnection continues automatically only when revalidation proves the same saved authorization. - The interaction enrichment now permits a null deadline. Existing string deadlines remain valid. The web client and shared validator change together. - This PR does not deploy to staging or perform a live Notebook publication. ## Model Used OpenAI Codex, GPT-6 family, with repository tools, code execution, and browser inspection. The runtime does not expose the exact model ID, context-window limit, or reasoning-mode label; those values are not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
251c6d5c69 |
fix(recovery): settle a remote workspace save whose process died mid-save (#20326)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshot
operations (`workspace_operations`, phase `remote_workspace_snapshot`).
Each save invocation is listed in `saveWork.activeIds` while it runs
> - When a controller process is killed during a save, its invocation
stays listed and the operation stays `running`
> - Nothing recorded which process ran the invocation, so the shared
ownership reconciler could not prove the owner gone and never settled
the save
> - The task's bounded retry then waited behind that save forever
> - This pull request records the owning process for each save
invocation, and lets the reconciler settle a save whose owner is proven
gone
> - The benefit is that a restart during a save no longer wedges the
task
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave on Daytona with a real cloned repository, two
tasks each started their first remote snapshot save (`boundary:
initial`, epoch 1). A SIGKILL hit the controller during the save, before
any upload. Each run ended `process_lost` and got a bounded retry. The
retries stayed `queued` forever, the snapshot operations stayed
`running` with `saveWork.activeIds: [<dead invocation>]` and
`quiescedAt: null`, and the sandbox leases stayed active. The ownership
reconciler reported the saves as owner-gone but had no way to settle
them.
**Expected behavior**
After a restart, recovery must settle a remote snapshot save whose
owning process is gone. It is marked failed with provenance, so the
task's retry can continue. A save that is still running must stay
untouched.
**Steps to reproduce**
1. Run a native task with a remote workspace (for example Daytona with a
`git_repo` project source).
2. Kill the controller (SIGKILL) while the first workspace save is in
flight, before its upload.
3. After the restart, the run ends `process_lost` and its retry is
promoted to `queued`, but the save operation stays `running` and the
retry never starts.
**Paperclip version or commit**
`master` at
|
||
|
Dotta
|
1b7fe33432 |
fix(workspaces): preserve import verification across process restarts (#20311)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online import preserves old local files before the web server can use a disposable filesystem. > - A parent import must authenticate saved child archives before it can omit their directories. > - The operation journal preserves import identity, but completed child checks existed only in process memory. > - A container restart therefore repeated downloads, decryption and restoration that had already succeeded. > - This pull request saves a versioned verification receipt on each immutable child snapshot. > - A replacement process can reuse those checks while it reacquires locks and verifies current local files. ## Linked Issues or Issue Description Refs #19854 for verification reuse within a live source fence. Refs #19873 for cohort import. Refs #20298 for the separate remote-hold admission fix. Searched related workspace verification and restart PRs; none provides durable child verification. **What happened?** Replacing a server container interrupts a long online import. The original operation and snapshot survive, but a later invocation downloads, decrypts and restores completed child archives again. The current cache ends with the source-lock owner. **Expected behavior** Completed immutable checks survive process and scratch loss. Resume keeps the same operation and snapshot, reuses only correctly bound verification, and publishes once. Current local files and Git state must still be checked under fresh source locks. **Steps to reproduce** 1. Import two nested child workspaces, then start the parent import. 2. Kill the importer after a child verification finishes. Remove its scratch directory. 3. Resume in a new process. Before this fix, the completed child is downloaded and restored again. **Paperclip version or commit** The regression fails on the previous migration verifier. The final branch includes master at `189fefb99183953be37f0d1084275fbb0f35d6e2`. **Deployment mode** Managed Linux server with PostgreSQL and explicit S3 storage configuration. ## What Changed - Add optional opaque storage namespace identity. S3 binds it to the explicit endpoint, region, bucket, normalized prefix and addressing style. Implicit endpoints retain full verification. - Save a versioned child-snapshot receipt only after canonical crypto, manifest and file restoration succeed and a second HEAD confirms the same strong ETag and size. - Bind reuse to the complete snapshot reference, key material, child lineage, source operation and storage namespace. Require a fresh strong HEAD and canonical validation that the current secret provider can unwrap the saved snapshot key. - Keep current directory/Git capture and source-owner checks on every invocation. Read-only verifier callers do not write checkpoints. - Merge checkpoint metadata under a short row lock without changing parent journal hashes, snapshot retention or update timestamps. Bound reads and writes. Lock timeouts, statement timeouts, deadlocks and serialization conflicts are safe cache misses. Other errors remain fatal. - Extend real PostgreSQL/Git/archive tests with process death, scratch removal, resume, changed bindings and single publication. ## Verification - Baseline RED: two import invocations read each child twice. With durable receipts, each child is read once across both invocations. - Real process regression passed on the final rebased head: SIGKILL the owned importer after one checkpoint, delete all scratch, and resume in a new Node process. It retains the same operation and snapshot, reads each child once across both processes, and records one publication. - 53 focused cases passed in 104.86 seconds on the final production bytes before rebase. They cover existing nested/cohort behavior, namespace identity, changed source/reference/key/lineage/ETag, unknown proof versions, weak validators and bounded row contention. - After clean rebase and stronger scratch/snapshot assertions, the real process case passed again in 18.54 seconds. Server TypeScript, module boundaries and diff checks passed on final head `9da68ee7ce4a778201ef85e21a11c49eeea7bc16`. Required CI supplies final-head broad coverage. - The initial restricted-network dependency setup failed; unchanged preflight passed with normal network access. Initial TypeScript failures in test pipe assertions were corrected. No failed attempt is counted as a pass. - Review found a missing current-master-key check. A real restart regression with all child checkpoints complete reproduced unsafe publication after switching to a different valid master key. Follow-up `7ea92c2abd9ffdb2e7dd81cecf62c8fc7a6e74a5` delegates to the canonical snapshot-key owner before both durable and in-memory cache hits. - The fixed process regressions passed 2/2. Ten key-authority and existing identity/source guard cases passed. The wrong-key process now refuses with zero additional child payload GETs, no publication, and unchanged original journal/snapshot rows. Server typecheck, module boundaries and diff checks passed on the follow-up. - Review also found that transient database failures could make the optional checkpoint cache stop a valid import. Read and save now fall back only for SQLSTATE `55P03`, `57014`, `40P01`, and `40001`. Connection loss, identity changes, crypto failures, programming errors and cancellation remain fatal. Reads use a read-only transaction with one-second lock and eight-second statement limits. - The checkpoint database-fault regression failed two cases before the fix. All four recoverable import cases then passed. The final selected run passed 21 cases, including fatal-error and cancellation controls, real row contention, both process-restart cases and key/identity guards. Server typecheck and boundaries passed. Fault injection is limited to the checkpoint database boundary; the importer, PostgreSQL, Git, archives, storage and publication remain real. - Latest master was merged without rewriting published history. Six real import/process-restart cases passed again on final head `8c8e4213a680ebb5c8d6864416bc3a4f271e1e76` in 38.03 seconds. [Required CI](https://github.com/paperclipai/paperclip/actions/runs/42658248222) finished with 54 successful checks and four intentional skips; Snyk passed. [Fresh review](https://github.com/paperclipai/paperclip/pull/20311#issuecomment-6745354381) is 5/5 with all four threads addressed and resolved. Hosted verification after normal release delivery remains pending. ## Risks - A stale receipt could skip authentication of a different object. Complete reference/lineage/namespace binding, fresh strong HEAD and row-locked save validation prevent that reuse. Missing or unsupported hints trigger full verification. - Mutable local files can change between attempts. Their source locks and fresh captures remain required; a checkpoint never certifies current local bytes. - Checkpoint persistence can contend with other snapshot owners. It has one-second lock and eight-second statement limits; the four classified transient SQL errors retain the normal fully verified path. Other database errors stop the import so lost source ownership cannot be hidden. - No schema, archive format, public API or saved-run identity change. Older snapshots without receipts remain readable and get full verification. Existing data and recovery obligations stay intact. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. Context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
099adb2728 |
fix(recovery): settle a remote workspace save that stopped without finishing (#20291)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshots.
Each save is a `workspace_operations` row with phase
`remote_workspace_snapshot`
> - The ownership registry lists that kind as in-progress work, but it
had no settle step
> - When a restart cut off a failing final save, the row stayed
`running` after its invocations had all stopped and its run had ended
> - The retained-source fallback refuses while any workspace operation
is running, so a task whose saved files could not be verified could
never continue from its last save
> - This pull request lets the ownership reconciler settle that row as
failed
> - The benefit is that the automatic fallback can continue such a task
without any user action
## Linked Issues or Issue Description
**What happened?**
A native run in a remote sandbox finished its turn. Its final workspace
save failed because the sandbox disk was full ("No space left on device"
in the snapshot copy). A graceful restart then interrupted the retry.
The run ended as `native_workspace_sync_out_retry_exhausted`. Its
`remote_workspace_snapshot` row kept `status: running` for six days,
with `saveWork: { activeIds: [], unknown: false, quiescedAt: <time> }`
and `exportCleanup: { state: "pending" }`. The retained-source fallback
checked the task and refused with `workspace_operation_running`, so the
task stayed blocked.
**Expected behavior**
A save with no active or unknown invocation, whose run is terminal, is
finished work. The reconciler must mark it failed, so the fallback can
continue from the last committed save. A save that might still be
running must stay untouched.
**Steps to reproduce**
1. Run a native task in a remote sandbox. Make its final save fail, then
restart the server during the retry, so the save's work is quiesced and
the operation stays `running`.
2. Let the task's recovery refuse with an unverifiable-source failure
(for example `allocation_ownership_ambiguous`).
3. The retained-source fallback refuses with
`workspace_operation_running` on every check, and the task never
continues.
**Paperclip version or commit**
`master` at
|
||
|
Dotta
|
5c73633a8f |
fix(workspaces): preserve remote recovery holds during online import (#20298)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace import preserves old controller-local files while unrelated remote work continues. > - A task can have an old local checkout and a separate durable remote workspace. > - A terminal remote run can retain a file-recovery hold on that task. > - The legacy source checker rejects remote references, so that hold blocks independent local imports. > - This pull request proves that the hold belongs only to the separate remote source and fences its finalizer during import. > - Old files can migrate while the remote recovery obligation stays intact. ## Linked Issues or Issue Description Refs #19873 and #19854 for the existing cohort and source-verification paths. Searched related online migration and remote recovery PRs; no duplicate fix was found. **What happened?** Online import rejects a legacy Git checkout with `online_workspace_source_scope_unproven` when a same-task failed run has a prepared reference to a separate durable remote workspace. The legacy run reader rejects remote references before the hold can be classified. **Expected behavior** Independent local files can import under their source fences. The separate remote workspace, dirty writer, allocation, accepted result and recovery hold must stay intact. **Steps to reproduce** 1. Create related local Git workspaces. 2. Record a terminal failed native run and an exhausted file-recovery hold for a separate durable remote workspace on the same task. 3. Import the local leaf online. The old checker refuses before capture. **Paperclip version or commit** Reproduced on master at `052c638b519ebb15f81e4fcd1a96adf6cd3b2f10`. **Deployment mode** Managed server with PostgreSQL and remote Runner execution. ## What Changed - Add a distinct remote-only preservation proof. Keep remote runs excluded from legacy export sources. - Use the canonical source inspector and prepared-reference matcher. Bind the run, finalizer, action, task, accepted result, authority, allocation and prior committed snapshot. - Acquire the held run's finalizer fence through the existing sorted lock path. Existing source-proof reinspection rejects changes before publication. - Preserve previous-release prepared references with an older baseline snapshot. Do not require that snapshot's epoch or source run to equal the current run. - Add real PostgreSQL import, authenticated cold restore, preservation, refusal, lock-contention and publication-race regressions. Document the behavior. ## Verification - Genuine baseline failure on unchanged product: the real importer refuses with `online_workspace_source_scope_unproven`. - Final focused regression file: 23 tests passed in 13.19 seconds through the canonical isolated PostgreSQL wrapper on Node 24.22.2. - 24 existing affected migration/recovery tests passed against the same final production bytes. Another 83 tests were explicitly excluded by the focused selection; no full-suite pass is claimed. - The positive imports local bytes, confirms the remote finalizer fence is held during upload, leaves all remote records unchanged, parks the original repository and authenticates/restores the archive. - Refusal and race cases cover malformed, foreign, same-local and mixed references; active owners/writers; changed allocation, result, task, action, authority and baseline; and finalizer contention. - Server TypeScript check, module-boundary check and `git diff --check` passed. - An intermediate run had one incorrect assertion that expected an explicit empty legacy-source array. The canonical journal omits that empty field. The final assertion accepts absence and still forbids any exported remote source. Final relevant results are 23 new plus 24 existing passes, rather than a claim that the earlier invocation passed in full. - Required CI passed on head `6a278e717db2b5f614b0a595f99bfd0fb5430227`: 55 successful checks and 4 intentional skips. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42650025033). Greptile reviewed that head at 5/5 with no actionable findings or open inline/review comments. Hosted continuation remains pending normal delivery. ## Risks - A false disjointness proof could admit an unsafe local import. Strict company/task/source binding, the canonical inspector, finalizer fencing and publication revalidation retain refusal on uncertainty. - This does not recover or settle remote files. Dirty authority and the original recovery obligation remain unchanged. - No schema, public API, archive format or saved-run identity changes. Existing persisted state carries forward. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
ac0d0374f2 |
perf(issue-detail): measure task opens on real instances (#20276)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open a task from their issues list to see what an agent is doing and to reply > - On a busy staging instance, the composer and the latest messages take 1.1–2.0 s to appear after a click. For one task with a slow run log, they took 13.9 s > - Local fixtures do not reproduce this. They do not have real network round trips, real server contention, or real object-store latency > - This pull request adds the measurement we need: a browser harness that times a task open on a real instance, and a Server-Timing entry on every API response that separates server time from network time > - It also records the staging baseline and the improvement plan > - The benefit is that each later performance change can be measured on real data with the same tool, before and after deploy ## Linked Issues or Issue Description **Problem or motivation** Opening a task from the issues list is slow on instances with real data. The composer and the latest chat messages can take seconds to appear. We could not measure this on staging in a repeatable way. We also could not tell server time from network and queueing time for most API requests. **Proposed solution** Add a reusable browser harness for task opens, and add a `paperclip_api` Server-Timing entry to every API response. **Alternatives considered** Local Playwright fixtures only. They do not reproduce real network round trips, server contention or object-store latency, so they did not show the delays that people see. Per-endpoint Server-Timing only. It does not cover the many other requests that run during a task open. **Roadmap alignment** This is measurement for existing task-loading behaviour. It does not add product features. The staging baseline and the measured causes are in `doc/plans/2026-11-09-task-loading-performance.md`. Follow-up pull requests change the loading behaviour itself, starting with #20277. ## What Changed - Added `server/src/middleware/api-server-timing.ts`. Every `/api` response now has a `paperclip_api;dur=<ms>` Server-Timing entry. The value is the server time from request arrival until headers are written, including admission waits. The entry is appended, so the existing `paperclip_issue` phases stay first. - Registered the middleware first in `server/src/app.ts`. - Added `tests/perf/issue-detail/staging-click-harness.js`. It clicks a task row and records, frame by frame, when the composer and the latest message become visible, layout shifts, movement of the latest message, and the requests that run before the page is ready. The summary labels each sample instead of using the task reference. It replaces query values and every path segment that contains a digit. A fetch that is still running when the sample ends is listed as pending. - Documented the harness in `tests/perf/issue-detail/README.md`. - Added the dated plan `doc/plans/2026-11-09-task-loading-performance.md` with the staging baseline, the causes, and the prioritized slices. ## Verification - `cd server && npx vitest run src/__tests__/api-server-timing.test.ts`. 3 tests pass. They cover the JSON responses, the time spent in earlier middleware, preserving route phases, and explicit `writeHead` calls. - `tests/perf/issue-detail/issue-detail.perf.spec.ts` reads the first `dur=` value. That value is still `paperclip_issue`. - The harness was used on staging to capture the baseline in the plan document. ## Risks - Low risk. The header has only a duration, with no request, user or company data. - The middleware wraps `res.writeHead` once for each request. This is the same pattern that `on-headers` uses. - Server-Timing is visible to same-origin scripts only. No `Timing-Allow-Origin` header is added. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6fd46c0025 |
perf(budgets): judge Paperclip's share of restart recovery; report the provider's session load (#20183)
## Thinking Path - Speed budgets protect the time to first visible agent activity, and restart recovery is one of the judged metrics (`native-speed-budgets.ts`, wave grader). - In the B1 concurrency-6 chaos wave (`w-20261109T045320-s9807`), Codex `restart_recovery` was p50 8.6 s and p95 11.2 s, against a budget of 6.5 s and 8.0 s. At concurrency 4 (K13) it was 3.8 s and 5.0 s. - The preparation spans on a copy of the kept database break it down: - Paperclip's share (ready → claim → prepare → runtime stage → runner launch) is a steady 1.2–2.0 s. - `provider.session.load`, the Codex app-server boot plus `thread/resume` of long saved sessions, takes 3.0–6.3 s, 65–75% of each recovery. - Resumes run concurrently, and the worktree lock and the startup reconciler are not on the path. - Judging the full latency therefore mixes our code with provider and session-size effects. Budget the part we control, so a regression in our code shows up regardless of session size or concurrency, and report the rest. ## Linked Issues or Issue Description **What happened?** `restart_recovery` breached its budget whenever many long provider sessions resumed at once, although Paperclip's own recovery work stayed within about 2 s. The breach did not point at anything Paperclip could fix, and a real regression in our code could hide inside the provider-load variance. **Steps to reproduce** 1. Run a local chaos wave at concurrency 6 with long-running Codex tasks and K1/K2/K4 faults (for example B1, seed 9807). 2. Compare `restart_recovery` in `speed.json` with `provider.session.load` in the recovered runs' run logs. **Expected behavior** The budget judges Paperclip's share of a recovery. The provider's session load and the full latency are reported but not judged. ## What Changed - `server/src/services/native-runtime/native-speed-budgets.ts`: - New budgeted metric `restart_recovery_control_plane`: from `ready` to the start of `provider.session.load`, or to the next provider event when nothing reloads (for example a reattached runner). - Its budget is 2.5 s / 4.0 s for claude, codex and opencode. Measured: 1.2–2.0 s per run at concurrency 6. - The full `restart_recovery` stays in the table for reporting. - Breach payloads carry `restartTotalMs` and `providerLoadMs`. - `server/src/services/native-runtime/native-run-trace.ts`: - The per-run check records when `provider.session.load` starts, keeps the recovery measurement, and judges it when the run finishes. - It judges after the deferred diagnostics drain, so a provider span committed after the provider event still counts. - It emits `restart_recovery_control_plane` breaches only. - `tests/restart-chaos/ttft-baseline-stats.ts`: each recovery sample gains `controlPlaneMs` and `providerLoadMs`. The provider-load start is the span event's write time minus its duration. - `tests/restart-chaos/wave-speed.ts`: - Three restart rows per adapter: `restart_recovery` (report-only), `restart_recovery_control_plane` (budgeted) and `restart_provider_load` (report-only). - All three have outage time removed. - `tests/restart-chaos/ttft-baseline.ts`: the recovery table adds a "Paperclip share" column. - Docs: `doc/run-log-events.md` (metric and payload) and the speed-budget table in `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`. ## Verification - `server`: `npx vitest run src/services/native-runtime/native-speed-budgets.test.ts`, 18 passed. Two new tests: - a recovery dominated by provider load does not breach; - a slow Paperclip share breaches with `restartTotalMs` and `providerLoadMs`, even when the provider span is committed after the provider event. The existing recovery tests moved to the new metric. All six were red before the trace change. - `tests/restart-chaos`: `npx vitest run --config tests/restart-chaos/vitest.config.ts wave-speed.test.ts ttft-baseline-stats.test.ts`, 29 passed. New tests cover the sample split and the grader rows (a judged share, a report-only provider load, zero breaches). - Server typecheck: no errors in the touched files. ## Risks - **Lower sensitivity to provider-side slowdowns.** A provider regression in session load no longer breaches. It stays visible in the report-only rows and in breach payloads. - **Runs with no `provider.session.load` span** (reattach, or an adapter that doesn't emit it) are judged on the full latency against the control-plane budget. That's conservative. - **Run-log consumers that filter on `metric: "restart_recovery"`** see `restart_recovery_control_plane` from now on. The doc notes the change date. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
95d476c57b |
docs(plans): record the measured seed experiment for per-turn uploads (#20139)
## Thinking Path - #20100 proposed sending only the changed files on per-turn follow-ups, with the repository seed held inside Daytona in a snapshot or a volume. It asked for one measured experiment before building. - Snapshot and volume operations were blocked by our Daytona key's permissions. A restricted key with those permissions now exists, so the experiment has run. - This PR records the numbers and the decision in the design doc. ## Linked Issues or Issue Description Refs #20100 **What happened?** The #20100 design's seed-location choice was unmeasured: snapshot build time and volume read speed were both open questions. **Steps to reproduce** 1. Run the SDK timing of direct upload, volume extract, and snapshot build and create on the paperclip seed (58.7 MB gzip) in Daytona `us`, 2 vCPU. **Expected behavior** The design doc states the measured winner, the per-turn saving, the one-time cost, and the work needed to build it. ## What Changed - `doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md`: - The status now reads: measured; snapshot chosen; not built. - New "Experiment results" table: - direct upload: 6.4 s per turn; - volume: 6.8 s per turn, which rules it out; - snapshot: about 2–3.5 s per turn, after a 177 s build per seed generation. - Findings: fork is unsupported, compression is exhausted, and the snapshot builder runs as `daytona` and does not extract archives. - New "Work to build the snapshot seed" list: lifecycle, sandbox creation, verification and fallback, expiry and cleanup, and rebuild policy. ## Verification Docs only. Every Daytona resource the experiment created was deleted, and an account-wide listing confirmed none remained. ## Risks None at runtime. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2a16be36ab |
docs(plans): per-turn follow-ups without re-uploading the repository (#20100)
## Thinking Path - Per-turn remote sandboxes stay the default while warm reuse (#20048) is on hold. - On the paperclip clone, a per-turn follow-up spends about 13 s in `workspace_sync.transfer`, though its changes pack in 0.5 s: every fresh sandbox receives the whole 70 MB (gzip) seed again, at about 5 MB/s from the controller. - The coordinator asked for a design that uploads only changed files into a fresh sandbox, with a full-upload fallback, measured against the spans. It also asked to build it only if the implementation is bounded and safe. ## Linked Issues or Issue Description Refs #20048 **What happened?** Per-turn follow-ups on large repositories wait about 13 s re-uploading the full workspace seed even when almost nothing changed (pack 0.54 s, transfer 13.3 s, time to first token about 31 s). **Steps to reproduce** 1. Run `tests/restart-chaos/ttft-baseline.ts --env daytona --workspace-repo <paperclip clone>` with per-turn sandboxes. 2. Compare `workspace_sync.pack` and `workspace_sync.transfer` on the follow-up turn. **Expected behavior** A follow-up uploads only its overlay into a sandbox that already holds the seed, falling back to the full upload when unsure. ## What Changed - `doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md`. It contains: - the measured span table and the size analysis (220 MB archive, 70 MB gzip, about 5 MB/s); - the expected saving: about 12–13 s per follow-up, time to first token about 31 s → about 18 s; - a design keyed by seed generation with a content-hashed manifest, using a Daytona snapshot (preferred) or a Daytona volume to hold the seed; - the overlay-only turn flow, with a verified fallback; - one bounded experiment to choose the seed location before building. - Not built yet: both seed locations need provider capabilities (snapshot build time, volume read speed) that must be measured first, so the implementation isn't bounded until the experiment picks one. ## Verification Docs only. ## Risks None at runtime. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c02fd8d427 |
docs(plans): salvage-then-retire design for kept restore-source and done-task warm sandboxes (#20087)
## Thinking Path - Paperclip keeps some remote sandboxes after their run: warm ones for reuse, and failed runs' stopped sandboxes as workspace-restore sources. - In the warm Daytona chaos waves for #20048, the real leaks were restore sources that nothing retires once a user Retry closes their recovery actions. Separately, a done task's suspended warm sandbox is only destroyed when its environment is deleted. - Retiring them early needs a provider-I/O salvage step: compare against the durable save, export anything unsaved to the issue, then destroy. That spans recovery, storage, attachments and teardown. The coordinator asked for a design note so it can be its own reviewed task instead of a rushed patch. ## Linked Issues or Issue Description Refs #20048, #20085, #20086 **What happened?** Closed restore sources and done-task warm sandboxes stay suspended until their environment is deleted, costing storage. No path compares them with the last durable save or exports their unsaved files before retirement. **Steps to reproduce** 1. A run fails; its sandbox is kept as a `retain_on_failure` restore source. 2. The user clicks Retry; a fresh run succeeds and the task is done. 3. The restore-source sandbox is never retired until the environment is deleted. **Expected behavior** After a clean compare, or a confirmed export of anything unsaved to the issue, the sandbox is destroyed. If salvage fails, it stays suspended and is retried. ## What Changed - `doc/plans/2026-11-08-restore-source-salvage-and-warm-retirement.md`: a design for one owned kind (`environment_kept_sandbox_retirement`) on the shared reconciler. It covers listing rules for both cases (24 h default retention for done-task warm sandboxes), a three-stage salvage state machine (compare, export, retire) with a provider-I/O step and receipt table, failure handling that never destroys without proof, and the test plan. ## Verification Docs only; no code or tests changed. ## Risks None at runtime. Additive documentation. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b32ef9f111 |
perf(budgets): judge time to first visible agent activity; keep time to first text as a reported metric (#20077)
## Thinking Path - The native speed budgets (plan invariant 8) judge how fast an agent responds. They judged `task.time_to_first_token`: time to the first deliverable **text**. - In the clean local restart storm (w-20261108T153612-s9401), Claude breached its TTFT budget (cold p50/p95 17.4/34.7 s against 16.5/30). The per-run spans showed this was not a control-plane wait: in all 49 cold runs (and 76% of warm), Claude started a tool call before writing any text. Its first visible activity came at 7.4 s p50 from run creation; first text came at 19.6 s. Codex never calls a tool first. - So first text charges model tool work as latency for tool-first adapters. What the user experiences as "the agent responded" is the first visible activity: a text delta, a tool-call start, or a thinking indicator. - The coordinator decided: budgets judge time to first visible agent activity from run creation, with the budget numbers unchanged; first text stays a reported metric with no budget for tool-first adapters. ## Linked Issues or Issue Description **What happened?** Local storm s9401: the Claude TTFT budget breached on every aggregate, while the control-plane steps (queue 0.05 s, prepare 0.3 s, runner startup ~4 s, first provider event 1.3–1.8 s) were well within budget. The rest of the first-text wait was Claude streaming tool input (about 5.5 s per call) and running tools before it wrote text. **Steps to reproduce** 1. Run a Claude task whose first action is a tool call (most real tasks). 2. Read the run log: `provider.time_to_first_agent_event` ends at the tool call; `task.time_to_first_token` ends many seconds later at the first text. 3. The speed budget judges the later one and reports a breach. **Expected behavior** The budget judges the first visible agent activity from run creation (`task.time_to_first_agent_activity`). Time to first text is still recorded and reported. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: when the first meaningful agent event commits (the existing `firstMeaningfulAgentEventKind`: text delta, tool-call start, or reasoning), record `task.time_to_first_agent_activity`. It uses the same detached anchor resolution as TTFT (accepted comment, answered question, durable wake request, or run creation), so it adds nothing to the turn path. - `server/src/services/native-runtime/native-speed-budgets.ts`: the metric is now named `time_to_first_activity` (was `ttft`), with identical budget numbers. The type comment documents the definition and why first text is unbudgeted. - `server/src/services/native-runtime/native-run-trace.ts`: the budget check judges `task.time_to_first_agent_activity` from run creation, keeping the existing input-wait handling. `task.time_to_first_token` is no longer budgeted. - `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`: report first activity from run creation (the budgeted value) next to TTFT. - Docs: `doc/observability.md` (new span) and `doc/plans/2026-11-07-ttft-baseline.md` (metric table). - Tests: - A tool-first turn records first activity at the tool call (5 s) while TTFT records first text (16 s). - A tool-first trace with slow first text but fast first activity is within budget. - The existing budget tests now judge the activity span. Observability path (AGENTS.md section 5.7): the new run-log span uses only the existing `eventKind` and `anchor` attributes. Its name survives run-log redaction intact; checked with `redactEventPayload`. The operator OTLP gate is unchanged. It is not Telemetry. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts`: 44/44 pass. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "first visible activity|first assistant text once|first-token timing"`: 7/7 pass. - `npx vitest run --root tests/restart-chaos`: 159/159 pass; `tsc` on `tests/restart-chaos` is clean. ## Risks - **A metric rename.** Consumers of `run.performance.budget_exceeded` see `metric: "time_to_first_activity"` instead of `"ttft"`. In-repo there are none besides the tests; Phase 3's chaos speed grade switches to the new span. - **A looser budget for slow first text.** Intentional: first text is reported, not judged. A regression in control-plane latency still shows up in first activity. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e8a3f1f6f2 |
feat(recovery): prove a replaced container's records gone through a fenced controller epoch (#20054)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, the control plane settles in-progress records (leases, claims, intents) only once it proves their owner process is gone > - The proof compares host and PID space, so a stamp from a replaced container or a rebooted machine stays unprovable forever, and its records never settle > - Self-hosted users on Docker, Kubernetes and Fly hit this on every redeploy; a Railway deployment with an exclusive volume is covered by `deployment_replaced` (#20026) > - This pull request adds a fenced controller epoch: each boot holds a database lease, and an expired boot is fenced so its records can be proven gone > - The benefit is that work stranded by a replaced container is recovered, without keeping an idle database awake ## Linked Issues or Issue Description Follow-up to #20026 and the Phase 2 durable-ownership data. Related: #19965 (owner stamps), #19883 (process space). No duplicate PR was found. **What happened?** A record stamped by a server process in another container reports `different_process_space`. A dead PID elsewhere is never proof, so the record stays `unprovable` and is never settled. Every redeploy of a self-hosted container leaves such records behind. **Expected behavior** A record owned by a server boot that was replaced (stopped cleanly, or lost its lease and was fenced) is proven gone and settled exactly once. A live boot's records stay with it. An idle server issues no recurring queries. **Steps to reproduce** 1. Run Paperclip in a container and start work, so an environment lease is stamped by that boot. 2. Replace the container. The new container has a new PID namespace. 3. The ownership reconciler reports `different_process_space` (unprovable) for the old lease, forever. **Paperclip version or commit** `master` before this PR. ## What Changed - New `controller_instances` table (migration 0449): boot id, epoch (bigserial), process space, pid, a lease on the database clock, `fenced_at` / `fenced_by_epoch`, `stopped_at`. - `controller-instance-lease.ts`, registered with the shared delivery-work coordinator (`controller-instance-lease` queue, runs during drain): - **Holds work → renews; idle → silent.** One upsert creates the row on first use and extends the lease only if the boot holds work: a stamp within the last 90 s, or a stamp whose ownership write may still be pending (held until the server's tracked work settles, capped at 10 minutes; a renewal that sees some other held record does not clear it, so a slow ownership write cannot be outlived), or an in-progress record held by its boot id. Those records are active environment leases, in-flight cleanup claims, sandbox creation intents, workspace finalize attempts, agent lifecycle steps, native finalizer and cleanup leases (`lease_holder` bound to the token), open native coordinator claims, and running legacy runs. Renewal runs every 30 s while the boot holds work. When it finds none, the lease lapses and the worker sleeps until the next stamp (`onControllerStamp`). - **No idle trailing statements.** When the server's last tracked work settles, the worker re-checks holdings at once, at most every 5 s (the rate limit also absorbs the worker's own run). So the last check happens with the work, not 30 s later. - **Quiesce, don't crash.** Each renewal has a 10 s bound. A failed or timed-out renewal applies an internal task drain, so no new work is dispatched. A recovered renewal resumes. The boot exits only after 60 s of failure (counted from the first failed attempt's start) while it still holds work, or at once when it finds itself fenced. - **Fence on demand, by a newer epoch.** For a stamp from another host or PID space, the prover reads that boot's row: - `stopped` or `fenced`: gone. - live lease: alive. - otherwise, the boot's last sign of life is the later of its lease expiry and the record's last write plus the TTL. Within 30 s of that, the owner stays `alive` (`controller_lease_grace`), so the settle watch keeps rechecking. After that, the prover fences the boot, but only with a lease row of its own (created on demand) and only an older epoch. - A graceful shutdown records `stopped_at`. - A reconciliation pass that settles nothing (`record: false` or `settle: false`) never fences and never creates the boot's row. - `owner-proof.ts`: these proofs apply only to server-held records. A stamp with a recorded process group stays unprovable, because agent processes can outlive their server. The reconciler passes each record's `updatedAt` to the proof. - `task-admission.ts`: - the quiesce drain never replaces an operator's drain, and has no idle owner, so HTTP stays open; - a `subscribeIdleWorkSettled` hook fires when the last tracked work finishes. - Off on a Railway deployment with an exclusive volume (it has its own replacement proof). `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=on|off` overrides this. A stateless Railway service keeps the lease. A server without its own lease row never fences. - `maintenance-server-routes.test.ts`: in the post-run quiet window, the lease may check once more that the run's records were released. The test now excludes those statements, then asserts that the lease issues nothing in a further window. - Docs: `doc/execution-semantics.md`; the Phase 2 data doc records the two known gaps. ## Verification - `controller-instance-lease.test.ts` (15 cases): - a replaced container with all Railway variables empty is fenced and settled exactly once over two passes; - a live lease is alive; - an owner in grace, and work written after an idle lapse, stay alive and watched; - an older epoch never fences a newer one; - no fencing without its own lease row (off or unregistered), and an idle prover creates its row on demand; - a stopped boot is gone; - process-group stamps stay unprovable; - idle issues zero statements, and one upsert renews only while records are held; - **a 45 s outage:** quiesced, no exit, resumes; - **a renewal that never answers:** bounded, and the boot exits on time; - **a 70 s outage:** exits at 60–65 s, and the successor recovers the work exactly once; - fenced: stops; - the settle re-check; - graceful stop; off on a volume-exclusive Railway deployment, on for a stateless one, and the on/off override; - a running legacy run keeps the lease live. - Red, with the proof's instance check removed: the replaced-container, live, stopped and successor cases fail. - Wider: task-drain, idle-sleep, idle-local-work, owner-proof and ownership suites, plus the maintenance quiet-window tests. Server low-memory typecheck is clean. ## Risks - **Clock skew.** The proof compares a record's `updatedAt` (often the writer's clock) with the lease (database clock). The 30 s grace absorbs ordinary skew. - **A new exit path exists.** A server whose database is unreachable for more than 60 s while it holds work now exits, after first quiescing. This is the intended safety backstop. - **Bounded trailing activity.** A lease that holds work issues one statement every 30 s, plus at most one check when its work settles. An idle server issues none. - **Boots from before this release** have no row, and their records stay unprovable. The board "mark owner gone" action for those follows in a separate PR. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c966334ec3 |
fix(recovery): settle a run-less lease once its stamped holder is proven gone (#20026)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane recovers in-progress records after a server restart through a shared ownership path > - Run-less sandbox leases (login, probe, harness verification) were released only by a five-minute idle sweep > - That sweep releases by age: it leaves a dead holder's sandbox for minutes and takes a live login session's sandbox > - This pull request settles a run-less lease only when its stamped holder is proven gone > - The benefit is no stuck sandbox after a restart, and no live session loses its sandbox ## Linked Issues or Issue Description Related: #19965 (named the owner of login/probe leases), #19964 (the run-lease settle this mirrors). No duplicate PR was found. **What happened?** The final Daytona chaos wave `w-20261108T083036-s9401` recorded I2 `environment_login_probe_lease` (`pid_dead`) on 2 records. These were the codex agent's harness-verification leases. A restart killed their holder. The leases stayed `active` until the five-minute idle sweep released them. The same sweep releases any run-less lease after five idle minutes, also when its holder is alive. **Expected behavior** A run-less lease is released as soon as its stamped holder is proven gone. A lease whose holder is alive is never released, however long it is idle. **Steps to reproduce** 1. Start a Daytona wave: `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image <image>` (K1/K4 faults). 2. Let a fault kill the server while an agent's harness verification holds its sandbox lease. 3. See the reconciler report I2 `environment_login_probe_lease` until the idle sweep releases it. **Paperclip version or commit** `master` at `8475826d2d`. ## What Changed - New `settleEnvironmentLoginProbeLease`, registered on `environment_login_probe_lease`. It runs only for a stamped holder that is proven gone, and compares and swaps on that exact stamp. - It moves the lease to `pending_cleanup` with the recovered-orphan reason, so the cleanup sweep destroys the sandbox in its next tick. - It leaves the lease alone when a legacy shutdown fence applies, or when another live lease holds the same provider resource. - The idle sweep for run-less leases now covers only legacy rows saved before holders were stamped (no `leaseController` key). Stamped rows are never released by age. - `holderUnchanged` is now exported from the run-lease settle so both settles share it. - The controller stamp (`leaseController`, and every other `ControllerProcess` stamp) now records the platform deployment when the platform names one (Railway: project, environment, service, deployment, volume). `controllerStamp` passes it to `proveOwnerGone`. A container redeploy changes the PID space, and the old stamp would read `different_process_space` (unprovable) forever. Now the existing `deployment_replaced` proof shows the old holder gone. This also covers `environment_run_lease_active` and the other kinds built on `controllerStamp`. - `currentDeploymentIdentity` in `owner-proof.ts` now delegates to `currentControllerDeployment`, so there is one parser. - Data doc: added the final Daytona wave, the six Daytona-only crash points (4 pass, 2 not reached, no violations), and the cause of the roughly 450 s codex TTFT. Runs waited behind the agent's harness verification, which restarted after each fault; boot itself took about 32 s. - Restart-chaos README: corrected which Daytona crash points a plain task reaches. ## Verification - Red, before the change: `environment-login-probe-lease-ownership.test.ts` failed 3 of 6. No settle happened, and the idle sweep released 3 live holders' leases. - Green: `pnpm exec vitest run` passes for `environment-login-probe-lease-ownership`, `environment-run-lease-ownership` and `heartbeat-orphaned-active-lease-sweep` (45/45), and for `environment-lease-owner-stamps`, `heartbeat-observed-workspace-preservation`, `ownership-reconciliation-report`, `heartbeat/recovery` and `workspace-recovery-warm-handoff`. - Redeploy: a stamp from a replaced Railway deployment settles with `deployment_replaced`. A stamp from the same deployment in another PID space stays `active`. This was red when `controllerStamp` dropped the deployment. - Live-owner counter-cases: the current process, another live PID, another PID space, and a lease re-stamped after the proof all stay `active`. A read-only pass never settles. - Not re-run live: the Daytona wave and crash points ran before this fix. The next Daytona wave should show no `environment_login_probe_lease` I2. ## Risks - A stamped run-less lease whose live holder never releases it, because of a bug in that process, is no longer released after five idle minutes. It is released once that process exits. This is the intended trade: never take a live holder's sandbox. - On an ephemeral server without a platform deployment identity (not Railway), a stamp from a replaced container is still unprovable. Such a run-less lease stays `active` until the provider expires the sandbox, and the reconciler reports it as unprovable. Before this PR, the five-minute sweep released it by age. This is the same open gap as unstamped legacy leases on an ephemeral server: there is no operator action yet. It is listed as a Phase 2 follow-up. - Legacy unstamped run-less leases keep the age fallback. These are rows written before #19965, and no current code writes them. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
68d5fb5a6e |
feat(chaos): check I6 and I8 per crash point; ready the Daytona points (#19984)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The on-demand crash-point suite (#19946) kills the control plane at named durable write boundaries and checks the plan's invariants > - Its first full run measured I1–I4 and I7, but I5, I6 and I8 were reported as not measured > - #19960 (generation stamping) and #19863 (speed budgets) now record what I6 and I8 need in the run log > - The six Daytona-only points have not run yet, and three of them cannot be reached by a plain task > - This pull request checks I6 and I8 per point, names the I5 gap, and readies the Daytona points for the coming Daytona wave > - The benefit is that every invariant is either asserted or a named gap, and a Daytona run explains any point it cannot reach ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5). Follows #19946 (the suite). Uses #19960 (event generations) and #19863 (speed budgets). **Problem** - The suite reported I5, I6 and I8 as "not measured". - A Daytona run would report three points as `not_reached` with no reason. - A Daytona run without its key or image failed only after a 3-minute boot. **Expected behavior** - I6 and I8 are checked at each point. - I5 is a named gap with its reason. - Daytona points say what they need, and a Daytona run checks its inputs before it boots. ## What Changed - `tests/restart-chaos/invariants.ts`: `runLogViolations` checks two invariants from the run log, scoped to the wave's company: - I6: a run that failed with an error that names a stale or earlier-generation event; - I8: each `run.performance.budget_exceeded` event, with the budget evidence (metric, measured, p95, adapter, start kind). Both are keyed by run or event id, so the per-point baseline diff charges each one to the point that produced it. I5 is reported as `notMeasured` with a reason (`I5_GAP`). - `packages/shared/src/crash-point-registry.ts`: an optional `prerequisite` on the three Daytona points a plain task cannot reach. These are the teardown claim (needs a failed release), the reusable destroy (needs a `reuse_by_environment` lease), and stop-and-retain. - `tests/restart-chaos/crash-points.ts`: - `--list` prints each point's `needs:`, and a `not_reached` result names it too; - `--env daytona` refuses to start without `DAYTONA_API_KEY` and an image. - Docs: the README and the plan describe the I5/I6/I8 coverage and the Daytona points. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/crash-points.test.ts`: 12 passed. - `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and `npx tsc --noEmit -p packages/shared` both pass. - I ran `runLogViolations` against a throwaway Postgres 18 with seeded rows, then stopped and deleted it. It returned: - I8 for the budget breach, with its evidence; - I6 for the run of this company that failed on `stale_generation_event`. It excluded the other company's stale run, the `process_lost` run and a non-budget event. - `node scripts/crash-points.mjs --list --env daytona` lists 19 points, with `needs:` on the three. `--env daytona` with no key or image fails at once with the missing names. - No harness run: the harness slot is held for other work. The next suite run exercises the new checks. ## Risks - Test harness only; no product code. - I6 is a proxy: it matches the run's error text and code. A failure that names a generation for another reason would be reported, with its evidence, for a person to judge. - I8 can flag breaches on a loaded host. Each breach carries its measured and budget values, so a load-caused breach is visible as one. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
46bf913fb9 |
docs(plans): record the Pi steer duplicate after a restart and its follow-up (#19982)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can steer a running Codex or Pi turn, and since #19970 a queued follow-up steers automatically > - Restart recovery (#19962) delivers a steered message that the model did not read before the restart, exactly once for Codex > - Pi gives the run log no read evidence for a steer, so after a restart a Pi steer that the model already read can arrive again > - The fix needs changes to the closure-pinned Pi distribution, which waits for the next Pi re-qualification > - This pull request records the limit and the follow-up plan so they are not lost > - The benefit is that the next Pi re-qualification can include the definitive fix ## Linked Issues or Issue Description **What is the documentation problem?** The Pi steering limit after a restart, and its planned fix, were only in pull request text and session notes. **Where?** `doc/plans/2026-11-07-pi-steer-restart-limit.md` (new). **What should it say?** Codex delivers an unread steered message exactly once after a restart. Pi can duplicate a message it already read, and never loses one. The note also gives the reasons, the follow-up plan (option A), its costs, and the rejected options. Refs #19962, Refs #19970 ## What Changed - New `doc/plans/2026-11-07-pi-steer-restart-limit.md`. It covers the limit, why Pi has no read evidence, the follow-up (forward Pi's native user `message_start` through pi-acp, acpx, the sidecar and Rust into a `userMessage` item), its re-qualification costs, and the rejected options. ## Verification - Documentation only. Read the note. The facts match `packages/paperclip-runner/src/steering-delivery.ts`, `patches/pi-acp@0.0.33.patch`, `pi-closure-pins.ts` and `rules/acpx-profile-identity.mdc`. ## Risks - None. Documentation only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
051834bdc8 |
feat(chaos): add the on-demand crash-point suite and its manual CI workflow (#19946)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart between two durable writes leaves an in-progress record owned by a dead process > - Random restart-chaos waves found these 20 ms–1 s windows only by luck > - #19937 added `crashPoint(name)`, and #19938 put it at 20 durable write boundaries > - Nothing yet runs a task, kills the server at a chosen point, restarts it, and checks the plan's invariants > - This pull request adds that on-demand suite, as a local command and as a manual CI workflow (not a required check, per Decision 2) > - The benefit is a repeatable work list: which boundary breaks which invariant, for Phases 2–4 ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5 and Decision 2). Refs #19861 (report-only reconciler used for I1–I7). Stacked on #19938 (call sites), which is stacked on #19937 (helper). **Problem** The named crash points cannot be exercised end to end. No command arms one point, runs real agent tasks across it, restarts, and grades the result. **Expected behavior** `node scripts/crash-points.mjs --points <glob> --kinds <id> --adapters <list>` runs one wave per selected point and reports each point as pass, fail (with failing R rules and violated invariants), not reached, or harness error. A `workflow_dispatch` workflow runs the same thing on demand. ## What Changed - **Runtime arming.** `PAPERCLIP_CRASH_POINT_CONTROL=<file>`, in the shared helper and the runner mirror. Each hit reads `<token> <name>[:<nth>]`, and a new token resets the count. It is test mode only: `NODE_ENV=production` ignores it, and the disarmed path is still one comparison. - **`tests/restart-chaos/main.ts --crash-point <a>:<nth>,<b>:<nth>,...`** (config, types, ledger): - one booted server and no Vite UI; - per point: take an invariant baseline, launch the point's batch, arm the point, then on rest disarm it and check invariants as violations that are new since the baseline; - every task is graded R1–R6 and T; - `crash-point.json` (v2) holds one entry per point. - **`tests/restart-chaos/supervisor.ts`:** - every spawn gets the control file; - a SIGKILL whose receipt names the child's pid is restarted like K1: the control file is emptied first, the owned tree is killed, and a replacement starts. A failed tree cleanup fails the fault; - the isolated instance pins `PAPERCLIP_IN_WORKTREE=false` (a `.paperclip/.env` above the checkout otherwise suppressed all scheduling); - `RESTART_CHAOS_HEALTH_TIMEOUT_MS`. - **`tests/restart-chaos/invariants.ts`:** - the Phase 1 report-only pass with `record: false`: two real-clock passes for I1 and I2, and one pass 15 minutes ahead for I3, I4 and I7; - spool kinds are read from the wave's instance; - an unfinished pass is a harness error; - each violation carries its wake and run state as evidence. - **`tests/restart-chaos/crash-points.ts` and `scripts/crash-points.mjs`:** - select points by glob, kind and environment, and run them in one wave; - a per-point outcome: pass, fail, not_reached or harness_error; - `SUMMARY.md` with a violation evidence column. - **`.github/workflows/crash-points.yml`**, manual only, under the paid-workflow contract: - a credential-free `authorize` job; - a `suite` job in `runner-e2e-paid` whose first step reauthorizes; - a frozen install with `--ignore-scripts` and no shared cache; - only the selected adapters' keys; - the workflow is added to both security-test allowlists. - **Docs:** "Crash-point suite" in `tests/restart-chaos/README.md`, and "Built (Phase 5)" plus the first work list in the plan. ## Verification - Focused tests: - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/crash-points.test.ts`; - `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15 tests, including runtime arming in a real child process; - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 72 tests; - `tests/runner-e2e/workflow-security.test.ts`: 19 tests. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Loop line: question "how long is one crash-point loop?" | loop: `node scripts/crash-points.mjs --points <one>` (codex A1) | elapsed 2.7 min end to end: boot 9 s, crash 9 s after arming, replacement ready in 6 s, rest about 2 min | result: one point alone takes about 3 min. Before this change it took about 15 min (one cold boot per point, Vite UI, host at load 77). - Full local run, 14 points, codex A1, 11.8 min in all: - every point was reached (the server killed itself at each one, and the replacement came back); - 11 pass; - `workspace.finalize.after_start_before_copyback` and `workspace.finalize.after_copyback_before_done` fail I2 and R6: a `workspace_finalize` operation stays running after the run succeeded; - `wake.claim.after_run_running_before_wakeup_claimed` reports I4. Its evidence shows the wake failed with "…retrying once", and the retry delivered the work, so this is the known refusal-regex false positive. - The 6 Daytona-only points were not run. ## Risks - Test harness and an on-demand workflow only. No product code changes. - `.github/**` needs a CODEOWNER review. The workflow uses provider keys, so it refuses non-default branches and actors that are not on the allowlist, like `runner-full-stack-e2e.yml`. - `PAPERCLIP_IN_WORKTREE=false` is pinned in every chaos wave, not only crash-point waves. A chaos instance is never a worktree dev instance, so scheduling must not be suppressed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b4842df1e4 |
docs(plans): Phase 2 Daytona wave results (#19947)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Phase 2 of the durable-ownership plan moves in-progress record kinds to one shared ownership path, guided by chaos data. > - The first data doc (#19927) covered local waves only. Remote runners, leases and sandboxes need a Daytona wave. > - This pull request adds the results of one Daytona wave to the data doc. > - The benefit is that Phase 2 and Phase 3 prioritize remote kinds from measured data. ## Linked Issues or Issue Description Refs #19927 (data doc), #19934 (claimed-wake settle), #19863 (speed budgets). **What is the problem?** The Phase 2 data doc had no remote data. **Where is it documented?** `doc/plans/2026-11-07-phase2-reconciler-data.md`. **What should change?** Add the Daytona wave's per-kind violations, settle latency, failures and speed numbers. ## What Changed - Added a "Daytona wave" section: 6 tasks with K1, K2 and two K4 outages of about 575 s. 5 of 6 passed, and R6 was clean. - Recorded the t03 failure: a Retry was cancelled by the company-wide accounting gate (Phase 3 scope). - Recorded the claimed-wake leak again (#19934 fixes it) and remote settle times of 76–107 s. - Corrected the earlier note about the image content-ID mismatch. ## Verification - Wave `w-20261108T012041-s9301`; the numbers come from a copy of the kept database and the read-only ownership sampler. - This is a docs-only change. ## Risks - Low. Docs only. The samples are small and the doc says so. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
f35eb1e5ef |
fix(observability): judge the TTFT budget from run creation; report the wait behind a live turn (#19941)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Invariant 8 of the durable-ownership plan says turns must start fast. #19863 added per-run speed budgets that write `run.performance.budget_exceeded` to the run log. > - The Phase 2 chaos data (#19927) showed follow-up TTFT about 3x over the claude budget. > - The cause is measurement scope. A follow-up comment posted during a live turn waits for that turn to end, and TTFT is anchored at the comment. The budgets came from idle follow-ups. > - This pull request judges the TTFT budget from run creation and reports the wait separately. > - The benefit is that budget breaches point at real start-up slowness, not at time spent behind a turn the user is already watching. ## Linked Issues or Issue Description Refs #19863 (speed budgets), #19927 (Phase 2 data), #19852 (startup sub-spans), #19888 (warm default). **What happened?** In two local storm waves, C1 follow-up runs breached the claude warm TTFT budget (p50 27.6–29.9 s against 9.5 s). Their spans show `comment.to_run_created` at 6–62 s (waiting for the live turn), then session resume at 1.4–3.8 s and submit-to-first-event at 2–5 s. Codex without the wait was about 4.8 s, the same as the warm-follow-up baseline. **What did you expect?** The budget should judge start-up latency, which is what its baseline measured. The wait behind a live turn should be visible but not judged. **Steps to reproduce** 1. Run a local chaos storm with C1 tasks. 2. Read `run.performance.budget_exceeded` rows with metric `ttft` for `issue_commented` runs, and compare them with each run's `comment.to_run_created` span. ## What Changed - `native-run-trace.ts`: the TTFT budget check measures from run creation (start of `heartbeat.queue`, or end of the ingress span). The TTFT span itself does not change. - `native-speed-budgets.ts`: breaches can carry `inputWaitMs` (TTFT only). - `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`: the wait behind a live turn is its own measured step (`inputWait`), next to the full TTFT and the TTFT from run creation. The TTFT table gains a "Wait behind live turn" column. The full comment-to-first-token time stays the user-visible metric. - `doc/run-log-events.md` and the plan's speed-budget section describe the rule. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts`: 43 pass, 2 new. `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/ttft-baseline-stats.test.ts`: 12 pass, 1 new. They cover a follow-up under budget after a 60 s wait (no breach), a follow-up over budget (breach with `inputWaitMs`), and queueing after run creation still judged. - Server typecheck passes. ## Risks - Low. This is a run-log-only change (AGENTS.md §5.7): no Telemetry and no span attributes change. - A slow wait before run creation is no longer a TTFT budget breach. The TTFT span still records it, and the breach payload names it. Admission holds after run creation are still judged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
17b9e10252 |
docs(plans): Phase 2 reconciler data; let chaos K4 holds reach 600 s (#19927)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Control-plane restarts leave in-progress records that each subsystem releases in its own way. The durable-ownership plan replaces those paths with one shared path. > - Phase 1 added a report-only reconciler. Its light local waves showed zero violations, and that proves little. > - Phase 2 moves kinds to the shared path. It needs data that shows which kinds really leak, and the speed numbers to protect. > - This pull request records that data from harsher waves. It also fixes the harness so that K4 outages past the reconnect grace qualify. > - The benefit is that Phase 2 moves kinds in order of measured risk, against a known speed baseline. ## Linked Issues or Issue Description Refs #19831 (durable-ownership plan), #19861 (report-only reconciler), #19863 (speed budgets), #19797 (K4 hold cap). **What is the problem?** Phase 2 needs per-kind invariant data and speed data from harsh restart waves. Only light local waves existed. In addition, a K4 hold over about 405 s always failed its fault receipt, so the 600 s cap from #19797 could not be used. **Where is it documented?** `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` (Phase 2 rows) and `tests/restart-chaos/README.md`. **What should change?** Record the data in a dated plan doc. Let chaos K4 holds run up to 600 s and still qualify. ## What Changed - Added `doc/plans/2026-11-07-phase2-reconciler-data.md`. It covers 5 unmasked local waves (two storms, a crash loop, two K4 runs). It gives per-kind violation counts with example rows, owner verdicts, TTFT, boot and restart recovery against the budgets, Phase 3 gate notes, and what is not covered (Daytona). - Added a short pointer to that doc in the durable-ownership plan. - `tests/restart-chaos`: new flag `--k4-hold-max-seconds` (120–600). It raises the upper end of the seeded K4 hold. The default range (120–300 s) does not change. - `tests/runner-e2e/controller-outage.ts`: the receipt's ready bound is now a parameter (`CONTROLLER_READY_BY_MS`, default 420 s). Runner suites keep 420 s. The chaos supervisor passes hold + 420 s, so a hold past the Runner's reconnect grace qualifies. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 12 files, 135 tests pass, including the new flag test. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/controller-outage.test.ts`: 36 pass, including the new long-hold receipt test. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and `tests/runner-e2e`: pass. - Live: the K4 wave at seed 9203 failed 3 of 6 tasks with the old bound (fault error "did not produce a qualifying replacement receipt"). It passed 6 of 6 with this fix, with holds of 294–579 s. ## Risks - Low risk. The changes are test harness code and docs only. The runner suites keep the 420 s ready bound by default. - The data is local only. The doc says which kinds the waves do not reach. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). The model ran the chaos waves and wrote the analysis and code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
99529f9d71 |
feat(observability): local speed budgets and a run-log budget check (#19863)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When someone messages an agent, the reply must start as fast as
possible, and restart-safety work must not slow it
> - The durable ownership plan makes this invariant 8 ("fast turns") and
asks Phase 1 to turn the Phase 0 baseline into p50/p95 budgets per
adapter
> - Without budgets, a regression in time to first token, agent boot or
restart recovery shows up only when someone reruns the baseline by hand
> - This pull request records the budgets and adds a per-run check that
writes a run-log row when a run exceeds its p95 budget
> - The benefit is that every later phase can show "no regression
against the budgets" from the run log, without new waits on the turn
path
## Linked Issues or Issue Description
Refs #19831 (plan, "Performance constraint", rule 4, and invariant 8).
Refs #19846 (the Phase 0 baseline these budgets come from). Stacked on
#19861.
**Problem**
The plan requires p50/p95 budgets per adapter for TTFT, agent boot and
restart recovery latency, and a check against them. Neither exists.
**Expected behavior**
Budgets are recorded in the plan and in code. A run that exceeds a p95
budget gets one run-log row that names the metric, the measurement and
the budget.
**Impact**
Phase 2–5 exit criteria can cite budget breaches from chaos waves and
staging. The check adds no wait before `turn.start`.
## What Changed
- New `server/src/services/native-runtime/native-speed-budgets.ts`: the
local budget table (codex, claude, opencode) for TTFT and agent boot
(cold and warm) and restart recovery latency. Each budget is 1.2× the
baseline, rounded up to 0.5 s. `evaluateSpeedBudget` judges one run
against p95. An unknown start kind uses the looser budget. Remote runs
and adapters without a baseline are not judged.
- `native-run-trace.ts`: the trace takes an optional `budget` subject.
It checks `task.time_to_first_token`, agent boot (end of
`heartbeat.queue` to the end of the first `runner.turn.submit` or
`runner.session.resume`) and restart recovery once each. It writes
`run.performance.budget_exceeded` through `defer`, so the write is
detached from the event path. `finish` now drains diagnostics that defer
more work.
- `native-session-executor.ts`: passes the adapter and environment to
the trace, labels cold or warm at the existing `provider_session` reuse
receipts, and checks restart recovery at a recovered run's first
provider event, measured from the startup recovery `ready` time.
- Docs: budget tables in
`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` ("Speed
budgets (Phase 1)"), and the new event in `doc/run-log-events.md`.
## Verification
- `npx vitest run
src/services/native-runtime/native-speed-budgets.test.ts
src/services/native-runtime/native-run-trace.test.ts` (in `server/`): 34
tests pass. They cover:
- every budget has p50 ≤ p95;
- the provider-to-adapter mapping;
- p95 judgement, and the looser budget when the start kind is unknown;
- one breach for a slow boot, and none for a second submit or a fast
TTFT;
- a deferred TTFT breach is written before the trace closes;
- restart recovery is checked once;
- remote runs and runs with no budget subject write nothing.
- `npx tsc --noEmit -p server`: no errors.
## Risks
- Low risk. This is a run-log change only (AGENTS.md §5.7, no extra
review). The OTLP and Telemetry paths are unchanged.
- The baseline samples are small (2–12 per cell) on a shared host.
Budgets carry 20% headroom, and the codex restart budget (8.0 s p95) is
tight. Expect some breach rows on a loaded machine. They are diagnostics
and change no behavior.
- Restart-recovery budgets use run 4 of the baseline, as merged in
#19846.
- Restart recovery is measured from the startup recovery `ready` time to
the first provider event, which matches the baseline definition.
- Previous-release story: no persisted run state changes. The new event
is additive.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
d5dd09c966 |
docs(plans): design turn-scoped ACPX MCP authority; correct the Claude baseline (#19912)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Follow-up messages must start fast. #19888 made Codex and OpenCode follow-ups 2.6–5× faster by keeping their runners warm. > - Claude (ACPX) did not gain, because ACPX relaunches the Claude process and reloads its session on every follow-up (3.2–4.2 s). > - That relaunch is a safety boundary: the ACPX MCP endpoint lives as long as the process and cannot attribute a tool call to a turn, so reaping the process is the proof against late calls. > - Removing it needs a new authority model, which needs a reviewed design before any code. > - This pull request adds that design and corrects the Claude baseline. ## Linked Issues or Issue Description Refs #19888 (warm default), #19846 (TTFT baseline, finding 4). **Docs issue** The TTFT docs did not explain why Claude follow-ups relaunch, what removing that would require, or the corrected Claude numbers. **What is wrong or missing** `doc/plans/2026-11-07-warm-follow-up-ttft.md` reported a Claude follow-up at 10.5 s, measured under heavy host load. A quieter re-measurement gives 6.3 s per-turn against 7.1 s warm. There was no design for removing the per-turn relaunch. **Proposed fix** A design document for turn-scoped ACPX MCP authority, and a corrected-baseline section in the warm follow-up doc. ## What Changed - New `doc/plans/2026-11-07-turn-scoped-acpx-mcp-authority.md`, which covers: - the measured cost breakdown; - how the current boundary works (tool bridge bearer, call-ID tombstones, `restart_idle_provider`, `run.attach`); - a threat model (late tool calls from a settled turn, replay, credential replay, cross-turn and cross-run confusion, restart replay); - three candidate designs with their safety arguments: turn-scoped credentials in a live session, a proof that no calls are in flight, and an in-process session reload with per-generation credentials, which is recommended for a spike; - restart and chaos implications, the expected TTFT gain, and next steps. - `doc/plans/2026-11-07-warm-follow-up-ttft.md`: corrected Claude baseline. Per_turn stays the default; the earlier 10.5 s was host noise. Finding 4 now points to the design and drops the idle-retirement idea, which would only bring warm level with per-turn. - Docs only. No code changes until Dotta has reviewed the design and a security review is done. ## Verification - The Claude numbers come from `ttft-baseline.ts --adapters claude --samples 6 --phases coldwarm --lifecycle <per_turn|warm>` on master `fc6e1b0346` (load average about 9). - The step breakdown comes from temporary timing in runnerd's ACPX backend. It was not committed. - `node scripts/check-maps.mjs` passes. ## Risks - Docs only. The design proposes no behavior change until it has been reviewed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fc6e1b0346 |
perf(native): keep local Codex and OpenCode runners warm by default (#19888)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When someone messages an agent, the turn must start fast. Time to first token (TTFT) is a first-class metric. > - The Phase 0 baseline (#19846) found that every local turn launches a new runner and loads the provider session again, because `per_turn` is the default lifecycle. That made a follow-up message slower than a new task. > - Warm runners keep the runner and provider session alive between runs. #19874 fixed the defect that broke warm OpenCode follow-ups, and #19879 bounds an idle warm runner whose controller dies. > - This pull request makes warm the default for local Codex and OpenCode agents, and migrates existing agents that only carried the old creation default. > - The benefit: follow-up TTFT p50 drops from 12.0 s to 4.7 s (Codex) and from 21.7 s to 4.2 s (OpenCode), with no new restart-chaos failures. ## Linked Issues or Issue Description Refs #19846 (TTFT baseline), #19852 (startup sub-spans), #19874 (warm completion contract), #19879 (warm reconnect grace). **What existing behavior does this improve?** The default runner lifecycle for native agents, and how a follow-up message starts. **Current behavior** Agent creation writes `lifecycleMode: "per_turn"` into every native agent. Every follow-up launches a new runner and restores the provider session: 4.2 s (Codex) and 16.3 s (OpenCode) at p50 before the turn starts. **Proposed behavior** New native agents store no lifecycle. On a local target, an unset lifecycle resolves to `warm` with the default 5-minute idle window for Codex and OpenCode. Other harnesses, remote targets and managed providers without warm support stay `per_turn`. An explicit choice always wins. Migration 0443 clears the stored creation default from existing agents, except where an edit set "Turn by turn" explicitly. **Reason and benefit** Measured locally (`doc/plans/2026-11-07-warm-follow-up-ttft.md`). Warm follow-ups are 2.6–5× faster for Codex and OpenCode, and restart recovery latency is equal or lower. ## What Changed - `native-lifecycle.ts`: an unset `agentMode` resolves to warm (5-minute idle window) for local `codex_app_server` and `opencode_server` agents. `runtime-selection.ts` passes `localTarget`. Explicit `per_turn` or `warm` wins. An admitted session keeps its persisted policy. ACPX Claude stays `per_turn`: it relaunches its provider process on every follow-up by design, so warm did not shorten its follow-up (10.7 s warm vs 10.5 s per_turn). - Creation and configuration writes no longer store `lifecycleMode: "per_turn"` for Codex, Claude or OpenCode. This covers `normalizePaperclipRunnerAdapterConfig` (runs on every create and update), `paperclipRunnerTransitionConfig` (switching to the native runner), `agent-runner-selection.ts`, `codex-local` `build-config.ts`, and the simplified new-agent setup (`paperclipRunnerCreationLifecycle` in `@paperclipai/shared`). Grok, Pi and Cursor keep their explicit value. - Agent form: the Runner lifecycle field shows the unset state as "Automatic". "Turn by turn" and "Warm session" remain. - Migration `0443_native_warm_follow_up_default.sql`: clears `lifecycleMode = 'per_turn'` from native Codex, OpenCode and Claude agents unless `agent_config_revisions` shows an edit that set it to `per_turn`. Each migrated agent gets an `agent.updated` activity entry with the old value (`actor_id = migration:native-warm-follow-up-default`). Creation records no revision, so a "Turn by turn" pick made at creation cannot be told apart from the old default. The product owner approved migrating those agents. Claude agents follow the default, which is `per_turn` today. - Restart chaos harness and TTFT driver: `--lifecycle per_turn|warm|default` (saved in the wave plan). `default` removes the catalog's pinned lifecycle, so a wave exercises the real default path. - Docs: `doc/DEVELOPING.md` "Runner lifecycle default", and the results in `doc/plans/2026-11-07-warm-follow-up-ttft.md`, which also documents baseline finding 4 (Claude's resumed turn-start gap is ACPX's per-turn process generation, not the provider). ## Verification - `server`: `native-lifecycle.test.ts` (new default cases: local Codex and OpenCode warm, Claude, remote, unknown target, unmeasured harness, managed Claude, admitted per-turn session, explicit choices, non-native adapter), `agent-runner-selection.test.ts`, `runtime-selection.test.ts` (335 pass). - `packages/db`: `native-warm-follow-up-default-migration.test.ts` on embedded Postgres. Default agents are cleared, the explicit edit is kept, other harnesses and legacy adapters are untouched, audit rows are written once, and a rerun is a no-op. Without the explicit-edit filter the test fails. `check:migrations` passes. - `codex-local` `build-config.test.ts` (65), `ui` `config-fields.test.tsx` (40), restart-chaos unit tests (134), `tsc --noEmit` for `server`. - `adapter-utils` `paperclip-runner-lifecycle-default.test.ts`: configuration writes and adapter conversion store no lifecycle for Codex, OpenCode and Claude, keep `per_turn` for the others, and preserve explicit choices. It fails without the fix. - End-to-end default check on this branch (`ttft-baseline.ts --lifecycle default`, codex and opencode, 2 samples): the agents read back with no stored lifecycle, and every follow-up reused the runner and provider session (no `runner.process.launch`). Follow-up TTFT was 3.6–4.8 s. This check found the per-write `per_turn` injection fixed here. - Restart chaos (codex and opencode, A1/B1/B5, K1/K2/K6, seed 4242), 8/8 PASS each with no open signatures: forced warm, forced per_turn, and `--lifecycle default` on this branch (agents created without a lifecycle). - `pnpm verify` selects broader CI (harness config is a global input), so required CI supplies the broad coverage. ## Risks - Behavior change: local Codex and OpenCode agents keep a runner and provider process for up to 5 minutes after a run. Instance sleep waits for that idle window. The process is reaped by the idle timer, by graceful shutdown, or by the #19879 reconnect grace if its controller dies. - Migration: agents whose "Turn by turn" was chosen at creation become Automatic. The activity entry records the old value; choosing "Turn by turn" restores it. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): a task session keeps the lifecycle policy it was admitted with, so sessions saved before this change stay `per_turn`. New sessions get the default. No run, checkpoint or launch state changes shape. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
5a70999306 |
Add a versioned online workspace migration command (#19882)
## Thinking Path > - Paperclip preserves the files and recovery state produced by its agents. > - Canonical online migration can preserve selected local workspaces while unrelated work continues. > - Operators sometimes need a newer migration owner before the serving web process can take that release. > - The existing packaged maintenance entry requires stopped writers. > - This pull request adds a finite online entry from one coherent reviewed release tree. > - The entry delegates capture and publication to the existing owners and keeps tool and serving identities separate. ## Linked Issues or Issue Description Refs #19854, #19855 and #19872. The duplicate search found related owner fixes and no separate versioned online entry. **What happened?** The finite online workspace importer was available as a library in the serving image. The packaged all-phase migration entry required maintenance mode. **Expected behavior** Run the reviewed online owner from a separate complete release tree without replacing the web process, stopping unrelated work or changing its release policy. **Deployment mode** A managed instance with its original local volume, database, existing encryption key and shared S3 storage. ## What Changed - Add strict inspect, single, cohort and exact single-journal resume requests with explicit company, workspace, deadline and concurrency bounds. - Verify the private tool root and its own build stamp separately from the serving source commit and deployment. - Reuse the nonmutating full migration-manifest inspector and require the additive controller host-identity column before publication. - Hydrate existing configuration and key bindings, then use the original ESM database pool and canonical storage, source locks and migration owners. - Keep multi-member inspection with the optional canonical cohort preflight in a read-only transaction with denied storage. Releases without that owner refuse cohort operations; single import and resume remain available. - Register the optional runtime bundle entry and document the supported full-tree source mode, resource admission and reconciliation. - Add hash-bound old-writer/reader compatibility and encrypted restore tests. ## Verification Final local revision: `c9345b72f27aee1158047dccd0b1060f9a0be6df`, on master `fbe689938c3bf8c5849d2cd56b9da90029db9389`, macOS arm64 with Node 24.22.2. - Entry and runtime packaging: **34/34 PASS**. - Real PostgreSQL/filesystem compatibility: **6/6 PASS**, disposable database wrapper exit 0 and cleanup complete. The frozen old source lock owner contends with the new entry; unrelated work proceeds. The old authority and snapshot readers decrypt and restore newly published bytes. - Pending and uploaded journal cases use the unchanged old persisted format and preserve operation/snapshot IDs through resume. - Full layered server source and test typecheck: **PASS**. - Module boundaries and `git diff --check`: **PASS**. - Actual source-mode module import under Node 24 succeeds without starting the application. - A separate temporary full-old-producer fixture did not get past module setup: 90 seconds under Vitest and a 15-second direct Node control. Neither is counted as an executed old-producer test or a product failure. The smaller frozen old-writer/reader proof and old-format journal cases are the local evidence. - Hosted resume of the actual saved operation and Linux artifact qualification remain operational acceptance gates. This PR does not claim those ran. - Review regression: a real successful import returned a null operation ID (genuine RED). The repaired entry binds the exact returned snapshot to its succeeded migration journal after commit (GREEN). Committed or superseded snapshots are accepted; a missing readback requires reconciliation rather than a blind retry. - Final affected check: **40/40 PASS** across the entry, packaging and actual PostgreSQL compatibility suites; wrapper exit 0 and cleanup complete. Full layered typecheck and boundaries pass after the user-requested rebase and current Runner rebuild. - A second real postcommit database-read fault also has genuine RED/GREEN proof. The succeeded journal, saved authority and original bytes remain intact, and every readback error now preserves the reconcile-required warning. - Final-head [CI run 42404116001](https://github.com/paperclipai/paperclip/actions/runs/42404116001): **PASS**, with all 59 check contexts terminal (55 success and 4 intentional skips). Fresh Greptile review is **5/5** on `c9345b72f27aee1158047dccd0b1060f9a0be6df`, with no unresolved threads; check and thread pagination are complete. ## Risks - The tool must come from one authenticated complete release tree with its own dependencies. A copied bundle chunk can retain old absolute paths and is not a portable release. - Concurrency is not resource admission. The operator must check memory, CPU and scratch headroom and start with one import or two cohort members. - Schema must be prepared by the existing schema owner. This entry never applies DDL. - Cancellation awaits the canonical owner. An SSH interruption does not prove the process stopped; durable journals must be reconciled before retry. - The entry preserves remote obligations and source files. It does not prove enrollment readiness or zero-downtime operation. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact serving snapshot and context-window size are not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
965ca91dd0 |
Capture legacy workspace cohorts under one source fence (#19873)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Workspace migration preserves local files as authenticated portable
snapshots.
> - Workspaces that share Git storage require one complete source fence.
> - Independent imports contend on that fence and cannot safely run as a
batch.
> - This change adds an opt-in cohort owner that captures members
concurrently and publishes them atomically.
> - Existing source, recovery, snapshot and publication checks remain
authoritative.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The administrative remote-workspace migration owner for workspaces in
one shared physical source component. Related changes: #19688, #19854
and #19855.
**Current behavior**
Each independent import acquires the complete source fence. Parallel
independent calls therefore contend instead of overlapping archive work.
A caller cannot use a surrounding transaction and assume that a
successful savepoint confirms the outer commit.
**Proposed behavior**
An explicit `migrateRemoteWorkspaceCohort` call selects one to eight
leaves from one proven component. It owns the full
source/lifecycle/copyback fence, uses the original pool for bounded
archive work, validates every member, publishes serially, and returns
success only after the outer commit.
**Reason and benefit**
This supplies a correctness-gated path for a controlled two-member
hosted qualification. It makes no wall-clock speedup claim. No running
app call site or scheduler selects cohorts automatically.
**Breaking changes**
The existing single-import API retains its guards and rejects a journal
owned by a cohort. New cohort journals require the same complete member
set on resume.
## What Changed
- Extract the existing reservation, capture, validation and publication
stages for reuse by both owners.
- Add an opt-in cohort API with default concurrency two and a maximum of
eight. Every member must share the same complete physical/Git/alias
component; selected roots must be leaves relative to each other.
- Persist exact cohort and operation membership in the existing journal.
Compare all selected fingerprints with the aggregate proof before the
first reservation under the complete fence, and recheck each member when
reserving its journal. Refuse foreign journals, incomplete or mixed
replay sets, lost fences and changed sources. Keep all authorities and
activity events unpublished if the outer commit fails.
- Share the merged nested-child verifier inside one complete fence.
Queue its prepare calls, retain its two-child bound and authenticated
immutable proof, and recheck local bytes and database bindings on every
pass.
- Document the API, exact-membership resume protocol, resource bounds
and required hosted qualification.
## Verification
- Integration head `fce1896010`: 32/32 PostgreSQL/filesystem cases
across the cohort owner, the complete merged nested-verifier suite, and
five existing parent-import cases.
- Final source-proof repair: 19/19 cohort-owner cases pass, including
real first- and last-member metadata changes before reservation,
changed-source partial-resume refusal, mixed local/imported refusal with
storage forbidden, cancelled/failed/finished admission, a late finished
timestamp, complete fences, queued child-proof reuse, cold restore,
cancellation, commit failure and exact replay. All local database checks
use the canonical owned wrapper, one Vitest worker and Node 24.22.2 on
macOS arm64/PostgreSQL 18.1.
- Final rebased head is based on
`fbe689938c3bf8c5849d2cd56b9da90029db9389`, including current
source-owner, database-schema and Runner contract changes. The 19-case
owner suite, full server layered source/tests/test-tree typechecks,
module boundaries and diff checks are rerun on that base; required CI
supplies the native Runner qualification.
- Selection-only `node scripts/local-verify.mjs --base
|
||
|
DottaandClaude Opus 5.5
|
6cfb69b4b0 |
docs(plans): Phase 0 ownership inventory and TTFT baseline (#19846)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart can land between two durable writes. It leaves an in-progress record owned by a dead process. About 69 restart-chaos PRs fixed that shape one subsystem at a time. > - The durable ownership plan (#19831) replaces those fixes with one owner stamp, one prover, write-ahead intents and one reconciler. Phase 0 must first list every in-progress record kind and measure how fast turns start today. > - Without the inventory, Phase 1 cannot choose one stamp or one proof. Without the baseline, no later phase can show that it kept turns fast (invariant 8). > - This pull request adds the Phase 0 inventory, the TTFT, boot and restart-recovery baseline, and the run-log-only driver that produced the baseline. > - The benefit is one reviewed map of 83 record kinds to plan sections A–G, and measured p50/p95 budgets per adapter. ## Linked Issues or Issue Description Refs #19831. Refs #19838 (the TTFT anchor that the baseline uses). **What is the problem?** Phase 0 of the durable ownership plan has no inventory of in-progress record kinds and no speed baseline. **What is the expected outcome?** Two reviewed documents in `doc/plans/`, plus a repeatable way to measure the baseline again in later phases. **Why does it matter?** Phase 1 cannot start without them, and every later phase's exit criteria compare against the baseline. ## What Changed - `doc/plans/2026-11-07-ownership-inventory.md`: - 83 durable in-progress record kinds (94 reader rows) across native runtime, environments, workspaces, accounting and the wake queue; - each kind with its table and column, writer, states, owner, release rule, proof, obligation, invariants and plan sections A–G; - the overlaps, the merge decisions and the zero-downtime upgrade record kinds. - `doc/plans/2026-11-07-ttft-baseline.md`: - TTFT, agent boot, pre-turn steps and restart recovery per adapter (codex, claude, opencode), local, cold and warm, p50/p95; - restart recovery from the overnight chaos evidence, the slowest steps ranked, and the gaps. - `tests/restart-chaos/ttft-baseline.ts`: an on-demand driver on the chaos supervisor. It reads the run log only, never Telemetry, and always stops its embedded Postgres. - `tests/restart-chaos/ttft-baseline-stats.ts` and its unit tests: pure percentile, boot, pre-turn ranking and recovery statistics. No runtime behavior changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/ttft-baseline-stats.test.ts` - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` - The baseline numbers come from the driver run named in the report. To reproduce them, follow the report's "Method" section. - The owner spot-checked the inventory's file:line claims at `f2d8a51027`. One example: `processLossRetryCount` has no writer, and `heartbeat/recovery.ts:2675` is its only reader. ## Risks - Low risk. This adds documentation and an opt-in test driver. Nothing runs in CI except the pure stats unit tests. - The driver spends a small amount of provider credit when someone runs it by hand. - The baseline sample is small (6 per cell). The local host was shared with other worktrees, so the p95 tails include load. The report states both. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use, code execution and five parallel read-only subagents for the inventory. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
54b83b15a3 |
Reuse authenticated nested workspace verification (#19854)
## Thinking Path > - Paperclip preserves the work that its agents produce. > - Portable workspace migration can save nested workspaces before their parent. > - The parent must prove that a committed child snapshot covers any files it omits. > - The existing owner downloads and restores the same immutable child archive before and after parent capture. > - The fresh local-file checks are required, but the repeated immutable archive check is avoidable within one live source fence. > - This pull request reuses only an authenticated archive proof with an unchanged reference and provider identity. > - It also bounds independent child branches at two workers while retaining every fresh local and Git check. ## Linked Issues or Issue Description Refs #19805 and #19688. A duplicate search found no separate fix for repeated child archive verification. **What existing behavior does this improve?** Online Git parent migration after nested workspaces have committed portable snapshots. **Subsystem affected** server/ — workspace migration and nested snapshot coverage. **Current behavior** Each parent verification pass downloads, authenticates and restores every child archive, then captures the child's current local tree. The second pass repeats the immutable download and restore. Independent child branches also run in sequence. A large parent can exhaust its existing budget before parent capture begins. **Proposed behavior** One private verifier belongs to the existing live source-owner transaction. It can reuse a completed archive proof only when the full snapshot reference, operation, child closure, ETag and byte length still match. Missing or weak ETags and non-cancellation HEAD errors keep full verification. Caller cancellation still stops work. A changed identity forces fresh authenticated readback. The owner still captures current local bytes and Git state in both passes. **Reason and benefit** Remove duplicate immutable work without treating an old local-file check as current authority. Two independent topmost branches can overlap. Recursive descendants stay serial within those branches, so the bound does not grow with depth. ## What Changed - Add a private per-fence verifier to the existing tree owner. It clears its proof map at the end of the import and refuses a different owner transaction. - Reuse only completed ciphertext, GCM, plaintext, and canonical restore verification with a matching fresh provider identity. - Retain all current database, operation, recursive coverage, local-file, Git and manifest checks on each pass. - Bound independent branches at two. Abort queued work on failure and await active peers before scratch cleanup or fence release. - Use the verifier only in the existing online Git importer. Keep the offline full-verification path and all capture deadlines unchanged. - Document resource admission, fallback behavior and the hosted acceptance limit. ## Verification - Genuine baseline RED through the real parent importer: child GET counts 2 + 2, canonical restores 4, maximum overlapping readbacks 1. The four fresh local child captures already pass. - Exact positive GREEN: child GET counts 1 + 1, canonical restores 2, maximum overlapping readbacks 2. All four fresh local child captures remain. - The fixture uses real encrypted archives and the canonical S3 provider's local HTTP HEAD parsing with actual payload-derived ETags. It uses fake fixture credentials only. - Final focused suite: **13/13 PASS**, 148.36 seconds, at `eaf17d3340`. This includes recursive max-two completion, failure/cancellation joining, identity/ref/content changes, same-fence alias reuse and stale-owner refusal. - Both affected existing migration suites: **256/256 PASS**, 201.84 seconds, at initial head `eaf17d3340`. - The HEAD-failure review repair has genuine two-case RED evidence and three passing failure/cancellation controls. Full layered source/test-tree typecheck and module boundaries pass at repaired head `763648adad`. Final exact-head controls: **21/21 PASS**, 86.63 seconds (all 16 new controls plus five existing parent cases). - The first extended recursive fixture correctly failed because it created a Git branch after taking an earlier snapshot. The fixture now creates all branches before any import; the source-change guard remains intact. - Exact-head CI: 56 success, four expected skips, and one neutral Paperclip Review. Required gates pass; Greptile is fresh 5/5. Its optional aggregate-handler cleanup recommendation is resolved with the intentional fail-closed rationale. - Local verification is macOS arm64 / Node26.4.0. The observed hosted parent still needs to complete within its unchanged budget after normal delivery. No hosted timing improvement is claimed here. ## Risks - Two simultaneous child verifications can use more I/O, memory and scratch space. Operators must count logical verifications, even in one process. - HEAD availability and provider identity are required for proof reuse. Providers without a usable ETag perform the full check. - Reuse lasts only for one live transaction. A retry, changed reference, changed provider identity, process exit or owner change requires fresh verification. - Faster verification does not waive changed child bytes or guarantee a specific hosted completion time. No provider turn, authority marker, source deletion or larger timeout is introduced. ## Model Used OpenAI GPT-6 through Codex, with repository and code-execution tools. The exact serving model revision and context-window size are not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
0f5ce5050e |
docs(plans): plan durable ownership and reconciliation for restart safety (#19831)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs must survive a control-plane restart: they must continue on their own, keep their work, and offer a Retry that works > - The 2026-11-06/07 restart chaos campaign found about 40 restart failures and merged about 69 fixes (#19469 through #19807) > - Most of these failures have one shape: a restart stops a process between two durable writes, and each subsystem then decides on its own whether the leftover record is safe to release > - Each fix closed one crash window, and about 12 fixes rebuilt the same "this owner is dead" proof > - This pull request adds a plan for one shared ownership record, write-ahead intents, one reconciler, and an on-demand crash-point suite > - The plan also makes time to first token and agent boot hard requirements, so restart safety never makes turns slow > - The benefit is that we fix the whole class of failure once, and we stop fixing one window at a time ## Linked Issues or Issue Description Refs #19696, #19713, #19717, #19724, #19741, #19747, #19784, #19807 **What existing behavior does this improve?** Recovery after a control-plane restart. Today each subsystem keeps its own in-progress records and its own release rules. **Subsystem affected** Native runtime, runner, environments and sandboxes, workspaces, accounting and budgets, and the wake queue. **Current behavior** A restart can leave a lease, a fence, a claim, a workspace operation, a receipt, or a deferred wake that belongs to a dead process. If no subsystem can prove that the owner is dead, the system fails closed. The task then needs a Retry, the company holds new work, or a run stays `running`. About 40 server files read a controller boot id, and about 25 files check process liveness in different ways. **Proposed behavior** One owner stamp and one liveness proof for every in-progress record. Intents are written before outside effects. One reconciler brings every record to a terminal state. Admission never waits on cleanup. Accepted work is delivered exactly once. Time to first token and agent boot are measured and stay within budgets. **Reason and benefit** Fewer restart failures, less duplicate code, and a fast first response after a restart. ## What Changed - Added `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`. The plan contains: - the five failure classes from the campaign, with example IDs and PR counts - eight invariants - a performance section with time-to-first-token, boot, and restart-latency metrics and rules - the design (sections A–G) - an on-demand crash-point suite - six phases - risks - the decisions that Dotta made on 2026-11-07 ## Verification - This change adds documentation only. No code changes. - Read the plan. Check that each failure class cites real merged PRs. - Check the code claims: - `git grep controller-process-identity -- server/src` shows three users. - `git grep task.time_to_first_token -- server/src` shows the existing metric. ## Risks - Low risk. The change adds one plan document and changes no code. - The plan proposes later behavior changes. The largest is the removal of the company-wide accounting gate. Each later phase will come as its own pull request with tests. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, git, GitHub CLI). The model coordinated the chaos campaign and wrote the plan from its evidence. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
af0fe8fe81 |
Keep personal MCP connections functional while idle tenants sleep (#19427)
## Thinking Path > - Paperclip manages work for AI agents. > - Personal MCP connections should remain enabled while an idle tenant sleeps. > - Unfinished OAuth, transfer and delivery work must retain its owner and deadline. > - Cloud can receive bounded metadata and incoming work while the tenant remains the company-scoped authorizer. > - Current identity, revision and delivery acknowledgments establish when ownership has transferred. > - This change preserves useful wake, retries and revocation while making settled ownership inspectable without writes. ## Linked Issues or Issue Description Refs: #19377 and #19750. Hosted companion: [paperclip-cloud#1217](https://github.com/paperclipai/paperclip-cloud/pull/1217). **What happened?** Enabled personal MCP grants and subscriptions could keep an otherwise idle tenant awake. Ignoring those records would lose discovery, authorization, delivery and revocation obligations. **Expected behavior** Unconsumed authorization codes, device requests, file handoffs and delivery work retain their original deadlines. Settled connections remain enabled and become quiet after the current external owner acknowledges responsibility. Incoming useful work resumes normal authorization and delivery; revocation still takes effect for the current company. **Steps to reproduce** Complete normal PKCE/device consent and company selection, synchronize client/device and revocation state, retain the connection while idle, then exercise useful tools/events and revocation. Sleep eligibility requires current incoming/deadline ownership throughout. ## What Changed - Add bounded signed client/device metadata and revocation synchronization with exact revision, identity and ACK checks, startup recovery and finite legacy-device cutover. - Preserve company selection, authorization and operation identity. Register grant work before its INSERT in the same transaction; retain rollback, expiry and retry behavior. - Require current managed identity/resource before hosted publication. Receipt-cleared re-enable must obtain a fresh source ACK before device creation. - Reuse read-only settings and inspection budgets; connect MCP owners to normal startup, drain and shutdown. - Bound metadata pages and the actual signed request body while preserving the default limits of other operations. - Generate migration 0440 after the merged Legacy migration. Preserve all 438 existing executable migrations and the journal prefix; retain the normal five newest snapshots. - Preserve the merged transaction-owner forwarding and regression coverage from #19750 through normal ancestry integration. ## Verification - Current own Shared/SDK, database, eval-kernel and Runner builds passed, together with all seven normal application TypeScript programs. Missing generated package exports/workspace links were restored through normal setup; the original setup failures were retained. No source, compiler option, test assertion or timeout was weakened. - Normal migration generation changed only the five intended MCP tables; its SQL is byte-identical to the reviewed MCP delta. Fresh reference databases and both retained master/Muse histories upgraded through 0440, preserving historical hashes, row fields, schema and idempotence. Direct assertions cover the Legacy null default and MCP null/zero defaults. Two selected history cases passed; the other 21 database cases were unselected. - All six current MCP lifecycle cases passed with zero skips: normal PKCE approval intent, ACK recovery after outage, mutation during publication, disable/re-enable with original expiry, company-deletion rollback/commit and key rotation. - All 81 checks in five current whole files passed with zero skips: OpenAPI 34, read-only settings 2, diagnostic catalog 25, PostgreSQL diagnostics 7 and dependency scheduling 13. PostgreSQL fixtures ran normal migrations before test imports. - Earlier canonical proof at App `cd1c8c6` / Cloud `9f67531` includes real 60-second rate and three 15-second quiet windows, useful work and three revokes. Later paired proof at App `43af514` / Cloud `914eb0f` plus its reviewed candidate includes normal DCR, PostgreSQL metadata, a 144,017-byte signed publication, actual ACK, cap/default-413 controls and a real 15-second quiet window. The relevant producer/signing/wire inputs remain byte-identical in this integration; shared changes are qualified above. These remain source-scoped observations, not a fresh full canonical run on the integrated tree. - All known test owners, process groups and OS sockets closed; an independent final PostgreSQL audit found no remaining test sessions. One dependency-test trace recorded 46 opens and 45 close events. Its cause is unknown; this is not claimed as proof of graceful closure. - Fresh exact-head remote checks passed: 59 successful and 4 skipped contexts, with a complete current 5/5 Greptile review and all seven review threads resolved. Targeted local checks do not constitute a new whole-repository test run. ## Risks App and Cloud must agree on identity, revision, namespace, deadlines and revocation. Partial rollout, unavailable authority or missing ACK must retain awake debt. Shared startup/shutdown and settings changed since the earlier canonical profile, so historical observations carry only within their stated source scope. No deployed Railway/Neon suspension, Cloud alarm, savings or complete hosted Dot/Muse acceptance is claimed. ## Model Used OpenAI `gpt-6.1-sol` produced code and verification, with a Codex coordinator reviewing and integrating the changes. The coordinator's exact runtime model ID and context-window sizes were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
scotttong
|
f5fb8ada3b | Restore chat contacts, agent pins, and scrollable recent navigation (#19685) | ||
|
DottaandPaperclip
|
d1306f338b |
fix(runtime): preserve settled failed harness backups online (#19715)
## Thinking Path
> - Paperclip coordinates agents, tasks, and their saved execution
state.
> - A replacement server needs native provider backups in shared
storage.
> - The online pre-copy owner accepts successful runs with a committed
result.
> - A sealed backup from a failed or cancelled run can still contain
user history.
> - The existing owner defers those files even after their source
ownership is settled.
> - This change adds a separate preservation owner with exact source and
publication checks.
> - The copy retains the failed outcome and original files while making
the backup readable from a fresh home.
## Linked Issues or Issue Description
Refs #19137 and #19108 for the existing pre-copy and encrypted backup
owners.
**What happened?**
Online pre-copy reports `source_not_successful` for a sealed legacy
backup from a settled failed or cancelled run. The provider files remain
tied to the old disk.
**Expected behavior**
Copy the exact recorded backup when the source has no remaining writer
ownership. Preserve the run status, warnings, result, allocation,
selected session, and local files.
**Steps to reproduce**
1. Create a lease-stamped suspended native backup.
2. Settle its source as a failed run with a terminal-failure coordinator
and no owner.
3. Keep an unrelated execution active.
4. Run canonical online pre-copy. The previous owner defers the source.
## What Changed
- Add a narrow preservation owner for terminal source files. It checks
the physical finalization lock, task, coordinator, run, selected
session, source lease, and source fingerprints.
- Recheck that owner before the encrypted upload receipt and
current-backup publication.
- Reuse the finite workspace keepalive during long source checks and
defer source-local invalid archives.
- Add optional exact-root stamp verification. A valid previous backup
cannot authorize a changed current payload. Existing callers keep their
fallback behavior.
- Bind the archive input to the checked owner and record an activity
event after publication.
- Keep pending object receipts after an uncertain or failed publication.
- Add focused PostgreSQL and filesystem controls. Update the existing
migration runbook.
## Verification
- Real PostgreSQL baseline on unchanged production: the new positive
case fails with `source_not_successful` instead of publishing the
backup.
- Same case passes after the change and decrypts/restores the exact
provider history.
- Final head
|
||
|
DottaandPaperclip
|
55c9c44e41 |
fix: preserve database-only failed controller evidence (#19714)
## Thinking Path > - Paperclip preserves failed work as well as successful work. > - Ephemeral enrollment must retain each selected controller source. > - A failed source can already exist entirely in an encrypted database checkpoint. > - Its original inline snapshot does not create a local filesystem dependency. > - The migration owner must prove that both legacy files are absent before accepting that case. > - This change preserves the failed evidence without creating execution authority. ## Linked Issues or Issue Description Refs: #19678. **What happened?** Startup still reports `native_controllers_require_migration` for a failed retained source with a valid database checkpoint, its original inline snapshot, and no local controller or supervisor. The online archive owner reports that the source is already absent. The existing database-only preservation case accepts only a missing inline snapshot. **Expected behavior** Accept the exact database-only failed evidence after full checkpoint and identity validation. Keep a present local payload blocked. Retain the failed run, original snapshot, coordinator, task pointer and queued work. **Steps to reproduce** 1. Store a valid encrypted checkpoint for a failed native run. 2. Keep its matching inline session snapshot and terminal-failure coordinator. 3. Remove both legacy controller files. 4. Check local controller obligations. 5. The old owner reports an obligation even though the evidence is already in the database. **Paperclip version or commit** The fix builds on current master `d5c4f7dd02`. **Deployment mode** Hosted native Runner with durable controller checkpoints. The local regression uses the existing real PostgreSQL and encrypted checkpoint fixture. ## What Changed - Add a narrow database-only failed-source preservation case to the existing migration owner. - Require the decoded checkpoint, protocol header, admitted execution, selected run and inline snapshot to match their exact scope. - Require both the controller and supervisor paths to be absent. A file or link keeps the local obligation blocked. - Retain the earlier null-snapshot predicate and all original failed execution evidence. Create no recovery metadata or executable authority. - Exercise the actual startup and maintenance consumers. Document the preservation boundary. ## Verification - Final revision `1b1da543dafd576e3c420657e3c2c6d0c882172d`: all checks are terminal, with 56 successful and four skipped, zero pending and zero failures. The [required CI aggregate](https://github.com/paperclipai/paperclip/actions/runs/42329501690/job/128275664414) passed. Build, types, server/Runner/serialized suites, end-to-end aggregate and Snyk passed. [Fresh exact-head review](https://github.com/paperclipai/paperclip/pull/19714#issuecomment-6701018015) is 5/5 with no open threads. - Review found that the first non-recovery assertion stopped at the queued fixture's missing current binding. The fixture now binds the exact native issue/session before calling the real resolver and asserts `unsupported:metadata_format` from the selected failed source. That corrected case passes in 4.58 seconds, with wrapper exit 0 and cleanup complete. The production predicate is unchanged. The original startup RED remains valid. The final source/test type graph, module boundaries and diff checks pass. Fresh revision CI and review remain required. - The actual startup regression failed before the source change: one case, 16.62 seconds, expected the absent-files obligation to be false but received true. The wrapper exited 1 and completed cleanup. This failure is retained. - The final focused owner run passed nine cases in 10.79 seconds. It covers the new DB-only case, controller/supervisor presence, every inline scope field, foreign checkpoint Runner/session identities, the unchanged null-snapshot case, and existing online/offline archival controls. The wrapper exited 0 and completed cleanup. - The new case verifies unchanged run, coordinator, checkpoint and task-session rows, retained queued work, no upload, and no invented recoverable session. - The full qualified server/test source type graph passes with db/shared/Runner source aliases. Module boundaries and diff checks pass. Required broad package builds and regression suites remain in CI. - Read-only source diagnosis validated one exact existing checkpoint with the canonical decoder and matching header, run, Runner and session bindings. No source profile or secret values were emitted. No hosted mutation, admission hold, cancellation, status change, or writers-stopped claim was performed. ## Risks - A false database-only classification could hide local bytes. Both exact legacy paths must be absent, and every protocol and database binding must match. - Corrupt or mismatched checkpoints still fail closed. Existing failed evidence does not become successful or executable. - The original null-snapshot case is unchanged. Present local files still require the existing preservation path. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d5c4f7dd02 |
fix: preserve shared Git sources during online workspace migration (#19688)
## Thinking Path > - Paperclip preserves the work produced by its agents. > - Remote workspace migration must preserve legacy Git history and local edits. > - Linked checkouts can share administration and objects with other tasks. > - A task lock alone cannot exclude writers of a shared or borrowed store. > - The online source owner must derive and retain the full physical ownership closure. > - This change reuses task, project, lifecycle and finalizer locks for that closure. > - Idle legacy Git work can then migrate while unrelated remote turns continue. ## Linked Issues or Issue Description Refs: #19642. **What happened?** Online migration keeps legacy Git sources blocked because several task checkouts can share a project base or a private Git cache. A read-only source observation found 160 pending Git records, 120 distinct paths and nine common stores. Its largest project cohort contained 126 records and 110 task owners. An ordinary-directory fence cannot preserve these sources by itself. **Expected behavior** Derive the physical source owners from stable Git metadata. Hold their existing source and lifecycle keys through publication. Retain all history and recovery obligations. Reject foreign aliases, unknown stores and changed sources. Let unrelated object-backed work continue. **Steps to reproduce** 1. Create a managed project checkout and linked task worktrees. 2. Keep a settled prepared recovery source on a sibling task. 3. Keep unrelated remote work running. 4. Request an online import of an idle leaf checkout. 5. The existing owner keeps the shared source blocked. **Paperclip version or commit** This branch builds on merged #19642 (`7ac99718a4`) and fresh master `fb3cf1a072`. It adds the Git closure. It does not add a writers-stopped shortcut. **Deployment mode** Hosted native Runner with remote workspace storage. Local qualification uses owned synthetic Git repositories, disposable PostgreSQL databases and the actual archive and storage owners. ## What Changed - Read bounded stable Git pointers, common-directory metadata and local alternates. Derive the complete physical source ownership cohort. Preserve all object stores. - Retain common and borrowed store owners in preparation, lifecycle, runtime, expiration and actual reopen paths. Keep company boundaries and the existing 128-key bound. - Hold exact sibling native recovery proofs under their existing physical finalizer keys. Recheck descriptors, allocations, recovery actions and source identities before capture and publication. Preserve original states and warnings. - Require leaf-first nested imports. Reuse the existing immutable child download, restore and source comparison before parent exclusion. - Bind the self-contained archive owner to the held Git namespaces. Keep its three walks, history normalization, fsck, bytes and digest format. Reject bare, detached, network, unknown or changed Git storage. - Check both foreign path fields. Keep foreign metadata private. Document the proof, bounds, source observation and remaining eligibility limits. ## Verification - Final revision `2934e41c923b93e16508b978c849b747c86be645`: all checks are terminal, with 56 successful and four skipped, zero failures and zero pending. The [required CI aggregate](https://github.com/paperclipai/paperclip/actions/runs/42326464408/job/128266279210) passed. Build, types, all server/Runner/serialized suites, end-to-end aggregate and Snyk passed. [Fresh exact-head Greptile review](https://github.com/paperclipai/paperclip/pull/19688#issuecomment-6699760104) is 5/5 with no open threads. GitHub reports MERGEABLE and CLEAN. - Review then found two correctness gaps: an ordinary managed directory inside an unrelated host repository was blocked, and a Git parent could miss a completed nested operation after its child snapshot became missing or deleted. Three actual-owner regressions reproduced both gaps in 9.70 seconds. The corrected owner passes 12 focused cases in 17.08 seconds. It imports and restores only the ordinary root bytes, preserves the outer repository, rejects both missing/deleted child receipts before a parent journal, and retains valid/changed child coverage and private-cache contention. Its wrapper exited 0 and cleanup completed. The first bootstrap attempt failed before tests ran because the host exhausted its 32 shared-memory segment limit; no shared services were stopped and no pass is claimed for that attempt. The final qualified source/test type graph, module boundaries and diff checks pass. - Receipt scans keep canonical path resolution. A lexical SQL prefix can omit children reached through symlinked ancestors. The observed full metadata closure completed in 1.17 seconds; no measured bottleneck justifies weakening that alias proof. Writer admission still rejects uncertain Git ownership before mutation. Expiration returns its typed skip, and reopen returns its structured failure. Runtime and lifecycle do not bypass an unknown source. No broad error-contract change was added. - Full CI at `720104ea975e` exposed five existing boundary cases across three server shards. Four exact cases reproduced locally as failures in 25.98 seconds. Reopen now preserves its structured failure when source inspection rejects the path, without reconstructing it. Runtime recognizes only the exact regular ignore-only `.git/info/exclude` fragment; migration still rejects that fragment. The original arbitrary-path and file-URL import/delete-host/resume fixtures keep both supported input forms and use the genuine offline setup path after their sole writer finishes. The production mode gate is unchanged. A new real-owner control proves the offline flag still rejects a running legacy database writer. - The final focused run passed all ten cases across four suites in 22.75 seconds, including all five CI failures, invalid-pointer admission, fragment rejection, offline writer exclusion and actual nested reopen lock ordering. Its wrapper exited 0 and cleanup completed. The known invalid-pointer case remains fail-closed; the separate unexpected-I/O diagnostic retains safe error metadata without command text or credentials. Existing Git stderr redaction coverage remains intact. The final metadata reader suite passed 20 cases in 2.68 seconds. The qualified server/test source type graph, module boundaries and diff checks pass. No production assertions or timeout defaults were weakened. Fresh CI and review are required for the repair revision. - A later review found that the ordinary early return still used only one foreign path field. The actual ordinary providerRef-only regression failed as incorrectly eligible. One source-path owner now enumerates both local locators for company/foreign ordinary and Git scans, layout checks, early admission, lifecycle/reopen/runtime and expiration/move/preparation retention. Capture and display keep their original preferred path. The actual owner canary passed 49 cases across two suites in 109.15 seconds. It covers cwd-only/ref-only/different/symlink locators, same-company mismatches, foreign owners, lifecycle ordering, Git/recovery imports and expiration/reopen. Three selected-locator fallback cases also failed before their guard; the final eight-case control passed in 31.05 seconds, including the configured alias and valid import/private-child controls. All wrappers exited 0 and completed cleanup. The metadata/locator suite passed 19 cases. The final source/test type graph, module boundaries and diff checks pass. - Review found two P1 gaps. Both configured-directory symlink cases failed before canonicalization. The actual parent import lost an already-imported child's unique private cache and failed its archive fence instead of retaining that owner. The corrected real-owner canary passed four cases in 20.78 seconds: configured symlink runtime provisioning, private-cache contention before the parent journal, successful parent publication after release, and unchanged/changed child coverage. The metadata suite passed 15 cases in 4.14 seconds. Both wrappers exited 0 and cleanup completed. - A fresh source read-only observation found 160 pending Git records and nine common stores. An additional exact receipt query found 1,243 completed migration receipts and 557 nested source paths under pending Git roots. All 126 records in the largest project were present in the captured Git metadata set. For all 110 present candidate workspace records, the derived closure fits the unchanged 128-key cap. The worst case needs 110 issue keys and one project key, with 17 keys of headroom. Its 173 imported children introduce no physical issue owner outside those 110 logical task owners. This checks the observed key bound only. It does not claim migration eligibility or a coherent snapshot. - The initial stacked CI did not run four Runner test lanes: trusted master passed new --package-lane/--native-server options to the old base lane script. The branch is now rebased onto actual merged master and retargeted before synchronization. It includes the canonical current lane parser without an unrelated Runner patch. Fresh new-head CI is required. - The actual importer/source owner run passed 36 cases across three suites in 187.75 seconds. It covers sibling prepared recovery, unrelated running remote work, common-cache contention, changed coordinator/allocation/path state during upload, exact nested coverage, dirty unpushed work, and existing expiration/reopen owners. Wrapper exit 0 and cleanup completed. - A separate previously hanging lifecycle-worker restore passed in 17.33 seconds with the unchanged 15-second test-body limit. An earlier run timed out in three expiration/reopen cases and blocked cleanup; its wrapper later exited 1. These failures remain recorded. They are not counted as passes. - Two boundary controls first failed: a foreign providerRef-only alias was incorrectly eligible, and actual reopen did not admit its distinct cache owner at the initial source boundary. The narrow guards now inspect both foreign locators and derive physical Git keys before the raw lifecycle transaction. The foreign alias and existing real nested reopen contention passed. The private-cache control first retained contradictory recorded source provenance and stayed fail-closed after release. The corrected present legacy-worktree fixture passed in 12.61 seconds. These failed attempts remain recorded; they are not passes. - The metadata reader and serialized/local archive controls use actual Git repositories. They compare archive bytes and digests, remove every original source, restore staged and unstaged work, and run fsck. The metadata reader passed 13 cases in 2.85 seconds. The local/serialized archive controls passed nine cases in 9.27 seconds. All processes exited 0. An additional actual importer control changed a relative commondir pointer to the absolute name of the same store during upload. Publication was rejected, the uploaded journal remained available, and original files stayed intact (one case, 5.86 seconds; wrapper exit 0 and cleanup complete). - The final rebased full local server/test type graph passes with db/shared/Runner source aliases because reused dependency declarations predate new master exports. The first rebased attempt reported those stale declaration fields/tables, and is not counted as a pass. Module boundaries and diff checks pass. Standard package builds, types and broad regression checks remain required in CI. - No hosted import, admission hold, cancellation, provider mutation, source filesystem write or shared-cache Git command was performed for this change. The source observation is read-only and is not a locked eligibility proof. ## Risks - A missing writer owner could make physical source exclusion incomplete. Existing mutation owners retain the same company/task/project keys, and uncertain metadata fails closed. - A project can contain a large cohort. Limits remain finite. A bound failure keeps the source blocked. - Bare, detached, unowned or network Git stores remain blocked online. Missing archived sources retain their existing authenticated reconstruction path. - Exact recovery proofs preserve authority and obligations. They do not infer provider termination or mark a prepared descriptor finalized. - This change does not claim all observed Git records are immediately eligible. Existing preparation, provider, recovery, runtime and source-change blockers remain. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7ac99718a4 |
fix(workspaces): fence legacy sources for online migration (#19642)
## Thinking Path > - Paperclip coordinates AI agents and preserves their saved work. > - Remote workspace storage replaces legacy host directories with durable snapshots. > - Several legacy workspace records can name one physical project directory. > - A task lock alone cannot exclude a new task that selects that directory. > - Terminal recovery records can retain valid sources after physical writers settle. > - This change fences the source through its real preparation and lifecycle owners. > - Idle ordinary sources can then migrate while unrelated remote work continues. ## Linked Issues or Issue Description **What happened?** Online migration rejects ordinary legacy directories when several workspace records name the same path. A source inspection found 45 aliases across three project roots with no root Git marker or previously imported nested child. The instance also had unrelated running remote work. Requiring the whole instance to drain prevented useful migration of these idle sources. Terminal native sources and superseded finalization attempts also remain blocked after their exact physical and durable owners settle. Their files, provider allocations and recovery records must remain available through the change of file authority. **Expected behavior** Migrate an idle ordinary source under a derived physical source fence. Let unrelated object-backed turns continue. Preserve provider and recovery obligations. Keep unknown writers, shared Git stores and uncertain source state blocked. **Steps to reproduce** 1. Create several local workspace records for one managed project directory. 2. Give the aliases different source tasks. Leave their local writers idle. 3. Keep a valid remote workspace turn running on another source. 4. Inspect or apply migration to an alias. The old owner returns `overlapping_workspace_requires_maintenance`. 5. Retain a terminal native prepared source with a null coordinator owner, or an old finalize attempt with an exact later committed receipt. The old online path keeps these sources blocked. **Paperclip version or commit** Current tested head: `7478c817f09f7eed396922c0f7831af07ae6ff14`, based on master `8621bcf53ee2e4bbefb7d193add42dd544ca06e8`. Related recovery work: #19131. **Deployment mode** Hosted native Runner with remote workspace storage. Qualification uses disposable local PostgreSQL databases, actual archive/storage owners and synthetic source trees. The source observations were read-only. ## What Changed - Retain the existing company/task source keys and add one physical managed-project key. Heartbeat source selection, materialization, reopen and lifecycle owners share it. Configured aliases of canonical task directories retain their original task key. - Fence runtime provision and each configured service CWD through both startup owners. Check all unsettled service CWDs for physical source overlap, even when workspace IDs differ. Runs with no services keep their filesystem-free path. - Derive a bounded ordinary alias closure. Lock every alias task and lifecycle key. Reject foreign companies, changed membership, unproven descendants, included Git metadata and other unresolved writer obligations. - Commit remote authority on the connection that holds source exclusion. Refresh a stale selected local workspace after admission if migration published remote authority. Publish activity after commit. - Admit exact terminal native recovery sources under the existing physical finalizer keys. Bind descriptors, allocations and owner checkpoints to journaled fingerprints. Recheck them before capture and publication. - Add a separate company-scoped online entry for exact superseded finalization receipts. Keep the offline entry's deployment prerequisites and keep inventory read-only. - Document the proof and the remaining Git/common-store extension. The original files stay in place. No new scheduler, deployment action or database migration is added. ## Verification - Review found that the real reopen path could hold lifecycle exclusion while waiting for a project source. The new actual nested-reopen regression failed before the repair: lifecycle lock count was 1 while another connection held the project fence. The repair acquires declared and physical project sources before reconstruction and lifecycle. The contention case, ordinary-fetch restore and runtime nested restore passed. The lifecycle-worker restore first completed in 16.64 seconds but exceeded its unchanged 15-second body limit; a bounded warm rerun passed at that same limit. Both wrappers exited and completed cleanup. The source type graph, module-boundary check and diff check pass. Final 7478 CI passed: 58 successful jobs, three skipped, one neutral and a successful Snyk context. Fresh Greptile is 5/5 for exact7478, with no unresolved threads. [Final CI](https://github.com/paperclipai/paperclip/actions/runs/42318150314/job/128240631573). - Full CI on `e4283b3452` passed the other server, build and type checks. One exposure owner case failed: two execution workspaces received one preview pair after source admission serialized their starts. The first local seven-case run reproduced that failure; adding the owner argument alone did not change the hash payload. The completed repair binds run-scoped reuse to its execution-workspace owner. All seven existing real-process/fake-broker cases then passed in 13.81 seconds, with wrapper exit 0 and cleanup complete. Public scope IDs and explicit project/agent sharing are unchanged. The final7478 head passed all required CI and fresh review gates. - The service/source/restore owner canary passed 25 cases across four suites before the final no-service fast-path guard (41.33 seconds). This includes actual project service processes with a different project ID and no execution-workspace ID. Their startup waits for source ownership, migration stays blocked while they run, and the source becomes eligible after their verified stop. The final three-case runtime canary passed at the final source inputs (18.70 seconds). It covers that fast path and both service-CWD variants. Both wrappers exited 0 and completed cleanup. - The boundary-repair canary passed 9 cases across four suites at `7c18f1c170`. It covers the real rehome/conversion gateway, project command provisioning, future task selection, stale imported authority, foreign-company binding, and existing standalone filesystem callers. The shared PostgreSQL wrapper exited 0 and completed cleanup. Duration: 94.15 seconds. - The P1 diagnostic passed 46 cases across five suites at the pre-rebase P1 revision. It covers the actual importer with an export warning created by `recordNativeFinalizationFailure`, strict prepared-source/action bindings, finalizer locks, source/lifecycle races, unrelated remote execution, and bare Git with external alternates. It uses the repository shared PostgreSQL wrapper, `--fsModuleCache`, and a local-only 90-second hook allowance. Duration: 245.44 seconds. No product timeout changed. - The canonical retained-directory compatibility check passed 22 cases across two suites. Missing archived task directories still import from authenticated restore points; they do not need a present-directory source proof. - Both initial P1 regressions were observed before the fix. The real finalizer warning made the importer ineligible. The old archive owner accepted a nested bare repository borrowing external objects. The new cases pass and preserve the original warning, prepared descriptor, allocations and files. - Prior attempts remain non-passes: the first 23-case canary passed on an earlier head; its exact-head rerun hit 30-second database setup/cleanup limits. A read of the owned PostgreSQL server found a template clone waiting on `DataFileExtend` for 58 seconds. A broader owner run was intentionally interrupted during slow existing heartbeat setup. One diagnostic invocation used an invalid CLI option and executed no tests. A later review canary passed 21 cases and failed the new real restore contention case: cwd-only admission missed the declared project of an external worktree. Its fixture also omitted the required file-repository authorization callback and produced an unhandled rejection after the failed assertion. The repaired run passed 22 cases, exited 0 and completed cleanup. These earlier failures are not reported as passes. - The recovery owner passed 37 PostgreSQL cases, including 14 new and 23 existing finalized-recovery cases. The operation reconciliation owner passed 40 cases. The initial archive run passed 103 cases and found a missing closed diagnostic entry; the corrected diagnostic and expanded Git rejection cases pass in the P1 run. - The full server type graph passes with the Runner source exports mapped locally because this checkout has no built Runner declarations. The latest module-boundary check and `git diff --check` pass. Standard package build/typecheck and broad final-head regression coverage remain required in CI. - The initial CI exposed missing admission scopes for standalone filesystem APIs and a changed foreign-company error. The latest repair keeps those APIs independent of a database and holds physical fences at their actual database-backed mutation owners. The branch also includes the existing master fix for unchanged Runner default-model assertions. New review findings and new-head CI are being checked. - The local canary command and bounded source selection are documented in `doc/plans/2026-11-06-online-ordinary-source-fence.md`. Source observations only read data. A fresh bounded metadata probe found 254 pending legacy records and zero unsettled local-process services. No live import, hold, cancellation, provider action or source filesystem write was performed for this change. ## Risks - Run-scoped reuse now includes execution-workspace ownership in its private key. Explicit project/agent sharing keeps its previous identity. - A missing writer owner could make a source proof unsafe. Admission derives physical aliases, includes future project selection and retains all existing source/lifecycle/finalizer locks. Unknown and changed state fail closed. - Cross-company physical alias checks use bounded minimal-path pages. Lexical-prefix filtering cannot safely replace them because a symlink can hide a foreign alias. Unknown paths or more than 10,240 foreign records leave the source explicitly unproven. More than 1,000 same-company local records also fail closed. - The project key can serialize local preparation within one project. A first project waits; additional project acquisition fails on contention to avoid a lock cycle. Remote authority keeps its remote preparation path. - Recovery preservation does not prove provider termination or successful file recovery. Existing provider allocations, checkpoints, task intent and recovery actions remain authoritative. - This slice does not finish shared Git migration. Git administrative/common-object/alternates closure and exact nested imported-child coverage remain the next owned extension. No global writers-stopped shortcut is used online. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
72f7b42fe3 |
Preserve settled failed controller state during online migration (#19678)
## Thinking Path
> - Paperclip controls agent work across web-server releases.
> - Volume-free servers need retained controller state in shared
storage.
> - Successful settled sources already support online migration.
> - Some retained failed sources have no accepted result and must stay
available for recovery inspection.
> - This change copies those sources under task, coordinator and
publication fences while unrelated work continues.
> - The original files, failed status and recovery warnings remain
unchanged.
## Linked Issues or Issue Description
Refs #19038. Refs #19631.
**What are you trying to improve?**
Preserve retained failed controller files without waiting for global
idle.
**What problem does this solve?**
The online producer skips terminal failed controllers. Their files can
still be needed for recovery inspection even when no accepted provider
artifact proves executable continuation.
**What would the improvement look like?**
A settled exact source can enter the existing encrypted archive store.
An accepted committed failed source can use the existing checkpoint
import. Busy, changed, unowned or foreign sources remain blocked.
## What Changed
- Add a source owner bound to the selected task session, run,
coordinator generation, admitted input and file digests.
- Check the task, physical finalizer lock, released coordinator,
canonical resume source and active related turns before upload and
before publication.
- Preserve failed sources through the existing encrypted archive format.
Do not treat archived bytes as execution authority or old-writer stop
proof.
- Allow the existing accepted committed checkpoint import for a finished
failed run without changing its status.
- Validate admitted input bindings before looking up local controller
files.
- Document the online preservation path and the final static writer
fence.
## Verification
- Real PostgreSQL: 125 cases passed across migration, controller
recovery and durable backup storage. One added assertion inspected an
unadmitted queued fixture; it was corrected while retaining the
queued-turn preservation and stale-publication assertions. The final
migration rerun passes 51/51; the other three affected suites pass
75/75.
- Genuine prepatch RED: online failed-source preservation and accepted
committed failed-source import both fail on the previous producer and
recovery owners.
- New cases cover independent restore after deleting the original
source, an expired but non-null coordinator owner, source mutation after
upload, a new related queued turn during upload, foreign input binding
and an accepted committed failed result.
- The full rebased server source and test-tree typecheck passes after
restoring its missing ignored workspace SDK link and building that
existing dependency.
- Module boundaries and whitespace checks pass.
- Rebased onto current master; both provider-artifact imports and the
new archive owner are preserved. Rebased type and module-boundary checks
pass. All six archive cases pass on the exact rebased head (39.68
seconds). All CI gates pass on exact head
|
||
|
DottaandPaperclip
|
6be71028ad |
Prepare online ephemeral enrollment through signed Cloud control (#19660)
## Thinking Path
> - Paperclip manages agent work through a control-plane web server.
> - A replacement web server needs shared durable data and the original
signing identity.
> - Existing enrollment exports that identity only after a global drain
and offline migration.
> - Online enrollment needs a separate preparation operation while agent
work continues.
> - This change adds a purpose-bound Cloud control operation with
existing data and key owners.
> - The result is an audited preparation receipt, with no permission to
stop agents or detach storage.
## Linked Issues or Issue Description
Refs #19038. Refs #19024.
**What are you trying to improve?**
Allow Cloud to prepare an already migrated tenant for volume removal
without the offline migrator or a global-idle gate.
**What problem does this solve?**
Cloud must preserve the original decision key before starting a
replacement with static ephemeral mode. A tenant session must never gain
key-export authority.
**What would the improvement look like?**
A fresh exact-action Cloud assertion obtains a private, deployment-bound
preparation receipt. Unfinished durable-data migration and invalid
replacement configuration block export.
## What Changed
- Add the signed `ephemeral-enrollment:prepare` control action and a
strict UUID-only POST route.
- Check existing durable-data eligibility and ephemeral configuration
before reading the original key. Never create a missing key.
- Audit only public preparation metadata. Keep HTTP diagnostics
secret-sensitive and the response non-cacheable.
- Bind the receipt to the attempt, stack, build commit, Railway
deployment, time, static mode and key digest.
- Always return `enrollmentReady: false`. Cloud still needs to arm a
replacement, verify old writers have stopped, repeat eligibility and
qualify native handoff before detaching the retained volume.
## Verification
- PASS: 80 focused tests across the preparation owner, signed route,
existing Cloud control, HTTP privacy and OpenAPI route coverage.
- PASS: 2 real PostgreSQL cases. Concurrent copies of one signed request
export once; the durable receipt still rejects reuse after clearing the
process fence. Normal run admission before skill pins is allowed;
missing durable skill installation remains blocked.
- Genuine prepatch RED: the same two PostgreSQL cases exported twice and
rejected ordinary admission. The fixed owner was restored automatically
after the comparison.
- PASS: server TypeScript check with workspace dependencies built.
- Fixed the first CI route-coverage failure by declaring this private
control endpoint in the existing explicit OpenAPI exclusion registry.
- Initial local HTTP execution lacked sandbox socket permission; the
reviewed run passed. Initial PostgreSQL fixtures enabled durable files
too late and reused an issue prefix; those fixture defects were
corrected before the red/green comparison.
- All required CI gates pass on
|
||
|
DottaandPaperclip
|
349b8b3f7b |
Migrate settled controller checkpoints while unrelated work runs (#19631)
## Thinking Path > - Paperclip manages work performed by AI agents. > - Server replacement must preserve retained Runner sessions. > - Older sessions can retain controller checkpoints on the server volume. > - The existing importer requires all execution writers to stop, including unrelated runs. > - This change adds a separate online path for successful, settled sources. > - It locks the source task and finalizer while it reads and publishes that source. > - Unrelated work continues. The offline importer keeps its existing authority. ## Linked Issues or Issue Description **What happened?** A retained controller checkpoint cannot be converted by the existing maintenance importer while another task runs. Requiring a busy server to become globally idle prevents online migration. **Expected behavior** An operator can convert a successful retained source after its finalizer releases ownership, while unrelated tasks continue. An expired owner remains an owner. Related active work and uncertain evidence remain blocked. **Steps to reproduce** 1. Retain a valid legacy controller for a successful run with an accepted result and a released committed finalizer. 2. Start a different task. 3. Run controller preparation. The old importer refuses because execution writers are active. Related preservation contracts: #19038 and #19024. This change does not replace workspace migration or authorize volume removal. ## What Changed - Add `prestageNativeControllers` with task, finalizer, and source-run row locks. Recheck related runs and exact checkpoint identity under those locks. - Allow legacy checkpoint material through normal settled-owner import checks. Keep offline authority separate. - Compare captured local controller and supervisor bytes before commit. A changed source rolls back publication. - Add `--controllers-only` to the existing finite native preparation entry. This avoids re-reading provider archives during controller conversion. - Add PostgreSQL cases for unrelated work, expired owners, related active work, source-lock contention, and source changes. - Document the finite command and its limits. ## Verification - Real PostgreSQL affected suites: 56 passed across controller migration and recovery. A first per-server run was interrupted because it was slow; the completed run used the repository's shared PostgreSQL test wrapper. - Initial online cases: 3 passed, covering unrelated work, an expired owner, and related active work. - Source lock and changed-file cases: 2 passed. Publication rolled back on a source change. - Final source-evidence regression: 2 passed. A first unproven source deferred and a later valid source imported; changed-file rollback remained intact. - Server TypeScript check passed with the SDK and Runner declarations built. - Server package build passed, including the Runner vendor checks and compiled command. - Runtime bundling suite: 10 passed. - Full repository CI and exact-head automated review remain required before merge. This draft is not staging acceptance. - PASS: 7 CLI cases after adding the controller-only dependency fixture and explicit-mode coverage. The first CI run exposed missing mock/export coverage; the fixture repair passed after rebase. - The unrelated model-default failures were fixed on master; this branch now includes that fix. ## Risks - A source task can wait briefly on its own migration lock. A one-second lock acquisition budget defers a contended source. No global admission hold is acquired. - Only successful sources with a released committed finalizer are eligible. Failed or unsettled sources stay deferred and require their preservation owners. - The operator must run the command on the host that owns the original files. The command does not certify enrollment readiness or detach any volume. - The existing offline path remains unchanged and still requires stopped writers. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
124b2cd0ad |
fix(server): certify background deadlines for idle tenants (#19001)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Background services own future checks, retries, and recovery work. > - An idle tenant can stop only when a durable wake covers its remaining duties. > - The current alarm catalog covers several domains but leaves these owners blocked. > - A saved future timestamp alone cannot prove that reconstruction or admitted work has settled. > - This pull request binds durable domain snapshots to the actual registered worker revisions. > - The tenant can hand off supported future duties and retain all checks for uncertain or active work. ## Linked Issues or Issue Description No matching public issue was found. Related work: [domain deadline handoff #18656](https://github.com/paperclipai/paperclip/pull/18656), [owned blocker diagnostics #18632](https://github.com/paperclipai/paperclip/pull/18632), and [completed history proof #18646](https://github.com/paperclipai/paperclip/pull/18646). The separate [native idle-session change #18988](https://github.com/paperclipai/paperclip/pull/18988) covers runtime retirement. [Far-future retry re-probes #13534](https://github.com/paperclipai/paperclip/pull/13534) changes recovery responsiveness after an upstream success; this change preserves existing retry decisions. **What happened?** Future status-card, external-object, skill-sync, watchdog, retry, and quota duties lacked complete owner-bound coverage in the idle deadline proof. A completed quota task could also retain its monitor policy and timestamp and remain blocked after the scheduler had disarmed it. **Expected behavior** A settled registered owner can hand off its exact supported future deadline. Active work, dirty reconstruction, transaction uncertainty, changed revisions, claims, unknown state, and unsupported duties keep the tenant awake. Completed history retains its data and stops recurring work when the owner has no remaining duty. **Steps to reproduce** 1. Produce a supported future duty through its normal service. 2. Let the registered worker finish reconstruction and admitted work. 3. Request an owned idle report with the negotiated domain deadline catalog. 4. Check the saved earliest deadline and generation before durable alarm registration. 5. Stop and reconstruct the worker. Check that ordinary admission and claims resume useful work once due. 6. Complete a quota task. Keep its saved policy and timestamp. Check that linked unfinished work still blocks sleep and that settled history does not. **Paperclip version or commit** The owned patch was qualified at `699c8569fa34338dd75cdc5731381ec1e2efdb45`. Publication head `491fac368ee72eaac8ae105d53e6bf6b305d8a6a` normally merges actual master `039e0dad2c51c80251e1302a19c5566e039c357a`. All 43 owned files are byte-identical. All 29 new upstream paths are preserved. The later normal current-master integration is `8c912e1ba4fc66397a7341ade59373af8075068f`, described separately below. **Deployment mode** Source checkout with real PostgreSQL 17 and a controlled local Node process. No deployed alarm, tenant suspension, database suspension, or fleet result is claimed. ## What Changed - Add registered-worker reconciliation revisions and invalidate them on stop. - Keep status-card, external-object, and skill deadline interpretation with their owners. - Reuse the watchdog classifier against the supplied database snapshot. - Validate the exact saved retry run and paired queued wake. - Validate legacy quota source bindings and the current assignee or reviewer policy. - Preserve common issue scope, dependency, sibling, and state checks for covered waits. - Compose normalized owner snapshots within the existing bounded deadline proof and recheck revisions after inspection. - Bind actual service and worker instances from server startup. Missing bindings preserve existing blockers. - Retain completed quota policy and timestamps while independently rejecting linked unfinished runs or wakes. - Block unconsumed budget evaluation inputs through the owning queue predicate. Accounting policy and settlement stay unchanged. - Bind completed quota history through fixed-size UUID-array parameters, preserve exact company/task/source associations, and guard external references before the run-ID join. - Batch watchdog subtrees and shared classifier facts without changing the 64-query or two-second proof budget; preserve exact company pairing for first-run evidence. - Extend production-path regression controls and document supported coverage and limits. ## Verification Current integration head: `688abc6cac6016b9fb50e5832f81e2975ed5d49f`. Fresh exact-head checks completed with **58 successes, three intentional skips and one neutral review**, including aggregate `ci / verify`. The complete current Greptile review is **5/5**. The complete current Ponytail verdict is **Ship (4/5)**, with only optional cleanup of an unused MCP proof input; both prior correctness findings are resolved. All review threads and nested pages were inspected. The normal master merges preserve the registered retry deadline owner and current startup completion log after the same readiness await. The earlier focused evidence below remains tied to its tested source revisions. Current-master integration `8c912e1ba4fc66397a7341ade59373af8075068f`: - Normally merge actual master `46fb933b0155d183ce64a8bdc4950d1b58c926f6`. Every nonoverlapping tree blob retains its owning parent across 14,044 paths. All 42 owner-only files and 15 critical binding/proof/readiness files remain byte-identical. Three automatic overlaps preserve the existing watchdog binding, conservative unavailable-reader fallback and optional actual owner arguments. The two scoped-deadline conflicts retain the entire upstream resource-reservation/mid-pass admission/abort algorithm inside the existing reconciliation revision/running/failure fence, with both parent test sets. - Actual TypeScript 7.0.2 server noEmit passed in 9.434 seconds, normal source declaration layers in 12.102 seconds, and four selected test/helper/config layers extending the actual CI configuration in 3.167 seconds. No compiler option weakening, heap override, donor declarations or SDK replacement stub. - Focused PG0 controls passed: 102 scheduler/proof/owner controls in 16.451 seconds, including all 21 scoped-deadline cases; 121 startup controls; and all 107 unchanged instance-settings route cases in 6.456 seconds. The latter first hit a local prewire-guard listener-registration race. Only the ignored guard was corrected to register the same actual OS-bound server port immediately after native listen returns. Closed, unknown, foreign, offhost and PG destinations remain blocked. Original failure evidence is retained; product code and assertions were unchanged. - The exact current UI policy command now reports zero findings (0.362 seconds); its seven scanner controls pass. The earlier dependent PR used an old scanner/source baseline under a newly enforcing trusted workflow and reported 397 findings. Carrying the genuine current master resolves that mismatch without editing unrelated UI sites or relaxing the gate. - Every meaningful worker checked its executing PID/thread/preload hash and matching receipt. PG0 controls opened no PostgreSQL sockets; all owned processes exited. Original dependency-cache/DNS, ignored-harness syntax/type and guard failures remain separate from product results. - This is fresh semantic and focused runtime qualification, not a local whole-repository or current-head PostgreSQL acceptance rerun. The upstream workspace-busy parked-retry/terminal-holder notification path remains unchanged by this merge and needs its separate current-graph producer check. Historical six-case/scale results below stay tied to their measured sources. A matching controlled default-cadence follow-up is in #19191; no fresh current-head two-cycle or deployed suspension claim follows from this merge. - All 59 exact-head check contexts now pass (55 success, one neutral, three intentional skips), plus the legacy security status. The complete edited Greptile review is 5/5 for this exact commit, with all root/nested pages inspected and no unresolved threads. Ponytail stopped without completing its new turn; it supplied no new verdict or finding. Prior actual findings remain addressed. Earlier scores/checks below belong to their explicit earlier commits. On frozen clean merged source `699c8569fa`: - Real PostgreSQL acceptance: **6/6 passed**,83.483 seconds wall. Actual retry producer creates the exact saved run/wake; stopped and reconstructed owner resumes through current admission, one durable start, real local process output and normal task completion. Includes cancellation, drain arrival, retained quota history with queued-run/wake negatives, watchdog review settlement and budget-input empty/pending/consumed control using the actual owner. - Affected coordinator, deadline, scoped-deadline, recovery and composed-proof controls: **78/78 in five files passed**,32.114 seconds wall, worker 1/no file parallelism. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: **passed**,17.467 seconds. Native TypeScript 7.0.2 and Node 24.22.2. - PostgreSQL tests use one dedicated test database, matching fixture/admin URLs, pool limit four, one worker, and no embedded fallback. Every run recorded unchanged source hashes and a clean checkout. - The eight-second real-clock retry is a domain execution check. It is below the Cloud minimum wake lead and is not a positive accepted hosted sleep cycle. - The completed quota-history product failure and original fixture/protocol failures were preserved; the owning-reader fix passed a real PostgreSQL red/green regression before the fresh merged six-case pass. - After runtime completion, the dedicated database had zero other clients and no owned runtime processes. All three qualification process PIDs exited. The coordinator independently audits and drops the database. - Comment-only cleanup has equal strict TypeScript parser ASTs across 15 files. The final lexical audit finds no added comments relative to actual master. `git diff --check` passed. - Earlier retry fixtures omitted responsible-user membership, advanced JavaScript time past the real PostgreSQL lease clock, or produced stdout without normal task completion. Corrected fixtures use real membership, an eight-second real deadline, and ordinary task comment/completion. The last case exposed legitimate disposition repair rather than duplicate retry promotion. - The quota and watchdog controls first ran before admitted work or the existing 30-second reconstruction duty settled. The stronger tests require an interim blocked report and then exact settled-owner proof. Production deadlines were not changed. The separate completed-quota-history defect was fixed in its reader and verified with linked unfinished-run and wake controls. - The status/object and skill owners reviewed the actual final startup, route and post-backup bindings. Their domain source evidence carries where token-equivalent. The master coordinator coalescing change was reviewed separately. Historical domain PostgreSQL results are not presented as fresh full-suite results on this head. The later master merge adds retry predecessor cleanup checks under the existing issue claim lock. The source audit confirms unchanged saved deadlines and run/wake identities. The three actual retry lifecycle/cancellation/drain cases passed on the batched candidate with this parent (31.817 seconds). Those local dependency controls preceded the current exact-head remote run. Native backup preservation and TestOps changes from master are retained. No pass is inferred for changed dependency behavior solely from unchanged owned files. Review repair published as `643f1e7400`: - The original reader returned `unknown` after seven actual watchdog reviews settled across seven companies. The unchanged composed-report regression passed after batching; the final reader passed in 61.841 seconds, including active queued-work and stopped-owner rejection. - A separate two-company red/green control rejects terminal-run context from another company as first-run grace evidence. Shared facts are serialized once, with per-root IDs and review summaries. - The affected recovery, proof and diagnostic unit controls passed 47/47 (14.575 seconds). Startup owner-contract fixtures passed 121/121 (10.589 seconds). - Diagnostic PostgreSQL controls passed7/7 (12.406 seconds), including the normal budget-input producer and its exact blocker reason. Initial empty-state setup uses the same cleanup as per-case teardown when a dedicated test database is reused. - Actual repository `pnpm exec tsc --noEmit -p server/tsconfig.json` passed 12.710 seconds with no heap override. Final source has no added code comments; whitespace checks pass. - The original query-budget failure, two-company discriminator, initial cross-suite fixture failure and an interrupted incorrect local-driver selector are retained separately. The interrupted run is not a pass. Independent cleanup found zero owned clients or runtime processes. Quota-scale repair published as `6696a37f8a`: - The exact 10,000-row completed-history case reproduced `MAX_PARAMETERS_EXCEEDED`:90,002 bound parameters exceeded the driver's 65,534 limit, and the composed report returned `unknown` despite settled owners. Original failure and actual driver diagnostics are retained. - Fixed pending-work queries use three typed UUID-array parameters plus the result limit, keeping exact company/issue/source tuple bindings. The primary run-ID column stays uncast; a guarded CASE validates the raw external reference before conversion. The 10,000-row cap and 64-query/two-second proof budget are unchanged. - The same 10,000-row composed-report case passed 18.860 seconds, with unfinished run/wake rejection, malformed-reference safety and conservative 10,001-row overflow. Its fixture derives retained history from the actual quota producer and normal task completion, expands valid persisted issue/source pairs, then starts the actual registered owners. - The existing actual quota producer case passed 14.187 seconds, including exact future deadline, retained completion history, unfinished work, real foreign-company source references and noncanonical/malformed references. - Affected owner/proof controls passed 22/22 (26.444 seconds). Production server noEmit passed 16.039 seconds. - CI on 643 identified TS7034/TS7005 in the settled-watchdog fixture's initially untyped collection. It is now explicitly typed. A separate affected-entry semantic check extending the repository's actual `server/src/__tests__/tsconfig.json` passed 15.093 seconds: six affected tests, two owner files, ambient declarations and current imported source, with unchanged strict/module/lib/path settings. The production server config excludes that test directory and is not presented as test-layer proof. No custom suppression or replacement SDK stub was added. - All runs were sequential with pool max 4, worker 1, source/config/command hashes and cleared inherited provider/LIVE/other-database variables. Cleanup found zero owned database clients or local runtime processes. At prior head `6696a37f8a5718cd0f79054969a0084462d41d47`, all required CI gates pass. The exact-head inventory has 55 successes, one neutral review and three intentional skips. Greptile is 5/5. The complete latest Ponytail verdict confirms the quota finding is fixed; its only remaining note is optional unused MCP proof state, not a behavior defect. All three review threads are resolved. The earlier 643 result remains historical. The repository-wide typecheck, tests, build and browser gates passed in the prior `6696a37f8a5718cd0f79054969a0084462d41d47` CI run. The new `8c912e1ba4fc66397a7341ade59373af8075068f` full CI also passes. They were not run as a full local workspace command in the focused PostgreSQL lease. Local evidence above uses bounded external databases and does not claim embedded-cluster, live-provider or deployed-alarm qualification. ## Risks - An unsupported or uncertain owner state must remain blocked. An optimistic exemption could lose work while compute is stopped. - Changes during reconstruction or inspection must invalidate the proof. A memory-only timer is not durable alarm coverage. - Current authorization, budgets, cancellation, accounting, and resource ownership still apply on dispatch. The proof does not grant execution authority. - The supported writer model uses in-process transaction signals and startup/resume reconstruction. Independent SQL writers and other replicas need an explicit wake contract. - Native, review, coalesced, and taskless retry variants remain conservative until their owner cases qualify. Reviewer quota and hosted alarm qualification remain open. - The shared catalog remains opt-in. Legacy callers and missing owner bindings retain their old blockers. ## Model Used OpenAI `gpt-6.1-sol` was the configured model for the delegated service implementations and reviews. OpenAI Codex, based on GPT-6, performed managing review and integration. Both used repository tools and code execution. The exact served backend snapshots and context-window limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass on the final merged head - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — exact current-head run passes - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — complete current-head review and all paginated threads inspected - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
871a9b3180 |
test(restart-chaos): add Phase 1 control-plane restart chaos harness (#19477)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane owns task execution, recovery and Retry for every agent run > - Server restarts were the largest remaining source of stalled tasks in the staging soak campaign > - The soak could only observe restarts that staging happened to perform, so a failure could not be repeated > - This pull request adds a script harness that causes the restarts on purpose and grades each task on fixed rules > - The benefit is a repeatable restart failure with a replay command and an evidence bundle for each fixer ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart test coverage. Today the runner-e2e restart cells test one fixed restart point per cell. They cannot run many tasks at once, cannot choose a random fault time, and cannot replay a failure. **Subsystem affected** Test tooling (`tests/restart-chaos/`). It reuses the runner-e2e process code in `tests/runner-e2e/`. Product code does not change. **Current behavior** Restart robustness is checked through a small set of fixed cells. A random restart during a real task has no defined pass rule. **Proposed behavior** A harness starts an isolated control plane and seeds a "Chaos Lab" company. It launches seeded workloads, injects K1, K2 or K4 faults at seeded times, and grades each task on rules R1 to R6 plus a deterministic task oracle. `--replay <wave>/<task>` repeats one task. **Reason and benefit** Fixers get a replay command and an evidence bundle for each failure signature. The first smoke wave already found three product signatures (dead Retry after a restart, a false over-budget stop after restarts, and an OpenCode turn killed mid-turn that ends as a provider failure). **Breaking changes** None. The change adds new test files, one entry in the test-project typecheck list and a gitignore entry. ## What Changed - Add `tests/restart-chaos/`: - `supervisor.ts` owns one isolated server. It reuses `replaceController`, the process-tree stopper, the environment isolation helpers, the config seed, the port reservation, the wrapper-owned Postgres and `server-entry.ts` from `tests/runner-e2e/`. It does not copy them. Faults: K1 (SIGKILL, immediate restart), K2 (SIGTERM, restart) and K4 (SIGKILL, 2 to 5 minute hold). Postgres and the controller runtime cache stay across a fault. - `seed.ts` creates the company, encrypted provider secrets, native Codex, ACPX Claude and OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project. The project is a git repository with a Python CLI, one failing test and a four-minute `bench.sh`. - `scheduler.ts` makes a seeded plan of workload, adapter, fault, fire time, hold and nonce for each task. It saves the plan so that `--replay` can run one task again. - `observer.ts` polls each issue and its runs. When a task stops and offers Retry, it sends `POST /api/issues/:id/retry` one time, with the same body as `useTaskRetry.ts`. - `grader.ts` grades R1 to R6 as pure functions. `workloads.ts` defines workloads A1, B1, B5 and C1 and their oracles. - `evidence.ts` and `ledger.ts` write an evidence bundle for each failure, a JSONL ledger and `SUMMARY.md`. All results go to the gitignored `tests/restart-chaos/results/`. Known provider key values are redacted. - Unit tests: `scheduler.test.ts` (determinism, replay and CLI parsing) and `grader.test.ts` (R1 to R6 and the oracles on fixture timelines). These tests need no credentials. - Add `tests/restart-chaos` to the test-project typecheck list (`scripts/typecheck-test-projects.mjs`), with a `tsconfig.layers.json` like `tests/runner-e2e`. The harness runs through direct script invocations, not `package.json` scripts. - After the first Greptile review: - A failed fault or an unexpected server exit now gets one bounded recovery. If the recovery fails, the wave fails and does not hang. - Incomplete cleanup is now a harness error, and the harness keeps the instance. - After the C1 follow-up comment, the observer needs a fresh rest window before it grades the task. - `--replay` now fires the recorded faults that crossed the task, at the same offsets, including the faults that other tasks requested. - A failed workspace-operation read is now a sweep error. It cannot give a clean rest. - R2 now checks that workspace files from before the first fault still exist at rest. - B5 now needs the completion record that `bench.sh` writes for each task. - After the second round of reviews: - An idle `todo` or `in_progress` task with no live run and no named wait is no longer treated as resting. The observer keeps watching it until the task resumes or passes the R5 stall limit. - A failed injected fault now fails the wave with exit 2. - T-oracle commands run agent-editable code. They now run under a macOS Seatbelt sandbox with a scratch `HOME` and no network, and they cannot read the operator's home directory. On other platforms these commands are refused unless the operator explicitly allows them. - R4 now counts every active run. A wake comment may be served again only after a run that did not succeed. - Evidence redaction now covers every environment variable name that ends in `KEY`, `TOKEN`, `SECRET` or `PASSWORD`. - If the pre-fault snapshot fails, only R2 for that task is left unchecked. The wave continues. - B5 also needs bench progress that the harness observed while it polled. - Add `doc/plans/2026-11-06-control-plane-restart-chaos.md` and its campaign ledger. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 tests pass. The tests include observer tests with a fake API, a real `bench.sh` run, and a sandbox test that cannot read a canary secret. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - `server/scripts/typecheck-layers.test.mjs`: 45 tests pass, including "covers every project under tests/". - A supervisor check with no credentials passed. K1 was ready again in about 6 seconds. K2 with a 3-second hold was ready again in about 9 seconds. Company data stayed readable. - One paid smoke wave ran: `--env local --adapters codex,claude,opencode --workloads A1,B1,C1 --faults K1 --tasks 3 --concurrency 2 --seed 7`. The harness worked from start to end, and all three tasks crossed a restart. The 0/3 result comes from product failures, which the campaign ledger records. A scan found no provider key in the results. - To run a wave: ```sh pnpm --filter @paperclipai/paperclip-runner build:runner-binaries set -a; source ~/.secrets; set +a node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude,opencode --concurrency 4 --tasks 8 --faults K1,K2,K4 --seed 42 node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay <wave>/<task> ``` ## Risks - Low product risk. The change adds test tooling only and does not change product code. - A wave spends provider money. The company budget hard cap (`--budget-cents`, default 3000) is the only spend limit. The plan's per-wave and per-night refusal is not implemented yet. - Known gaps: - `--env daytona` is reserved. It fails with a TODO in `seed.ts`. The Daytona sandbox sweep for R6 is not implemented yet. - The project requests isolated git-worktree workspaces. A B1 task still ran in the shared fixture repository. Concurrent B1 and B5 tasks can share files. The oracles use marker files for each task. - Only K1, K2 and K4 at seeded random times are available. Boundary triggers (B1 to B16), other faults and the UI probe come in Phase 2. - The paid smoke did not exercise B5 or K4. - On macOS, B1 oracle commands need `python3` outside the operator's home directory, for example `/usr/bin` or Homebrew. On Linux they are refused until a sandbox is added. - The B5 proof is best-effort against a deliberately adversarial agent, because that agent can write to the workspace. - Startup phase timings come from polling `/api/ready`. The server records its own timings only as OpenTelemetry spans. - `pnpm verify` and the CI Vitest jobs do not select these unit tests. The root Vitest projects cover workspace packages only. CI typechecks the files, but the tests run only through the vitest command above. - After the smoke, rest detection and the review fixes changed. A recovery action that the board owns now counts as a rest, and R6 reports it. A paid wave did not run again after these changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code as an agent with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
c0214826b3 |
fix(workspaces): preserve pending repository requests during import (#19439)
## Thinking Path > - Paperclip manages agent work and its saved files. > - An ephemeral web server needs remote workspace authority before the old volume can detach. > - A task can have a repository request that has never prepared a checkout. > - The offline importer currently treats that pending request as unfinished file publication. > - This blocks migration even when the target does not exist and there is no saved commit, branch or failure. > - This pull request preserves that request in the database and imports the existing files separately. > - The next authorized sandbox preparation can acquire the repository through its existing owner. ## Linked Issues or Issue Description **What happened?** On Core `253bf071f0a496554b61fff897ad3f11afaf8766`, offline migration of a volume-backed workspace fails with `repository_preparation_unresolved` for an untouched pending repository request. The requested directory has never existed. The request has no pinned commit, branch or failure. **Expected behavior** Preserve the pending request and all existing files. Do not label the absent checkout as part of a saved archive. Keep partial publication and active preparation blocked. **Steps to reproduce** Create a local task workspace. Add an unmaterialized pending repository request. Run the canonical offline importer after stopping writers. The unchanged importer rejects the request before preserving the workspace. Related migration work: #19429. A GitHub search found no duplicate pending-repository import fix or matching public issue. ## What Changed - Admit an untouched pending request only at the existing offline writer fence and after proving its target is absent. - Keep the database request unchanged and exclude it from materialized snapshot descriptors. - Recheck file absence and source identity before publication. Reject a request that becomes ready during capture. - Cover cold filesystem loss, real sandbox-side Git acquisition, and eleven blocked states. Document the preservation contract. ## Verification - Red: the new real PostgreSQL/filesystem regression fails on the unchanged importer with `repository_preparation_unresolved`. - Green: both affected migration/lifecycle suites pass, **236 tests**. The cold test deletes the predecessor directory, restores saved bytes, and acquires the pending repository through the full admission owner, checking ready state and the pinned commit. Real Git runs against a local HTTP origin through an exact-URL redirect at the sandbox transport boundary. It does not claim live Daytona or GitHub qualification. - Full workspace typecheck and build pass at the final head. Both affected suites pass **236 tests**, including the full admission owner and eleven blocked states. All final-head CI checks pass: **54 success, 2 skipped, 1 neutral**. Greptile is **5/5**, with no unresolved review threads. - The full local test command failed in `recovers lost inline and check responses without duplicate GitHub publications` in the chat integration suite. That exact case passes in a fresh isolated invocation. Its cause remains unclassified; this is not a full-local-suite pass claim. The original failure is retained alongside the isolated result. - No candidate code has been applied to the serving instance. ## Risks - The exception applies only to offline migration and a pending request with no target, commit, branch or failure. Unreadable paths, symlinks, partial directories and active/repair states retain their blockers. - Requests retain normal repository authorization and acquisition behavior. This change neither fetches repositories during migration nor marks requests ready. - A changed materialized source aborts publication and leaves original files intact. ## Model Used OpenAI GPT-6 through Codex, with reasoning and tool use. The exact deployment variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
253bf071f0 |
fix(workspaces): preserve resolved pending merges in snapshots (#19429)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task workspaces must survive replacement of the web server. > - Workspace snapshots therefore preserve both files and portable Git state. > - A resolved merge can still await its merge commit. > - The archive rejected this state, so an existing workspace could not migrate. > - This pull request saves and checks the complete supported merge state. > - The benefit is that migration preserves the user's choice to continue or abort the merge. ## Linked Issues or Issue Description **What happened?** A legacy workspace contained a resolved merge with `MERGE_HEAD`, `MERGE_MODE`, `MERGE_MSG`, `ORIG_HEAD`, and `AUTO_MERGE`. Its index had no unresolved entries. The snapshot owner refused the operation. This kept the original safe but blocked migration. **Expected behavior** Save this supported state without changing the source. Restore it without the original repository, borrowed object store, or linked worktree. Retain the merge parents, selected index, later edits, and Git's normal safety checks. **Steps to reproduce** 1. Merge two branches that conflict. 2. Resolve and stage the conflict. Leave the merge uncommitted. 3. Capture the workspace. The base implementation rejects the merge marker. Related prior work: #19310 preserves an inactive rebase bookmark. No duplicate open merge-preservation PR was found. This is a workspace preservation fix, not a new roadmap feature. ## What Changed - Admit only resolved pending merges with valid, bounded metadata and available objects. - Retain and verify the merge mode, message, parents, original head, automatic merge tree, and index. - Verify copied metadata and object closure before publication. Keep the before/after source checks. - Keep unresolved conflicts, autostash, rerere, mixed operations, malformed state, and unsafe links blocked. - Test real ordinary, linked, and borrowed repositories through local and serialized capture. Delete all original sources before continuing or aborting the restored merge. - Document the supported state and the remaining refusal cases. ## Verification - Six new continuation cases fail against the unchanged base at the pending-operation guard. - The first candidate restored and continued the merges, but six abort assertions failed. The fixture rewrote the worktree without updating Git's index cache. It now first verifies that abort protects later edits, uses normal `git restore` to discard those fixture edits, then aborts. No archive assertion was removed. - The final archive and environment suites pass all 98 tests at `3c87d13d1b`, including linked-metadata protections and pending merges in ordinary, linked, borrowed, shallow, partial, and SHA-256 repositories. All 12 local/serialized continuation cases pass after source removal. - Full repository build and typecheck pass at `1544cc4a90`. The build first found a nullability error; an explicit rejection return fixed it. - The first full local test attempt stopped before tests ran because omitted package postinstall steps left PostgreSQL library links absent. Direct `initdb --version` reproduced the missing-library failure. Restoring the package links made that probe pass; the full test rerun is active. - The initial typecheck also found missing excluded-plugin SDK links after the same install. Running the repository postinstall link helper corrected this, and the full typecheck now passes. - All current-head CI gates pass at `3c87d13d1b`: 54 success, 2 skipped, 1 neutral. Fresh Greptile is 5/5 and no review threads remain open. - Review requested shallow and SHA-256 coverage. The current commit adds both plus partial-clone coverage; all 98 archive tests and the full server typecheck pass. - All six local chat shards pass. The monolithic local general-suite run reports one viewer-permission test failure and remains active. That exact test passes alone, and its full 505-test file also passes alone. The original failure cause is unresolved; this is not a full local-suite PASS. The complete current-head CI suite is green. ## Risks - This widens capture eligibility only for the checked resolved-merge state. No migration commits or aborts the source operation. - A missing object or changed metadata still blocks publication and retains the source. - Unresolved merges and other operation formats remain unsupported. Their preservation guard is unchanged. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, and code execution. The runtime does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with the available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task id - [x] I have run the affected tests locally and they pass; full-suite limitations are recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented the risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c1d567ff6a |
fix(workspaces): isolate archived nested Git reconstruction (#19409)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace import must preserve archived task files and Git history. > - A saved nested checkout can be a linked worktree whose branch belongs to its old absolute path. > - Archived import reconstructs that checkout in a new scratch directory. > - Ordinary linked recovery rejects the old branch at that new path. It can also write into a cache used by other import workers. > - This pull request reconstructs archived nested checkouts as private standalone repositories. > - Each import preserves saved files and extra history without changing shared worktree registrations or branches. ## Linked Issues or Issue Description **What happened?** Archived workspace import fails when a saved nested linked checkout has a branch owned by the old path. The new scratch path fails the linked-worktree ownership check. The existing cache lock is process-local, so separate import processes also share mutable Git state without a common lock. **Expected behavior** Archived import must create an independent portable copy of the saved checkout. It must retain the saved commit, branch name, staged edits, working files, ignored recovery files, settings, and extra saved commits. It must keep the source cache unchanged and enforce source authorization. **Steps to reproduce** 1. Create a managed nested linked checkout with an unpushed commit and staged and unstaged edits. 2. Save an encrypted restore point, including ignored files and an extra history tip. 3. Restore its saved contents under a different absolute workspace path. 4. Before this repair, the new path cannot adopt the branch owned by the old path. **Paperclip version or commit** The real Git regression fails before the repair at `fc1f5f3052`. The final branch includes the merged interrupted-upload and nested-source authorization fix from #19396. **Deployment mode** Archived remote workspace migration with real Git repositories, disposable PostgreSQL, and encrypted object storage. Local integration tests use the filesystem storage provider. Builds on merged #19402 and #19396. Related #19374 concerns interrupted clone staging; it does not implement private archived reconstruction. No duplicate open repair was found. ## What Changed - Enable private repository reconstruction only for archived migration. Ordinary linked recovery keeps its path ownership guard. - Add an object-only source to the managed checkout owner. A verified cache supplies only the saved commit and authenticated bundle prerequisites through scoped fetches. The private copy includes no unrelated task refs or objects, hard links, object alternates, or current cache files. - Fetch missing authenticated bundle prerequisites before importing a saved bundle in both private reconstruction and ordinary linked recovery. A pruned cache can use the approved remote; unavailable history still blocks restore. - Keep every authenticated restore point's extra commit under a private ref in root and nested private copies so portable archive capture and garbage collection retain it. Ordinary recovery keeps its existing history and expiration rules. - Share the verified, bounded ignore-file copy between root and nested reconstruction. Sources remain unchanged. - Retain source authorization before and after reconstruction and the final source-row locks from #19396. - Test concurrent private copies, source rejection, foreign task refs and objects, ordinary linked prerequisite recovery, a real migration, and cold restore after removing all original repositories. Test hard links, oversized excludes, unsafe info paths, and replacement during a descriptor read. ## Verification - Final affected run: 292 tests pass across ten suites. These cover real Git repositories, encrypted restore points, disposable PostgreSQL migration, portable archives, the restore-point ledger, source rejection, and concurrent copies. - Full repository typecheck and build pass after the scoped-object repair. Final server typecheck and full build also pass after restricting preserved refs to archive mode. - RED then GREEN: a linked checkout at a different path; a pruned bundle prerequisite still available from the approved remote; foreign task refs and objects entering the archive; root extra history disappearing after cold restore; and archive-only refs blocking ordinary standalone expiration. - Two concurrent private copies retain the saved head, branch, staged and unstaged edits, ignored recovery bytes, settings, and modes. Source refs, index, worktree registrations, and working files remain unchanged. - Delete the original checkout, cache, remote, and private copies after portable capture. Cold restore retains the saved files and extra history after garbage collection. No alternates or unrelated task objects remain. - Ordinary linked recovery retains its path ownership check and recovers pruned prerequisites. New and existing ordinary standalone destinations have no archive-only refs and remain eligible for expiration after normal object cleanup. - Unsafe exclude tests reject hard links, oversized files, nonregular paths, info-directory substitution, and replacement during the descriptor read. They preserve the target and replacement source bytes. - Preserve the failed attempts: an initial migration fixture changed its home after constructing the source path. Later expiration fixture attempts used a filtered current-head tip, omitted the local-source callback, or used cleanup that generated Git server-info files or left packed unreachable objects. Correct those setup assumptions before the actual regressions fail. Final affected suites pass. - Two earlier local full-suite attempts were interrupted before editing valid review findings. Neither is reported as a full-suite PASS. Final-head full CI passes; review is 5/5 with no open findings. ## Risks Archived imports use additional private disk space for independent Git objects. Missing, changed, or unsafe sources still block import. The repair does not weaken the default linked-worktree ownership check, change stored formats, or update live provider state. The shared checkout addition is optional and has no effect for existing callers that do not supply an object source. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
92452524f4 |
fix(server): recover authenticated empty retention checkpoints (#19417)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Ephemeral server enrollment must preserve existing task files. > - Archived task directories can depend on encrypted retention checkpoints. > - A missing object blocks migration even when its authenticated contents were exactly empty. > - Metadata counters alone cannot prove that no saved files existed. > - This change verifies the exact candidate against the original encrypted key envelope before normal migration publication. > - The benefit is lossless migration of this bounded case without resetting task history or weakening ordinary restore. ## Linked Issues or Issue Description **What happened?** Migration stops at an archived empty directory when its retained encrypted object is missing. The original encrypted key envelope still authenticates the payload hash, byte length, company, workspace, checkpoint ID, and generation. **Expected behavior** Administrative migration can reconstruct those exact original bytes when storage confirms absence and every authenticated check passes. It must keep missing nonempty, expired, corrupt, or ambiguous sources blocked. **Steps to reproduce** 1. Capture and encrypt an empty task directory. 2. Retain its reference and encrypted key envelope. 3. Remove the old directory and storage object. 4. Run the normal remote workspace migration. **Paperclip version or commit** The regression fails before this change at `205ffcb606`. Related preservation work is in #19396, #19402, and #19409. No duplicate open fix was found. **Deployment mode** Administrative migration to external workspace storage. Tests use real encryption, real filesystem storage, and disposable PostgreSQL. ## What Changed - Let the encryption owner verify a candidate against the existing scoped encrypted key envelope. Normal ciphertext validation keeps its previous order and checks. - Share the task-directory encoder with an exact empty payload constructor. - Add a default-off loader option for confirmed missing empty checkpoints. Only the administrative migration caller enables it. - Keep the original checkpoint reference and object unchanged. Publish the recovered bytes through the existing snapshot and source-fence owners. - Reject forged counters, wrong identities, missing nonempty data, present corruption, unreadable present objects, unknown absence, and expiry. ## Verification - Final affected run: all 238 tests pass across five crypto, snapshot, retention, and migration suites. Full repository typecheck and build pass; server typecheck and full build also pass after the cancellation repair. - RED then GREEN: exact missing-empty recovery and cancellation of a stalled HEAD through the real storage-service boundary. Cancellation retains the caller's original abort reason. - Disposable PostgreSQL migration publishes a cold-restorable empty archive. Archived status, task status, task binding, original reference, and the absent original source stay unchanged. Replay keeps one canonical snapshot. - Negative cases reject forged empty counters, an authenticated directory or file, every foreign identity field, expiry, present corruption, unreadable present objects, unknown absence, and missing nonempty data. Ordinary restore keeps its previous failure behavior. - Initial verification retained one new fixture-path failure after 145 passes, and a typecheck rejected the fixture's widened schema literal. Correct the fixture to use the returned staging path and its explicit reference type. The final complete affected run passes. - The new HEAD request runs only after a failed read with the administrative option enabled. It uses the existing cancellation contract and adds no polling or background work. - Final-head full CI passes; review is 5/5 with no open findings. This PR does not claim deployed enrollment or a live migration result. ## Risks Recovery requires the original valid encrypted key envelope and an exact format match. A format change or unavailable key continues to block migration. Existing ciphertext hash, size, authentication, and ordinary restore behavior remain enforced. This change has no schema, provider-session, API, or UI contract changes. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
958f9aa9f3 |
feat(issues): limit agent-created task trees to parent and child (#19312)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents split work into tasks. A task can have a parent task. > - Today agents can nest tasks to any depth: parent, child, grandchild, and deeper. > - Deep agent-made trees are hard to follow, and an agent that works on a child task makes grandchildren by default. > - This pull request limits agent-created trees to two levels: a top-level task and its children. > - A request for a grandchild does not fail. The server puts the new task beside the requester and tells the agent. > - Board users keep full control. They can still nest to any depth. Existing deep trees stay valid. > - The pull request also records which task created each new task, so the audit trail stays complete when the parent changes. ## Linked Issues or Issue Description No public issue exists. This is the feature description. Related: #16853 (budgets that include descendant work) also reasons about task trees; this change does not alter its traversal. **Subsystem affected** Issues (tasks): create, child create and move in the issue service and routes. Agent tools, chat suggestions and creation provenance use these paths. **Problem or motivation** Agents create subtasks as deep as they want. An agent that runs on a child task gets that child as the default parent, so its follow-ups become grandchildren. The trees become deep and hard to read. The parent link is also the only record of which task created a follow-up, so that record is lost when the parent changes. **Proposed solution** Limit agent-created trees to a parent and its children. When an agent asks for a grandchild, put the new task under the requester's parent and return `parentAdjusted`. Do not reject the request. Keep board users unrestricted. Record the originating task in a new server-set `origin_issue_id` column. Keep `origin_run_id` as the originating run. **Alternatives considered** Reject grandchild requests with an error. We did not do this, because agents then fail mid-task. Add a database constraint. We did not do this, because existing deep trees must stay valid and nesting can return later. ## What Changed - Add `MAX_AGENT_TASK_HIERARCHY_DEPTH` (2) and the `IssueParentAdjustment` type in `packages/shared`. - Add `resolveNewTaskParent` and `planIssueParentChange` in `server/src/services/issues.ts`. `create`, `createChild` and `update` use the agent limit by default (`parentDepthPolicy: "flatten"`). - The issue create, `/children` and PATCH routes pass `allow_nested` for board users. For agents, the routes check authorization against the parent that the server actually writes. - The requester keeps its behavior when the server moves a task beside it: `blockParentUntilDone` blocks the requester, and the new task inherits the requester's workspace and privacy. - The server never moves a task above a low-trust boundary root that contains the requester. - These subsystems keep any depth, because they find their work by parent: watchdog reviews, monitor recovery, email-channel placement, and routines that a person configured. - An agent move of a task that has children skips only the parent change and returns `reason: "has_children"`. - Add `issues.origin_issue_id` (nullable, `on delete set null`, indexed) in migration `0434`. Only the server sets it. The `createdFrom` detail field, the `createdFromIssueId` filter and task-creation provenance read it first. - Chat prompts now let a chat agent create one task, a task with children, or separate tasks. The conversation is never a parent. - The runner `create_task` tool accepts an optional `parentTaskId`. A chat agent can create a top-level task and then put subtasks under it. The run must be able to read and change the parent. A conversation is never a parent. The depth limit still applies. Project tools map the field to `parentId` through the issue route. The protocol manifest, semantic contracts and eval fixture are regenerated. - Watchdog runs that have an active watchdog keep their requested parent on all create paths, because their access is limited to the watched subtree. - Fix a bug that the test drive found: the chat (MCP) `create_task` ignored the requested `status`, so a `backlog` task became `todo` and started to run. It now uses the same rule as the native tool. - Update `skills/paperclip`, `doc/SPEC-implementation.md`, and add the plan `doc/plans/2026-11-05-two-level-task-hierarchy.md`. ## Verification - `cd server && npx vitest run src/__tests__/agent-task-hierarchy-depth.test.ts src/__tests__/agent-task-hierarchy-depth-routes.test.ts`: 19 new tests for service and route behavior, all pass. - Related suites (64 files: issues service, thread interactions, privacy, watchdogs, provenance, runner tool authority, decisions, routines, project tools): 1208 tests pass. Two tests changed because of the intended behavior and pass now. Two `email-channels.integration` tests fail locally because the test cannot find a bundled skill file from this worktree path. This change does not touch that code. - `packages/db`: migration safety, snapshot drift and journal tests pass. - Typecheck passes for `server`, `packages/shared`, `packages/db`, `cli` and `ui` (local `ui` errors come only from generated Storybook catalog files that are absent). - Local test drive on an isolated instance with real Claude agents: 29 of 29 use cases pass, including an agent on a child task (follow-ups placed beside it, with the notice) and a chat that creates a top-level task with three subtasks. - I did not run the full `pnpm test:run` or `pnpm build` locally. CI runs them. ## Risks - Behavior change for agents: some follow-ups that were grandchildren are now siblings. The parent of the requester is woken when all its children finish, which now includes these tasks. - The runner `create_task` input contract adds one optional field, so the protocol manifest and the pack contract change. The tool output does not change: it shows the actual `parentId`, and the activity log records `parentAdjusted`. - Migration `0434` adds a nullable column, a foreign key and an index. It does not backfill rows. - `IssueCreationSource.run` is now nullable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
TonioandClaude Opus 5.5
|
9e9a628a57 |
feat(ui): turn Artifacts V1 on for every instance and hide its toggle (#19164)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Artifacts show the output of agent work on the Artifacts page, in the panel Artifacts tab, and in the task thread > - #17228 added the Artifacts V1 presentation behind an experimental setting that is off by default > - The presentation is now ready for all users, so an opt-in toggle is no longer necessary > - A new default alone does not reach all instances, because each experimental write persists every flag > - This pull request removes the toggle and turns V1 on for every instance > - The benefit is one artifact presentation for all users, with an off switch for hosting operators ## Linked Issues or Issue Description Refs #17228 Refs #17263 **Is your feature request related to a problem? Please describe.** Artifacts V1 shows only on instances where an admin turned on an experimental setting. Other users see the previous cards. **Describe the solution you would like** Show Artifacts V1 on all instances. Remove the toggle from Experimental settings. **Describe alternatives you have considered** A new default without a change to the hook. This does not work, because many instances have a stored false that nobody chose, and no toggle would remain to change it. ## What Changed - Removed the Artifacts V1 card from Instance Settings, Experimental. - Changed `enableArtifactsV1` to default on in the schema, the feature catalog, and the settings service. - Changed `resolveArtifactsV1Enabled` to ignore a stored false. It honours false only when the value comes from managed config. - Kept the flag and the previous cards, so a hosting operator can hold V1 off. - Kept the key in the Cloud staging list, so a managed false does not hold V1 off on staging. - Updated unit tests, the Artifacts V1 plan, and the directions-lab Today reference. - Moved the attachment and upload e2e specs and the runner download helper to the V1 rows. - Added V1 tests for a Markdown review: expansion, deep link, and open, raw and download actions. ## Verification - `cd ui && npx vitest run`: one test file failed before its fix. That file tests the previous cards, and it now mocks the hook to off. I reran it with the other changed UI tests, and they pass. I did not repeat the full UI run after the fix. - `npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/text-attachment-tabs.spec.ts tests/e2e/artifact-uploads.spec.ts tests/e2e/artifact-tab-arrival.spec.ts`: 10 tests pass on V1. - `tests/e2e/cloud-auth.spec.ts` failed once in CI on a task heading. It does not touch artifacts, and 10 local runs pass. - `cd server && npx vitest run src/__tests__/instance-settings`: 183 tests pass. - `cd packages/shared && npx vitest run src/feature-catalog.test.ts src/validators/instance.test.ts`: 35 tests pass. - `pnpm run check:token-gates` and the `ui` and `server` typechecks pass. I did not run the full repository check. - In the artifact directions lab, with no stored settings, the Today right panel renders the V1 rows. - I did not test this on a live instance. After merge, confirm on staging that the Experimental page has no Artifacts V1 card and that all three surfaces show V1. ## Risks - All users see V1 after the next release, not only staging users. To revert, revert this pull request. - Users cannot turn V1 off. Only a hosting operator can, through managed config, and not on Cloud staging. - An instance with a managed `enableArtifactsV1: false` stays on the previous cards. The cloud harness source has no reference to this key, but I cannot read the live fleet configuration. - V1 surfaces render before settings load. An operator-held instance can briefly show V1 before the previous cards. - Admins who use the API can still write the flag. The write has no effect on unmanaged instances. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
f7a4c5678f |
feat(runtime): pre-copy completed harness history before enrollment (#19137)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server needs provider history in shared object
storage.
> - Offline enrollment must preserve old backups before it can detach a
volume.
> - A large copied dataset still exceeded the maintenance deadline with
parallel transfers.
> - Most of its bytes belong to successful turns with released runtime
authority.
> - This pull request pre-copies that history while unrelated executions
continue.
> - The final offline migration still verifies every source and grants
enrollment authority.
## Linked Issues or Issue Description
Refs #19108, #19038, #18943.
**What happened?**
A copied instance has 1,953 local harness backups and 54.8 GB of
declared source bytes. A resumed parallel migration still reached its
25-minute deadline. It cannot safely enroll by moving all history during
maintenance.
**Expected behavior**
Copy eligible completed history while the server remains online. Defer
active, failed or changed sources. Keep all existing data and final
enrollment checks.
**Steps to reproduce**
1. Retain many local native harness backups on a volume-backed instance.
2. Attempt a full offline conversion within the maintenance deadline.
3. Observe that most time moves historical bytes rather than current
execution state.
## What Changed
- Add finite `native-harness-prestage.mjs` preparation from a normal
runtime build. Require explicit opt-in and shared object storage.
Process one source at a time with a 90-minute deadline.
- Use the recorded released runtime owner of each successful, finished
turn. Do not assert that all writers stopped.
- Recheck checkout, related queued/running work, successful result
ownership, session, Runner, exact lease stamp and unchanged backup at
database publication fences.
- Defer source changes, malformed source JSON and unsafe backup roots.
Log each such source and its reason. Keep database failures and
cancellation fatal. Keep completed objects valid for retry. Never return
an enrollment-ready receipt.
- Reuse the bounded migration source walker. Keep offline admission and
final verification unchanged.
- Keep private lifecycle state, task ownership and provider policies
unchanged. Document the operator path and its limits.
## Verification
- A zero-source scan now checks the existing database obligation query.
A known mismatch blocks with a source-missing receipt; empty
fresh/already-durable scans and nonempty deferred scans remain valid
advisory results. The prior command fails the mismatch regression. All
16 command and runtime-bundle tests pass for this follow-up.
- At head `7ba3dbc7aa`, 87 focused command, database-timeout, storage,
archive, publication and runtime-bundle tests passed with real
PostgreSQL and filesystem restoration.
- The current follow-up passes all 68 backup and archive tests. A source
renamed after preparation fails the new regression on the previous head,
then defers and permits the next backup with this fix. An archive output
ENOENT remains fatal. Existing database-error and cancellation tests
pass.
- New race tests fail against the previous storage owner: a changed
backup, queued turn or revoked lease otherwise publishes.
- Tests prove unrelated executions stay running, same-task work defers,
failed history stays deferred, retry verifies existing objects, and
private bytes restore after deleting the old home.
- Command review fixes require the configured existing vault key, retain
it for the process, set database lock/statement/idle bounds, close
clients on cancellation, and cap shutdown at ten seconds. Abort
immediately sets failure status and prints one blocked receipt. A real
child with real shortened timers reproduces the old missing-receipt exit
and verifies the fix. Tests also use a real PostgreSQL lock.
- Malformed manifest or runner JSON and symlink replacement fail the new
preparation and publication-fence regressions on the previous code. The
fixed command defers each damaged source and copies the next eligible
backup. Separate tests keep database SyntaxError and cancellation fatal.
Offline migration still rejects damaged sources.
- The bundle test runs the emitted preparation entry with plain Node
without starting the app.
- A prior copied-data experiment completed with 1,738 new publications,
nine verified current objects and 206 deferred sources. Later checks
found different database and filesystem capture times: 43 source runs
were absent from the older database snapshot, and one required
historical manifest was absent from the retained filesystem archive.
This experiment does not prove a consistent restore or enrollment
readiness.
- Full current-head CI at
|
||
|
DottaandPaperclip
|
f5f5184f23 |
fix(workspaces): preserve retained branches and private commit sources (#19147)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers must preserve existing task files before
volume removal.
> - An old private checkout can be absent while its recorded branch
still exists.
> - A deleted branch can also retain its exact commit in the original
private object source.
> - The importer currently rejects these surviving sources.
> - This pull request saves the verified committed tree and records its
exact source proof.
> - It keeps task state, branch ownership and recovery obligations
unchanged.
## Linked Issues or Issue Description
Refs #19138, #19144, #18826.
**What happened?**
Existing Git workspace records can have missing checkout folders and
retained branch objects. The importer returns
`local_source_unsafe_or_missing`. A successful deletion receipt can also
have a valid private object source that the importer overlooks in favor
of a missing or changed project cache.
**Expected behavior**
Save the surviving committed tree from the exact recorded branch or
deletion receipt. Reject competing owners, live worktree registrations,
changed branches and invalid source identities. Do not use the current
default branch as a substitute.
**Steps to reproduce**
1. Retain an isolated workspace branch in its managed project cache.
2. Remove its clean checkout but keep the branch and original workspace
record.
3. Attempt workspace migration.
4. Alternatively, retain a deleted commit in its recorded private source
while its shared source is unavailable.
## What Changed
- Pin a retained branch in its verified company and project Git cache.
- Require the recorded private path, no competing workspace owner and no
surviving Git worktree registration or index authority.
- For a user-owned branch, require its exact successful unforced removal
receipt after archival. Preserve the branch ownership metadata.
- Prefer a verified recorded private object source for successful
deletion receipts when that source still exists.
- Recheck branch and receipt proof before publication. Preserve
compatibility with existing migration journals.
- Allow missing active records only through the existing offline writer
fence. Keep their status unchanged.
- Add byte-restoration and rejected-authority tests. Document the source
limits.
- Revalidate and honor interrupted journals' original Git source path.
Reject detached registrations and retained orphan index data. Reuse the
cleanup producer command formatter for paths containing spaces.
## Verification
- 172 focused migration, concurrency and isolated-source tests pass with
real PostgreSQL and Git repositories.
- The previous importer fails the retained-branch and offline-active
restoration cases. The previous source selection also fails the
private-cache and clean user-owned receipt cases.
- Tests remove the old home and restore the pinned commit and task file
bytes. New shared default-branch work is excluded.
- Tests reject competing owners, registered worktrees, missing branch
ownership, deleted branches, changed branch tips, forced removal and
revoked receipts.
- Read-only re-inventory of 79 actual failed source records makes 78
eligible with committed owner overlays. One older local test repository
remains blocked. This is source-overlay qualification, not full
released-image acceptance.
- A separate bounded archive replay completed six imports before its
25-minute deadline. That timing failure remains open; it does not
establish complete migration.
- Local server typecheck has no changed-file diagnostics. Stale linked
declarations prevent a clean local whole-server result. Full clean CI
remains required.
- Serving enrollment and empty-home recovery remain separate acceptance
gates.
- Latest rebased head
|
||
|
DottaandPaperclip
|
c8013d06ad |
fix(workspaces): preserve task-only warnings during offline migration (#19144)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers must save existing workspaces before volume
removal.
> - Offline migration checks file ownership and recovery authority.
> - The same gate also rejects task assignment and configuration
warnings.
> - Those warnings do not require the old filesystem.
> - This pull request preserves known task-only warnings while it saves
files.
> - File recovery and execution obligations keep their existing
blockers.
## Linked Issues or Issue Description
Refs #18943, #19038, #19138.
**What happened?**
Offline workspace migration refuses tasks with assignment,
configuration, disposition or deliberate-wait warnings. It returns
`workspace_recovery_unresolved` even when the warning has no file or
execution recovery authority.
**Expected behavior**
Save local workspace bytes while retaining the task warning unchanged.
Continue to block unknown recovery kinds, file sources, export failures
and execution recovery work.
**Steps to reproduce**
1. Retain a local workspace on a task with an active assignment or
configuration warning.
2. Stop all app writers through the offline maintenance entry.
3. Attempt workspace migration.
## What Changed
- Admit four known task-only warning kinds during offline file
migration.
- Keep file recovery markers, system recovery, native failure markers,
export-failure causes and unknown kinds blocked unless their exact
legacy source is already preserved.
- Keep the existing live-migration recovery exclusion.
- Recheck warning evidence before publication. Do not resolve warnings
or change their owners.
- Add restoration, rejection and publication-race tests. Update the
migration plan.
## Verification
- 112 focused migration and concurrency tests pass with real PostgreSQL
and filesystem archives.
- The four task-only restoration cases fail against the previous
importer with `workspace_recovery_unresolved`.
- Tests delete the old home, restore its saved bytes and compare each
warning row unchanged.
- File and execution markers, unknown kinds and export failures remain
blocked.
- A task warning that gains file recovery during upload prevents
publication.
- Local server typecheck has no changed-file diagnostics. Stale linked
dependency declarations prevent a clean local whole-server result. Clean
full CI remains required before merge.
- The PR was rebased onto the merged Git preservation fix after
resolving a plan-document conflict. The rebased head passed full PR CI
(41943153422). Current head
|