mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
f4a3f85b89b2dae7889b0a9c7fee07ed47b554cb
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
46dde54f52 |
fix(cloud): migrate run evidence during online preparation (#22337)
Drive bounded run-log and provider trace migration pages during online durable-data preparation. Share the page owner and settled-native archive rules with background migration, preserve active-source and storage-integrity guards, and report fresh page results. Keep canceled transfers owned and tracked until the underlying operation settles. Retain page cursors across the preparation budget. Verified 52 focused tests, including embedded PostgreSQL and object storage, server typechecking, and all 58 successful PR checks. Greptile 5/5 with no unresolved review comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
440f96247b |
fix(cloud): resolve or name the remaining online move blockers (#22294)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A hosted server can move its durable data to shared storage while it serves work, and then run without a volume > - A bounded online pass moves the data, and a readiness check names what still blocks the move > - Some servers stay blocked after many passes, and the pass reports only `skills:blocked` or `native_harness:source_not_successful`, or reports nothing for run logs > - Operators cannot see why, so they cannot tell recoverable data from data that needs a maintenance step > - This pull request recovers the cases that are safe to recover online and gives every other case a precise code > - The benefit is fewer blocked moves and a clear next step for each one that stays blocked ## Linked Issues or Issue Description Refs #19719 **What happened?** A serving server that runs the online durable-data pass many times stays ineligible. The pass reports `skills:blocked` without a reason, `native_harness:source_not_successful` for several different causes, and no reason at all for run logs that stay on the local disk. **Expected behavior** The pass moves every item that it can move safely while the server serves work. For each item that stays, the pass reports a code that names the cause. **Steps to reproduce** 1. Install a catalog skill on a volume-backed server. Remove its installed files and do not keep a saved version. 2. Keep a settled native run whose local log file has no final size or digest and whose events do not prove full coverage. 3. Run the online durable-data pass. 4. See that the skill is reported only as `skills:blocked` and the run log stays local with no reason in the pass result. **Additional context** The settled native log archive path from #19719 had no online caller. ## What Changed - Skill installation conversion returns a `code` for each blocked installation (for example `source_files_unavailable`, `saved_version_incomplete`, `catalog_origin_unavailable`, `legacy_source_unsafe`, `source_changed`). The pass reports `skills:<code>`. An unknown cause stays `skills:blocked`. - A spent budget during conversion now stops the page. Before, it was reported as a blocked skill and the page moved past it. - A catalog skill whose files are gone converts from the bundled catalog package only when the catalog content hash equals the recorded origin hash and the last audit recorded the installed bytes as equal to that origin with no local changes. - A modified catalog skill whose reset origin is gone gets its origin version from the bundled catalog package only when the catalog content hash equals the recorded origin hash. - Native harness pre-copy replaces `source_not_successful` with `source_run_active`, `source_run_unfinished`, `source_coordinator_missing`, `source_coordinator_unsettled`, `source_session_not_selected` and `source_lease_missing`. The settled-source rules do not change. - The run evidence worker archives a settled native log by run id when the canonical import cannot prove event coverage. The existing settled-source guard still decides. The worker records the reasons from its last completed sweep, and the pass reports them as `run_logs:<code>` and `provider_traces:<code>`. - `migrateRunEvidence` accepts a `runId` filter. ## Verification - `cd server && npx vitest run src/__tests__/managed-skill-content.test.ts` (embedded PostgreSQL): new catalog recovery, hash mismatch, unproven install and budget abort tests fail before the change and pass after it. - `npx vitest run src/services/native-runtime/native-harness-online-preservation.integration.test.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: the new reason tests fail before the change and pass after it. - `npx vitest run src/__tests__/run-evidence-migration-work.test.ts src/services/native-log-archive.integration.test.ts`: the worker archive test fails before the change and passes after it. The integration test shows a real settled source that is blocked as `native_log_source_coverage_unproven` and then archived by run id. - `npx vitest run src/services/online-durable-data-migration.test.ts src/services/online-durable-data-migration-postgres.test.ts` - `pnpm --filter @paperclipai/server typecheck` ## Risks - A settled native log with unproven coverage now moves to the read-only historical archive instead of staying local. The archive keeps the exact bytes. It does not claim canonical event coverage. This is the same result that the offline path gives. - The catalog fallback reads bundled or pinned catalog bytes. Each file is checked against its pinned SHA-256, and the package must match the recorded origin hash. - Pass deferral codes change. Readers that match `native_harness:source_not_successful` must use the new codes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9b16044bf1 |
fix(cloud): move idle local workspaces in the online durable-data pass (#22270)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A hosted server can move to volume-free storage only when no durable data stays on its local volume > - The online durable-data pass moves that data while the server keeps serving, one bounded and resumable pass at a time > - The pass does not move local execution workspaces, so servers with local workspaces stay ineligible (`workspaces_require_migration`) > - Online-safe owners for workspaces already exist: the online remote workspace migration and the online workspace operation reconciliation > - This pull request adds workspace phases to the pass that call those owners, and defers busy or unsupported workspaces with a reason > - The benefit is that idle workspaces move with no outage, and operators can see which workspaces cannot move and why ## Linked Issues or Issue Description **What happened?** The online durable-data pass never moves local execution workspaces. A server with local workspaces keeps the `workspaces_require_migration` blocker after every pass. **Expected behavior** The pass moves each idle local workspace to shared storage through the online owner. It defers a busy or unsupported workspace and reports the reason. **Steps to reproduce** 1. Run the online durable-data pass on a server that has a local `local_fs` execution workspace. 2. Read the pass result and the eligibility blockers. 3. The workspace stays local and `workspaces_require_migration` stays. **Deployment mode** Hosted server that moves from a local volume to volume-free storage. ## What Changed - Add two phases to the online durable-data pass, after `runtime_skill_assets` and before the native phases: - `workspace_operations`: pages companies that have local workspaces and calls `reconcileOnlineWorkspaceOperations` for each. It counts `workspaceOperations` and defers each held operation as `workspace_operations:<reason>`. - `workspaces`: pages local workspaces (same filter as the eligibility check), deepest path first so nested workspaces go before their parents. For each workspace it reads `inventoryRemoteWorkspaceMigration`. If the workspace is eligible, it calls `migrateRemoteWorkspace` in online mode (never `writersStopped`) with the pass signal and a 1 s source-lock wait. - A workspace that the owner refuses is deferred with each owner reason as `workspaces:<reason>`, for example `workspaces:active_execution`, `workspaces:task_binding_required`, `workspaces:workspace_provider_unsupported`, `workspaces:workspace_source_busy`, or `workspaces:execution_rows_busy`. An unknown error becomes `workspaces:owner_failed` and its message is not echoed. The page continues with the next workspace. - Both phases use a cursor and a limit that the pass can halve after a budget timeout, like the other phases. - Add `listLocalWorkspaceCompanies` and `listLocalWorkspaces` next to `localWorkspaceDependencyWhere`. - The pass stops waiting for a workspace owner at the budget. It reports `workspaces:owner_busy` until that owner settles. ## Verification - `cd server && npx vitest run src/services/online-durable-data-migration.test.ts src/services/online-durable-data-migration-postgres.test.ts` - Unit tests with injected owners: phase order, online mode and lock wait, deferral codes, owner refusal under locks, cursor resume across budget-limited calls, and page halving that keeps finished workspaces. Seven tests fail without the change and pass with it. - Embedded PostgreSQL test with the real owners: an idle managed-project workspace moves to a committed snapshot and its files restore from it. A workspace with a running task run is deferred as `workspaces:active_execution`, and its record, files, and operations stay unchanged. The test fails without the change and passes with it. - Embedded PostgreSQL test for the workspace and company cursors: ten workspaces in two companies, with equal and different path lengths, across pages of three. Each workspace appears once, deepest first. - `pnpm verify` selected 2934 of 3802 test files (almost the full suite). It was not run to the end. Required CI covers the full suite. ## Risks - A workspace move that takes longer than one pass budget (60 s) stops at the budget. The workspace keeps its in-progress move marker, so new runs for that workspace are refused until a later pass resumes the move. The pass resumes the same page on the next call. If one workspace times out twice at page size 1, the pass reports `workspaces:stalled` and goes on; the next cycle resumes it again. A very large workspace can therefore stay blocked for runs across passes. The operator online workspace tool can resume it with a longer deadline. - A workspace with no proven online source uses the owner's lifecycle lock, which waits without a timeout. The pass does not wait for it past the budget. The pass returns at the budget, and the owner stops at its next abort check after it gets the lock. Until that owner settles, later passes skip the workspace phase and report `workspaces:owner_busy`, so two moves never run at the same time. - Workspaces that can never move online (for example unsupported providers or workspaces without a task binding) are reported each pass and are not changed. This change makes no product decision for them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |