mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 14:19:00 +01:00
fe72bc5f81dd3bf2722daf5bf714aa8d451b8374
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
ae1f781cac |
fix(native): preserve web files without settling remote recovery (#19814)
## Thinking Path > - Paperclip manages work that AI agents perform for users. > - Native sessions retain files on the web server and state in a remote provider. > - Replacing the web server must preserve its files without changing remote recovery authority. > - The existing maintenance owner requires destroyed-provider receipts even when all web writers have settled. > - This can keep the old web volume required only because a remote recovery obligation remains unresolved. > - This pull request adds a closed web-byte evidence mode to the existing preservation owner. > - The benefit is durable web-file coverage with unchanged remote references, failures and continuation policy. ## Linked Issues or Issue Description Refs #19748, #19806. A search for native harness filesystem evidence and open web/harness changes found no duplicate implementation. **What happened?** A settled successful native source can retain a raw controller, a separate warm supervisor and provider-home files while its historical external lease has no termination receipt. The existing filesystem-independence owner refuses before it can preserve these web files. One retained skill file can also remain physically present after its version row was deleted. **Expected behavior** Preserve every nonexcluded web file as authenticated evidence. Keep missing files absent and deleted skill versions deleted. Leave VM references, recovery actions, failures and execution authority unchanged. Validate the published web-byte receipt on an empty replacement using database bindings only. **Steps to reproduce** Use the included successful-source fixture with a settled web owner, an unresolved external lease, real controller/supervisor files and a retained skill whose version row is deleted. The baseline rejects at its destroyed-provider guard. The new explicit maintenance mode preserves the files and validates on a replacement. A separate fixture uses a prior successful source, a failed retained last run, a genuinely absent controller, a warm supervisor and 267 durable checkpoint parts. **Paperclip version/commit** Baseline `384fa2b7be`; final branch rebased onto `585c885168`. Current head: `990210d7e9`. **Deployment mode** Original-volume maintenance followed by an ephemeral replacement. No hosted operation or provider request is part of this PR. ## What Changed - Add explicit `web_evidence` maintenance mode and a separate `paperclip.native-harness-web-evidence.v1` receipt. Keep the existing terminated-provider mode and old manifest shapes. - Archive the closed canonical session-root inventory, the separate warm supervisor and exact admitted skill manifests through the existing encrypted `original_volume_source_evidence` owner. Preserve bytes and modes. Keep canonical credential exclusions. - Preserve local skill edits and deleted-version files as forensic evidence. Create no skill version, executable checkpoint, provider backup or recovery authority. - Require original-volume capability, settled web writers, physical/finalizer locks, exact task/source/cohort/input bindings and final byte/mode/membership comparisons. Keep genuinely absent raw controllers absent. - Bind allocation and stamp identity while allowing later external status and cleanup observations. Validate replacements from durable database and archive bindings without the original anchor, filesystem or storage reads. ## Verification - Node 24.22.2, PostgreSQL 18.1, macOS arm64. The unchanged baseline fails at the real destroyed-provider guard. The final revision passes all 85 owner, original-volume capability and unchanged continuation-policy cases in 21.14 seconds. - Tests perform real encryption, persistence through the S3 storage interface backed by the owned local fixture, authenticated restore and exact byte/mode comparisons. The 281-byte file remains present while its version row remains absent. The evidence archive stays separate from the provider-session backup scope. - The supervisor-only case preserves 267 database checkpoint parts, keeps the raw controller absent and validates without original-volume identity. Existing strict-mode and old receipt controls pass. - Negative controls cover active web writers, expired but owned finalizers, physical owners, foreign controller/supervisor identities, missing manifests, symlinks, changed bytes/modes/membership/source during upload, changed durable bindings, incomplete receipt coverage and invalid late publication. Concurrent attempts retain one valid winner and one audit event. - Server TypeScript, module boundaries, diff checks and private-data scans pass. The verification planner's first invocation used a wrong script path; its canonical invocation refused a plan because fixture inputs changed during selection. Neither attempt counts as a pass. Required CI will provide broader application coverage. - Earlier fixture and treatment failures remain recorded. They exposed a missing helper, JSONB property-order instability, missing separate-supervisor mode coverage and fixture timing/source/count errors. The final same owner suite passes without weakened assertions. - A canonical cleanup audit found two additional observation fields: the cleanup controller and manual-attempt ID. Their real publication updates reproduced false receipt invalidation on the first PR head. The narrow correction keeps WEB coverage valid through those observations while changed allocation identity still fails. The exact same 85-case suite passes afterward. The final `990210d7e9` head has fresh Greptile 5/5 with no comments or unresolved threads. Required CI run [42370607882](https://github.com/paperclipai/paperclip/actions/runs/42370607882) passes: 55 successful checks and four intended skips, plus successful Snyk. The old head's CI was superseded by this correction and is not counted as a pass. From `server/`: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts src/services/native-runtime/native-harness-original-volume.test.ts src/services/native-runtime/native-harness-filesystem-independence-continuation.integration.test.ts --hookTimeout 90000 --maxWorkers 1 --fsModuleCache ``` ## Risks This changes a maintenance trust boundary. It proves web-byte preservation, not provider recovery or termination. External recovery obligations and existing resume failures remain. Unknown layouts, missing manifests, active local owners and changed authority fail closed. Invalid receipts are not automatically replaced. The inventory has independent limits of 128 entries, 16 MiB per file and 64 MiB total. Larger valid data stays on the original volume and blocks this mode. Canonical credential exclusions remain excluded. Evidence uses the existing encrypted ledger and retention roots; it does not become an executable backup. Old receipts remain byte-compatible. Startup adds bounded per-retained-scope database reads. Measure the affected cohort's actual serving startup cost before enrollment. Validation adds no filesystem or object read, worker or recurring poll. Live source qualification and publication remain separate after review, merge and normal delivery. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and PostgreSQL/filesystem test execution. The runtime does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f673bd3cb5 |
fix(native): verify legacy harness filesystem independence (#19748)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native sessions retain provider checkpoints and recovery state across turns. > - Server replacement must preserve every remaining local data dependency. > - A legacy backup stamp can outlive a destroyed provider and an absent local backup. > - Surviving skill files can already have exact immutable package copies. > - Raw controller files also contain stream events that database checkpoints omit. > - This pull request records verified filesystem independence under the original-volume maintenance owner. > - Replacement startup checks durable bindings while existing resume failures and policies remain intact. ## Linked Issues or Issue Description Refs #19715, #19729, #19733. **What happened?** Startup treats every retained legacy backup stamp without a current backup row as a local filesystem dependency. This also blocks sessions whose provider was already destroyed and whose remaining local skill bytes have exact durable copies. **Expected behavior** An authorized maintenance owner can prove that an exact retained source has no uncovered local harness payload. Startup can use that proof after replacement. The proof must preserve failed recovery state and must not invent a provider backup. **Steps to reproduce** 1. Retain a terminal native session, its checkpoint, and its legacy backup stamp. 2. Record exact canonical destroyed-provider receipts. Keep only admitted immutable skill files on the original host. 3. Publish the owner-verified independence receipt. The old startup predicate still blocks; the updated predicate validates its durable bindings. ## What Changed - Add an internal original-volume capability. It requires a private nonce anchor, independently reviewed attachment evidence, current reserved runtime identity, and a witness that is at most 120 seconds old. - Verify retained and selected-source inputs, checkpoints, leases, destroyed-provider receipts, released ownership, and exact immutable skill references. - Inspect canonical persistence roots and compare surviving skill bytes and modes. Refuse any provider directory present outside the current persistence profile, including an old Codex home after a preconfigured-home switch. Unknown data, backups, history, owners, or source changes still block publication. - Preserve raw controller and supervisor bytes through the existing encrypted evidence archive owner. Match recovery records to the database checkpoint, authenticate the exact archive row, and compare the original bytes again before publication. The archive grants no executable authority. - Publish one audited receipt with source locks and a final comparison. Keep original files, stamps, warnings, statuses, and session pointers. A late capture retains a valid published winner and refuses an invalid receipt. - Validate published receipts from current database bindings on replacement hosts. Add no startup filesystem probe, object read, provider call, or recurring worker. ## Verification - Genuine baseline regression: restore the original startup predicate while keeping the actual receipt owner. The positive PostgreSQL/filesystem case fails with expected `false`, actual `true` after valid publication. Restore the changed predicate; the case passes. - Final rebased head `e78f4b4424`: 42 affected PostgreSQL/filesystem cases pass. This includes 32 owner guards, three paired continuation cases, six existing failed-source archive controls, and one existing lease-verified backup migration control. The other 113 tests in the selected existing files were explicitly unselected. - Genuine controller baseline: the unchanged prior owner publishes no raw archive and accepts missing checkpoint parts. Both new regressions fail. The repair passes encrypted byte-for-byte controller/supervisor restore, missing checkpoint, foreign source, active source, and changed-after-upload controls. - Genuine omitted-provider baseline: the unchanged owner accepts old Codex, OpenCode, and ACPX history after a preconfigured-home switch. All three refusal cases fail. The fix passes those cases and permits preconfigured execution with no omitted directory. - Genuine late-publication baseline: both concurrent winner tests fail before the repair. The repair keeps the existing valid receipt, refuses an invalid one, and keeps one audit event. A delayed raw upload also retains the first authenticated archive and its receipt. - Capability controls: 12 passed, including empty replacement refusal, stale or future evidence, changed deployment, missing volume, path aliases, changed anchor, and current remount identity. - The actual server-to-Runner boundary preserves explicit recovery policy on original and empty homes. The separate actual CREATED-sandbox preparation callback preserves checkpoint identity and policy without a backup. Provider I/O and final transport construction are controlled. - Warm sessions retain the same canonical refusal on both homes, make no backend/provider attempt, and retain the saved checkpoints. - Server TypeScript, module boundary checks, diff checks, and a private-data scan pass. The Runner TypeScript build was refreshed after rebase to include two new upstream exports. - Fresh Greptile review is 5/5 with no unresolved threads on `e78f4b44240b0fc33c2fd7cae2d5b4a2578e3153`; check `128330103712` completed successfully. Required CI remains pending. No live preservation or enrollment action ran. Focused command from `server/`: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts src/services/native-runtime/native-harness-filesystem-independence-continuation.integration.test.ts src/services/native-runtime/native-controller-migration.integration.test.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts --testNamePattern 'owner-verified|legacy harness continuation|preserves a failed source online|migrates a lease-verified' --hookTimeout 90000 --maxWorkers 1 --fsModuleCache ``` ## Risks - This proves that no uncovered harness payload remains local. It does not recover provider history or certify that no other copy exists. Existing governed-wait and warm-session failures remain. - Unknown providers, retained owners, extra payloads, missing references, or changed source bindings fail closed. Invalid receipts require investigation; this change does not repair or replace them automatically. - The maintenance capability requires independently reviewed Railway attachment evidence. The original anchor remains stable across normal releases; each new staging invocation needs current deployment and mount evidence. - Startup validation adds bounded per-scope database reads. It pages candidate scopes and caps each source history at 500 rows. Measure the affected cohort's actual startup latency on the qualified serving build before enrollment. - Raw archives use the existing backup ledger, encryption, retention, and restore machinery. They preserve evidence only. They do not repair missing provider history or grant a new resume policy. - Local tests control provider and transport boundaries. Live staging remains a separate step after review, merge, normal delivery, and a fresh original-volume capability. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and local code/test execution. The exact runtime variant and context-window limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
72f7b42fe3 |
Preserve settled failed controller state during online migration (#19678)
## Thinking Path
> - Paperclip controls agent work across web-server releases.
> - Volume-free servers need retained controller state in shared
storage.
> - Successful settled sources already support online migration.
> - Some retained failed sources have no accepted result and must stay
available for recovery inspection.
> - This change copies those sources under task, coordinator and
publication fences while unrelated work continues.
> - The original files, failed status and recovery warnings remain
unchanged.
## Linked Issues or Issue Description
Refs #19038. Refs #19631.
**What are you trying to improve?**
Preserve retained failed controller files without waiting for global
idle.
**What problem does this solve?**
The online producer skips terminal failed controllers. Their files can
still be needed for recovery inspection even when no accepted provider
artifact proves executable continuation.
**What would the improvement look like?**
A settled exact source can enter the existing encrypted archive store.
An accepted committed failed source can use the existing checkpoint
import. Busy, changed, unowned or foreign sources remain blocked.
## What Changed
- Add a source owner bound to the selected task session, run,
coordinator generation, admitted input and file digests.
- Check the task, physical finalizer lock, released coordinator,
canonical resume source and active related turns before upload and
before publication.
- Preserve failed sources through the existing encrypted archive format.
Do not treat archived bytes as execution authority or old-writer stop
proof.
- Allow the existing accepted committed checkpoint import for a finished
failed run without changing its status.
- Validate admitted input bindings before looking up local controller
files.
- Document the online preservation path and the final static writer
fence.
## Verification
- Real PostgreSQL: 125 cases passed across migration, controller
recovery and durable backup storage. One added assertion inspected an
unadmitted queued fixture; it was corrected while retaining the
queued-turn preservation and stale-publication assertions. The final
migration rerun passes 51/51; the other three affected suites pass
75/75.
- Genuine prepatch RED: online failed-source preservation and accepted
committed failed-source import both fail on the previous producer and
recovery owners.
- New cases cover independent restore after deleting the original
source, an expired but non-null coordinator owner, source mutation after
upload, a new related queued turn during upload, foreign input binding
and an accepted committed failed result.
- The full rebased server source and test-tree typecheck passes after
restoring its missing ignored workspace SDK link and building that
existing dependency.
- Module boundaries and whitespace checks pass.
- Rebased onto current master; both provider-artifact imports and the
new archive owner are preserved. Rebased type and module-boundary checks
pass. All six archive cases pass on the exact rebased head (39.68
seconds). All CI gates pass on exact head
|