mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 12:33:05 +01:00
b05d3db2b3b19142c12170e795832b961e6212b2
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
f7a4c5678f |
feat(runtime): pre-copy completed harness history before enrollment (#19137)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server needs provider history in shared object
storage.
> - Offline enrollment must preserve old backups before it can detach a
volume.
> - A large copied dataset still exceeded the maintenance deadline with
parallel transfers.
> - Most of its bytes belong to successful turns with released runtime
authority.
> - This pull request pre-copies that history while unrelated executions
continue.
> - The final offline migration still verifies every source and grants
enrollment authority.
## Linked Issues or Issue Description
Refs #19108, #19038, #18943.
**What happened?**
A copied instance has 1,953 local harness backups and 54.8 GB of
declared source bytes. A resumed parallel migration still reached its
25-minute deadline. It cannot safely enroll by moving all history during
maintenance.
**Expected behavior**
Copy eligible completed history while the server remains online. Defer
active, failed or changed sources. Keep all existing data and final
enrollment checks.
**Steps to reproduce**
1. Retain many local native harness backups on a volume-backed instance.
2. Attempt a full offline conversion within the maintenance deadline.
3. Observe that most time moves historical bytes rather than current
execution state.
## What Changed
- Add finite `native-harness-prestage.mjs` preparation from a normal
runtime build. Require explicit opt-in and shared object storage.
Process one source at a time with a 90-minute deadline.
- Use the recorded released runtime owner of each successful, finished
turn. Do not assert that all writers stopped.
- Recheck checkout, related queued/running work, successful result
ownership, session, Runner, exact lease stamp and unchanged backup at
database publication fences.
- Defer source changes, malformed source JSON and unsafe backup roots.
Log each such source and its reason. Keep database failures and
cancellation fatal. Keep completed objects valid for retry. Never return
an enrollment-ready receipt.
- Reuse the bounded migration source walker. Keep offline admission and
final verification unchanged.
- Keep private lifecycle state, task ownership and provider policies
unchanged. Document the operator path and its limits.
## Verification
- A zero-source scan now checks the existing database obligation query.
A known mismatch blocks with a source-missing receipt; empty
fresh/already-durable scans and nonempty deferred scans remain valid
advisory results. The prior command fails the mismatch regression. All
16 command and runtime-bundle tests pass for this follow-up.
- At head `7ba3dbc7aa`, 87 focused command, database-timeout, storage,
archive, publication and runtime-bundle tests passed with real
PostgreSQL and filesystem restoration.
- The current follow-up passes all 68 backup and archive tests. A source
renamed after preparation fails the new regression on the previous head,
then defers and permits the next backup with this fix. An archive output
ENOENT remains fatal. Existing database-error and cancellation tests
pass.
- New race tests fail against the previous storage owner: a changed
backup, queued turn or revoked lease otherwise publishes.
- Tests prove unrelated executions stay running, same-task work defers,
failed history stays deferred, retry verifies existing objects, and
private bytes restore after deleting the old home.
- Command review fixes require the configured existing vault key, retain
it for the process, set database lock/statement/idle bounds, close
clients on cancellation, and cap shutdown at ten seconds. Abort
immediately sets failure status and prints one blocked receipt. A real
child with real shortened timers reproduces the old missing-receipt exit
and verifies the fix. Tests also use a real PostgreSQL lock.
- Malformed manifest or runner JSON and symlink replacement fail the new
preparation and publication-fence regressions on the previous code. The
fixed command defers each damaged source and copies the next eligible
backup. Separate tests keep database SyntaxError and cancellation fatal.
Offline migration still rejects damaged sources.
- The bundle test runs the emitted preparation entry with plain Node
without starting the app.
- A prior copied-data experiment completed with 1,738 new publications,
nine verified current objects and 206 deferred sources. Later checks
found different database and filesystem capture times: 43 source runs
were absent from the older database snapshot, and one required
historical manifest was absent from the retained filesystem archive.
This experiment does not prove a consistent restore or enrollment
readiness.
- Full current-head CI at
|
||
|
DottaandPaperclip
|
d941ed3879 |
feat(server): verify existing-volume ephemeral enrollment (#18529)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud web servers can now keep durable state in PostgreSQL and S3.
> - Existing servers still have files and a decision signing key on
their volume.
> - The volume must stay attached until every data owner verifies its
migration.
> - This pull request adds a dedicated maintenance process for that
transition.
> - Cloud can preserve the original key and remove the volume only after
a matching receipt.
## Linked Issues or Issue Description
Refs #18518 and #18526. Existing-volume enrollment is a required step in
the zero-downtime web deployment work. This is the Core maintenance
side; the Cloud transition is a separate private change.
## What Changed
- Package a standalone maintenance entry that starts no normal app
routes or execution workers.
- Coordinate the existing agent-file, run-evidence and skill migration
owners. Follow every page and verify shared bytes before final
eligibility.
- Require stopped writers. Refuse active execution, missing local bytes,
local workspaces and external instructions.
- Read the existing decision key without generating a replacement.
Preserve normal key creation for ordinary server startup.
- Require an attempt token for status and export. Export the original
key only after successful migration. Keep status and errors free of
secret material.
- Document the Cloud ownership boundary, deadline, retry and retention
requirements.
## Verification
- Full workspace typecheck and build passed before the review fixes. The
final server TypeScript compilation passed after correcting Drizzle's
optional predicate type.
- Production runtime bundling passed: 2,702 inputs, 122 chunks,
including the maintenance executable.
- The focused suite passed 21 cases using PostgreSQL and real
S3-compatible storage. It covers multi-page migration of 26 agents,
exact reads after deleting the predecessor home, corrupt-object
rejection, active/missing-data refusal, and the bundled plain-Node
process with authenticated export and clean SIGTERM exit.
- Regression cases refuse unsaved terminal working copies, require
explicit stopped/cleaned receipts for unavailable copies, clean 101
expired traces through the owning service, and enforce the deadline
after migration succeeds.
- Initial integration attempts exposed fixture cleanup ordering
problems; corrected runs pass. CI also caught a Drizzle predicate type
error, now fixed and compiled locally.
- All repository CI gates passed on
|
||
|
DottaandPaperclip
|
006d5691a6 |
feat(server): support volume-free web server handoff (#18518)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
27709a99b1 |
perf(scripts): emit a server module beside the runtime bundle entry (#16765)
The runtime bundle now also has a server entry point, dist-runtime/server.mjs: the bundled dist/index.js without the argv shim. index.mjs loads it, so importing server.mjs loads the server code without starting it, and a compile cache warmed from it covers exactly the files a boot loads. bundleRuntime reports it as serverModule. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
e30550ef39 |
refactor: remove code comments added by recent PRs (#16681)
CONTRIBUTING.md says PRs add no code comments, but #16485, #16586, #16621 and #16672 added some. This removes the 215 comment-only lines git blame on master attributes to those merge commits, across 12 code files, plus one blank line the removal left doubled. Rewritten (not added) comments in scripts/assert-cloud-image-sentry.mjs and comments in .github workflows are left alone. No code changes. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
27ab902d3d |
perf(docker): start the server from a precompiled runtime bundle (#16672)
The production image started the server through the tsx loader, which resolved about 3,100 modules and transpiled about 1,200 workspace TypeScript files on every boot (7.6-20 s before listen on a live instance). A build:runtime step now bundles server/dist/index.js with esbuild, compiling in the workspace packages, and the image starts node server/dist-runtime/index.mjs. Server dependencies stay bare so runtime resolution (including the cloud image's optional peers) is unchanged; each module keeps its original import.meta location; dynamic imports stay split; the entry presents the original dist/index.js as argv[1]. The avatar worker loads through tsx's API when the main thread has no loader, tini runs as a subreaper (-s), and PR CI boots the bundle with scripts/server-boot-smoke.sh. Co-Authored-By: Paperclip <noreply@paperclip.ing> |