mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 07:55:47 +01:00
630ed6624c71c26617a1e0f3c5e97384ece2caae
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dotta
|
82dd3bb702 |
Preserve unstarted queued turns during offline enrollment (#18887)
## Thinking Path > - Paperclip manages AI agents and their work. > - Existing-volume enrollment moves server files to durable object storage. > - Enrollment stops application deployments before it changes file authority. > - A task drain keeps unstarted turns queued in the database. > - The migration rejected every queued row, so a busy instance could never finish enrollment without discarding requests. > - This change preserves unstarted queue intent and continues to reject running or previously started executions. > - Users keep their pending turns and saved files through the conversion. ## Linked Issues or Issue Description **What happened?** A preserving enrollment found queued turns after task admission stopped. The workspace, instruction-file, and historical-log guards treated these rows as live writers. The drain also prevented the rows from starting, so waiting could not satisfy the migration guard. **Expected behavior** After every old deployment has stopped, maintenance can move files while unstarted queued turns remain unchanged. The replacement claims those turns through normal admission. Actual executions and retained execution ownership continue to block migration. **Steps to reproduce** 1. Queue a turn behind another task. 2. Start a task drain and let the current task finish. 3. Stop the application deployments and run offline enrollment. 4. Observe `active_execution_requires_drain` for the unstarted queued turn. **Paperclip version or commit** Reproduced on the migration contract through #18864. This branch starts at `ab2d2098a2764c511978d1c3e7a3fc5dfcef07f4`. Related: #18826, #18835, #18841, #18864. Searches for queued enrollment and queued migration found no duplicate fix. ## What Changed - Share one offline execution predicate across workspace, agent-file, and historical-log migration. - Reuse the existing pre-dispatch evidence predicate; queued turns with execution markers, saved checkpoints, source events, coordinators, or environment leases remain blockers. - Keep task bindings and normal live-migration restrictions unchanged. - Exercise normal replacement admission after deleting the old filesystem: the same queued run reads restored workspace bytes and instructions, completes its original wake, and cannot execute twice. - Document the offline boundary and queued-turn behavior. ## Verification - **111 tests passed** across the final 110-case migration suite and assembled queued-turn heartbeat proof, with real PostgreSQL and local S3. This includes workspace/log migration, queued ownership blockers, ordinary agent-file controls, and the actual bundled maintenance process. - The maintenance process preserves the queued row while archiving historical output. The new assembled heartbeat test creates a normal wake under drain, migrates without changing either row, deletes the original filesystem, and resumes through normal admission. It reads saved workspace bytes and the restored instruction bundle, persists one successful native result, completes the same wake, and verifies a second sweep does not duplicate execution. Only sandbox transport/provider execution are scripted; no live model or Daytona calls. - An earlier optional run-evidence worker integration invocation was skipped without its dedicated environment; it is outside the changed offline path. - Server typecheck, server build, and runtime bundle passed. - The first maintenance test exposed the separate agent-file guard; that guard now uses the same offline predicate and the test passes. - Initial local typecheck attempts lacked CLI dependency links in this isolated checkout. Restoring dependencies fixed setup without source changes. - Full repository checks run in CI. The first head lost spot runners in Build and serialized tests; one unchanged idle-deadline fixture crossed UTC midnight and failed its timestamp assertion. Final-head CI passed on run 41744678994 after retrying a spot-runner shutdown. Greptile reviewed the exact final head at 5/5; all inline findings are resolved. The separate Paperclip Review app remains queued. Hosted enrollment remains pending release qualification and compatibility verification for the existing provider configuration. ## Risks The offline predicate relies on Cloud stopping every old application deployment. It does not make online migration safe. Running executions always block. Requeued executions with retained ownership also block. No queued request, task binding, or recovery state is removed or rewritten. No database schema changes. ## Model Used OpenAI Codex, GPT-6 based, with reasoning and tool-assisted code execution. The exact deployed model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Dotta
|
049c1eacf5 |
Preserve historical native logs during ephemeral enrollment (#18864)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers need retained run history in shared storage.
> - Some old native runs have surviving local logs but no completion
digest or verifiable native source coverage.
> - Enrollment must preserve those bytes without calling them a complete
native transcript.
> - This pull request adds a read-only archive path to offline
enrollment.
> - Users can still read old output after the original server filesystem
is removed.
## Linked Issues or Issue Description
**What happened?**
Offline enrollment blocks on interrupted runs and unfinalized historical
native logs. The current importer requires canonical native-event
coverage even when the original local file survives.
**Expected behavior**
Preserve the exact surviving file in shared storage. Keep the original
run status, completion metadata, and source events. Do not allow writers
or native repair to modify archived history.
**Steps to reproduce**
Use an S3-configured instance with a terminal native run whose local log
lacks a final size or digest. Stop all app deployments and run ephemeral
maintenance. Before this change, interrupted status or unproven native
coverage prevents enrollment.
Related preservation work: #18826, #18835, #18841. This is a bounded
migration fix, not a new live logging protocol.
## What Changed
- The run-evidence migration owner can archive unfinalized native local
files during offline enrollment.
- Require the offline migration environment, zero active or queued runs
across the instance, and expired controller/finalizer leases. Check
global quiescence at each page boundary. Recheck each run's ownership
and metadata before publication.
- Verify stable local file identity and metadata, any recorded
size/digest, uploaded prefixes, and remote SHA-256.
- Store raw history in a separate `<run>.archive` journal with source
cursor zero and backend `s3_archive`. Leave completion fields and native
events unchanged.
- The existing log store reads archives but rejects begin, append,
enqueue, flush, and finalize. Native source repair does not run for
archives.
- Keep the default background importer's coverage checks. Continue to
reject incomplete mirrors and corrupt files.
- Add real PostgreSQL/S3 and packaged maintenance coverage. Correct a
pre-existing large-chunk fixture: the current verifier can prove exact
compactor output, so the negative case now uses a different canonical
source.
## Verification
- Passed on final head
|
||
|
DottaandPaperclip
|
2a16dcf629 |
fix(workspaces): preserve shared directory trees during enrollment (#18835)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A volume-free web server needs existing task files in durable object
storage.
> - Historical workspace records can share a physical folder or contain
other task folders.
> - A task-level migration hold cannot fence every alias, and capturing
a parent recursively duplicates nested task data.
> - Offline enrollment imports children first and verifies their durable
coverage before omitting them from a parent archive.
> - Snapshot retention preserves those child versions for as long as a
covering parent depends on them.
> - Existing files, task bindings, and archived states remain intact
through the storage migration.
## Linked Issues or Issue Description
Builds on merged #18826.
**What happened?**
The per-workspace importer could capture a shared project folder while
another workspace record still owned the same path. It also recursively
included runtime-created nested task workspaces in each parent import.
Packaged enrollment rejected existing workspace data instead of invoking
its migration owner.
**Expected behavior**
Require offline maintenance for overlapping paths. Import nested tasks
before their parents, verify every excluded directory against its saved
snapshot, preserve dependencies through retention, and resume
interrupted imports without replacing existing remote authority.
**Steps to reproduce**
Create a project workspace and another task workspace below its
`.paperclip/worktrees/` directory. Try importing either while ordinary
writers are running, then run offline enrollment. Interrupt a parent
import after upload and resume it.
## What Changed
- Add physical-overlap admission checks and an offline entry-point
guard, including global running and queued executions.
- Import pending records in leaf-first pages through the existing
migration owner.
- Verify nested snapshots by downloading, restoring, and comparing the
original source before and after parent upload; check transitive
dependencies.
- Publish parent dependency receipts under child snapshot row locks.
Retention pins those children and revisits them when the parent is
deleted.
- Index imported-child dependency lookups with a partial GIN index; a
PostgreSQL query-plan regression covers 2,500 unrelated snapshots.
- Propagate cancellation through capture and object storage operations
and preserve existing retry journals.
- Wire workspace migration into packaged maintenance and document its
preservation contract.
## Verification
- The original overlap and duplicate-parent regressions failed before
implementation.
- 73 migration and retention tests pass against real PostgreSQL and
files, including interrupted uploads, resumed batches, equal-path
aliases, changed or missing child data, nested descendants, and active
or queued executions.
- Six S3-compatible storage integration tests pass, including migration
through the maintenance owner and restoration of both parent and child
after deleting the original filesystem.
- The standalone process case was initially skipped because the worktree
had no bundle. After rebuilding the server and runtime bundle, that test
passed against PostgreSQL and S3: authenticated ready/export receipts,
preserved decision secret, blocked application routes, and clean SIGTERM
shutdown.
- Server typecheck, server build, runtime bundle build, and
module-boundary checks passed. Full repository CI passed on
|
||
|
DottaandPaperclip
|
d941ed3879 |
feat(server): verify existing-volume ephemeral enrollment (#18529)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud web servers can now keep durable state in PostgreSQL and S3.
> - Existing servers still have files and a decision signing key on
their volume.
> - The volume must stay attached until every data owner verifies its
migration.
> - This pull request adds a dedicated maintenance process for that
transition.
> - Cloud can preserve the original key and remove the volume only after
a matching receipt.
## Linked Issues or Issue Description
Refs #18518 and #18526. Existing-volume enrollment is a required step in
the zero-downtime web deployment work. This is the Core maintenance
side; the Cloud transition is a separate private change.
## What Changed
- Package a standalone maintenance entry that starts no normal app
routes or execution workers.
- Coordinate the existing agent-file, run-evidence and skill migration
owners. Follow every page and verify shared bytes before final
eligibility.
- Require stopped writers. Refuse active execution, missing local bytes,
local workspaces and external instructions.
- Read the existing decision key without generating a replacement.
Preserve normal key creation for ordinary server startup.
- Require an attempt token for status and export. Export the original
key only after successful migration. Keep status and errors free of
secret material.
- Document the Cloud ownership boundary, deadline, retry and retention
requirements.
## Verification
- Full workspace typecheck and build passed before the review fixes. The
final server TypeScript compilation passed after correcting Drizzle's
optional predicate type.
- Production runtime bundling passed: 2,702 inputs, 122 chunks,
including the maintenance executable.
- The focused suite passed 21 cases using PostgreSQL and real
S3-compatible storage. It covers multi-page migration of 26 agents,
exact reads after deleting the predecessor home, corrupt-object
rejection, active/missing-data refusal, and the bundled plain-Node
process with authenticated export and clean SIGTERM exit.
- Regression cases refuse unsaved terminal working copies, require
explicit stopped/cleaned receipts for unavailable copies, clean 101
expired traces through the owning service, and enforce the deadline
after migration succeeds.
- Initial integration attempts exposed fixture cleanup ordering
problems; corrected runs pass. CI also caught a Drizzle predicate type
error, now fixed and compiled locally.
- All repository CI gates passed on
|
||
|
DottaandPaperclip
|
006d5691a6 |
feat(server): support volume-free web server handoff (#18518)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |