mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
DottaandPaperclip
965ca91dd0
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Workspace migration preserves local files as authenticated portable
snapshots.
> - Workspaces that share Git storage require one complete source fence.
> - Independent imports contend on that fence and cannot safely run as a
batch.
> - This change adds an opt-in cohort owner that captures members
concurrently and publishes them atomically.
> - Existing source, recovery, snapshot and publication checks remain
authoritative.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The administrative remote-workspace migration owner for workspaces in
one shared physical source component. Related changes: #19688, #19854
and #19855.
**Current behavior**
Each independent import acquires the complete source fence. Parallel
independent calls therefore contend instead of overlapping archive work.
A caller cannot use a surrounding transaction and assume that a
successful savepoint confirms the outer commit.
**Proposed behavior**
An explicit `migrateRemoteWorkspaceCohort` call selects one to eight
leaves from one proven component. It owns the full
source/lifecycle/copyback fence, uses the original pool for bounded
archive work, validates every member, publishes serially, and returns
success only after the outer commit.
**Reason and benefit**
This supplies a correctness-gated path for a controlled two-member
hosted qualification. It makes no wall-clock speedup claim. No running
app call site or scheduler selects cohorts automatically.
**Breaking changes**
The existing single-import API retains its guards and rejects a journal
owned by a cohort. New cohort journals require the same complete member
set on resume.
## What Changed
- Extract the existing reservation, capture, validation and publication
stages for reuse by both owners.
- Add an opt-in cohort API with default concurrency two and a maximum of
eight. Every member must share the same complete physical/Git/alias
component; selected roots must be leaves relative to each other.
- Persist exact cohort and operation membership in the existing journal.
Compare all selected fingerprints with the aggregate proof before the
first reservation under the complete fence, and recheck each member when
reserving its journal. Refuse foreign journals, incomplete or mixed
replay sets, lost fences and changed sources. Keep all authorities and
activity events unpublished if the outer commit fails.
- Share the merged nested-child verifier inside one complete fence.
Queue its prepare calls, retain its two-child bound and authenticated
immutable proof, and recheck local bytes and database bindings on every
pass.
- Document the API, exact-membership resume protocol, resource bounds
and required hosted qualification.
## Verification
- Integration head `fce1896010`: 32/32 PostgreSQL/filesystem cases
across the cohort owner, the complete merged nested-verifier suite, and
five existing parent-import cases.
- Final source-proof repair: 19/19 cohort-owner cases pass, including
real first- and last-member metadata changes before reservation,
changed-source partial-resume refusal, mixed local/imported refusal with
storage forbidden, cancelled/failed/finished admission, a late finished
timestamp, complete fences, queued child-proof reuse, cold restore,
cancellation, commit failure and exact replay. All local database checks
use the canonical owned wrapper, one Vitest worker and Node 24.22.2 on
macOS arm64/PostgreSQL 18.1.
- Final rebased head is based on
`fbe689938c3bf8c5849d2cd56b9da90029db9389`, including current
source-owner, database-schema and Runner contract changes. The 19-case
owner suite, full server layered source/tests/test-tree typechecks,
module boundaries and diff checks are rerun on that base; required CI
supplies the native Runner qualification.
- Selection-only `node scripts/local-verify.mjs --base
68db7ed9e3 --explain` exceeded its
60-second observation bound before producing a plan. It supplies no test
pass; required CI must supply broad coverage.
- Genuine replay RED: a committed two-member cohort accepted a
one-member replay. The repaired owner requires its full
succeeded-journal/snapshot membership.
- Genuine terminal-journal RED: read-only preflight accepted a
cancelled, finished saved member, and apply created two pending
snapshots and the missing peer journal before refusing later. The repair
rejects that state before preflight eligibility or snapshot work and
rechecks unfinished state before publication.
- Genuine source-proof race REDs: early workspace metadata drift could
strand a partial journal. A first repair checked each member only when
reserving it; an additional last-member regression showed that an
earlier journal could still be written before noticing the changed
member. The final owner compares the complete preimage in the existing
fenced preflight before the first reservation and retains
per-reservation checks. Both first- and last-member cases now leave zero
journals/snapshots and permit a fresh attempt. Changed sources after a
genuine partial reservation still refuse.
- Coverage includes parallel uploads, cold restore with originals
removed, complete source/copyback fences, exact partial-reservation
resume, actual read-only preflight with storage forbidden,
foreign/changed journals, peer failure and cancellation, fence
connection loss, and a real deferred outer-commit failure.
- Prior head `70bc2740d0` passed all CI after one explicitly bounded
same-head retry of server shard 4. Its first attempt observed an
unrelated heartbeat/plugin test still running at a five-second wait;
automatic test retry then hit the fixed-name environment uniqueness
constraint. The test and 1,100 relevant owner files matched the nearest
successful base gate. The one job retry passed the target test in
3,491ms and all 3,045 shard tests. Original failures remain recorded;
their initial cause is unknown. This does not supply CI acceptance for
the later repair head.
- Earlier prototype evidence remains historical: 286 PostgreSQL controls
and twelve cold restores passed on its prior base. Six macOS timing runs
were inconclusive under contention, including slower cohort runs. They
are not speedup or hosted acceptance evidence.
- Hosted acceptance remains a two-member same-component leaf canary with
no imported children, followed by same-host timing, peak-memory and
scratch-headroom observations before considering a larger batch.
Run the focused owner file from `server/` with the canonical temporary
PostgreSQL owner:
```sh
node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/remote-workspace-cohort-migration.test.ts --fsModuleCache --maxWorkers 1 --hookTimeout 90000
```
Final head `112ffeb20c272c6ef5e1e0956be889c85511f0d0` passed [required
CI](https://github.com/paperclipai/paperclip/actions/runs/42402942287)
with 56 successful checks and four skips; the aggregate is green.
[Greptile reviewed this exact head at
5/5](https://github.com/paperclipai/paperclip/pull/19873#issuecomment-6710780943),
and a fresh paginated review-thread read has zero unresolved findings.
## Risks
- Concurrent archive work increases CPU, memory, scratch space, database
and storage load. Admission must count logical member workers and up to
two nested-child workers, not Node processes. Eight-member use remains
unqualified.
- The complete cohort fence lasts until all started work joins. Slow
callbacks can delay cancellation and associated source writers.
- The exact journal exception is internal and binds company, cohort,
workspace and operation IDs. Wrong or changed membership fails closed;
operators must inspect unknown outcomes before resuming the same full
set.
- Existing remote recovery holds, provider obligations, source files and
history remain preserved. A successful import does not settle those
independent obligations.
## Model Used
OpenAI GPT-6 (Codex), with reasoning, code execution and repository
tools. This runtime does not expose a more specific backend model ID or
context-window size.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>