mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 23:43:16 +01:00
fix(server): complete volume-free controller admission and recovery (#18543)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its volume-free web server must admit remote warm execution without
depending on a local disk.
> - Fresh run rows retain the initial legacy label during workspace
preparation.
> - The native runtime selector commits the execution mode later.
> - The earlier guard rejected valid remote runs before that selection.
> - This PR checks storage during preparation and the resolved lifecycle
during runtime selection.
> - Agent-file recovery also has to reconstruct disposable cache paths
from the retained remote directory.
> - The deployment test removes the complete predecessor home, including
immutable caches, to enforce that contract.
## Linked Issues or Issue Description
Refs #18518, #18529, #18533.
**What happened?**
A real Codex/Daytona deployment test failed during setup because the
volume-free workspace guard checked the initial run label. After fixing
admission, controller retirement first failed on an immutable skill
cache. Removing that cache exposed a production recovery failure: B
rejected the saved agent-directory path from A, even though the remote
execution identity was unchanged. The next live attempt recovered that
directory and transferred the same runner, but adoption then tried to
recreate provider launch assets from the deleted controller home.
**Expected behavior**
Allow remote workspace preparation before native selection. Reject
actual legacy and per-turn execution at the runtime boundary. Delete the
entire stopped predecessor home, including immutable caches, without
following links to shared dependencies. B must reconstruct its local
agent-file projection from the exact registered remote lease while
preserving the save baseline and unsaved remote edits. Reattaching an
already-running provider must not prepare a new launch or require the
old launch files. The full backend must recover the admitted
instructions from the matching durable run checkpoint.
**Steps to reproduce**
Run the explicit
`rolling-web-deployment.runner-codex.daytona.recover-controller` Product
E2E cell with a complete server/UI build, disposable PostgreSQL/S3, and
an owned Daytona fixture. The new boundary regressions also reproduce
the premature guard and missing runtime rejection without a model call.
**Paperclip version or commit**
The failures were observed on
`247552fe0719dcc92f7b42c50b658be6f624986d`. This branch is based on
`e122478368`.
## What Changed
- Validate Runner and remote workspace authority before runtime
selection without trusting the initial run label.
- Reject resolved legacy execution and local-source fallback, preserving
the native warm-policy check before admission.
- Rebind remote agent-file caches on ephemeral controllers and retrieve
their validated contents without uploading over the retained execution.
Preserve pending saves when the remote collector is unavailable.
- Restore over-quota remote directories without applying save quotas to
run admission; ordinary saves still enforce quotas.
- Skip provider launch asset materialization when adopting an existing
Runner. A real-process regression deletes the old skill/instruction
sources before reattachment.
- Recover admitted system instructions through Runner-owned checkpoint
validation for the same run, session and context digest. Fresh launches
still compose current instructions.
- Make the attention-retention test advance past its database-assigned
deadline, fixing a clean-master clock mismatch.
- Serialize remote collection and cleanup across controllers with an
operation-scoped Postgres advisory lock; fence stale cache-path updates.
- Keep synthetic deployment probe tasks in a separate disposable company
so they cannot block the business completion oracle.
- Share the launcher's directory cleanup with rolling retirement. Reject
a symlink root and skip linked descendants.
- Test admission with real PostgreSQL transactions and cleanup with real
read-only directories and external symlinks.
- Update developer documentation and the canonical qualification record.
Broader deployment acceptance remains in progress.
## Verification
- `pnpm -r typecheck` and `pnpm build` passed.
- All 156 workspace preparation, runtime selection and ephemeral
contract tests passed. Two new regressions failed before the source fix.
- Seven focused Product E2E cleanup/replacement tests and `pnpm
test:e2e:runner:typecheck` passed.
- `pnpm test:run` stopped at an unrelated 15-second chat integration
timeout after 338 passes. That exact test passed in isolation without
increasing its timeout. The later full-suite shards did not run.
- `pnpm test:e2e:runner:unit` reported 5,898 passes and 13 failures.
Clean master reproduced 11 failures in unchanged onboarding
expectations, operating receipts, Pi catalog assertions, workflow
bootstrap counts and embedded PostgreSQL startup. The other two workflow
assertions were fixed by the intervening master commit. These failures
are retained, not counted as passes.
- All 122 workspace-preparation tests pass after the reviewed
host-fallback fix. The first broader attempt exposed a test environment
leak; restoring the ephemeral flag between cases resolved it.
- Agent-directory and instruction regressions passed 179 tests. Five
focused cache-loss cases pass: different home, same path on an empty
filesystem, foreign lease, concurrent controller, and missing collector.
- The live fixture at `a18f6b85a3` gracefully stopped A and deleted its
entire home, then failed the 420-second remote reattachment deadline on
the saved local agent-directory path. That failure is retained; exact
owned sandbox cleanup was independently confirmed. The next live attempt
at `e14ca5f3e9` restored the cache and transferred the same runner, but
failed while rematerializing launch assets from A. Both failures are
retained and their owned sandboxes confirmed absent. No active-overlap,
ten-deploy, or production-canary acceptance is claimed.
- All 71 related Runner adoption/recovery cases pass, including the new
removed-context regression (failed before the source fix).
- The revised full agent-directory suite passed 110 cases and failed
three cleanup timing cases. All 20 cleanup/cache-loss cases then passed
in isolation with unchanged timeouts, including over-quota recovery.
- Server and Runner TypeScript checks pass after the latest corrections;
the live retry remains pending.
- The live test at `67714af9ff` adopted the same remote execution after
A was deleted, then failed while the full backend read the retired
instruction bundle. All owned sandboxes were independently confirmed
absent. Its regression now passes with actual backend recovery and no
new thread or turn. All 59 backend factory tests and focused checkpoint
identity tests pass. Runner and server TypeScript checks pass; the exact
committed server build passes. A fresh paid-model retry followed.
- The attention-retention CI timeout reproduced on clean master. The
deadline correction passes all 53 tests in that file with unchanged
timeouts.
- The real `cd690de4d3` run transferred the same runner/session after
deleting A, completed, and completed the next revision. During handoff,
372 reads and 38 saves had zero failures; all 38 saves were verified.
The full attempt was interrupted because the probe's open task
contaminated the business company oracle. This remains incomplete
evidence. The scope regression fails before the fixture fix, then
passes; all nine deployment support tests and E2E typecheck pass. All
three owned sandboxes were confirmed absent and temporary
credentials/storage were cleaned up. A clean corrected attempt is
running.
## Risks
- The runtime guard must still reject legacy and per-turn execution
before dispatch. Tests cover these refusals and a valid resolved warm
policy.
- Directory permission changes apply only during cleanup of stopped,
test-owned homes. Traversal does not follow symlinks. This changes no
production cache permissions.
- Cache reconstruction must not change remote execution authority or
rebase the original save baseline. Tests verify foreign-lease rejection,
preserved receipts, and absence of local-to-remote upload.
- Postgres locking is scoped to active remote file work and adds no idle
polling; local/durable-server locking remains unchanged.
- No schema or public API changes. External conversation exemptions
remain unchanged.
## Model Used
OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution
and live integration-test tools. The exact runtime model variant and
context-window size are not exposed in this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Latest real qualification at f5915fdc33: both empty-home replacements
completed with 523 reads, 53 saves and 53 verified saves, zero probe
failures. The active remote execution survived and completed once; saved
project bytes and the later useful answer passed. The full workflow
remains FAILED: the later continuation reset its native/provider session
and the public run row lacked the native terminal contract. These newly
exposed compatibility/projection gaps are tracked in the next
implementation slice; this PR is not a claim of full Railway
qualification. All owned Daytona sandboxes were independently confirmed
absent after cleanup.
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
e122478368
commit
7751fc3107
29 files changed
+682
-60
No files matched your search
@@ -4174,6 +4174,23 @@ capabilities, stable keys, operating limits, automatic history migration and qua
|
||||
Do not remove a web volume until its complete retained evidence inventory and
|
||||
the independent disposable-server requirements are satisfied.
|
||||
|
||||
## Volume-free execution admission
|
||||
|
||||
With `PAPERCLIP_EPHEMERAL_SERVER=true`, workspace preparation requires Paperclip
|
||||
Runner and remote workspace storage. A fresh run still has its initial `legacy`
|
||||
label at this phase; that label is not the selected execution mode. The runtime
|
||||
selector checks the resolved native warm lifecycle before admission and rejects
|
||||
legacy or per-turn execution. External conversation providers keep their existing
|
||||
exemption because they own no task filesystem. Source resolution checks this
|
||||
contract again before a configured host folder can fall back to local storage.
|
||||
|
||||
A replacement controller reconstructs an active remote agent-directory cache
|
||||
from the registered sandbox, including unsaved files. The remote directory,
|
||||
lease and save baseline remain unchanged; current canonical files are never
|
||||
uploaded over the retained execution. Remote collection and cleanup serialize
|
||||
through Postgres across controller homes. A missing remote collector retains a
|
||||
pending save instead of treating an empty local cache as the agent's files.
|
||||
|
||||
## Cloud startup configuration
|
||||
|
||||
Cloud stores nonsecret startup choices in a private PostgreSQL singleton.
|
||||
|
||||
@@ -1,7 +1,68 @@
|
||||
# Zero-downtime web deployments
|
||||
|
||||
Date: 2026-10-31. Status: runtime and Cloud overlap policy merged; qualification
|
||||
and existing-volume enrollment remain in progress. No hosted acceptance claim.
|
||||
Date: 2026-10-31. Status (2026-11-01 14:05 UTC): existing-volume staging enrollment
|
||||
is complete and its read/save/file-integrity checks pass. Active-agent recovery,
|
||||
natural database idling and the ten-deploy staging campaign remain unqualified.
|
||||
No production canary yet.
|
||||
|
||||
Latest critical path: Core #18543 includes runtime admission, immutable-cache
|
||||
retirement, remote agent-file projection, and adoption without rematerializing
|
||||
launch assets. The live test at `67714af9ff` transferred the same runner/session to
|
||||
controller generation two after deleting A's entire home, then failed in the full
|
||||
backend constructor: composing system instructions still read A's retired bundle.
|
||||
All three owned Daytona sandboxes were independently confirmed absent; the child
|
||||
credential was revoked and the test object bucket deleted. This remains failed
|
||||
evidence, not a successful deployment qualification.
|
||||
|
||||
The backend now accepts admitted instructions from a validated same-run Runner
|
||||
checkpoint. Runner owns checkpoint/template inspection; Core binds it to the
|
||||
current run, normalized session and context digest. Staged attachments select the
|
||||
validated target template. Fresh runs and ordinary later continuations still
|
||||
compose current instructions. The deleted-bundle reproduction failed before the
|
||||
fix, then passed with real backend session recovery and no new thread/turn start.
|
||||
All 59 backend factory tests and the focused checkpoint validation tests pass.
|
||||
At `cd690de4d3`, the real Codex/Daytona run survives removal of A's entire home,
|
||||
reattaches the same runner/session under generation two, and succeeds. Its next
|
||||
revision also succeeds. The handoff records 372 reads, 38 saves and 38 verified
|
||||
saves with zero failures. Full qualification still fails: the supervisor created
|
||||
an unrelated open probe task in the business company, so the unchanged all-tasks-
|
||||
done oracle could never advance to the second replacement. The attempt was
|
||||
interrupted and retained as incomplete. A regression reproduces this harness
|
||||
scope defect; probes now use their own disposable company. The oracle is unchanged.
|
||||
Exact-head review of the recovery fix is 5/5 with no unresolved threads; new
|
||||
fixture checks and another bounded live attempt are next.
|
||||
|
||||
Review also identified save quotas incorrectly applied during cache recovery.
|
||||
Recovery uses the existing inbound remote projection and preserves path/link
|
||||
validation without enforcing save quotas on admission. Ordinary saves still reject
|
||||
oversized changes. Related transport recovery tests passed 71 cases; directory
|
||||
cleanup/cache-loss tests passed 20 cases after three timing failures in a broad
|
||||
run. A separate CI attention-retention timeout reproduced on clean master: its
|
||||
fixed clock fell behind database-assigned updatedAt. Advancing across the saved
|
||||
deadline fixes it; all 53 tests in that file pass with original timeouts.
|
||||
|
||||
Cloud #1102 merged as `461c4aea28` and is serving on the staging controller.
|
||||
The original enrollment completed using its original qualified image, a retained
|
||||
detached volume, and the migrated decision key. The post-enrollment probe passed
|
||||
20 reads and three saves with no HTTP/WebSocket failures and verified all 34 files.
|
||||
The original browser tab reconnected and persisted a task title. Server logs later
|
||||
showed old Agents chunks returned 404 and navigation recovered via a page reload:
|
||||
this is migration continuity evidence, not uninterrupted old-tab asset loading.
|
||||
|
||||
Two eight-minute natural database-idle observations failed. A separate bounded SQL
|
||||
attribution window saw no other app-database sessions; it is diagnostic stimulus,
|
||||
not idle proof, and does not establish the source of provider activity. Automatic
|
||||
idle sleep was correctly refused because this development fixture is not eligible
|
||||
for the free/demo policy. Manual sleep with force disabled passed: the registry
|
||||
reports sleeping, Railway has no active web deployment or attached volume, and
|
||||
Neon is idle. The normal organization entry started wake, and Railway became healthy on the
|
||||
same volume-free service. Post-wake proof passed 21 reads, three saves, a connected
|
||||
WebSocket and all 34 file hashes. The browser displayed the retained task and saved
|
||||
a new title, independently read back through the API. Automatic return from the
|
||||
refreshing entry page was not verified; the browser navigated to the tenant after
|
||||
readiness. Wake took several minutes while existing shared UI assets were checked.
|
||||
Private receipts remain under the task's live-test, staging enrollment and hosted
|
||||
probe directories; no credential or live target identity belongs here.
|
||||
|
||||
## Outcome and endpoint
|
||||
|
||||
@@ -63,10 +124,11 @@ Measure one-time setup separately from repeated attempts. Keep failed evidence.
|
||||
|
||||
| Item | State | Next action |
|
||||
| --- | --- | --- |
|
||||
| Prior prerequisites | #18403 now merged; current source fetched | Characterize remaining local authority paths |
|
||||
| Core implementation | `/private/tmp/paperclip-zero-downtime-20261031`, durable files, readiness/drain and migration/cleanup implemented in draft PR #18518 | Cross-process commands, browser/assets, active Runner qualification |
|
||||
| Cloud implementation | `/private/tmp/paperclip-cloud-zero-downtime-20261031`, persisted ephemeral shape and image/variable gates implemented in draft PR #1096 | Complete lifecycle guards, test and deploy qualification |
|
||||
| Staging / production | Not selected or deployed | Record exact target, owner, artifact and campaign before action |
|
||||
| Prior prerequisites | #18403 and all five storage workstreams merged | Retain the ephemeral filesystem contract |
|
||||
| Core implementation | #18518, #18526, #18529, #18530, #18531 and #18533 merged; local packaged migration, browser overlap, cold restoration and durable command proofs recorded below | Complete live remote execution, accounting and later continuation qualification |
|
||||
| Cloud implementation | #1096, #1098, #1099, #1100, #1101 and #1102 merged; #1102 serving on staging | Verify repeated Railway overlap behavior after Core recovery qualification |
|
||||
| Staging | Original enrollment complete; migrated data and signing key preserved; volume detached and retained | Diagnose idle behavior, verify sleep/wake and repair, then qualify ten app-only replacements |
|
||||
| Production | No canary mutation performed | Proceed only after the staging and active-agent acceptance gates pass |
|
||||
|
||||
Keep PRs, tested revisions, artifact identities, evidence paths, deployment targets
|
||||
and unresolved gates here. A green CI/build/deploy status alone is not acceptance.
|
||||
@@ -528,3 +590,196 @@ and after logs are `/private/tmp/paperclip-zdt-buffer-test-{before,after}-pg.log
|
||||
A first local attempt failed before tests because embedded PostgreSQL could not
|
||||
start; the existing external test database enabled the comparison. The failed
|
||||
post-merge shard was retried once while the assertion fix proceeds through review.
|
||||
|
||||
### 2026-11-01 11:07 UTC package compatibility prerequisite
|
||||
|
||||
The post-merge source retry passed. The final Cloud composition for Core
|
||||
`1d348bbef0` is `cloud-41298718800-1`, digest
|
||||
`sha256:800cb8578368c63e33bdd048786ec66b3e7a9902f0f221d7fe41ea3c35134020`.
|
||||
It passed the real-volume migration and empty-home rehearsal with 121 reads,
|
||||
25 saves and zero failures; receipt:
|
||||
`/private/tmp/paperclip-zdt-container-drdrQS/evidence.json`.
|
||||
|
||||
The forward/rollback manifest comparison exposed a real packaging blocker.
|
||||
The Agent Computer bundle differed only in its embedded Runner source revision
|
||||
and source map, despite unchanged Runner inputs, SDKs and dependency lock.
|
||||
The exact-package gate correctly requires maintenance for those two images.
|
||||
Do not use them as proof of an app-only overlap pair.
|
||||
|
||||
The Cloud source-selection owner now selects the last first-parent Runner input
|
||||
change when the older reviewed ROM pin differs, then proves complete input
|
||||
equality. The plugin and ROM builder share that selection and keep temporary
|
||||
history alive through archive verification. Actual clean checkouts of both
|
||||
released Core commits select `66e8fab9447aa21350f87ddec79fcd027f3e23ad`.
|
||||
A real shallow-checkout probe selected the same source in 25.6 seconds without
|
||||
modifying the caller. The focused suite passed 32 cases; full Cloud tests passed
|
||||
4,830 cases with 129 gated skips and no failures. Rebuilt package equality and
|
||||
hosted overlap remain unverified. Private receipts are
|
||||
`/private/tmp/paperclip-zdt-canonical-source-{proof,shallow-proof}.json`.
|
||||
|
||||
Core's assertion fix is in #18533, reviewed at 5/5. One CI shard received a
|
||||
runner shutdown signal, so that cancelled shard and its downstream aggregate
|
||||
gate were retried. The original cancellation log remains at
|
||||
`/private/tmp/paperclip-zdt-buffer-pr-cancelled-job.log`.
|
||||
No hosted enrollment has occurred. After the controller became quiet, the
|
||||
qualified maintenance image was added to the staging allowlist without an
|
||||
automatic deployment. An explicit controller configuration restart was then
|
||||
submitted with a mutation-confirmed deployment identity. The prior busy-campaign
|
||||
and pending-default checks produced no mutations. Private receipt:
|
||||
`/private/tmp/paperclip-zdt-controller-allowlist-RzRa7k/receipt.json`.
|
||||
The paid-agent credential blocker and all remaining acceptance criteria remain.
|
||||
|
||||
|
||||
### 2026-11-01 11:15 UTC hosted enrollment response correction
|
||||
|
||||
Core #18533 merged as `247552fe0719dcc92f7b42c50b658be6f624986d`; Cloud
|
||||
#1099 merged as `1c5c5133c2d3740fdb716b26c31f038add273b55`. Both latest
|
||||
heads passed all checks and review at 5/5 with no unresolved findings. Real
|
||||
ROM-input generation also verified the complete source archive and Linux
|
||||
dependency lock in 53.1 seconds from a clean checkout.
|
||||
|
||||
The staging controller configuration restart succeeded with unchanged source
|
||||
`316163b28`. The first real enrollment request at 11:08 UTC failed before
|
||||
provider mutation with `Enrollment requires the owned Railway service domain`.
|
||||
The registry correctly retains maintenance routing. The original web deployment
|
||||
and attached volume were not stopped or changed. Railway's scoped domain query
|
||||
returns the matching service/environment but a null `projectId`. Reproducing
|
||||
that response in the existing lifecycle test fails identically. The correction
|
||||
accepts the nullable project field from the project-scoped query while rejecting
|
||||
explicit project, service or environment mismatches before provider mutation.
|
||||
The original attempt identity is retained for retry after reviewed deployment.
|
||||
Private failure receipt: `/private/tmp/paperclip-zdt-enrollment-controller-logs.json`.
|
||||
No hosted overlap deployment or production canary has passed yet.
|
||||
|
||||
|
||||
### 2026-11-01 11:25 UTC exact release pair qualified
|
||||
|
||||
Cloud #1100 merged as `abcb017123013f2baa215fd665b4092495de00a1` after
|
||||
4,831 local tests, 129 explicit skips, typecheck, golden-path smoke and all
|
||||
latest-head CI checks passed, with 5/5 review and no unresolved findings. Its
|
||||
automatic staging controller deployment is starting; enrollment retry waits for
|
||||
that serving revision. The original held deployment and attached volume were
|
||||
independently confirmed unchanged after the first failure.
|
||||
|
||||
The rebuilt pair now has byte-identical Cloud bundles, SDKs and dependency lock:
|
||||
`cloud-41300941768-1` (Core `1d348bbef0`) and `cloud-41301234904-1` (Core
|
||||
`d941ed3879`). Both select the same canonical Runner source. The production
|
||||
`ephemeralPackageOverlap` function accepts this pair. There is no database diff
|
||||
between the two Core commits. Exact manifest receipt:
|
||||
`/private/tmp/paperclip-zdt-canonical-pair-proof.json`.
|
||||
|
||||
Each exact image passed its own packaged migration, complete predecessor-volume
|
||||
deletion and two empty-home replacements against real PostgreSQL and S3: 118
|
||||
reads/24 saves and 116 reads/24 saves respectively, no failures, all owned
|
||||
resources cleaned. Receipts: `/private/tmp/paperclip-zdt-container-woB7ci/evidence.json`
|
||||
and `/private/tmp/paperclip-zdt-container-tMUSnD/evidence.json`. Both images were
|
||||
added to the staging allowlist with deployment skipped, under the normal quiet
|
||||
guards, before the reviewed controller rollout. The repeated hosted experiment
|
||||
will invoke the existing fleet workflow driver with exact release references and
|
||||
a single pinned fixture; it retains campaign/release/serving verification while
|
||||
avoiding repeated GitHub scheduler waits. No hosted success is inferred from
|
||||
these local receipts. Active-agent credentials remain blocked as previously noted.
|
||||
|
||||
|
||||
### 2026-11-01 11:34 UTC maintenance entrypoint qualification
|
||||
|
||||
The corrected controller is serving `abcb017123013f2baa215fd665b4092495de00a1`.
|
||||
The first prepare reached the durable `preparing` phase and stopped the old
|
||||
deployment, but Railway's remove call timed out. A read-only provider check
|
||||
confirmed removal and retention of the attached volume, then the same attempt
|
||||
resumed successfully to `migrating`. Its healthy maintenance process reported
|
||||
`blocked` during configuration; no detach was attempted.
|
||||
|
||||
Railway's start-command override replaces ENTRYPOINT, whereas the original local
|
||||
Docker rehearsal replaced only CMD. The maintenance process therefore ran as root
|
||||
and refused the original UID-1000-owned signing key. An explicit local entrypoint
|
||||
override reproduced the same blocked status and captured root process identity
|
||||
and key ownership, without reading key contents into evidence. Failed receipt:
|
||||
`/private/tmp/paperclip-zdt-container-5nGSUs/evidence.json`.
|
||||
|
||||
Cloud #1101 preserves Core's init and entrypoint wrapper in the maintenance
|
||||
command. With actual entrypoint replacement, the corrected packaged rehearsal
|
||||
passed original-key preservation, volume deletion and later empty-home recovery:
|
||||
116 reads, 24 saves, no failures, cleanup confirmed in
|
||||
`/private/tmp/paperclip-zdt-container-POFr0C/evidence.json`. Twelve lifecycle tests,
|
||||
root typecheck, the full 4,831-pass suite and ordinary golden-path smoke passed.
|
||||
The PR is awaiting CI/review; hosted preparation will resume with the same
|
||||
attempt after deployment. The original live volume remains attached and retained.
|
||||
No hosted overlap or active-agent acceptance has passed.
|
||||
|
||||
### 2026-11-01 11:48 UTC reviewed entrypoint fix
|
||||
|
||||
Cloud #1101 merged as `76f2dedf076ed139af3a7136e988c6748589d50b` with
|
||||
all latest-head checks passing and no unresolved review findings. The exact image
|
||||
currently held in staging also passed the corrected Railway-entrypoint rehearsal:
|
||||
118 reads, 24 saves, zero failures, original signing authority retained, predecessor
|
||||
volume deleted and all owned test resources cleaned. Private receipt:
|
||||
`/private/tmp/paperclip-zdt-container-vyIrsD/evidence.json`.
|
||||
|
||||
Automatic staging controller workflow `41302710585` passed preflight and is
|
||||
waiting for the ordinary controller coordination gate. The concurrent app campaign
|
||||
has completed 16 of 17 targets with one still running; the pinned qualification
|
||||
fixture is excluded. No cancellation or coordination bypass is permitted. The
|
||||
serving controller remains `abcb017123013f2baa215fd665b4092495de00a1`.
|
||||
Enrollment must resume with its original attempt identity after the new controller
|
||||
is serving. The active-model credential remains unavailable; no paid-agent result
|
||||
or hosted overlap acceptance is inferred from the local container proofs.
|
||||
|
||||
|
||||
### 2026-11-01 12:34 UTC live enrollment and runtime admission findings
|
||||
|
||||
Cloud #1101 reached staging at 12:05 UTC. The owning enrollment attempt completed
|
||||
migration and authenticated eligibility: 34 agent objects, 12 skill versions,
|
||||
zero blockers. The original signing key was stored before the volume detached.
|
||||
The original volume remains retained. Final app boot failed because Railway left
|
||||
the maintenance start command unchanged when the provider sent null.
|
||||
|
||||
The same attempt exposed delayed detach reads, automatic maintenance deployment
|
||||
after detach, and successful deployment stops whose API responses timed out.
|
||||
Cloud #1102 handles those provider transitions with bounded ownership-checked
|
||||
reconciliation, an empty-string start-command reset, and app reconfiguration on
|
||||
same-attempt boot retry. Root typecheck, 4,835 tests (129 gated skips), ordinary
|
||||
routing/sleep smoke and 352 focused provider tests passed. Review/deployment is
|
||||
pending. Tenant routing remains in maintenance; no ten-roll acceptance is claimed.
|
||||
|
||||
The user supplied a working credential location; the OpenAI authentication gate
|
||||
is resolved without exposing the key. Two live attempts exposed premature runtime
|
||||
admission: fresh run rows still say legacy while workspace preparation runs,
|
||||
before the native runtime selector commits its decision. Workspace preparation
|
||||
must validate storage authority; runtime selection must validate the resolved
|
||||
native warm lifecycle. Two focused regressions failed before this source fix;
|
||||
all 156 workspace/runtime/ephemeral-contract tests passed after it. Full typecheck
|
||||
and build passed; the full test suite and rebuilt live cell are running.
|
||||
|
||||
A third live attempt mistakenly used the old compiled server and reproduced the
|
||||
old guard. That retained failure is not evidence against the source fix. The
|
||||
launcher now records the compiled server hash and rejects that stale guard.
|
||||
All three completed attempts revoked their owned Daytona credentials and deleted
|
||||
their disposable object buckets. No paid-agent acceptance has passed yet.
|
||||
|
||||
|
||||
### 2026-11-01 12:43 UTC recovery fix merged and live cleanup boundary
|
||||
|
||||
Cloud #1102 merged as `461c4aea28a1ffc31b1175db42158338dd9de1ae` after
|
||||
4,853 passing tests (129 gated skips), typecheck, routing/sleep smoke, latest-head
|
||||
required CI and 5/5 review with no unresolved threads. The optional security
|
||||
scanner was inconclusive because its sandbox could not start; a rerun request
|
||||
returned 404. Automatic staging deployment is workflow `41307729362`.
|
||||
|
||||
The rebuilt live cell completed two native Codex/Daytona runs. Its first web
|
||||
replacement then failed while deleting an immutable skill-cache directory in the
|
||||
stopped predecessor home. The attempt remains failed, with partial estimated LLM
|
||||
cost $0.1072965 and incomplete usage coverage. The shared harness cleanup now
|
||||
makes stopped owned directories removable without following symlinks; seven
|
||||
focused regressions and the E2E typecheck pass. No receipt is accepted without
|
||||
whole-home deletion. Its leaked remote fixture is being reconciled through exact
|
||||
company/environment labels and recorded lease IDs; the first delete was refused
|
||||
while the sandbox was stopping. Original attempt evidence is retained.
|
||||
|
||||
Required broad checks exposed separate failures: `pnpm test:run` stopped at a
|
||||
15-second chat integration timeout (338 passed); that exact test passed in
|
||||
isolation without changing its limit. Product E2E unit checks reported 5,898
|
||||
passes and 13 failures in untouched onboarding expectations, operating receipts,
|
||||
Pi catalog assertions, workflow security assertions and embedded PostgreSQL
|
||||
startup. These are not counted as passes; baseline attribution is pending.
|
||||
Workspace typecheck/build and all 156 affected Core tests remain passing.
|
||||
@@ -21,6 +21,8 @@ import {
|
||||
} from "./runtime-context.js";
|
||||
|
||||
export interface CodexNativeSessionBackendOptions {
|
||||
/** Admitted instructions from a validated same-run checkpoint, supplied by the recovery owner. */
|
||||
retainedSystemInstructions?: string;
|
||||
/** Effective provider environment, including the assigned workspace boundary. */
|
||||
environment?: NodeJS.ProcessEnv;
|
||||
observability?: import("../drivers/agent-observability.js").ProviderObservabilityLaunch;
|
||||
@@ -149,7 +151,7 @@ function createTransportBackedNativeSessionBackend(
|
||||
...nativeTaskConstraints(input),
|
||||
];
|
||||
|
||||
const baseInstructions = nativeSystemInstructions(input);
|
||||
const baseInstructions = options.retainedSystemInstructions ?? nativeSystemInstructions(input);
|
||||
const transportFactory = options.transportFactory;
|
||||
return new HarnessDriverBackend(
|
||||
new CodexAppServerDriver({
|
||||
|
||||
@@ -360,6 +360,36 @@ describe("native backend factory", () => {
|
||||
await session.close({ reason: "test complete" });
|
||||
});
|
||||
|
||||
it("recovers with admitted instructions after the old bundle is removed", async () => {
|
||||
const input = preparedExecution(execution());
|
||||
if (!("runtimeContext" in input)) throw new Error("fixture requires runtime context");
|
||||
const retainedSystemInstructions = composeNativeSystemInstructions(input.runtimeContext, "Keep assigned instructions.");
|
||||
const original = await createNativeSessionBackend(input, {
|
||||
codexTransportFactory: () => new FakeCodexTransport(),
|
||||
environment: { ...process.env, PAPERCLIP_WORKSPACE_CWD: WORKSPACE },
|
||||
}).openSession({
|
||||
identity: { runId: "run", sessionId: "session", companyId: "company", agentId: "agent", issueId: "issue" },
|
||||
workingDirectory: WORKSPACE,
|
||||
});
|
||||
const snapshot = await original.snapshot();
|
||||
await original.close({ reason: "controller replacement" });
|
||||
rmSync(input.runtimeContext.instructions.bundle.rootPath, { recursive: true });
|
||||
const transport = new FakeCodexTransport();
|
||||
transport.readResponse = { thread: { id: transport.threadId, sessionId: transport.providerSessionId, cwd: WORKSPACE, turns: [] } };
|
||||
const factory = vi.fn(() => transport);
|
||||
expect(() => createNativeSessionBackend(input, { codexTransportFactory: factory })).toThrow();
|
||||
expect(factory).not.toHaveBeenCalled();
|
||||
const backend = createNativeSessionBackend(input, {
|
||||
retainedSystemInstructions, codexTransportFactory: factory,
|
||||
workingDirectoryAuthority: "remote_runner", environment: { PAPERCLIP_WORKSPACE_CWD: WORKSPACE },
|
||||
});
|
||||
const recovered = await backend.recoverSession!({ ...snapshot, activeTurnId: null }, { signal: new AbortController().signal });
|
||||
expect(recovered).toMatchObject({ recovered: true });
|
||||
expect(factory).toHaveBeenCalledWith(expect.objectContaining({ baseInstructions: retainedSystemInstructions }));
|
||||
expect(transport.calls.some(call => call.method === "thread/start" || call.method === "turn/start")).toBe(false);
|
||||
if (recovered.recovered) await recovered.session.close({ reason: "test complete" });
|
||||
});
|
||||
|
||||
describe("runner-prepared launch working directory", () => {
|
||||
// The transport reports the prepared directory; it exists only on the VM.
|
||||
class PreparedLaunchTransport extends FakeCodexTransport {
|
||||
|
||||
@@ -68,6 +68,7 @@ export function createNativeSessionBackend(
|
||||
}
|
||||
if (options.codexTransportFactory) {
|
||||
return createRunnerdNativeSessionBackend(input, {
|
||||
retainedSystemInstructions: options.retainedSystemInstructions,
|
||||
completionFeedback: options.completionFeedback,
|
||||
runnerInstanceId: options.runnerInstanceId,
|
||||
onSpawn: options.onSpawn,
|
||||
|
||||
@@ -33,6 +33,7 @@ import {
|
||||
decodeDurablePrpCheckpoint,
|
||||
inspectDurablePrpCheckpointHeader,
|
||||
inspectStagedRunAttachment,
|
||||
inspectRetainedRunInstructions,
|
||||
inspectStagedRunAttachmentHeader,
|
||||
type DurablePrpCheckpoint,
|
||||
durableRecoveryInternals,
|
||||
@@ -55,6 +56,33 @@ const identity: DurableRecoveryIdentity = {
|
||||
const expectedRunnerVersion = "0.3.0";
|
||||
const expectedRunnerDigest = `sha256:${"a".repeat(64)}`;
|
||||
|
||||
it("recovers only the matching run's admitted instructions from a valid checkpoint", async () => {
|
||||
const root = mkdtempSync(resolve(tmpdir(), "runner-retained-instructions-"));
|
||||
const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest });
|
||||
const digest = "b".repeat(64);
|
||||
const expected = { runId: identity.runId, normalizedSessionId: identity.normalizedSessionId, runtimeContextDigest: digest };
|
||||
const template = { provider: { instructions: "Exact admitted instructions.\n", runtimeContext: { aggregateDigest: digest } } };
|
||||
try {
|
||||
core.persistRunAttachTemplate(template);
|
||||
expect(inspectRetainedRunInstructions(core.store.state, expected)).toBe(template.provider.instructions);
|
||||
for (const field of ["runId", "normalizedSessionId", "runtimeContextDigest"] as const) {
|
||||
expect(inspectRetainedRunInstructions(core.store.state, { ...expected, [field]: "foreign" })).toBeNull();
|
||||
}
|
||||
expect(inspectRetainedRunInstructions({ ...core.store.state, commands: null }, expected)).toBeNull();
|
||||
expect(inspectRetainedRunInstructions({ ...core.store.state, runAttachTemplate: { provider: { ...template.provider, instructions: "" } } }, expected)).toBeNull();
|
||||
const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" };
|
||||
const targetTemplate = { provider: { instructions: "Target instructions", runtimeContext: { aggregateDigest: "c".repeat(64) } } };
|
||||
core.stageRunAttachment(target, targetTemplate);
|
||||
const targetExpected = { ...expected, runId: target.runId, runtimeContextDigest: "c".repeat(64) };
|
||||
expect(inspectRetainedRunInstructions(core.store.state, targetExpected)).toBe("Target instructions");
|
||||
expect(inspectRetainedRunInstructions(core.store.state, { ...targetExpected, runtimeContextDigest: digest })).toBeNull();
|
||||
expect(inspectRetainedRunInstructions(core.store.state, expected)).toBe(template.provider.instructions);
|
||||
const corrupted = structuredClone(core.store.state);
|
||||
corrupted.stagedRunAttachment!.templateDigest = `sha256:${"f".repeat(64)}`;
|
||||
expect(inspectRetainedRunInstructions(corrupted, targetExpected)).toBeNull();
|
||||
} finally { await core.stop(); rmSync(root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
it("persists an undispatched source-to-target attachment before bootstrap and preserves it after cache loss", async () => {
|
||||
const root = mkdtempSync(resolve(tmpdir(), "runner-staged-attachment-"));
|
||||
const target = { ...identity, runId: "00000000-0000-4000-8000-000000000002", turnId: "next-turn", itemId: "next-item" };
|
||||
|
||||
@@ -923,6 +923,26 @@ export function inspectStagedRunAttachment(state: unknown, expectedTarget?: Dura
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
/** Recover admitted instructions without opening the retired controller's assets.
|
||||
* Inspection is evidence only; the caller still authorizes the runner and lease. */
|
||||
export function inspectRetainedRunInstructions(state: unknown, expected: {
|
||||
runId: string; normalizedSessionId: string; runtimeContextDigest: string;
|
||||
}): string | null {
|
||||
try {
|
||||
if (!isRecord(state) || !exactIdentity(state.identity) || !isStoredCoreState(state, state.identity)) return null;
|
||||
const matches = (identity: DurableRecoveryIdentity) => identity.runId === expected.runId
|
||||
&& identity.normalizedSessionId === expected.normalizedSessionId;
|
||||
const staged = state.stagedRunAttachment;
|
||||
const template = staged && matches(staged.targetIdentity)
|
||||
? stagedAttachmentTemplate(state)
|
||||
: matches(state.identity) ? state.runAttachTemplate : null;
|
||||
if (!isRecord(template) || !isRecord(template.provider)) return null;
|
||||
const { instructions, runtimeContext } = template.provider;
|
||||
return isRecord(runtimeContext) && runtimeContext.aggregateDigest === expected.runtimeContextDigest
|
||||
&& typeof instructions === "string" && instructions.trim().length > 0 ? instructions : null;
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
function isStoredCoreState(
|
||||
value: unknown,
|
||||
identity: DurableRecoveryIdentity,
|
||||
|
||||
@@ -39,6 +39,7 @@ export {
|
||||
RUNNER_PRP_PROTOCOL_MAX,
|
||||
inspectWarmRunTransition,
|
||||
inspectStagedRunAttachment,
|
||||
inspectRetainedRunInstructions,
|
||||
inspectStagedRunAttachmentHeader,
|
||||
type StagedRunAttachment,
|
||||
type DurablePrpControlPlaneOptions,
|
||||
|
||||
@@ -20,7 +20,7 @@ import {
|
||||
} from "node:fs/promises";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { execFileSync, spawn } from "node:child_process";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { bindAcpxAgentFiles } from "../drivers/acpx/agent-files-binding.js";
|
||||
import { resolveRunnerProviderAssetsRoot } from "../drivers/acpx/provider-assets-root.js";
|
||||
import { createServer } from "node:http";
|
||||
@@ -7646,9 +7646,14 @@ it("rotates PRP authority in place for a warm cross-run attachment", async () =>
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it("recovers an active second warm business run with its opaque attachment identity", async () => {
|
||||
it("recovers an active second warm business run after deleting its controller context assets", async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), "runnerd-warm-second-active-"));
|
||||
const firstRoot = join(root, "first-controller"), nextRoot = join(root, "next-controller");
|
||||
const assetRoot = join(root, "retired-controller-assets");
|
||||
const skillRoot = join(assetRoot, "skill"), instructionRoot = join(assetRoot, "instructions");
|
||||
await mkdir(skillRoot, { recursive: true }); await mkdir(instructionRoot, { recursive: true });
|
||||
await writeFile(join(skillRoot, "SKILL.md"), "# Pinned skill\n");
|
||||
await writeFile(join(instructionRoot, "AGENTS.md"), "Pinned instructions\n");
|
||||
const server = createServer();
|
||||
let authority: DurablePrpControlPlane | null = null, generation = 0;
|
||||
let checkpoint: durableControlPlane.DurablePrpCheckpoint | undefined;
|
||||
@@ -7670,6 +7675,7 @@ it("recovers an active second warm business run with its opaque attachment ident
|
||||
runId: "run-warm-first-healthy", normalizedSessionId: "session-warm-second-active", turnId: "turn-first", itemId: "item-first" };
|
||||
const options = { runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex,
|
||||
codexArgs: fakeCodexArgs(root, "--emit-tool-call", "--durable-tool-ids"), stateDirectory: firstRoot,
|
||||
runtimeContext: assignedRuntimeContext(skillRoot, instructionRoot),
|
||||
prpIdentity: initialIdentity, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 }, controlPlaneRegistration: registration };
|
||||
const first = createCapabilityRunnerdCodexTransport(options);
|
||||
const within = async <T>(label: string, promise: Promise<T>) => {
|
||||
@@ -7707,6 +7713,8 @@ it("recovers an active second warm business run with its opaque attachment ident
|
||||
await activeCore.flushCheckpoint();
|
||||
await within("controller detach", first.detachControllerForRestart());
|
||||
expect(() => process.kill(runnerPid!, 0)).not.toThrow();
|
||||
await rm(assetRoot, { recursive: true, force: true });
|
||||
expect(existsSync(assetRoot)).toBe(false);
|
||||
checkpoint!.restoreSessionDirectory(nextRoot);
|
||||
const noSpawn = vi.fn(() => { throw new Error("Duplicate Runner launch"); });
|
||||
const openedThread = opened.thread as Record<string, unknown>;
|
||||
@@ -7758,6 +7766,7 @@ it("recovers an active second warm business run with its opaque attachment ident
|
||||
if (runnerPid) { try { process.kill(-runnerPid, "SIGKILL"); } catch {} }
|
||||
server.closeAllConnections();
|
||||
if (server.listening) await new Promise<void>(done => server.close(() => done()));
|
||||
for (const controller of [firstRoot, nextRoot]) await releaseMaterializedNativeRuntimeSkills(join(controller, "codex-home", "skills"));
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
@@ -6046,16 +6046,16 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
this.#assertStartupOpen();
|
||||
// Route and immutable endpoint admission precedes every launch-material
|
||||
// write. A refused transition leaves its retained home/context untouched.
|
||||
if (runtimeContext !== null && !this.options.prepareLaunch && this.#sourceRecoveryIdentity === null) {
|
||||
if (runtimeContext !== null && preparesProviderLaunch && this.#sourceRecoveryIdentity === null) {
|
||||
writeFileSync(
|
||||
localRuntimeContextPath,
|
||||
`${JSON.stringify(runtimeContext)}\n`,
|
||||
{ mode: 0o600 },
|
||||
);
|
||||
}
|
||||
if (provider === "aws_agentcore" && !this.options.prepareLaunch)
|
||||
if (provider === "aws_agentcore" && preparesProviderLaunch)
|
||||
mkdirSync(codexHome, { recursive: true, mode: 0o700 });
|
||||
if (!this.options.prepareLaunch && provider === "codex" && this.options.homeMode !== "preconfigured" && this.#sourceRecoveryIdentity === null) {
|
||||
if (preparesProviderLaunch && provider === "codex" && this.options.homeMode !== "preconfigured" && this.#sourceRecoveryIdentity === null) {
|
||||
await releaseMaterializedNativeRuntimeSkills(
|
||||
resolve(localCodexHome, "skills"),
|
||||
);
|
||||
|
||||
@@ -14,8 +14,8 @@ import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { agentFileStore, fileHash, inspectAgentFile, snapshotAgentFiles, MAX_AGENT_FILE_BYTES, MAX_AGENT_DIRECTORY_BYTES, MAX_AGENT_DIRECTORY_ENTRIES } from "../services/agent-file-store.js";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { agentFileStore, fileHash, inspectAgentFile, inspectAgentDirectory, snapshotAgentFiles, MAX_AGENT_FILE_BYTES, MAX_AGENT_DIRECTORY_BYTES, MAX_AGENT_DIRECTORY_ENTRIES } from "../services/agent-file-store.js";
|
||||
import { agents, companies, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, environmentLeases, environments, agentInstructionWorkingCopies, agentInstructionRevisions, agentInstructionHeads, createDb, fileStorageDb } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js";
|
||||
@@ -902,6 +902,87 @@ describe("persistent agent directories", () => {
|
||||
expect((await fs.stat(active.localRoot)).isDirectory()).toBe(true);
|
||||
});
|
||||
|
||||
it.each(["new-home", "same-path-empty", "foreign-lease", "other-controller", "missing-collector", "over-quota"] as const)("recovers remote agent files after controller cache loss: %s", async scenario => {
|
||||
const copy = await run({ warm: true });
|
||||
const environmentId = randomUUID(), leaseId = randomUUID(), remoteCwd = "/fixture/retained-task";
|
||||
const executionRoot = path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", agentId, copy.runId);
|
||||
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox" });
|
||||
await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, heartbeatRunId: copy.runId,
|
||||
provider: "daytona", providerLeaseId: "original-sandbox", status: "active", metadata: { remoteCwd } });
|
||||
await db.update(agentInstructionWorkingCopies).set({ location: `remote:${environmentId}`, executionRoot,
|
||||
receipt: { ...copy.receipt, cleanup: { leaseId, remoteCwd } } }).where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
const saved = (await copies.get(companyId, copy.runId))!;
|
||||
const replacementHome = path.join(home, `replacement-${randomUUID()}`);
|
||||
const remoteFiles = await fs.mkdtemp(path.join(home, "remote-checkpoint-"));
|
||||
await fs.mkdir(path.join(remoteFiles, "files", "policy"), { recursive: true });
|
||||
await fs.writeFile(path.join(remoteFiles, "files", entryFile), "Remote instructions changed during the turn");
|
||||
await fs.writeFile(path.join(remoteFiles, "files", "unsaved.txt"), "Only the running sandbox has this");
|
||||
await fs.rm(path.dirname(copy.localRoot), { recursive: true, force: true });
|
||||
if (scenario === "over-quota") {
|
||||
const handle = await fs.open(path.join(remoteFiles, "files", "too-large.bin"), "w");
|
||||
try { await handle.truncate(MAX_AGENT_FILE_BYTES + 1); } finally { await handle.close(); }
|
||||
}
|
||||
const transport = vi.spyOn(executionTargetTools, "prepareAdapterExecutionTargetRuntime").mockImplementation(async options => ({
|
||||
target: options.target!, workspaceRemoteDir: options.workspaceRemoteDir!, runtimeRootDir: options.runtimeRootDir!,
|
||||
assetDirs: {}, additionalSourceDirs: {}, additionalSourceFailures: [], workspaceSyncSnapshot: null,
|
||||
restoreWorkspace: async () => { await fs.cp(path.join(remoteFiles, "files"), options.workspaceLocalDir, { recursive: true }); },
|
||||
}));
|
||||
const ephemeralBefore = process.env.PAPERCLIP_EPHEMERAL_SERVER;
|
||||
process.env.PAPERCLIP_EPHEMERAL_SERVER = "true";
|
||||
if (scenario === "new-home") process.env.PAPERCLIP_HOME = replacementHome;
|
||||
const remote = { kind: "remote" as const, transport: "sandbox" as const, environmentId,
|
||||
leaseId: scenario === "foreign-lease" ? randomUUID() : leaseId, remoteCwd, runner: { execute: vi.fn() } };
|
||||
try {
|
||||
if (scenario === "missing-collector") {
|
||||
const retained = await copies.collectStopped({ companyId, runId: copy.runId });
|
||||
expect(retained).toMatchObject({ state: "pending_collection", errorCode: "INSTRUCTION_REMOTE_COLLECTOR_UNAVAILABLE", baseHash: saved.baseHash });
|
||||
expect(retained?.receipt?.baseline).toEqual(saved.receipt?.baseline);
|
||||
expect(retained?.nextAttemptAt).toBeInstanceOf(Date);
|
||||
expect(transport).not.toHaveBeenCalled();
|
||||
return;
|
||||
}
|
||||
if (scenario === "other-controller") {
|
||||
await db.transaction(async tx => {
|
||||
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`agent-directory:${companyId}:${agentId}`}, 0))`);
|
||||
await expect(copies.prepare({ ...target(), runId: copy.runId, cwd: remoteCwd, target: remote, warm: true }))
|
||||
.rejects.toThrow("Agent directory operation is already active");
|
||||
expect(transport).not.toHaveBeenCalled();
|
||||
});
|
||||
return;
|
||||
}
|
||||
const preparing = copies.prepare({ ...target(), runId: copy.runId, cwd: remoteCwd, target: remote, warm: true });
|
||||
if (scenario === "foreign-lease") {
|
||||
await expect(preparing).rejects.toThrow("Agent directory recovery lease changed");
|
||||
expect(transport).not.toHaveBeenCalled();
|
||||
return;
|
||||
}
|
||||
const recovered = (await preparing)!;
|
||||
expect(recovered.executionRoot).toBe(executionRoot);
|
||||
expect(recovered.baseHash).toBe(saved.baseHash);
|
||||
expect(recovered.receipt).toEqual(saved.receipt);
|
||||
if (scenario === "new-home") {
|
||||
expect(recovered.localRoot.startsWith(replacementHome)).toBe(true);
|
||||
await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
} else expect(recovered.localRoot).toBe(copy.localRoot);
|
||||
expect(await fs.readFile(path.join(recovered.localRoot, "unsaved.txt"), "utf8")).toBe("Only the running sandbox has this");
|
||||
expect(await fs.readFile(path.join(recovered.localRoot, entryFile), "utf8")).toBe("Remote instructions changed during the turn");
|
||||
expect(transport).toHaveBeenNthCalledWith(1, expect.objectContaining({ adapterKey: "agent-files-recovery", workspaceRemoteDir: executionRoot,
|
||||
workspaceBaseline: { exclude: [".paperclip-runtime"], entries: new Map() }, workspaceInboundMode: "adopt_remote" }));
|
||||
if (scenario === "over-quota") {
|
||||
expect((await fs.stat(path.join(recovered.localRoot, "too-large.bin"))).size).toBe(MAX_AGENT_FILE_BYTES + 1);
|
||||
await expect(inspectAgentDirectory(recovered.localRoot)).rejects.toThrow();
|
||||
}
|
||||
expect(transport).toHaveBeenCalledWith(expect.objectContaining({ workspaceLocalDir: recovered.localRoot, workspaceInboundMode: "adopt_remote" }));
|
||||
await copies.prepare({ ...target(), runId: copy.runId, cwd: remoteCwd, target: remote, warm: true });
|
||||
expect(transport).toHaveBeenCalledTimes(2);
|
||||
} finally {
|
||||
process.env.PAPERCLIP_HOME = home;
|
||||
if (ephemeralBefore === undefined) delete process.env.PAPERCLIP_EPHEMERAL_SERVER;
|
||||
else process.env.PAPERCLIP_EPHEMERAL_SERVER = ephemeralBefore;
|
||||
transport.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("retains remote cleanup across restart and failed retries until the original lease is cleaned", async () => {
|
||||
const copy = await run();
|
||||
const environmentId = randomUUID(), leaseId = randomUUID(), remoteCwd = "/fixture/task";
|
||||
|
||||
@@ -161,7 +161,10 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000;
|
||||
expect(await db.select().from(issueRelations).where(and(eq(issueRelations.issueId, child.id),
|
||||
eq(issueRelations.relatedIssueId, parent.id)))).toHaveLength(1);
|
||||
await settle(); expect(coordinator!.nextWakeAt()).not.toBeNull();
|
||||
await vi.advanceTimersByTimeAsync(delay + 2000); await archived("review", child.id); await quiet();
|
||||
// SQL assigns updatedAt using the database clock, independently of the fake
|
||||
// scheduler clock. Cross the saved deadline rather than a fixed duration.
|
||||
await vi.advanceTimersByTimeAsync(Math.max(0, child.updatedAt.getTime() + delay - Date.now()) + 2000);
|
||||
await archived("review", child.id); await quiet();
|
||||
expect(await db.select().from(decisionArchiveNotificationOutbox).where(eq(decisionArchiveNotificationOutbox.sourceId, child.id))).toHaveLength(1);
|
||||
});
|
||||
|
||||
|
||||
@@ -26,14 +26,15 @@ it("never connects the listener with migration privileges, another tenant, or th
|
||||
});
|
||||
it("rejects disk-dependent execution and instructions while retaining remote warm execution", () => {
|
||||
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
|
||||
const remote = { externalConversation: false, native: true, adapterType: "paperclip_runner", pluginTaskWorkspace: false, remoteAuthority: true };
|
||||
const remote = { externalConversation: false, adapterType: "paperclip_runner", pluginTaskWorkspace: false, remoteAuthority: true };
|
||||
expect(() => assertEphemeralWorkspaceExecution(remote)).not.toThrow();
|
||||
expect(() => assertEphemeralWorkspaceExecution({ ...remote, remoteAuthority: false, pluginTaskWorkspace: true })).not.toThrow();
|
||||
for (const patch of [{ native: false }, { remoteAuthority: false }, { adapterType: "process" }]) {
|
||||
for (const patch of [{ remoteAuthority: false }, { adapterType: "process" }]) {
|
||||
expect(() => assertEphemeralWorkspaceExecution({ ...remote, ...patch })).toThrow("warm Paperclip Runner");
|
||||
}
|
||||
expect(() => assertEphemeralNativeLifecycle("warm")).not.toThrow();
|
||||
expect(() => assertEphemeralNativeLifecycle("per_turn")).toThrow("warm Paperclip Runner");
|
||||
expect(() => assertEphemeralNativeLifecycle(undefined)).toThrow("warm Paperclip Runner");
|
||||
const agent = { id: "agent", companyId: "company", name: "Agent", adapterConfig: {} };
|
||||
expect(() => assertEphemeralAgentInstructions(agent)).not.toThrow();
|
||||
expect(() => assertEphemeralAgentInstructions({ ...agent, adapterConfig: { instructionsBundleMode: "external", instructionsRootPath: "/old-volume" } })).toThrow("managed agent instructions");
|
||||
|
||||
@@ -3,7 +3,7 @@ import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
import { and, eq, inArray, isNull, sql } from "drizzle-orm";
|
||||
import { agents, environmentLeases, environments, agentInstructionWorkingCopies as copies, type Db } from "@paperclipai/db";
|
||||
import { agents, environmentLeases, environments, agentInstructionWorkingCopies as copies, withDedicatedDbConnection, type Db } from "@paperclipai/db";
|
||||
import { syncDirectoryToSsh, restoreWorkspaceFromSshExecution } from "@paperclipai/adapter-utils/ssh";
|
||||
import { prepareAdapterExecutionTargetRuntime, runAdapterExecutionTargetShellCommand, type AdapterExecutionTarget, type PreparedAdapterExecutionTargetRuntime } from "@paperclipai/adapter-utils/execution-target";
|
||||
import { withDirectoryMergeLock, directorySnapshotSha256, parseDirectorySnapshot, serializeDirectorySnapshot, type DirectoryMergeLockOperation } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
@@ -18,6 +18,7 @@ import type { Environment, EnvironmentLease } from "@paperclipai/shared";
|
||||
import { agentDirectoryBaselineDigest, agentDirectoryProbeProgram, observeLocalAgentDirectory } from "./agent-directory-probe.js";
|
||||
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import { cachedAgentFileManifest, captureAgentFileCheckpoint, checkpointBaseline, checkpointSnapshot, type AgentFileManifest } from "./agent-file-checkpoints.js";
|
||||
import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
|
||||
type Copy = typeof copies.$inferSelect;
|
||||
@@ -54,7 +55,16 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
});
|
||||
}
|
||||
const ownerFile = (row: Copy) => path.join(path.dirname(row.localRoot), "owner.json");
|
||||
const disposableRemoteCopy = (row: Copy) => process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && row.location.startsWith("remote:");
|
||||
function projectionRoot(row: Copy): string {
|
||||
const origin = String(row.receipt?.materializationRunId ?? row.receipt?.directoryRunId ?? row.runId);
|
||||
if (!/^[a-zA-Z0-9_-]{1,128}$/.test(origin)) throw conflict("Invalid agent directory owner");
|
||||
return path.join(path.dirname(resolveManagedInstructionsRoot({ id: row.agentId, companyId: row.companyId, name: "", adapterConfig: {} })), "file-sync", "runs", origin, "live");
|
||||
}
|
||||
async function owns(row: Copy) {
|
||||
// Remote ownership is durable. A marker on a discarded controller cannot
|
||||
// establish ownership of the retained sandbox on its replacement.
|
||||
if (disposableRemoteCopy(row)) return row.state !== "superseded" && !row.receipt?.retainedByRunId;
|
||||
if (row.receipt?.warm !== true) return true;
|
||||
return await fs.readFile(ownerFile(row), "utf8").then(s => JSON.parse(s).runId === row.runId).catch(error => error.code === "ENOENT" && row.processStoppedAt !== null);
|
||||
}
|
||||
@@ -159,7 +169,7 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
? path.posix.join(input.target.remoteCwd, ".paperclip-runtime", "agent-files", input.agentId, origin)
|
||||
: localRoot;
|
||||
const location = input.target?.kind === "remote" ? `remote:${input.target.environmentId ?? ""}` : "local";
|
||||
if (row && (row.localRoot !== localRoot || row.executionRoot !== executionRoot || row.agentId !== input.agentId || row.location !== location)) throw conflict("Agent directory belongs to a different execution environment");
|
||||
if (row && ((!disposableRemoteCopy(row) && row.localRoot !== localRoot) || row.executionRoot !== executionRoot || row.agentId !== input.agentId || row.location !== location)) throw conflict("Agent directory belongs to a different execution environment");
|
||||
if (row && !serialHeld) {
|
||||
// Restart preparation can replace a completed copy under the same run ID.
|
||||
// Join cleanup's lock before resetting its receipt or creating new bytes.
|
||||
@@ -168,7 +178,10 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
return serial(row, () => prepareCopy(input, true), "agent_directory_prepare");
|
||||
}
|
||||
if (row && !completed.has(row.state) && row.state !== "preparing") {
|
||||
if (input.target?.kind === "remote" && !transports.has(key(row))) transports.set(key(row), await transport(row, input.target, true));
|
||||
if (input.target?.kind === "remote") {
|
||||
if (disposableRemoteCopy(row)) await restoreProjection(row, input.target);
|
||||
if (!transports.has(key(row))) transports.set(key(row), await transport(row, input.target, true));
|
||||
}
|
||||
return row;
|
||||
}
|
||||
// Register ownership before creating any bytes, so a crash during the copy
|
||||
@@ -241,6 +254,39 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
}
|
||||
}
|
||||
|
||||
async function restoreProjection(row: Copy, target: AdapterExecutionTarget) {
|
||||
const exists = await fs.lstat(row.localRoot).then(stat => {
|
||||
if (!stat.isDirectory() || stat.isSymbolicLink()) throw conflict("Agent directory projection is not a directory");
|
||||
return true;
|
||||
}, error => {
|
||||
if (error.code === "ENOENT") return false;
|
||||
throw error;
|
||||
});
|
||||
if (exists) return;
|
||||
const lease = await ownedRemoteLease(row);
|
||||
const cleanup = row.receipt?.cleanup as { leaseId?: string; remoteCwd?: string } | undefined;
|
||||
if (target.kind !== "remote" || !lease || target.leaseId !== lease.id || lease.status !== "active"
|
||||
|| lease.releasedAt || hasRemoteTerminationReceipt(lease) || cleanup?.remoteCwd !== target.remoteCwd
|
||||
|| row.location !== `remote:${target.environmentId}`) throw conflict("Agent directory recovery lease changed");
|
||||
// Reconstruct a private mirror without applying save quotas to admission.
|
||||
// An empty merge baseline downloads all remote bytes without uploading any;
|
||||
// the original save baseline remains unchanged in Postgres.
|
||||
await fs.mkdir(row.localRoot, { recursive: true, mode: 0o700 });
|
||||
let projection: PreparedAdapterExecutionTargetRuntime | undefined;
|
||||
try {
|
||||
projection = await prepareAdapterExecutionTargetRuntime({ target, runId: row.runId, adapterKey: "agent-files-recovery",
|
||||
workspaceLocalDir: row.localRoot, workspaceRemoteDir: row.executionRoot,
|
||||
runtimeRootDir: sandboxTransferRoot(row, target.remoteCwd), syncWorkspace: true,
|
||||
workspaceInboundMode: "adopt_remote", workspaceBaseline: { exclude: [".paperclip-runtime"], entries: new Map() },
|
||||
workspaceGitSnapshot: null, workspaceRepositories: [], workspaceFileMode: "all", workspaceExclude: [".paperclip-runtime", ".paperclip-runtime/**"] });
|
||||
await projection.restoreWorkspace();
|
||||
await inspectAgentDirectory(row.localRoot, false);
|
||||
if (row.receipt?.warm === true) await fs.writeFile(ownerFile(row), JSON.stringify({ runId: row.runId }));
|
||||
} catch (error) {
|
||||
await fs.rm(row.localRoot, { recursive: true, force: true });
|
||||
throw error;
|
||||
} finally { await projection?.cleanupWorkspaceSnapshot?.(); }
|
||||
}
|
||||
async function retrieve(row: Copy, target?: AdapterExecutionTarget | null) {
|
||||
if (row.location !== "local") {
|
||||
let runtime = transports.get(key(row));
|
||||
@@ -356,6 +402,14 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
}
|
||||
async function checkpoint(row: Copy, target?: AdapterExecutionTarget | null, stopped = false): Promise<Copy> {
|
||||
if (!await owns(row) || row.state === "superseded") return row;
|
||||
target ??= transports.get(key(row))?.target;
|
||||
if (row.location !== "local" && target?.kind !== "remote") {
|
||||
// A missing controller cache is not the remote directory. Retain the
|
||||
// pending save until its exact remote collector or destruction proof exists.
|
||||
return patch(row, { ...(stopped ? { state: "pending_collection", processStoppedAt: row.processStoppedAt ?? new Date() } : {}),
|
||||
errorCode: "INSTRUCTION_REMOTE_COLLECTOR_UNAVAILABLE", errorMessage: "The remote agent directory is retained; its collector is unavailable. No save or discard is claimed.",
|
||||
nextAttemptAt: stopped ? new Date(Date.now() + 30_000) : null });
|
||||
}
|
||||
let failure: unknown;
|
||||
for (let attempt = 0; attempt < 3; attempt++) {
|
||||
let captured: Awaited<ReturnType<typeof captureAgentFileCheckpoint>> | undefined;
|
||||
@@ -575,13 +629,29 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
return result.exitCode === 0 && !result.timedOut;
|
||||
}
|
||||
async function serial<T>(row: Copy, fn: (current: Copy) => Promise<T>, operation: DirectoryMergeLockOperation): Promise<T> {
|
||||
// Duplicate stop callbacks and restart recovery must not race while moving
|
||||
// the stopped working copy. This lock is outside the writable tree.
|
||||
return withDirectoryMergeLock(path.resolve(row.localRoot, "../../.."), async () => {
|
||||
const current = await get(row.companyId, row.runId);
|
||||
if (!current) throw notFound("Agent directory copy not found");
|
||||
return fn(current);
|
||||
}, process.env, operation);
|
||||
const run = async () => {
|
||||
const lockRoot = path.resolve(disposableRemoteCopy(row) ? projectionRoot(row) : row.localRoot, "../../..");
|
||||
if (disposableRemoteCopy(row)) await fs.mkdir(lockRoot, { recursive: true, mode: 0o700 });
|
||||
return withDirectoryMergeLock(lockRoot, async () => {
|
||||
let current = await get(row.companyId, row.runId);
|
||||
if (!current) throw notFound("Agent directory copy not found");
|
||||
if (disposableRemoteCopy(current) && current.localRoot !== projectionRoot(current)) {
|
||||
current = await patch(current, { localRoot: projectionRoot(current) });
|
||||
if (current.localRoot !== projectionRoot(current)) throw conflict("Agent directory projection changed");
|
||||
transports.delete(key(current));
|
||||
}
|
||||
return fn(current);
|
||||
}, process.env, operation);
|
||||
};
|
||||
if (!disposableRemoteCopy(row)) return run();
|
||||
// Different controllers have different cache roots. Serialize their active
|
||||
// collection/cleanup operations in Postgres, without holding row locks over I/O.
|
||||
return withDedicatedDbConnection(db, dedicated => dedicated.transaction(async tx => {
|
||||
const [lock] = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`agent-directory:${row.companyId}:${row.agentId}`}, 0)) as acquired,
|
||||
(select setting::bigint from pg_settings where name = 'idle_in_transaction_session_timeout') as idle_timeout_ms`);
|
||||
if (lock?.acquired !== true) throw conflict("Agent directory operation is already active");
|
||||
return withWorkspaceLockKeepalive(tx as unknown as Db, Number(lock.idle_timeout_ms), run);
|
||||
}));
|
||||
}
|
||||
return { prepare: (input: Parameters<typeof prepareCopy>[0]) => prepareCopy(input), hasChanges, adopt, canReuse, recoverUnavailable,
|
||||
recoverStoppedRemote: (row: Copy) => serial(row, current => current.receipt?.retainedByRunId ? Promise.resolve(current) : recoverStoppedRemote(current), "agent_directory_collect"),
|
||||
|
||||
@@ -121,7 +121,7 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
||||
if (signalsCleanup) await notifyDirectoryCleanup(tx, { companyId: row.companyId, runId: row.runId });
|
||||
if (signalsCollection) await notifyInstructionCollection(tx, { companyId: row.companyId, runId: row.runId });
|
||||
const [updated] = await tx.update(copies).set({ ...values, updatedAt: new Date() }).where(and(
|
||||
scope(row.companyId, row.runId), eq(copies.state, row.state), eq(copies.attempts, row.attempts), eq(copies.baseHash, row.baseHash),
|
||||
scope(row.companyId, row.runId), eq(copies.state, row.state), eq(copies.attempts, row.attempts), eq(copies.baseHash, row.baseHash), eq(copies.localRoot, row.localRoot),
|
||||
sql`${copies.receipt}->>'retainedByRunId' IS NOT DISTINCT FROM ${typeof row.receipt?.retainedByRunId === "string" ? row.receipt.retainedByRunId : null}`,
|
||||
requireDue ? or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())) : undefined, condition,
|
||||
)).returning();
|
||||
|
||||
@@ -328,6 +328,17 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
|
||||
expect(await coordinators(run.id)).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("rejects resolved legacy execution on an ephemeral server before runtime admission", async () => {
|
||||
const { input, run } = await fixture("process");
|
||||
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
|
||||
expect(input.nativeRuntimeResolution.kind).toBe("legacy");
|
||||
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("warm Paperclip Runner");
|
||||
expect((await read(run.id)).runtimeMode).toBe("legacy");
|
||||
expect(await coordinators(run.id)).toHaveLength(0);
|
||||
expect(input.onNativeLifecycleSelected).not.toHaveBeenCalled();
|
||||
expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["warm", "per_turn"] as const)("checks resolved %s policy for ephemeral admission", async mode => {
|
||||
const { input, run } = await fixture();
|
||||
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
|
||||
|
||||
@@ -267,6 +267,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
|
||||
const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config;
|
||||
const runOptions = input.recovery;
|
||||
const { providerTraceRequested, providerTraceCapture } = input.trace;
|
||||
if (nativeRuntimeResolution.kind !== "native" && !isDotRun) assertEphemeralNativeLifecycle(undefined);
|
||||
let nativeExecution: NativeExecutionInput | null = null;
|
||||
let getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
|
||||
let nativeRunnerInstanceId: string | null = null;
|
||||
|
||||
@@ -52,7 +52,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: Db;
|
||||
let home: string;
|
||||
const persistenceKeys = ["DATABASE_URL", "PAPERCLIP_STORAGE_PROVIDER", "PAPERCLIP_SECRETS_MASTER_KEY", "PAPERCLIP_SECRETS_MASTER_KEY_FILE"];
|
||||
const persistenceKeys = ["PAPERCLIP_EPHEMERAL_SERVER", "DATABASE_URL", "PAPERCLIP_STORAGE_PROVIDER", "PAPERCLIP_SECRETS_MASTER_KEY", "PAPERCLIP_SECRETS_MASTER_KEY_FILE"];
|
||||
let persistenceBefore = new Map<string, string | undefined>();
|
||||
const configureRemotePersistence = () => Object.assign(process.env, { DATABASE_URL: database.connectionString,
|
||||
PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture" });
|
||||
@@ -170,14 +170,15 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ kind: "empty", repoUrl: null, branchMode: null, fresh: false },
|
||||
{ kind: "empty", repoUrl: null, branchMode: null, fresh: false, ephemeral: true },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: null, fresh: false },
|
||||
{ kind: "git", repoUrl: "ssh://git@github.com/example/remote.git", branchMode: null, fresh: false },
|
||||
{ kind: "git", repoUrl: "git@github.com:example/remote.git", branchMode: null, fresh: false },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "existing", fresh: false },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: false },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true },
|
||||
] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh) without materializing a controller checkout", async ({ kind, repoUrl, branchMode, fresh }) => {
|
||||
] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh) without materializing a controller checkout", async (scenario) => {
|
||||
const { kind, repoUrl, branchMode, fresh } = scenario;
|
||||
const { input, source, issue, envOrchestrator } = await fixture();
|
||||
const [savedEnvironment] = await db.insert(environments).values({ name: `Remote ${randomUUID()}`,
|
||||
driver: "sandbox", status: "active", config: { provider: "daytona" } }).returning();
|
||||
@@ -227,10 +228,13 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
|
||||
throw new Error("reached remote acquisition");
|
||||
});
|
||||
const configured = { DATABASE_URL: database.connectionString, PAPERCLIP_STORAGE_PROVIDER: "s3",
|
||||
PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture" };
|
||||
PAPERCLIP_SECRETS_MASTER_KEY: "remote-workspace-preparation-fixture",
|
||||
PAPERCLIP_EPHEMERAL_SERVER: "ephemeral" in scenario ? "true" : "false" };
|
||||
const previous = new Map(Object.keys(configured).map(key => [key, process.env[key]]));
|
||||
Object.assign(process.env, configured);
|
||||
try {
|
||||
// Fresh runs are still marked legacy until the runtime selector owns promotion.
|
||||
expect(input.run.runtimeMode).toBe("legacy");
|
||||
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached remote acquisition");
|
||||
expect(resolve).not.toHaveBeenCalled();
|
||||
expect(repositoryPreparation).not.toHaveBeenCalled();
|
||||
@@ -268,7 +272,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
|
||||
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("auto-enrollment leaves a configured local folder on the host authority", async () => {
|
||||
it.each([false, true])("configured local folders retain host authority only on durable servers (ephemeral=%s)", async ephemeral => {
|
||||
const { input, envOrchestrator } = await fixture();
|
||||
input.agent.adapterType = "paperclip_runner";
|
||||
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
|
||||
@@ -277,8 +281,11 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
|
||||
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
|
||||
configureRemotePersistence();
|
||||
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("local folder retained"));
|
||||
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("local folder retained");
|
||||
expect(resolve).toHaveBeenCalledOnce();
|
||||
process.env.PAPERCLIP_EPHEMERAL_SERVER = String(ephemeral);
|
||||
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(ephemeral
|
||||
? "Volume-free servers require warm Paperclip Runner execution with remote workspace storage." : "local folder retained");
|
||||
if (ephemeral) expect(resolve).not.toHaveBeenCalled();
|
||||
else expect(resolve).toHaveBeenCalledOnce();
|
||||
expect(await db.select().from(executionWorkspaces)).toHaveLength(0);
|
||||
expect(acquire).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
@@ -241,7 +241,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
|
||||
&& !projectExecutionWorkspacePolicy?.storageProviderPluginId,
|
||||
existingWorkspace: reusableExistingExecutionWorkspace ?? existingExecutionWorkspace,
|
||||
});
|
||||
assertEphemeralWorkspaceExecution({ externalConversation: isDotRun, native: run.runtimeMode === "native",
|
||||
assertEphemeralWorkspaceExecution({ externalConversation: isDotRun,
|
||||
adapterType: agent.adapterType,
|
||||
pluginTaskWorkspace, remoteAuthority });
|
||||
const assertRemotePreparationOwned = run.runtimeMode === "native" ? () => assertNativeSessionPreparationAuthority({
|
||||
@@ -322,7 +322,13 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
|
||||
issue: issueRef, agent, projectId: issueRef!.projectId, repoRef: null,
|
||||
})) : null),
|
||||
});
|
||||
if (sourceState.authority === "host") remoteAuthority = false;
|
||||
if (sourceState.authority === "host") {
|
||||
remoteAuthority = false;
|
||||
// Source resolution can retain a configured host folder. Check the
|
||||
// resolved authority before materializing it or acquiring a sandbox.
|
||||
assertEphemeralWorkspaceExecution({ externalConversation: isDotRun,
|
||||
adapterType: agent.adapterType, pluginTaskWorkspace, remoteAuthority });
|
||||
}
|
||||
else {
|
||||
remoteSourceState.value = sourceState;
|
||||
if (!enrolledRemoteAuthority) await assertRemotePreparationOwned();
|
||||
|
||||
@@ -145,6 +145,7 @@ import {
|
||||
executeNativeSession,
|
||||
applyNativeSessionGoalControl,
|
||||
inspectStagedRunAttachment,
|
||||
inspectRetainedRunInstructions,
|
||||
inspectWarmRunTransition,
|
||||
parseNativeExecutionInput,
|
||||
parsePaperclipQuestionSet,
|
||||
@@ -12545,6 +12546,19 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
: null;
|
||||
const stagedRecovery = input.restartRecovery?.kind === "reattach_remote_runner"
|
||||
? await verifyStagedAttachmentRestart(input) : null;
|
||||
const retainedSystemInstructions = input.restartRecovery?.kind === "reattach_remote_runner"
|
||||
&& "runtimeContext" in input.execution
|
||||
&& input.execution.schema !== "paperclip.native-execution-input.v6"
|
||||
&& input.execution.schema !== "paperclip.native-execution-input.v8"
|
||||
? inspectRetainedRunInstructions(JSON.parse(readBoundedNativeFile(
|
||||
resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
|
||||
"native_runner_recovery_instructions_unavailable").toString("utf8")), {
|
||||
runId: input.execution.binding.runId,
|
||||
normalizedSessionId: nativeSessionKey(input.execution),
|
||||
runtimeContextDigest: input.execution.runtimeContext.aggregateDigest,
|
||||
})
|
||||
: undefined;
|
||||
if (retainedSystemInstructions === null) throw new Error("native_runner_recovery_instructions_unavailable");
|
||||
let stagedRecoveryPending = stagedRecovery !== null;
|
||||
const recoveryUsageIdentity = stagedRecovery?.sourceIdentity ?? input.remoteWarmRecovery?.identity ?? durableIdentity;
|
||||
const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity
|
||||
@@ -12647,6 +12661,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
return current.execute(call);
|
||||
};
|
||||
const backend = createNativeSessionBackend(input.execution, {
|
||||
retainedSystemInstructions,
|
||||
// The plugin resolves the VM directory while preparing the launch. Core keeps
|
||||
// its host environment out of the plugin and records the prepared directory.
|
||||
runnerInstanceId: prior?.runnerInstanceId ?? effectiveRunnerInstanceId, environment: {},
|
||||
@@ -13981,6 +13996,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
runId: input.execution.binding.runId, prepareRemote: prepareRemoteRunner });
|
||||
}
|
||||
const backend = createNativeSessionBackend(runnerExecution, {
|
||||
retainedSystemInstructions,
|
||||
runnerInstanceId:
|
||||
retainedTransition?.runnerInstanceId ?? input.runnerInstanceId,
|
||||
environment: effectiveRunnerEnvironment,
|
||||
|
||||
@@ -76,11 +76,13 @@ export function shouldUseRemoteWorkspaces(input: {
|
||||
* remote controller may replace its local projection; host-owned execution
|
||||
* cannot make that promise. External conversation providers own no task disk. */
|
||||
export function assertEphemeralWorkspaceExecution(input: {
|
||||
externalConversation: boolean; native: boolean; adapterType: string;
|
||||
externalConversation: boolean; adapterType: string;
|
||||
pluginTaskWorkspace: boolean; remoteAuthority: boolean;
|
||||
}) {
|
||||
if (process.env.PAPERCLIP_EPHEMERAL_SERVER !== "true" || input.externalConversation) return;
|
||||
if (!input.native || input.adapterType !== "paperclip_runner"
|
||||
// Runtime selection has not promoted a fresh run yet. Validate storage here;
|
||||
// the runtime selector separately admits only its resolved warm native policy.
|
||||
if (input.adapterType !== "paperclip_runner"
|
||||
|| (!input.pluginTaskWorkspace && !input.remoteAuthority)) {
|
||||
throw unprocessable("Volume-free servers require warm Paperclip Runner execution with remote workspace storage.",
|
||||
{ code: "ephemeral_execution_configuration_required" });
|
||||
@@ -89,7 +91,7 @@ export function assertEphemeralWorkspaceExecution(input: {
|
||||
|
||||
/** The runtime selector calls this after environment, credentials and saved
|
||||
* execution policy have resolved. Raw agent settings are not lifecycle authority. */
|
||||
export function assertEphemeralNativeLifecycle(mode: "warm" | "per_turn") {
|
||||
export function assertEphemeralNativeLifecycle(mode: "warm" | "per_turn" | undefined) {
|
||||
if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && mode !== "warm") {
|
||||
throw unprocessable("Volume-free servers require warm Paperclip Runner execution with remote workspace storage.",
|
||||
{ code: "ephemeral_execution_configuration_required" });
|
||||
|
||||
+1
@@ -95,6 +95,7 @@ export const RUNNER_PRP_PROTOCOL_MAX = runner.RUNNER_PRP_PROTOCOL_MAX;
|
||||
export const runnerCodexDynamicToolsFit = runner.runnerCodexDynamicToolsFit;
|
||||
export const inspectWarmRunTransition = runner.inspectWarmRunTransition;
|
||||
export const inspectStagedRunAttachment = runner.inspectStagedRunAttachment;
|
||||
export const inspectRetainedRunInstructions = runner.inspectRetainedRunInstructions;
|
||||
export const inspectStagedRunAttachmentHeader = runner.inspectStagedRunAttachmentHeader;
|
||||
export type StagedRunAttachment = import("@paperclipai/paperclip-runner").StagedRunAttachment;
|
||||
export const readRunnerdArtifactBinding = runner.readRunnerdArtifactBinding;
|
||||
|
||||
@@ -100,9 +100,12 @@ The separate explicit-only `rolling-web-deployment` suite uses the same syntheti
|
||||
projectless workflow, four-admission budget, thirty-minute limit, and independent
|
||||
run/file oracles. Its supervisor starts an independent empty-home server, waits
|
||||
for `/api/ready`, moves ingress, sends SIGTERM to the predecessor, and deletes
|
||||
that predecessor's whole home and build. Continuous ordinary task reads and
|
||||
that predecessor's whole home and build. Cleanup makes read-only cache directories
|
||||
removable only after shutdown, without following dependency symlinks. Continuous ordinary task reads and
|
||||
comment saves must have zero failures, and every accepted save must be read back
|
||||
exactly once. Forced process cleanup fails graceful qualification even if the
|
||||
exactly once. Probe tasks live in a separate disposable company, so they cannot
|
||||
block the business workflow's all-tasks-done oracle or consume its agent budget.
|
||||
Forced process cleanup fails graceful qualification even if the
|
||||
successor recovers the execution. The read-only observer shares the controllers'
|
||||
disposable encryption key. It repeats replacement after completion before the retained-session
|
||||
follow-up. The five-minute outage suites are unchanged.
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
import { chmod, lstat, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { expect, it } from "vitest";
|
||||
import { makeDirectoryTreeRemovable } from "./directory-cleanup.js";
|
||||
|
||||
it("removes a retired home with immutable bundles without changing linked dependencies", async () => {
|
||||
const root = await mkdtemp(path.join(tmpdir(), "paperclip-retired-home-"));
|
||||
const home = path.join(root, "controller"), external = path.join(root, "external");
|
||||
const bundle = path.join(home, "runtime-context-assets/bundles/digest");
|
||||
try {
|
||||
await mkdir(bundle, { recursive: true }); await mkdir(external);
|
||||
await writeFile(path.join(bundle, "SKILL.md"), "immutable skill");
|
||||
await writeFile(path.join(external, "keep"), "dependency");
|
||||
await chmod(bundle, 0o555); await chmod(external, 0o555);
|
||||
await symlink(external, path.join(home, "node_modules"));
|
||||
await makeDirectoryTreeRemovable(home);
|
||||
await rm(home, { recursive: true });
|
||||
await expect(lstat(home)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect((await lstat(external)).mode & 0o777).toBe(0o555);
|
||||
expect(await readFile(path.join(external, "keep"), "utf8")).toBe("dependency");
|
||||
const link = path.join(root, "not-an-owned-home");
|
||||
await symlink(external, link);
|
||||
await expect(makeDirectoryTreeRemovable(link)).rejects.toThrow("not a symlink");
|
||||
expect((await lstat(external)).mode & 0o777).toBe(0o555);
|
||||
} finally {
|
||||
await makeDirectoryTreeRemovable(root); await rm(root, { recursive: true });
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,13 @@
|
||||
import { chmod, lstat, readdir } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
|
||||
/** Cleanup of a stopped, test-owned tree. Never follow links to dependencies
|
||||
* or external workspaces when making immutable bundle directories removable. */
|
||||
export async function makeDirectoryTreeRemovable(directory: string): Promise<void> {
|
||||
const stat = await lstat(directory);
|
||||
if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Cleanup requires an owned directory, not a symlink");
|
||||
await chmod(directory, 0o700);
|
||||
const entries = await readdir(directory, { withFileTypes: true });
|
||||
await Promise.all(entries.filter(entry => entry.isDirectory() && !entry.isSymbolicLink())
|
||||
.map(entry => makeDirectoryTreeRemovable(path.join(directory, entry.name))));
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { makeDirectoryTreeRemovable } from "./directory-cleanup.js";
|
||||
import { readUpgradeArtifacts, verifyUpgradeArtifact, upgradeArtifactKey } from "./upgrade-artifacts.js";
|
||||
import { recoverWorkerTiming } from "./worker-timing.js";
|
||||
import { openEvalOperations, type EvalOperations } from "./operations.js";
|
||||
@@ -118,17 +119,6 @@ function secret(bytes = 32) {
|
||||
return randomBytes(bytes).toString("base64url");
|
||||
}
|
||||
|
||||
async function makeDirectoryTreeRemovable(directory: string): Promise<void> {
|
||||
await chmod(directory, 0o700);
|
||||
const entries = await readdir(directory, { withFileTypes: true });
|
||||
await Promise.all(
|
||||
entries
|
||||
.filter((entry) => entry.isDirectory() && !entry.isSymbolicLink())
|
||||
.map((entry) =>
|
||||
makeDirectoryTreeRemovable(path.join(directory, entry.name)),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
function stopProcessGroup(pid: number, signal: NodeJS.Signals = "SIGTERM") {
|
||||
try {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { makeDirectoryTreeRemovable } from "./directory-cleanup.js";
|
||||
/** Credential-free full-server overlap characterization. No model or Runner
|
||||
* continuity is claimed. Requires an explicitly selected disposable S3 service.
|
||||
* node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/rolling-deployment-smoke.ts */
|
||||
@@ -208,7 +209,7 @@ try {
|
||||
proxy.routeTo(b.origin);
|
||||
await delay(500);
|
||||
await a.stop({ requireGraceful: true });
|
||||
await rm(a.home, { recursive: true, force: true });
|
||||
await makeDirectoryTreeRemovable(a.home); await rm(a.home, { recursive: true });
|
||||
await delay(1_000);
|
||||
report.handoffElapsedMs = Math.round(performance.now() - switchedAt);
|
||||
if (browser) await check("Existing browser reconnects and opens an old lazy chunk after A removal", async () => {
|
||||
@@ -270,7 +271,7 @@ try {
|
||||
if (fixture.seeded) assert.equal((await api(b.origin, `${route}?path=AGENTS.md`)).content, "Durable original");
|
||||
}
|
||||
});
|
||||
await b.stop({ requireGraceful: true }); await rm(b.home, { recursive: true, force: true });
|
||||
await b.stop({ requireGraceful: true }); await makeDirectoryTreeRemovable(b.home); await rm(b.home, { recursive: true });
|
||||
const c = await start("C", false); proxy.routeTo(c.origin);
|
||||
await check("Fresh C restores files after both homes are deleted, even with adoption opt-in disabled", async () => {
|
||||
assert.equal((await api(c.origin, `${fileRoute}?path=reference.txt`)).content, content);
|
||||
|
||||
@@ -4,17 +4,17 @@ import { createLocalEncryptedCipher } from "../../server/src/secrets/local-encry
|
||||
const mocks = vi.hoisted(() => ({
|
||||
start: vi.fn(), observer: vi.fn(), stop: vi.fn(), close: vi.fn(), writeFile: vi.fn(),
|
||||
}));
|
||||
vi.mock("node:fs/promises", () => ({ mkdir: async () => {}, writeFile: mocks.writeFile,
|
||||
vi.mock("node:fs/promises", () => ({ mkdir: async () => {}, rename: async () => {}, writeFile: mocks.writeFile,
|
||||
readFile: async () => JSON.stringify({ database: { mode: "postgres", connectionString: "postgres://fixture" },
|
||||
secrets: { localEncrypted: { keyFilePath: "/fixture/secrets/master.key" } } }) }));
|
||||
vi.mock("../../packages/shared/src/config-schema.js", () => ({ paperclipConfigSchema: { parse: (value: unknown) => value } }));
|
||||
vi.mock("./rolling-deployment-process.js", () => ({ startRollingDeploymentProcess: mocks.start }));
|
||||
vi.mock("./rolling-deployment-proxy.js", () => ({ startRollingDeploymentProxy: async () => ({ origin: "http://127.0.0.1:1", routeTo() {}, close: mocks.close }) }));
|
||||
vi.mock("./controller-recovery-observer.js", () => ({ createControllerRecoveryObserver: mocks.observer }));
|
||||
vi.mock("./server-restart-request.js", () => ({ readRestartRequest: async () => ({ requestId: "key-check", rollingDeployment: true, recoveryBinding: {} }) }));
|
||||
vi.mock("./server-restart-request.js", () => ({ readRestartRequest: async () => ({ requestId: "key-check", rollingDeployment: true, recoveryBinding: { companyId: "business-company" } }) }));
|
||||
import { superviseRollingDeployment } from "./rolling-deployment-supervisor.js";
|
||||
|
||||
afterEach(() => { vi.unstubAllEnvs(); vi.clearAllMocks(); });
|
||||
afterEach(() => { vi.unstubAllEnvs(); vi.unstubAllGlobals(); vi.clearAllMocks(); });
|
||||
|
||||
it.each([undefined, "prior-disposable-key"])("shares the server key with checkpoint observation and restores %s on failure", async prior => {
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", prior);
|
||||
@@ -40,3 +40,24 @@ it.each([undefined, "prior-disposable-key"])("shares the server key with checkpo
|
||||
expect(mocks.close).toHaveBeenCalledOnce();
|
||||
expect(process.env.PAPERCLIP_SECRETS_MASTER_KEY).toBe(prior);
|
||||
});
|
||||
|
||||
|
||||
it("keeps open deployment probe tasks out of the agent workflow's company", async () => {
|
||||
mocks.start.mockResolvedValue({ child: { exitCode: null, signalCode: null }, configPath: "/fixture/config.json",
|
||||
origin: "http://127.0.0.1:2", waitForReady: async () => {}, stop: mocks.stop });
|
||||
mocks.observer.mockResolvedValue(undefined);
|
||||
const routes: string[] = [];
|
||||
vi.stubGlobal("fetch", vi.fn(async (url: string) => {
|
||||
const route = new URL(url).pathname;
|
||||
routes.push(route);
|
||||
if (route === "/api/companies") return new Response(JSON.stringify({ id: "probe-company" }));
|
||||
if (route === "/api/companies/probe-company/issues") throw new Error("probe fixture boundary reached");
|
||||
throw new Error(`Probe modified business workflow: ${route}`);
|
||||
}));
|
||||
await expect(superviseRollingDeployment({ temporaryRoot: "/fixture", repositoryRoot: "/fixture",
|
||||
configPath: "/fixture/config.json", port: 1, executionIds: ["rolling"], cancelled: () => false,
|
||||
environment: { PAPERCLIP_STORAGE_PROVIDER: "s3" },
|
||||
})).rejects.toThrow("probe fixture boundary reached");
|
||||
expect(routes).toEqual(["/api/companies", "/api/companies/probe-company/issues"]);
|
||||
expect(mocks.stop).toHaveBeenCalledOnce();
|
||||
});
|
||||
@@ -1,3 +1,4 @@
|
||||
import { makeDirectoryTreeRemovable } from "./directory-cleanup.js";
|
||||
import { randomBytes, randomUUID } from "node:crypto";
|
||||
import { mkdir, readFile, rename, rm, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
@@ -76,7 +77,9 @@ export async function superviseRollingDeployment(input: {
|
||||
const writes: Array<{ id: string; body: string }> = [];
|
||||
try {
|
||||
if (!probeIssueId) {
|
||||
const companyId = request.recoveryBinding?.companyId ?? (await api<{ id: string }>("/companies", "POST", { name: "Synthetic deployment probes" })).id;
|
||||
// The business oracle observes every task in its company. Keep the
|
||||
// unassigned read/save probe outside that workload and its agent budget.
|
||||
const companyId = (await api<{ id: string }>("/companies", "POST", { name: "Synthetic deployment probes" })).id;
|
||||
const issue = await api<{ id: string }>(`/companies/${companyId}/issues`, "POST", {
|
||||
title: "Synthetic deployment read/save probe", description: "Unassigned fixture for ordinary user saves during deployment.", status: "todo",
|
||||
});
|
||||
@@ -103,7 +106,7 @@ export async function superviseRollingDeployment(input: {
|
||||
waitForReady: async replacement => { await replacement.server.waitForReady(); },
|
||||
switchTraffic: replacement => { current = replacement.server; proxy.routeTo(current.origin); },
|
||||
stopPrevious: () => previous.stop({ requireGraceful: true }),
|
||||
deletePreviousHome: () => rm(previous.home, { recursive: true }),
|
||||
deletePreviousHome: async () => { await makeDirectoryTreeRemovable(previous.home); await rm(previous.home, { recursive: true }); },
|
||||
beforeHandoff: observer?.beforeHandoff,
|
||||
afterPreviousStopped: observer?.stoppedAfterHandoff,
|
||||
afterHandoff: observer ? (receipt, record) => observer.recovered(receipt, record, input.cancelled) : undefined,
|
||||
|
||||
Reference in new issue
Block a user