Preserve native sessions through legacy workspace enrollment (#18943)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native session retains its workspace and provider session across
turns.
> - An ephemeral web server must restore those files without its
predecessor's disk.
> - Legacy enrollment changes the workspace authority and can change
both workspace addresses.
> - The retained provider backup checks those addresses as part of its
identity.
> - This pull request preserves the exact addresses in a verified
migration receipt.
> - The benefit is that the next turn can use the original session after
enrollment.

## Linked Issues or Issue Description

Refs #18932, merged as `612d9931a6`. This pull request builds on that
all-Runner durability change. Related: #18918 fixes concurrent workspace
preservation locks. This change covers identity preservation during
offline enrollment.

## What Changed

- The native workspace migration owner records the selected session
bindings in the successful workspace migration operation.
- Runtime selection retains the original logical address only for the
exact agent, task and session.
- Remote workspace preparation restores the original provider address.
It refuses conflicting targets, forged receipts and occupied unowned
roots.
- The environment owner includes the migration operation in the reusable
lease fingerprint. Warm policy stays warm and receives a fresh
allocation through normal lease cleanup.
- Exact migration evidence permits the existing verified provider backup
to restore into that fresh warm allocation.
- Preserve verified provider allocation files through a normalized,
lease-bound digest receipt. Unknown files and altered markers remain
refused; interrupted publication is retryable.
- Retain legacy managed-instruction session fingerprints using the exact
imported content hash and recorded paths, preserving all other
configuration and credential checks.
- Migrate existing local skill archive/manifest objects through upload
claims, readback verification and receipt compare-and-swap; retain
original objects and IDs.
- Add explicit two-turn browser qualification for every released native
Daytona profile, with warm and per-turn variants. The production
maintenance entry runs offline. The old home is deleted only after a
verified export and successor readiness. The oracle checks provider
identity, native session, addresses, reference bytes, question and final
document.

## Verification

- Full repository typecheck passed at `b12ecd823e`. The production
session fix passed 198 session/credential tests. Full repository build
passed at `94bb917947`. The final Rust-only provider fix passed 57 ACPX
provider tests and the exact Linux release build.
- Workspace and transport checks passed: 117 transport/workspace tests,
six final real PostgreSQL migration boundary cases, 178 environment
tests, 539 Daytona tests (eight skips), and the native warm-backup
admission control.
- Local skill migration passed eight PostgreSQL/S3 tests (one existing
opt-in skip). Failed readback retains the source. Successful readback
still works after deletion of the old home.
- Latest Product E2E typecheck and 38 focused harness tests pass.
Earlier catalog/admission/report checks passed. Twelve unrelated
support-test failures reproduce on unchanged base `1a39dea9aa`; these
are not reported as passing.
- Live Codex per-turn at `2d5bb9a6d1` retained the native and provider
sessions, both workspace paths and exact reference bytes after deletion
of the old home. It completed the task and delivered its saved document
and original reference. Its original failure was an output-count grading
defect. The corrected grader passes all 21 checks against retained,
hashed evidence; the original result remains unchanged.
- Live OpenCode per-turn and Codex warm at harness `94bb917947` pass all
21 product checks. Their launcher cleanup failed because a read-only
diagnostic observer kept its disposable database open. The observer now
closes before ordinary database cleanup; a no-provider reproduction
confirms the fix. All owned sandboxes were independently verified
absent, child keys revoked and buckets removed. Original failed campaign
dispositions remain unchanged.
- Claude at `94bb917947` kept its sessions and reference bytes but
omitted the final clickable document link. That user-delivery failure
remains recorded. The fixture now explicitly asks for the link and
records migration evidence before checking navigation. The fresh Claude
attempt at `b12ecd823e` passed both strict migration checks: exact
provider/native session, both paths, original file bytes and
independently downloaded attachment. Its full browser workflow failed
again: it requested unnecessary confirmation, used the answer sentence
as the reference, and omitted the clickable document link. This is
retained as a failed user-workflow result, not a full pass.
- Earlier live failures exposed and reproduced the skill-object,
fresh-allocation marker and managed-instruction fingerprint bugs. The
fixes retain original identity checks. No provider session reset or
accounting bypass was used.
- The real Grok continuation at `b12ecd823e` failed before provider
execution because its immutable-instruction attach policy rejected
relocated registered file copies. A no-provider regression reproduced
the exact rejection. `b05ba3728a` selects the existing strict
authenticated asset-path policy for Grok; content digests, unknown
context fields, session/workspace identity and same-run mutations remain
enforced. The regression and all 57 ACPX provider controls pass. The
Grok browser migration now passes all 21 checks with the b05 Linux
Runner, including provider identity, exact reference bytes and final
user output; the launcher also reports successful cleanup.
- `24fabf1d81` applies the same policy to released ACPX Codex after a
second no-provider reproduction. All 58 ACPX provider controls and the
final Linux release build pass. Its live browser migration passes all 21
checks on the final binary, including exact original session/file
preservation and completed user output.
- The full native-session executor suite passes all 1,121 tests after
updating its existing lease database fixture for the migration-receipt
join. The prior CI failure was this unsupported test-double method; the
production guard still rejects ordinary unproven recovery. CI and
Greptile 5/5 passed at `ca1e0d5fe6`. The first CI attempt lost several
workers and had one rotated-recovery transport failure; its seven
descriptor cases passed locally, and the single unchanged retry passed.
The original failure remains recorded. Staging enrollment has not
started. Initial offline enrollment and subsequent rolling availability
are separate checks.

- The final review follow-up adds real runtime-selection PostgreSQL
cases for exact, different-session and absent receipts, plus
executor-level warm restoration that copies actual backup files and
verifies provider identity and the new lease stamp. Both complete
affected suites pass (1,172 tests), and full repository typecheck
passes. Removing either production migration connection independently
makes its new positive case fail; both source files were restored
byte-for-byte after these negative controls.

- Final commit `0971d373a3` passes all 63 checks (57 successes, five
expected skips, one neutral); Greptile is 5/5 and every review thread is
resolved.

## Risks

- An identity conflict blocks enrollment and keeps the original files.
It does not silently reset the session.
- A new migration generation retires a reusable lease through its
existing owner. The original sandbox must finish normal cleanup before
reuse proceeds.
- Initial enrollment has an offline step. The separate rolling suites
prove uninterrupted deploy behavior after enrollment.
- This adds no schema migration and changes no provider credentials or
accounting gate.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model build and context
window are not exposed in this session. The agent used reasoning,
terminal tools, code edits, PostgreSQL/filesystem tests and isolated
live browser qualification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-04 04:43:56 -06:00
1 parent cf51936587
commit 84aa989bf9
40 files changed
+1177 -66

No files matched your search

+6
View File
@@ -661,3 +661,9 @@ run-attributed tool receipts on local and Daytona execution. Its four native
Claude/Codex cells are explicit-only. Remote registrations remain unqualified
until retained live attempts prove them. It uses the normal Product E2E report
and keeps synthetic feedback distinct from actionable product findings.
# Legacy workspace preservation
The explicit [legacy workspace migration Product E2E suites](../tests/runner-e2e/README.md#legacy-workspace-migration)
verify a retained session through the production offline migration and an empty
replacement controller. They complement the rolling deployment suites and do
not claim uninterrupted service during initial enrollment.
@@ -0,0 +1,119 @@
# Preserve native sessions when enrolling a legacy workspace
A retained native session binds both its Core workspace address and the provider's
sandbox address. Migrating files to object storage must preserve these addresses;
a new `.paperclip-workspaces` directory alone is not session continuity.
The offline workspace owner records selected native source bindings in its
successful migration operation. It checks the source again before publication.
The workspace points to that operation; arbitrary metadata is insufficient.
Each binding retains company/workspace, agent, task, normalized session, source
run, logical address, and the shared provider address. Conflicting or missing
identity blocks publication without deleting source files.
Runtime selection uses the original logical address only for the exact retained
session. The remote workspace owner restores at the original provider address,
refuses another base path or an occupied unowned root, and may publish into a
verified empty directory on a newly allocated sandbox. The environment owner
includes the migration operation in its reusable lease fingerprint, so existing
warm policy obtains a fresh allocation and then reuses it normally. A verified
migration binding permits restoration of the exact retained warm backup into
that new allocation. Existing backup digest and identity checks remain required.
Verification covers real PostgreSQL, filesystem restore after removal of the old
controller workspace, unchanged provider bytes, warm and per-turn policy, forged
receipts, wrong targets, occupied roots, and reusable-lease replacement. The live
acceptance gate is an ordinary agent continuation through the packaged offline
migration, followed by a rolling deployment with active ownership transfer.
Named staging enrollment remains pending until this proof and the PR gates pass.
The first live enrollment also found that already-packaged local skill objects
were skipped by the older database-body migration. The skill storage owner now
copies each verified local archive and manifest to a new S3 upload claim, reads
back their exact bytes, then publishes the new receipt with a compare-and-swap.
Package and version IDs stay unchanged. Failed copies retain the original
receipt, local objects and durable upload journal. The maintenance entry pages
through this owner before its existing package and version verification.
Fresh provider allocations can contain provider-owned runtime files. The plugin
lease metadata exposes a bounded `workspaceAllocationFiles` receipt containing
the exact provider lease, remote root, relative paths and SHA-256 hashes. Daytona
owns its sentinel encoding; workspace restore only consumes that normalized
receipt. The restore owner admits only the declared regular files, preserves
those bytes alongside the restored workspace, and refuses unknown files, changed
hashes and links. A parked allocation directory makes interrupted publication
retryable. Providers without the optional receipt retain empty-root behavior.
The all-Runner prerequisite merged as `612d9931a6`. The second live Codex migration
at `469d28314f` verified all ten local skill objects, the original workspace and
provider backup, and reached a ready maintenance receipt. Its next turn exposed
the allocation-marker restore failure; that attempt remains failed. The exact
owned sandboxes, temporary credentials, database and object bucket were cleaned.
The original task remains staging enrollment, active handoff, and admission of
new work with existing warm/per-turn policies; staging has not yet been enrolled.
The `91db767eb8` live attempt successfully restored the workspace and preserved
its reference bytes, but the next turn incorrectly selected a fresh provider
session. Its recorded admission reason was an adapter-configuration change.
A no-provider reproduction isolated the cause: importing unchanged managed
instructions creates a durable head, after which normal fingerprints omit the
old local instruction paths. Session preparation now compares the complete old
fingerprint using the verified imported content hash and the still-recorded
paths. It does not waive content, model, credential or other configuration
changes. Ordinary future turns retain the location-independent fingerprint.
Migration qualification also waits for a live question to become a saved
handoff and for its task lock to release before requesting offline enrollment.
The maintenance owner's active-run rejection remains unchanged.
The same failed attempt also exposed a fixture conflict: it requested a generated
reference file but forbade delivering it, and completion stayed blocked. The
migration journey now asks for that exact reference as an attachment after the
answer, alongside the confirmation. The byte/session/path preservation oracle
is unchanged; initial delivery remains forbidden and final task completion is
still required. This is fixture setup, not a relaxation of product completion.
Live Codex at `2d5bb9a6d1` retained the same native and provider session, both
workspace addresses and exact reference bytes after old-home destruction. It
finished the task and delivered its confirmation and reference attachment.
The original campaign failed one generic output-count check because it counted
the explicitly requested reference as a second primary output. The migration
owner now grades that attachment separately and excludes only its independently
verified original bytes from the primary-output count; duplicates and unrelated
files remain failures. Regrading the retained evidence passes all 21 checks.
The original failed disposition remains recorded. Forty-four grader/readiness
controls pass; further live provider and warm-policy qualification follows.
OpenCode per-turn and Codex warm at harness `94bb917947` passed all 21 product
checks. Their diagnostic observer held the disposable database open at cleanup;
the observer now closes on the final case receipt, and an ordinary PostgreSQL
drop reproduces both the failure and fix without a provider. Owned sandboxes,
keys and buckets were independently cleaned. Original campaign failures remain
recorded. Full repository build passed at `94bb917947`; full repository typecheck
and 38 focused harness tests passed at `b12ecd823e`.
The next Grok migration preserved the workspace and session selection but failed
at the Runner's strict attachment check. Imported instructions and skills had
new registered copy paths. Grok still used immutable-instruction attachment,
while Claude already allowed only the authenticated canonical asset-path suffix
to move. A no-provider Grok regression reproduces that exact rejection. Grok now
uses the same narrow asset-path policy. Content, skill digests, workspace/session
identity, unknown context policy, same-run mutations and unauthenticated run
rotation remain rejected. All 57 ACPX provider tests pass after the fix. A fresh
Linux Runner and real Grok continuation must still verify the provider boundary.
The released ACPX Codex profile had the same immutable-path default. Its
no-provider regression also reproduces the attachment rejection, and the same
narrow policy fixes it. All 58 ACPX provider tests pass. Unknown profiles retain
their prior restrictive default. Claude at `b12ecd823e` passed both strict
migration assertions, including the downloaded reference, but its full browser
workflow failed on an unnecessary confirmation and incorrect booking text.
That failed user-flow result remains recorded separately from preservation.
Fresh Grok (`b05ba3728a`) and ACPX Codex (`24fabf1d81`) migration campaigns each
passed all 21 checks, including real provider continuation and cleanup. Default
regression coverage now exercises the actual runtime-selection database path
with matching, mismatched, and absent migration receipts. An executor-level
test restores real backup files into a created warm lease only for the exact
migrated session, verifies the retained provider identity, and checks the new
lease's backup stamp. Neither test replaces the migration decision helper.
@@ -4115,6 +4115,18 @@ mod tests {
authenticated_run_grant_attachment("claude");
}
#[cfg(unix)]
#[test]
fn grok_attachment_rotates_only_authenticated_asset_paths() {
authenticated_run_grant_attachment("grok");
}
#[cfg(unix)]
#[test]
fn codex_attachment_rotates_only_authenticated_asset_paths() {
authenticated_run_grant_attachment("codex");
}
#[cfg(unix)]
fn authenticated_run_grant_attachment(agent: &str) {
let directory = temporary_directory(&format!("{agent}-cross-run-attach"));
@@ -4164,7 +4176,7 @@ mod tests {
"mcp": {"assignmentSetId": "assignment-1", "digest": "e".repeat(64), "bindingId": "old-run-binding"},
"futurePolicy": {"companyId": "company-1"},
});
if agent == "claude" {
if matches!(agent, "claude" | "grok" | "codex") {
descriptor_value["instructions"] = json!(format!(
"Pinned prompt. Custom entry mentioning /old-bundle.\n\n{}",
registered_asset_suffix(&descriptor_value["runtimeContext"]).unwrap()
@@ -4301,7 +4313,7 @@ mod tests {
json!("/new-bundle");
descriptor_value["runtimeContext"]["skills"][0]["bundle"]["rootPath"] = json!("/new-skill");
descriptor_value["runtimeContext"]["mcp"]["bindingId"] = json!("new-run-binding");
if agent == "claude" {
if matches!(agent, "claude" | "grok" | "codex") {
descriptor_value["instructions"] = json!(format!(
"Pinned prompt. Custom entry mentioning /old-bundle.\n\n{}",
registered_asset_suffix(&descriptor_value["runtimeContext"]).unwrap()
@@ -12,7 +12,7 @@ pub(crate) enum RunAttachmentPolicy {
pub(crate) fn run_attachment_policy(agent: &str) -> RunAttachmentPolicy {
match agent {
"cursor" | "copilot" | "pi" | "kimi" => RunAttachmentPolicy::AuthenticatedRunGrants,
"claude" => RunAttachmentPolicy::AuthenticatedAssetPaths,
"claude" | "grok" | "codex" => RunAttachmentPolicy::AuthenticatedAssetPaths,
_ => RunAttachmentPolicy::ImmutableInstructions,
}
}
@@ -39,11 +39,17 @@ mod tests {
run_attachment_policy("claude"),
RunAttachmentPolicy::AuthenticatedAssetPaths
);
for agent in ["codex", "grok", "unknown"] {
assert_eq!(
run_attachment_policy(agent),
RunAttachmentPolicy::ImmutableInstructions
);
}
assert_eq!(
run_attachment_policy("grok"),
RunAttachmentPolicy::AuthenticatedAssetPaths
);
assert_eq!(
run_attachment_policy("codex"),
RunAttachmentPolicy::AuthenticatedAssetPaths
);
assert_eq!(
run_attachment_policy("unknown"),
RunAttachmentPolicy::ImmutableInstructions
);
}
}
@@ -1,3 +1,4 @@
import { createHash } from "node:crypto";
import { execFileSync } from "node:child_process";
import { promises as fs } from "node:fs";
import os from "node:os";
@@ -1347,7 +1348,7 @@ describe("Daytona sandbox provider plugin", () => {
});
});
it("acquires a lease from a created sandbox", async () => {
it.each([true, false])("records allocation bytes for reuseLease=%s", async reuseLease => {
process.env.DAYTONA_API_KEY = "host-key";
const sandbox = createMockSandbox();
mockCreate.mockResolvedValue(sandbox);
@@ -1363,7 +1364,7 @@ describe("Daytona sandbox provider plugin", () => {
config: {
image: "node:20",
timeoutMs: 300000,
reuseLease: true,
reuseLease,
},
});
@@ -1374,13 +1375,21 @@ describe("Daytona sandbox provider plugin", () => {
shellCommand: "bash",
sandboxId: "sandbox-123",
remoteCwd: "/home/daytona/paperclip-workspace",
reuseLease: true,
reuseLease,
workspaceSentinel: {
path: "/home/daytona/paperclip-workspace/.paperclip-runtime/reusable-sandbox-lease.json",
result: "written",
},
},
});
const uploaded = sandbox.fs.uploadFile.mock.calls.find(call => call[1] ===
"/home/daytona/paperclip-workspace/.paperclip-runtime/reusable-sandbox-lease.json")?.[0];
expect(Buffer.isBuffer(uploaded)).toBe(true);
expect(lease?.metadata?.workspaceAllocationFiles).toEqual({ providerLeaseId: "sandbox-123",
remoteCwd: "/home/daytona/paperclip-workspace", files: [{
relativePath: ".paperclip-runtime/reusable-sandbox-lease.json",
sha256: createHash("sha256").update(uploaded as Buffer).digest("hex"),
}] });
expect(sandbox.fs.createFolder).toHaveBeenCalledWith(
"/home/daytona/paperclip-workspace/.paperclip-runtime",
"755",
@@ -33,6 +33,7 @@ import type {
PluginEnvironmentGetInteractiveSetupParams,
PluginEnvironmentInteractiveSetupSession,
PluginEnvironmentLease,
PluginEnvironmentWorkspaceAllocationFiles,
PluginEnvironmentProbeParams,
PluginEnvironmentProbeResult,
PluginEnvironmentRealizeWorkspaceParams,
@@ -197,6 +198,7 @@ interface DaytonaDriverConfig {
type WorkspaceSentinelResult = {
path: string;
token: string | null;
allocationFiles?: PluginEnvironmentWorkspaceAllocationFiles;
runId?: string;
providerLeaseId?: string;
result: "written" | "matched" | "missing" | "mismatch" | "skipped";
@@ -633,23 +635,26 @@ async function writeWorkspaceSentinel(input: {
return { path: sentinelPath, token: null, result: "skipped" };
}
await input.sandbox.fs.createFolder(path.posix.dirname(sentinelPath), "755");
const content = Buffer.from(JSON.stringify({
version: 1,
token,
companyId: input.params.companyId,
environmentId: input.params.environmentId,
agentId: input.params.agentId,
executionWorkspaceId: input.params.executionWorkspaceId,
...(input.params.executionWorkspaceId ? {} : { projectlessIssueId: input.params.issueId }),
adapterType: input.params.adapterType ?? null,
provider: "daytona",
writtenAt: new Date().toISOString(),
}, null, 2), "utf8");
await input.sandbox.fs.uploadFile(
Buffer.from(JSON.stringify({
version: 1,
token,
companyId: input.params.companyId,
environmentId: input.params.environmentId,
agentId: input.params.agentId,
executionWorkspaceId: input.params.executionWorkspaceId,
...(input.params.executionWorkspaceId ? {} : { projectlessIssueId: input.params.issueId }),
adapterType: input.params.adapterType ?? null,
provider: "daytona",
writtenAt: new Date().toISOString(),
}, null, 2), "utf8"),
content,
sentinelPath,
input.timeoutSeconds,
);
return { path: sentinelPath, token, result: "written",
allocationFiles: { providerLeaseId: input.sandbox.id, remoteCwd: input.remoteCwd,
files: [{ relativePath: WORKSPACE_SENTINEL_RELATIVE_PATH, sha256: createHash("sha256").update(content).digest("hex") }] },
...(!input.config.reuseLease ? { runId: input.params.runId!, providerLeaseId: input.sandbox.id } : {}) };
}
@@ -734,6 +739,7 @@ function leaseMetadata(input: {
...(input.config.disk != null ? { disk: input.config.disk } : {}),
...(input.config.gpu != null ? { gpu: input.config.gpu } : {}),
...(input.workspaceSentinel ? { workspaceSentinel: input.workspaceSentinel } : {}),
...(input.workspaceSentinel?.allocationFiles ? { workspaceAllocationFiles: input.workspaceSentinel.allocationFiles } : {}),
};
}
+1
View File
@@ -180,6 +180,7 @@ export type {
PluginEnvironmentProbeParams,
PluginEnvironmentProbeResult,
PluginEnvironmentLease,
PluginEnvironmentWorkspaceAllocationFiles,
PluginEnvironmentAcquireLeaseParams,
PluginEnvironmentResumeLeaseParams,
PluginEnvironmentReleaseLeaseParams,
+6
View File
@@ -618,6 +618,12 @@ export interface PluginEnvironmentProbeResult {
metadata?: Record<string, unknown>;
}
export interface PluginEnvironmentWorkspaceAllocationFiles {
providerLeaseId: string;
remoteCwd: string;
files: Array<{ relativePath: string; sha256: string }>;
}
export interface PluginEnvironmentLease {
providerLeaseId: string | null;
metadata?: Record<string, unknown>;
@@ -17,6 +17,7 @@ import {
} from "@paperclipai/adapter-utils/ssh";
import {
activityLog,
workspaceOperations,
costEvents,
agents,
builtInManagedResources,
@@ -238,6 +239,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
await stopSshEnvLabFixture(path.join(root, "state.json")).catch(() => undefined);
await rm(root, { recursive: true, force: true }).catch(() => undefined);
}
await db.delete(workspaceOperations);
await db.delete(activityLog);
await db.delete(environmentLeases);
await db.delete(issues);
@@ -4441,7 +4443,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
});
});
it("does not resume released reusable plugin sandbox leases after provider config drift", async () => {
it.each(["provider config", "workspace migration"])("does not resume released reusable plugin sandbox leases after %s drift", async drift => {
const pluginId = randomUUID();
const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment();
const providerConfig = {
@@ -4488,12 +4490,13 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
await insertReusableProject(projectId, companyId);
await insertReusableExecutionWorkspace(executionWorkspaceId, companyId, projectId);
let acquisitions = 0;
const workerManager = {
isRunning: vi.fn((id: string) => id === pluginId),
call: vi.fn(async (_pluginId: string, method: string, params: any) => {
if (method === "environmentAcquireLease") {
return {
providerLeaseId: `lease-${params.config.image}`,
providerLeaseId: `lease-${params.config.image}${++acquisitions > 1 ? "-replacement" : ""}`,
metadata: {
provider: "fake-plugin",
image: params.config.image,
@@ -4532,13 +4535,27 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
});
await runtimeWithPlugin.releaseRunLeases(runId);
if (drift === "workspace migration") {
const operationId = randomUUID();
await db.insert(workspaceOperations).values({ id: operationId, companyId, executionWorkspaceId,
phase: "remote_workspace_migration", status: "succeeded", metadata: {
schema: "paperclip.remote-workspace-migration.v1", snapshotId: randomUUID(), nativeSessionContinuity: {
schema: "paperclip.native-workspace-migration.v1", remoteCwd: "/workspace", sessions: [{
sourceRunId: runId, agentId, issueId: randomUUID(), sessionId: randomUUID(), logicalCwd: "/old/controller/workspace",
}],
},
} });
await db.update(executionWorkspaces).set({ metadata: { nativeWorkspaceMigration: {
schema: "paperclip.native-workspace-migration.v1", operationId,
} } }).where(eq(executionWorkspaces.id, executionWorkspaceId));
}
const nextRunId = randomUUID();
await insertRunningManualRun(nextRunId, companyId, agentId);
const updatedEnvironment: Environment = {
...environment,
config: {
...providerConfig,
image: "template-b",
image: drift === "provider config" ? "template-b" : "template-a",
},
};
await environmentService(db).update(environment.id, {
@@ -4557,7 +4574,8 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
},
});
expect(second.lease.providerLeaseId).toBe("lease-template-b");
expect(second.lease.providerLeaseId).toBe(`lease-${updatedEnvironment.config.image}-replacement`);
expect(second.lease.metadata?.reusableSandboxLease).not.toEqual(first.lease.metadata?.reusableSandboxLease);
expect(workerManager.call).not.toHaveBeenCalledWith(
pluginId,
"environmentResumeLease",
@@ -23,6 +23,11 @@ import { verifyAgentFileObjects } from "../services/agent-file-maintenance.js";
import { getStorageService } from "../storage/index.js";
import { resolveDecisionSigningSecret } from "../services/decision-signing.js";
import { getRunLogStore } from "../services/run-log-store.js";
import { companySkillService } from "../services/company-skills.js";
import { skillPackageStore } from "../services/skill-package-store.js";
import { skillInstallationMaintenance } from "../services/skill-installation-maintenance.js";
import { migrateLocalSkillPackageObjects } from "../services/skill-package-storage-migration.js";
import { companySkills, companySkillRepositorySnapshots } from "@paperclipai/db";
// Explicit real-dependency qualification. Never contacts off-host storage in
// the default test path; use a disposable S3-compatible service and PostgreSQL.
@@ -55,6 +60,8 @@ describe.skipIf(!process.env.PAPERCLIP_TEST_S3_ENDPOINT)("existing-volume enroll
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId));
await db.delete(agents).where(eq(agents.companyId, companyId));
await db.delete(companySkills).where(eq(companySkills.companyId, companyId));
await db.delete(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.companyId, companyId));
await db.delete(companies).where(eq(companies.id, companyId));
let token: string | undefined;
do {
@@ -100,6 +107,46 @@ describe.skipIf(!process.env.PAPERCLIP_TEST_S3_ENDPOINT)("existing-volume enroll
await s3.send(new PutObjectCommand({ Bucket: bucket, Key: file.objectKey, Body: "corrupt" }));
await expect(verifyAgentFileObjects(db, getStorageService(), AbortSignal.timeout(30_000))).rejects.toThrow(/agent_file_object_(size|digest)_mismatch/);
}, 120_000);
it("preserves original skill objects and identity when destination readback fails", async () => {
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "local_disk");
const skills = await companySkillService(db).list(companyId);
const local = getStorageService(), store = skillPackageStore(db, local);
const before = await Promise.all(skills.map(async skill => {
const version = await store.version(companyId, skill.id, skill.currentVersionId!);
return { skillId: skill.id, version: version!, files: await store.versionFiles(version!) };
}));
expect(before.length).toBeGreaterThan(0);
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
const remote = getStorageService();
await expect(migrateLocalSkillPackageObjects(db, { source: local, destination: { ...remote,
getObject: async () => { throw new Error("Synthetic destination readback failed"); } }, writersStopped: true,
signal: AbortSignal.timeout(30_000) })).rejects.toThrow("Synthetic destination readback failed");
for (const point of before) {
expect((await store.descriptor(companyId, point.version.packageSnapshotId!)).storage?.provider).toBe("local_disk");
expect(await store.versionFiles(point.version)).toEqual(point.files);
}
}, 120_000);
it("verifies bundled skill migration after removal of the old server home", async () => {
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "local_disk");
const skills = await companySkillService(db).list(companyId), local = skillPackageStore(db);
const before = await Promise.all(skills.map(async skill => {
const version = await local.version(companyId, skill.id, skill.currentVersionId!);
return { skillId: skill.id, version: version!, files: await local.versionFiles(version!) };
}));
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
const result = await migrate();
expect(result.eligibility.eligible).toBe(true);
expect(result.counts.skillObjects).toBe(before.length);
await fs.rm(home, { recursive: true, force: true });
const remote = skillPackageStore(db);
for (const point of before) {
const version = await remote.version(companyId, point.skillId, point.version.id);
expect(version?.packageSnapshotId).toBe(point.version.packageSnapshotId);
expect(await remote.versionFiles(version!)).toEqual(point.files);
}
expect((await skillInstallationMaintenance(db).verifyVersions({ companyId, limit: 50 })).results
.every(row => ["remote_storage", "database_release_snapshot"].includes(row.status))).toBe(true);
}, 120_000);
it("migrates overlapping task folders through maintenance and restores both from S3 after volume loss", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "73".repeat(32));
const parent = path.join(home, "project"), child = path.join(parent, ".paperclip/worktrees/task");
@@ -1,3 +1,4 @@
import { createHash } from "node:crypto";
import { execFile as execFileCallback } from "node:child_process";
import fs from "node:fs/promises";
import os from "node:os";
@@ -2732,6 +2733,28 @@ describe("effective run session config freshness", () => {
expect(freshness(await metadata("/replacement-controller", "sha256:entry-v1", "skill-version-2")).changedCategories).toContain("runtimeSkills");
});
it("retains a legacy managed-instruction session after verified import removes the old home", async () => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), "legacy-instruction-session-"));
const body = "Preserve this exact agent identity and instructions.\n";
await fs.writeFile(path.join(root, "AGENTS.md"), body);
const config = { model: "gpt-5.4-mini", instructionsBundleMode: "managed", instructionsRootPath: root,
instructionsFilePath: path.join(root, "AGENTS.md"), instructionsEntryFile: "AGENTS.md" };
const before = await buildSessionConfigMetadata({ adapterType: "paperclip_runner", effectiveAdapterConfig: config });
await fs.rm(root, { recursive: true });
const managedInstructions = { entryFile: "AGENTS.md", contentHash: "sha256:" + createHash("sha256").update(body).digest("hex") };
const metadata = (contentHash = managedInstructions.contentHash, model = config.model, retainManagedInstructionPaths = false) =>
buildSessionConfigMetadata({ adapterType: "paperclip_runner", effectiveAdapterConfig: { ...config, model },
managedInstructions: { ...managedInstructions, contentHash }, retainManagedInstructionPaths });
const after = await metadata();
const decide = (current: SessionConfigMetadata, compatible: SessionConfigMetadata[] = []) => resolveTaskSessionConfigFreshness({
hasTaskSession: true, configuredModel: config.model, taskSessionParams: sessionParamsWithConfigMetadata(before),
configMetadata: current, compatibleConfigMetadata: compatible });
expect(decide(after)).toMatchObject({ reset: true, changedCategories: ["adapterConfig"] });
expect(decide(after, [await metadata(undefined, undefined, true)]).reset).toBe(false);
expect(decide(await metadata("sha256:changed"), [await metadata("sha256:changed", undefined, true)]).reset).toBe(true);
expect(decide(await metadata(undefined, "different-model"), [await metadata(undefined, "different-model", true)]).reset).toBe(true);
});
it("does not read unbounded legacy instructions paths for config fingerprints", async () => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-session-fingerprint-"));
const instructionsPath = path.join(root, "AGENTS.md");
@@ -1,3 +1,7 @@
import { loadConfig } from "../config.js";
import { createStorageService } from "../storage/service.js";
import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js";
import { migrateLocalSkillPackageObjects } from "../services/skill-package-storage-migration.js";
import { heartbeatRuns, type Db } from "@paperclipai/db";
import { offlineMigrationExecutionWhere } from "../services/offline-migration-execution.js";
import { getStorageService } from "../storage/index.js";
@@ -73,6 +77,15 @@ export async function migrateEphemeralServerData(db: Db, input: {
const packages = skillPackageMaintenance(db);
const skills = skillInstallationMaintenance(db);
phase("skill_objects");
const localSkillStorage = createStorageService(createLocalDiskStorageProvider(loadConfig().storageLocalDiskBaseDir));
after = undefined;
for (;;) {
const page = await migrateLocalSkillPackageObjects(db, { source: localSkillStorage, destination: storage, writersStopped: true, signal, after });
counts.skillObjects = (counts.skillObjects ?? 0) + page.migrated;
if (page.inspected < 50 || !page.after) break;
after = page.after;
}
for (const [name, migrate] of [["skill_packages", packages.migrateSnapshots], ["skill_versions", packages.migrateVersions]] as const) {
phase(name);
after = undefined;
@@ -1,3 +1,4 @@
import { readNativeWorkspaceMigration } from "./native-runtime/native-workspace-migration.js";
import { HttpError } from "../errors.js";
import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js";
import { createOwnedSandbox, recoverSandboxCreations, recoverableTaskCreation } from "./sandbox-creation-operations.js";
@@ -964,6 +965,8 @@ async function buildReusableSandboxLeaseFingerprint(input: {
companyId: input.companyId,
environment: input.environment,
});
const workspaceMigration = input.executionWorkspaceId
? await readNativeWorkspaceMigration(input.db, { companyId: input.companyId, id: input.executionWorkspaceId }) : null;
return createEffectiveRunConfigFingerprints({
lease: {
companyId: input.companyId,
@@ -972,6 +975,7 @@ async function buildReusableSandboxLeaseFingerprint(input: {
driver: input.environment.driver,
},
executionWorkspaceId: input.executionWorkspaceId,
...(workspaceMigration ? { workspaceMigrationGeneration: workspaceMigration.operationId } : {}),
...(input.executionWorkspaceId ? {} : { projectlessIssueId: input.issueId }),
agentId: input.agentId,
adapterType: input.adapterType,
@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { activityLog, agents, companies, completionContracts, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { activityLog, agents, companies, completionContracts, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
@@ -82,6 +82,54 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
const coordinators = (id: string) => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, id));
const sandbox = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", reusableLeaseConfigured: false };
it.each(["matching", "different-session", "absent"])("persists the migrated logical cwd only for the receipt's exact session: %s", async receipt => {
const sameSession = receipt === "matching";
const { input, run, agent, issue } = await fixture();
const sessionId = randomUUID(), operationId = randomUUID();
const logicalCwd = "/paperclip/legacy/task-workspace";
const marker = { schema: "paperclip.native-workspace-migration.v1", operationId };
const [workspace] = await db.insert(executionWorkspaces).values({
companyId: agent.companyId, sourceIssueId: issue.id, name: "Migrated workspace",
mode: "shared_workspace", strategyType: "project_primary", cwd: "/tmp/ephemeral/task-workspace",
providerType: "cloud_sandbox", metadata: receipt === "absent" ? {} : { nativeWorkspaceMigration: marker },
}).returning();
await db.insert(workspaceOperations).values({
id: operationId, companyId: agent.companyId, executionWorkspaceId: workspace!.id,
phase: "remote_workspace_migration", status: "succeeded", metadata: {
schema: "paperclip.remote-workspace-migration.v1", snapshotId: randomUUID(),
nativeSessionContinuity: { schema: marker.schema, remoteCwd: sandbox.remoteCwd, sessions: [{
sourceRunId: randomUUID(), agentId: agent.id, issueId: issue.id,
sessionId: sameSession ? sessionId : randomUUID(), logicalCwd,
}] },
},
});
input.workspace.persistedExecutionWorkspace = workspace!;
input.workspace.executionWorkspace.cwd = logicalCwd;
input.workspace.executionTarget = sandbox;
const prior = await selectHeartbeatRuntime(db, input);
if (!prior.selected || !prior.nativeExecution) throw new Error("Expected prior native selection");
const priorInput = { ...prior.nativeExecution, session: { ...prior.nativeExecution.session, normalizedSessionId: sessionId } };
const priorProfile = (await read(run.id)).runnerProfileJson;
await db.update(heartbeatRuns).set({ status: "succeeded", nativeSessionId: sessionId,
runnerProfileJson: { ...priorProfile, nativeExecutionInput: priorInput, sessionCheckpoint: {
backendKind: "runner", driverKind: "codex_app_server", sessionId: "retained-provider-thread",
providerSessionId: "retained-provider-thread", identity: { ...priorInput.binding, sessionId },
} },
}).where(eq(heartbeatRuns.id, run.id));
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: agent.companyId, agentId: agent.id,
issueId: issue.id, scopeKind: "issue", status: "running", nativeSessionId: sessionId,
createdAt: new Date(run.createdAt.getTime() + 1000),
}).returning();
input.run = nextRun!;
input.workspace.executionWorkspace.cwd = workspace!.cwd!;
const selected = await selectHeartbeatRuntime(db, input);
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected migrated native selection");
expect(selected.nativeExecution.session.normalizedSessionId).toBe(sessionId);
expect(selected.nativeExecution.workspace.cwd).toBe(sameSession ? logicalCwd : workspace!.cwd);
expect((await read(nextRun!.id)).runnerProfileJson?.nativeExecutionInput).toEqual(selected.nativeExecution);
expect((await read(nextRun!.id)).nativeSessionId).toBe(selected.nativeExecution.session.normalizedSessionId);
});
it.each([false, true])("persists the Cursor provider API placement and completion authority without a Computer lease (ephemeral: %s)", async ephemeral => {
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", String(ephemeral));
const { input, run, issue } = await fixture();
@@ -1,3 +1,4 @@
import { readNativeWorkspaceMigration, migratedNativeExecutionCwd } from "../native-runtime/native-workspace-migration.js";
import { resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js";
import { readClaimedNativeStopRecovery } from "../native-runtime/native-cancellation-request.js";
import { museRunnerBroker } from "../muse-runner-broker.js";
@@ -762,6 +763,8 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
codexCliVersion: codexCliVersion!,
});
}
const workspaceMigration = persistedExecutionWorkspace?.metadata?.nativeWorkspaceMigration
? await readNativeWorkspaceMigration(db, { companyId: agent.companyId, id: nativeExecutionWorkspaceId }) : null;
const buildExecution = ({ normalizedSessionId, resumedSession }: { normalizedSessionId: string; resumedSession: boolean }) =>
buildNativeExecutionInput({
agentKeyId: agentIdentity?.keyId,
@@ -817,7 +820,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
// transient workspace to the run id so the native input remains durable and replayable
// without fabricating a project-scoped execution_workspaces row.
id: nativeExecutionWorkspaceId,
cwd: executionWorkspace.cwd,
cwd: migratedNativeExecutionCwd(workspaceMigration, { agentId: agent.id, issueId: issueRef.id, sessionId: normalizedSessionId }) ?? executionWorkspace.cwd,
repoUrl: executionWorkspace.repoUrl,
repoRef: executionWorkspace.repoRef,
branchName: executionWorkspace.branchName,
@@ -152,7 +152,14 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre
: null,
};
const sessionConfigMetadata = await buildEffectiveRunSessionConfigMetadata(sessionConfigMetadataInput);
let compatibleConfigMetadata: EffectiveRunSessionConfigMetadata[] = [];
const compatibleConfigMetadata: EffectiveRunSessionConfigMetadata[] = [];
const importedManagedInstructions = Boolean(taskSession && sessionConfigMetadataInput.managedInstructions?.contentHash
&& readConfigFingerprintFromSessionParams(taskSession.sessionParamsJson)?.fingerprint !== sessionConfigMetadata.fingerprint);
if (importedManagedInstructions) {
compatibleConfigMetadata.push(await buildEffectiveRunSessionConfigMetadata({
...sessionConfigMetadataInput, retainManagedInstructionPaths: true,
}));
}
if (managedAiRuntime && aiBinding && taskSession &&
readConfigFingerprintFromSessionParams(taskSession.sessionParamsJson)?.fingerprint !== sessionConfigMetadata.fingerprint) {
const revisions = requestedAiBinding?.mode === "router"
@@ -166,7 +173,12 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre
storedIdentity: taskSession.sessionParamsJson?.paperclipAiCredentialIdentity,
sessionIdentity: managedAiRuntime.sessionIdentity, credentialIdentity: managedAiRuntime.identity, revisions,
});
compatibleConfigMetadata = await Promise.all(candidates.map(candidate => buildEffectiveRunSessionConfigMetadata({ ...sessionConfigMetadataInput, ...candidate })));
compatibleConfigMetadata.push(...await Promise.all(candidates.flatMap(candidate => [
buildEffectiveRunSessionConfigMetadata({ ...sessionConfigMetadataInput, ...candidate }),
...(importedManagedInstructions ? [buildEffectiveRunSessionConfigMetadata({
...sessionConfigMetadataInput, ...candidate, retainManagedInstructionPaths: true,
})] : []),
])));
}
const configuredModel =
readConfiguredModelFromAdapterConfig(runtimeConfig);
+2 -1
View File
@@ -2839,6 +2839,7 @@ export async function buildEffectiveRunSessionConfigMetadata(input: {
effectiveAdapterConfig: Record<string, unknown>;
managedAiHome?: string;
managedInstructions?: { entryFile: string; contentHash: string | null } | null;
retainManagedInstructionPaths?: boolean;
agentRuntimeConfig: unknown;
issueOverrides: unknown;
workspaceConfig: unknown;
@@ -2858,7 +2859,7 @@ export async function buildEffectiveRunSessionConfigMetadata(input: {
...(input.managedInstructions.contentHash !== null ? { contentHash: input.managedInstructions.contentHash } : {}),
} : await resolveInstructionsConfigFingerprintMetadata(input.effectiveAdapterConfig);
const effectiveAdapterConfig = { ...managedAiSessionFingerprintConfig(input.effectiveAdapterConfig, input.managedAiHome) };
if (input.managedInstructions) {
if (input.managedInstructions && !input.retainManagedInstructionPaths) {
delete effectiveAdapterConfig.instructionsRootPath;
delete effectiveAdapterConfig.instructionsFilePath;
}
@@ -44,6 +44,7 @@ import {
budgetReservations,
heartbeatRuns,
environmentLeases,
executionWorkspaces,
heartbeatRunEvents,
issueThreadInteractions,
issues,
@@ -2677,6 +2678,14 @@ describe("verified native harness backups", () => {
).toBe(false);
});
it("requires verified migration continuity before restoring a retained warm session into a fresh sandbox", () => {
const acquisition = { acquisitionOutcome: "created" as const, reusableLeaseConfigured: true, backupAvailable: true };
expect(shouldRestoreNativeHarnessBackupIntoSandbox(acquisition)).toBe(false);
expect(shouldRestoreNativeHarnessBackupIntoSandbox({ ...acquisition, migratedSessionContinuation: true })).toBe(true);
expect(shouldRestoreNativeHarnessBackupIntoSandbox({ ...acquisition, migratedSessionContinuation: true, backupAvailable: false })).toBe(false);
expect(shouldRestoreNativeHarnessBackupIntoSandbox({ ...acquisition, migratedSessionContinuation: true, acquisitionOutcome: "resumed" })).toBe(false);
});
it("accepts a complete digest-matched backup and rejects corruption", async () => {
const root = await mkdtemp(join(tmpdir(), "paperclip-harness-backup-"));
try {
@@ -6660,6 +6669,7 @@ function leaseDb(
: [];
const query = {
then: Promise.resolve(rows).then.bind(Promise.resolve(rows)),
leftJoin: () => query,
where: () => query,
orderBy: () => query,
for: () => query,
@@ -16811,6 +16821,100 @@ describe("runnerd provider runtime wiring", () => {
expect(runnerEnvironment).toMatchObject(paths);
});
it.each([true, false])("restores a migrated warm backup into a created lease only for its exact session (matching: %s)", async matching => {
const remoteCwd = join(isolatedStateDirectory, "remote");
const current: NativeExecutionInputV1 = { ...execution,
binding: { ...execution.binding, companyId: randomUUID(), agentId: randomUUID(), issueId: randomUUID(),
runId: randomUUID(), executionWorkspaceId: randomUUID() },
workspace: { ...execution.workspace, cwd: "/paperclip/legacy/task-workspace" },
session: { ...execution.session, normalizedSessionId: randomUUID(), lifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } },
};
const operationId = randomUUID(), runnerInstanceId = "migrated-warm-runner";
const receipt = { operationId,
workspaceMetadata: { nativeWorkspaceMigration: { schema: "paperclip.native-workspace-migration.v1", operationId } },
metadata: { schema: "paperclip.remote-workspace-migration.v1", snapshotId: randomUUID(), nativeSessionContinuity: {
schema: "paperclip.native-workspace-migration.v1", remoteCwd, sessions: [{ sourceRunId: randomUUID(),
agentId: current.binding.agentId, issueId: current.binding.issueId,
sessionId: matching ? current.session.normalizedSessionId : randomUUID(), logicalCwd: current.workspace.cwd,
}],
} },
};
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
const db = leaseDb(current, {}, {}, updates), ordinarySelect = db.select.bind(db);
db.select = ((fields?: Record<string, unknown>) => ({ from: (table: unknown) => {
if (table !== executionWorkspaces && table !== environmentLeases) return ordinarySelect(fields as never).from(table as never);
const rows = table === executionWorkspaces ? [receipt] : [{ metadata: {}, providerLeaseId: "sandbox-created",
companyId: current.binding.companyId, heartbeatRunId: current.binding.runId }];
const query = { then: Promise.resolve(rows).then.bind(Promise.resolve(rows)),
leftJoin: () => query, where: () => query, limit: () => query };
return query;
} })) as unknown as Db["select"];
const remoteExecute = vi.fn(async (command: { command: string; args?: string[] }) => {
if (command.args?.[0] === "--build-metadata") return { exitCode: 0, timedOut: false, stderr: "", stdout: JSON.stringify({
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2,
durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"],
}) };
if (command.args?.[0] === "--version") return { exitCode: 0, timedOut: false, stdout: "codex-cli 0.160.0", stderr: "" };
if (command.args?.[1]?.includes("base64")) {
try { return { exitCode: 0, timedOut: false, stdout: execFileSync("sh", command.args, { encoding: "utf8" }), stderr: "" }; }
catch { return { exitCode: 1, timedOut: false, stdout: "", stderr: "" }; }
}
return { exitCode: 0, timedOut: false, stdout: "", stderr: "" };
});
// The sandbox boundary copies actual backup bytes; migration admission and
// the executor's manifest, restored identity, and lease stamp checks stay real.
const syncIn = vi.fn(async (operations: Array<{ files: Array<{ sourcePath: string; targetPath: string }> }>) => {
for (const operation of operations) for (const file of operation.files) {
expect(file.targetPath.startsWith(`${remoteCwd}/`)).toBe(true);
await mkdir(dirname(file.targetPath), { recursive: true });
await cp(file.sourcePath, file.targetPath, { recursive: true });
}
});
await createRunnerdBackend({ db, execution: current, runnerInstanceId, runnerIngressAuthorized: true,
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd, environmentId: "environment",
leaseId: "lease-created", providerKey: "daytona", reusableLeaseConfigured: true,
effectiveCapabilities: { runnerWebSocketIngress: true },
sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "sandbox-created" },
runner: { execute: remoteExecute, syncIn },
} as never,
});
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise<void> };
const backupRoot = join(options.stateDirectory!, "failover-backups/current");
for (const directory of resolveNativeHarnessPersistenceProfile(current).directories)
await mkdir(join(backupRoot, directory.name), { recursive: true });
const runnerState = durableRunnerState({ runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId,
runnerInstanceId, environmentLeaseId: "lease-created" }, "suspended");
await writeFile(join(backupRoot, "runner/runner-state.json"), JSON.stringify(runnerState));
await writeFile(join(backupRoot, "runner/codex-provider-state.json"), JSON.stringify({
schema: "paperclip.runner.codex-provider-state.v1", lifecycle: "prepared", threadId: "retained-provider-thread",
providerSessionId: null, activeProviderTurnId: null, config: { provider: "codex", driver: "codex_app_server" },
}));
await writeFile(join(backupRoot, "codex-home/sentinel.txt"), "retained provider home");
await writeFile(join(backupRoot, "manifest.json"), JSON.stringify(buildNativeHarnessBackupManifest({
backupRoot, execution: current, runnerInstanceId, sourceProviderLeaseId: "prior-sandbox",
providerSessionIdentity: { providerSessionId: "retained-provider-thread", providerBackendSessionId: null, providerSessionIdentity: null },
})));
if (matching) {
await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined();
const remoteSession = join(remoteCwd, ".paperclip-runtime/paperclip-runner/sessions",
createHash("sha256").update(current.session.normalizedSessionId!).digest("hex"));
expect(JSON.parse(await readFile(join(remoteSession, "runner/runner-state.json"), "utf8"))).toEqual(runnerState);
expect(await readFile(join(remoteSession, "filesystem/codex-home/sentinel.txt"), "utf8")).toBe("retained provider home");
expect(updates.filter(update => update.table === environmentLeases)).toEqual([expect.objectContaining({
values: expect.objectContaining({ metadata: { nativeHarnessBackup: expect.objectContaining({
normalizedSessionId: current.session.normalizedSessionId, runnerInstanceId,
sourceProviderLeaseId: "prior-sandbox", authorizedProviderLeaseId: "sandbox-created",
}) } }),
})]);
} else {
await expect(options.prepareExternalRunnerState()).rejects.toThrow("backup_without_reusable_lease");
expect(syncIn).not.toHaveBeenCalled();
expect(updates.filter(update => update.table === environmentLeases)).toEqual([]);
}
});
it("archives failover evidence with an explicitly replaced provider session", async () => {
const remoteCwd = join(isolatedStateDirectory, "remote");
const remoteExecute = vi.fn(async (command: { command: string; args?: string[] }) => {
@@ -1,3 +1,4 @@
import { canRestoreMigratedNativeSession } from "./native-workspace-migration.js";
import { prepareCursorCloudRuntime } from "./cursor-cloud-runtime.js";
import { resolveLocalRuntimeApiUrl } from "../../runtime-api.js";
import { providerFailureFromError } from "../../vendor/paperclip-runner/index.js";
@@ -6638,10 +6639,11 @@ export function shouldRestoreNativeHarnessBackupIntoSandbox(input: {
acquisitionOutcome: "created" | "resumed" | "replacement" | null;
reusableLeaseConfigured: boolean | null | undefined;
backupAvailable: boolean;
migratedSessionContinuation?: boolean;
}): boolean {
return (
input.acquisitionOutcome === "created" &&
input.reusableLeaseConfigured === false &&
(input.reusableLeaseConfigured === false || input.migratedSessionContinuation === true) &&
input.backupAvailable
);
}
@@ -13880,6 +13882,11 @@ async function createRunnerdBackendWithinSessionClaim(
reusableLeaseConfigured:
remoteTarget.reusableLeaseConfigured,
backupAvailable,
migratedSessionContinuation: sandboxLeaseAcquisition?.outcome === "created" && backupAvailable
&& remoteTarget.reusableLeaseConfigured === true && await canRestoreMigratedNativeSession(input.db, {
...input.execution.binding, workspaceId: input.execution.binding.executionWorkspaceId,
sessionId: nativeSessionKey(input.execution), logicalCwd: input.execution.workspace.cwd, remoteCwd: remoteTarget.remoteCwd,
}),
});
await input.trace?.record({
name: "sandbox.lease.acquisition",
@@ -0,0 +1,83 @@
import path from "node:path";
import { and, eq, sql } from "drizzle-orm";
import { executionWorkspaces, heartbeatRuns, workspaceOperations, type Db } from "@paperclipai/db";
import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
import { findNativeSessionResumeRun } from "./native-session-resume.js";
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
const SCHEMA = "paperclip.native-workspace-migration.v1";
const object = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
const absolute = (value: unknown): value is string => typeof value === "string" && value !== "/"
&& !value.includes("\0") && path.posix.isAbsolute(value) && path.posix.normalize(value) === value;
const uuid = (value: unknown): value is string => typeof value === "string" && /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(value);
type Workspace = Pick<typeof executionWorkspaces.$inferSelect, "companyId" | "id">;
type SessionBinding = { sourceRunId: string; agentId: string; issueId: string; sessionId: string; logicalCwd: string };
export type NativeWorkspaceMigration = { schema: typeof SCHEMA; remoteCwd: string; sessions: SessionBinding[] };
export async function inspectNativeWorkspaceMigration(db: Db, workspace: Workspace): Promise<NativeWorkspaceMigration | null> {
const rows = await db.execute<{ agentId: string; issueId: string; sessionId: string; createdAt: Date }>(sql`
select s.agent_id as "agentId", r.native_issue_id as "issueId", r.native_session_id as "sessionId", r.created_at as "createdAt"
from agent_task_sessions s join heartbeat_runs r on r.id = s.last_run_id
and r.company_id = s.company_id and r.agent_id = s.agent_id
where s.company_id = ${workspace.companyId} and s.adapter_type = 'paperclip_runner'
and r.native_issue_id is not null and r.native_session_id is not null
and r.runner_profile_json#>>'{nativeExecutionInput,binding,executionWorkspaceId}' = ${workspace.id}
order by s.id limit 1001`);
if (rows.length > 1000) throw new Error("native_workspace_migration_session_limit");
const sessions: SessionBinding[] = [];
let remoteCwd: string | null = null;
for (const row of rows) {
const source = await findNativeSessionResumeRun(db, { companyId: workspace.companyId, agentId: row.agentId,
issueId: row.issueId, normalizedSessionId: row.sessionId, beforeCreatedAt: new Date(row.createdAt), includeProgressBarrier: true });
if (!source) continue;
const execution = parseNativeExecutionInput(source.runnerProfileJson.nativeExecutionInput);
const [saved] = await db.select({ sync: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'` })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, workspace.companyId), eq(heartbeatRuns.id, source.id)));
const sync = readNativeWorkspaceSyncReference(saved?.sync);
if (execution.binding.executionWorkspaceId !== workspace.id || execution.binding.companyId !== workspace.companyId
|| execution.binding.agentId !== row.agentId || execution.binding.issueId !== row.issueId
|| execution.session.normalizedSessionId !== row.sessionId || !absolute(execution.workspace.cwd)
|| !sync || sync.workspaceId !== workspace.id || !absolute(sync.remoteCwd)) throw new Error("native_workspace_migration_identity_unproven");
if (remoteCwd && remoteCwd !== sync.remoteCwd) throw new Error("native_workspace_migration_remote_path_conflict");
remoteCwd = sync.remoteCwd;
sessions.push({ sourceRunId: source.id, agentId: row.agentId, issueId: row.issueId, sessionId: row.sessionId, logicalCwd: execution.workspace.cwd });
}
return remoteCwd ? { schema: SCHEMA, remoteCwd, sessions } : null;
}
export async function readNativeWorkspaceMigration(db: Pick<Db, "select">, workspace: Workspace) {
const [row] = await db.select({ workspaceMetadata: executionWorkspaces.metadata, metadata: workspaceOperations.metadata, operationId: workspaceOperations.id })
.from(executionWorkspaces).leftJoin(workspaceOperations, and(
eq(workspaceOperations.companyId, executionWorkspaces.companyId), eq(workspaceOperations.executionWorkspaceId, executionWorkspaces.id),
sql`${workspaceOperations.id} = case when ${executionWorkspaces.metadata}->'nativeWorkspaceMigration'->>'operationId' ~ '^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$'
then (${executionWorkspaces.metadata}->'nativeWorkspaceMigration'->>'operationId')::uuid end`,
eq(workspaceOperations.phase, "remote_workspace_migration"), eq(workspaceOperations.status, "succeeded"),
)).where(and(eq(executionWorkspaces.companyId, workspace.companyId), eq(executionWorkspaces.id, workspace.id)));
if (!row?.workspaceMetadata?.nativeWorkspaceMigration) return null;
const migration = object(row.metadata?.nativeSessionContinuity);
if (object(row.workspaceMetadata.nativeWorkspaceMigration).schema !== SCHEMA || !row.operationId || row.metadata?.schema !== "paperclip.remote-workspace-migration.v1" || !uuid(row.metadata.snapshotId)
|| migration.schema !== SCHEMA || !absolute(migration.remoteCwd) || !Array.isArray(migration.sessions)
|| !migration.sessions.length || migration.sessions.length > 1000
|| !migration.sessions.every(value => {
const session = object(value);
return [session.sourceRunId, session.agentId, session.issueId, session.sessionId].every(uuid) && absolute(session.logicalCwd);
})) throw new Error("native_workspace_migration_receipt_invalid");
return { ...(migration as NativeWorkspaceMigration), operationId: row.operationId };
}
export function nativeWorkspaceMigrationMarker(operationId: string) {
return { schema: SCHEMA, operationId };
}
export function migratedNativeExecutionCwd(migration: NativeWorkspaceMigration | null, input: { agentId: string; issueId: string; sessionId: string }) {
return migration?.sessions.find(session => session.agentId === input.agentId && session.issueId === input.issueId
&& session.sessionId === input.sessionId)?.logicalCwd ?? null;
}
export async function canRestoreMigratedNativeSession(db: Db, input: {
companyId: string; workspaceId: string; agentId: string; issueId: string; sessionId: string; logicalCwd: string | null; remoteCwd: string;
}) {
if (input.logicalCwd === null) return false;
const migration = await readNativeWorkspaceMigration(db, { companyId: input.companyId, id: input.workspaceId });
return migration?.remoteCwd === input.remoteCwd && migratedNativeExecutionCwd(migration, input) === input.logicalCwd;
}
@@ -0,0 +1,18 @@
import type { PluginEnvironmentWorkspaceAllocationFiles } from "@paperclipai/plugin-sdk";
import type { EnvironmentLease } from "@paperclipai/shared";
export function workspaceAllocationFiles(lease: EnvironmentLease, remoteCwd: string) {
const raw = lease.metadata?.workspaceAllocationFiles;
if (raw === undefined) return [];
const receipt = raw as PluginEnvironmentWorkspaceAllocationFiles | null;
if (!receipt || receipt.providerLeaseId !== lease.providerLeaseId || receipt.remoteCwd !== remoteCwd
|| !Array.isArray(receipt.files) || receipt.files.length > 8
|| receipt.files.some(file => !file || typeof file.relativePath !== "string"
|| !/^\.paperclip-runtime\/[a-zA-Z0-9_-]+\.json$/.test(file.relativePath)
|| file.relativePath === ".paperclip-runtime/workspace-restored.json"
|| typeof file.sha256 !== "string" || !/^[a-f0-9]{64}$/.test(file.sha256))
|| new Set(receipt.files.map(file => file.relativePath)).size !== receipt.files.length) {
throw new Error("remote_workspace_allocation_receipt_invalid");
}
return receipt.files;
}
@@ -22,6 +22,8 @@ import { readNativeWorkspaceSyncReference } from "./native-runtime/native-worksp
import { isLegacyWorkspaceTaskRelated, readLegacyWorkspaceRecoveryAllocations, LEGACY_WORKSPACE_RECOVERY_MIGRATION, type LegacyWorkspaceRecoverySource } from "./legacy-workspace-recovery-migration.js";
import { offlineWorkspaceMigrationBlockers, prepareNestedWorkspaceImports, validateImportedWorkspaceChildren, workspaceMigrationLayoutBlockers } from "./remote-workspace-migration-tree.js";
import { inspectNativeWorkspaceMigration, nativeWorkspaceMigrationMarker } from "./native-runtime/native-workspace-migration.js";
const SCHEMA = "paperclip.remote-workspace-migration.v1";
type Workspace = typeof executionWorkspaces.$inferSelect;
type ReadDb = Pick<Db, "select">;
@@ -233,6 +235,7 @@ export async function migrateRemoteWorkspace(input: {
const current = await inspect(tx as unknown as Db, input);
if (current.authority) return { workspace, authority: current.authority, operation: null, source: current.source, sourceFingerprint: "", issueBinding: current.issueBinding, retainedDirectory: null, archivedGitSource: null, legacyRecoverySources: [] as LegacyWorkspaceRecoverySource[] };
if (current.reasons.length) throw new Error(`remote_workspace_migration_blocked:${current.reasons.join(",")}`);
const nativeSessionContinuity = await inspectNativeWorkspaceMigration(tx as unknown as Db, workspace);
const source = remoteWorkspaceSourceSchema.parse(input.source ?? current.source);
const sourceFingerprint = fingerprint(workspace);
const marker = current.marker;
@@ -241,6 +244,7 @@ export async function migrateRemoteWorkspace(input: {
const [operation] = await tx.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, workspace.companyId),
eq(workspaceOperations.executionWorkspaceId, workspace.id), eq(workspaceOperations.id, marker.operationId), eq(workspaceOperations.phase, "remote_workspace_migration")));
if (!operation) throw new Error("remote_workspace_migration_operation_missing");
if (canonical(operation.metadata?.nativeSessionContinuity ?? null) !== canonical(nativeSessionContinuity)) throw new Error("native_workspace_migration_source_changed");
// Older operations only admitted the task's current workspace. Preserve
// that binding when resuming an import journaled before this field existed.
const issueBinding = Object.hasOwn(operation.metadata ?? {}, "sourceIssueBinding")
@@ -254,6 +258,7 @@ export async function migrateRemoteWorkspace(input: {
const [operation] = await tx.insert(workspaceOperations).values({ companyId: workspace.companyId, executionWorkspaceId: workspace.id,
issueId: workspace.sourceIssueId, cwd: workspace.cwd ?? workspace.providerRef, phase: "remote_workspace_migration", status: "running",
metadata: { schema: SCHEMA, sourceFingerprint, source, sourceIssueBinding: current.issueBinding,
...(nativeSessionContinuity ? { nativeSessionContinuity } : {}),
nestedWorkspaceScope: { version: 1, state: "pending" },
...(current.legacyRecoverySources.length ? { legacyRecoverySources: current.legacyRecoverySources } : {}),
...(current.archivedGitSource ? { archivedGitSource: gitSourceProof(current.archivedGitSource) } : {}),
@@ -347,9 +352,12 @@ export async function migrateRemoteWorkspace(input: {
|| saved.sourceOperationId !== child.operationId) throw new Error("nested_workspace_snapshot_changed");
}
if (canonical(validation.legacyRecoverySources) !== canonical(reserved.legacyRecoverySources)) throw new Error("remote_workspace_migration_source_changed");
const continuity = await inspectNativeWorkspaceMigration(tx as unknown as Db, workspace);
if (canonical(continuity) !== canonical(operation.metadata?.nativeSessionContinuity ?? null)) throw new Error("native_workspace_migration_source_changed");
const authority: RemoteWorkspaceAuthority = { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: ref.id,
source: reserved.source, location: null, writerRunId: null, dirty: false };
const metadata: Record<string, unknown> = { ...workspace.metadata, remoteWorkspace: authority }; delete metadata.remoteWorkspaceMigration;
if (continuity) metadata.nativeWorkspaceMigration = nativeWorkspaceMigrationMarker(operation.id);
if (reserved.legacyRecoverySources.length) {
metadata.legacyWorkspaceRecoveryMigration = { schema: LEGACY_WORKSPACE_RECOVERY_MIGRATION, operationId: operation.id };
for (const source of reserved.legacyRecoverySources) {
@@ -161,6 +161,53 @@ describe("remote workspace transport safety with real sandbox commands", () => {
} finally { await captured.cleanup(); }
});
it.each(["owned", "changed", "unrelated", "symlink", "interrupted"])("preserves only verified allocation files during a migrated restore: %s", async scenario => {
await fs.writeFile(path.join(workspace, "saved.txt"), "retained user work");
const { remote, execute } = target();
const captured = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() });
try {
const root = path.join(scratch, "allocated"), relativePath = ".paperclip-runtime/reusable-sandbox-lease.json";
const bytes = JSON.stringify({ version: 1, token: "exact-allocation-token", provider: "daytona" });
await fs.mkdir(path.join(root, ".paperclip-runtime"), { recursive: true });
const sentinel = path.join(root, relativePath);
await fs.writeFile(sentinel, bytes);
const input = { target: remote, remoteCwd: root, archivePath: captured.archivePath, manifestDigest: captured.manifestDigest,
restorationIdentity: "d".repeat(64), allowEmptyRoot: true,
allocationFiles: [{ relativePath, sha256: createHash("sha256").update(bytes).digest("hex") }] };
if (scenario === "changed") await fs.writeFile(sentinel, "foreign marker");
if (scenario === "unrelated") await fs.writeFile(path.join(root, ".paperclip-runtime", "unrelated.txt"), "keep");
if (scenario === "symlink") {
const outside = path.join(scratch, "outside-marker"); await fs.writeFile(outside, bytes);
await fs.unlink(sentinel); await fs.symlink(outside, sentinel);
}
if (["changed", "unrelated", "symlink"].includes(scenario)) {
await expect(restoreRemoteWorkspaceDirectory(input)).rejects.toThrow();
expect(await fs.readFile(sentinel, "utf8")).toBe(scenario === "changed" ? "foreign marker" : bytes);
expect(await fs.lstat(path.join(root, "saved.txt")).catch(() => null)).toBeNull();
return;
}
if (scenario === "interrupted") {
const run = execute.getMockImplementation()!; let interrupt = true;
execute.mockImplementation(async command => {
if (interrupt && command.args?.[1]?.includes("remote_workspace_restore_failed")) {
interrupt = false;
return run({ ...command, args: command.args.map((arg, index) => index === 1
? arg.replace("try{await fs.rename(staged,root);}", "throw Error('interrupted');try{await fs.rename(staged,root);}") : arg) });
}
return run(command);
});
await expect(restoreRemoteWorkspaceDirectory(input)).rejects.toThrow();
expect(await fs.readFile(path.join(root + ".allocation-" + input.restorationIdentity, relativePath), "utf8")).toBe(bytes);
}
await restoreRemoteWorkspaceDirectory(input);
expect(await fs.readFile(sentinel, "utf8")).toBe(bytes);
expect(await fs.readFile(path.join(root, "saved.txt"), "utf8")).toBe("retained user work");
await fs.writeFile(path.join(root, "saved.txt"), "later user edits");
await restoreRemoteWorkspaceDirectory(input);
expect(await fs.readFile(path.join(root, "saved.txt"), "utf8")).toBe("later user edits");
} finally { await captured.cleanup(); }
});
it("keeps a completed restore available when the staging cleanup reply is lost", async () => {
await fs.writeFile(path.join(workspace, "saved.txt"), "durable synthetic files\n");
const { remote, execute } = target();
@@ -279,7 +279,8 @@ export async function captureRemoteWorkspaceArchive(input: {
}
export async function restoreRemoteWorkspaceDirectory(input: {
target: RemoteWorkspaceTarget; remoteCwd: string; archivePath: string; manifestDigest: string; restorationIdentity: string;
target: RemoteWorkspaceTarget; remoteCwd: string; archivePath: string; manifestDigest: string; restorationIdentity: string; allowEmptyRoot?: boolean;
allocationFiles?: Array<{ relativePath: string; sha256: string }>;
}) {
if (!/^[a-f0-9]{64}$/.test(input.restorationIdentity)) throw new Error("remote_workspace_restore_identity_invalid");
const id = input.restorationIdentity;
@@ -292,25 +293,54 @@ export async function restoreRemoteWorkspaceDirectory(input: {
}]);
// Publish the restored folder and its identity together. A lost successful
// reply can then adopt the same files without repeating a destructive restore.
const restore = String.raw`const fs=require('node:fs/promises'),path=require('node:path'),cp=require('node:child_process');
(async()=>{const [helper,archive,root,digest,identity]=process.argv.slice(1);
if(!/^[a-f0-9]{64}$/.test(identity))throw Error('invalid_identity');
const prior=await fs.lstat(root).catch(e=>{if(e.code==='ENOENT')return null;throw e});
if(prior){
if(!prior.isDirectory()||prior.isSymbolicLink()||await fs.realpath(root)!==root)throw Error('unsafe_root');
const runtime=path.join(root,'.paperclip-runtime'),marker=path.join(runtime,'workspace-restored.json');
const dir=await fs.lstat(runtime);if(!dir.isDirectory()||dir.isSymbolicLink())throw Error('unsafe_receipt');
const file=await fs.lstat(marker);if(!file.isFile()||file.isSymbolicLink()||file.nlink!==1||file.size>16384)throw Error('unsafe_receipt');
const restore = String.raw`const fs=require('node:fs/promises'),path=require('node:path'),cp=require('node:child_process'),crypto=require('node:crypto');
(async()=>{const [helper,archive,root,digest,identity,allowEmpty,rawFiles]=process.argv.slice(1),files=JSON.parse(rawFiles);
if(!/^[a-f0-9]{64}$/.test(identity)||!Array.isArray(files)||files.length>8)throw Error('invalid_identity');
for(const file of files)if(!/^\.paperclip-runtime\/[a-zA-Z0-9_-]+\.json$/.test(file.relativePath)
||file.relativePath==='.paperclip-runtime/workspace-restored.json'||!/^[a-f0-9]{64}$/.test(file.sha256))throw Error('invalid_allocation');
if(new Set(files.map(file=>file.relativePath)).size!==files.length)throw Error('invalid_allocation');
const stat=async p=>fs.lstat(p).catch(e=>{if(e.code==='ENOENT')return null;throw e});
const directory=async p=>{const s=await fs.lstat(p);if(!s.isDirectory()||s.isSymbolicLink()||await fs.realpath(p)!==p)throw Error('unsafe_root')};
const allocation=async p=>{
await directory(p);const names=await fs.readdir(p);
if(names.length===0&&files.length===0)return [];
if(names.length!==1||names[0]!=='.paperclip-runtime'||!files.length)throw Error('occupied_root');
const runtime=path.join(p,'.paperclip-runtime');await directory(runtime);
const contents=await fs.readdir(runtime),expected=files.map(file=>path.basename(file.relativePath));
if(contents.length!==expected.length||contents.some(name=>!expected.includes(name)))throw Error('occupied_root');
return Promise.all(files.map(async file=>{const name=path.join(p,file.relativePath),s=await fs.lstat(name);
if(!s.isFile()||s.isSymbolicLink()||s.nlink!==1||s.size>16384)throw Error('unsafe_allocation');
const content=await fs.readFile(name);if(crypto.createHash('sha256').update(content).digest('hex')!==file.sha256)throw Error('allocation_changed');
return {name:path.basename(name),content};}));
};
const parked=root+'.allocation-'+identity;
if(await stat(parked)){if(allowEmpty!=='true')throw Error('unexpected_allocation');await allocation(parked);if(!await stat(root))await fs.rename(parked,root);}
const prior=await stat(root);if(prior)await directory(root);
const marker=path.join(root,'.paperclip-runtime','workspace-restored.json');
if(prior&&await stat(marker)){
await directory(path.dirname(marker));const file=await fs.lstat(marker);
if(!file.isFile()||file.isSymbolicLink()||file.nlink!==1||file.size>16384)throw Error('unsafe_receipt');
const receipt=JSON.parse(await fs.readFile(marker,'utf8'));
if(receipt.identity!==identity||receipt.manifestDigest!==digest)throw Error('workspace_owner_changed');return;
if(receipt.identity!==identity||receipt.manifestDigest!==digest)throw Error('workspace_owner_changed');
if(await stat(parked)){await allocation(parked);await fs.rm(parked,{recursive:true});}return;
}
if(prior&&allowEmpty!=='true')throw Error('occupied_root');
const retained=prior?await allocation(root):[];
if(!prior&&files.length)throw Error('allocation_missing');
const staged=await fs.mkdtemp(root+'.restoring-');
await require('node:util').promisify(cp.execFile)(process.execPath,[helper,'restore',archive,staged,digest],{maxBuffer:1048576});
const runtime=path.join(staged,'.paperclip-runtime');await fs.mkdir(runtime,{mode:0o700});
await fs.writeFile(path.join(runtime,'workspace-restored.json'),JSON.stringify({identity,manifestDigest:digest}),{flag:'wx',mode:0o600});
if(await fs.realpath(path.dirname(root))!==path.dirname(root))throw Error('unsafe_root');await fs.rename(staged,root);
try{
await require('node:util').promisify(cp.execFile)(process.execPath,[helper,'restore',archive,staged,digest],{maxBuffer:1048576});
const runtime=path.join(staged,'.paperclip-runtime');await fs.mkdir(runtime,{mode:0o700});
for(const file of retained)await fs.writeFile(path.join(runtime,file.name),file.content,{flag:'wx',mode:0o600});
await fs.writeFile(path.join(runtime,'workspace-restored.json'),JSON.stringify({identity,manifestDigest:digest}),{flag:'wx',mode:0o600});
if(await fs.realpath(path.dirname(root))!==path.dirname(root))throw Error('unsafe_root');
if(prior){await allocation(root);if(await stat(parked))throw Error('occupied_parking');await fs.rename(root,parked);}
try{await fs.rename(staged,root);}catch(e){if(prior&&!await stat(root))await fs.rename(parked,root);throw e;}
if(prior){await allocation(parked);await fs.rm(parked,{recursive:true});}
}finally{await fs.rm(staged,{recursive:true,force:true});}
})().catch(()=>{process.stderr.write('remote_workspace_restore_failed');process.exitCode=1});`;
await command(input.target, "restore", ["-e", restore, staged.helper, staged.archive, input.remoteCwd, input.manifestDigest, input.restorationIdentity]);
await command(input.target, "restore", ["-e", restore, staged.helper, staged.archive, input.remoteCwd,
input.manifestDigest, input.restorationIdentity, String(input.allowEmptyRoot === true), JSON.stringify(input.allocationFiles ?? [])]);
// Publish the restored location before cleanup can fail, stall, or lose its
// reply. The owner journals this receipt and then reclaims disposable staging.
return { operationId, stagingRoot: parent, cleanupPending: true };
@@ -1,4 +1,7 @@
import { randomUUID } from "node:crypto";
import { parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js";
import { readNativeWorkspaceMigration, migratedNativeExecutionCwd, canRestoreMigratedNativeSession } from "./native-runtime/native-workspace-migration.js";
import { buildNativeHarnessBackupManifest, verifyNativeHarnessBackup } from "./native-runtime/native-session-executor.js";
import { createHash, randomUUID } from "node:crypto";
import { execFile, spawn } from "node:child_process";
import net from "node:net";
import { promisify } from "node:util";
@@ -8,7 +11,7 @@ import path from "node:path";
import { pathToFileURL } from "node:url";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { and, eq, sql } from "drizzle-orm";
import { activityLog, agents, agentWakeupRequests, applyPendingMigrations, closeRegisteredClients, companies, createDb, environmentLeases, environments,
import { activityLog, agents, agentTaskSessions, agentWakeupRequests, applyPendingMigrations, closeRegisteredClients, companies, createDb, environmentLeases, environments,
executionWorkspaces, executionWorkspaceSnapshots, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, livenessDb, nativeRunFinalizations, projects, projectWorkspaces, workspaceOperations } from "@paperclipai/db";
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
@@ -90,6 +93,7 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, ids.company));
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, ids.company));
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, ids.company));
await db.delete(agentTaskSessions).where(eq(agentTaskSessions.companyId, ids.company));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, ids.company));
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, ids.company));
await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.companyId, ids.company));
@@ -437,6 +441,101 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
try { expect(await fs.readFile(path.join(browse.root, "report.md"), "utf8")).toContain("useful saved result"); } finally { await browse.cleanup(); }
}, 60_000);
it.each(["per_turn", "warm", "occupied", "foreign-target", "foreign-allocation", "forged-receipt"])("preserves retained native workspace identity through migration: %s", async scenario => {
const original = path.join(root, "old-controller", "workspace"); await fs.mkdir(original, { recursive: true });
await fs.writeFile(path.join(original, "saved.txt"), "retained work");
const remote = await target("legacy-provider-root");
const [local] = await db.insert(executionWorkspaces).values({ companyId: ids.company, sourceIssueId: ids.issue,
mode: "isolated_workspace", strategyType: "project_primary", providerType: "local_fs", name: "Retained session",
cwd: original, providerRef: original }).returning();
const sessionId = randomUUID(), runnerInstanceId = randomUUID();
const execution = parseNativeExecutionInput({ schema: "paperclip.native-execution-input.v1",
provider: { kind: "codex", model: null }, binding: { companyId: ids.company, agentId: ids.agent,
issueId: ids.issue, runId: ids.run, executionWorkspaceId: local.id },
task: { identifier: "RWP-1", title: "Retained work", description: null, prompt: "Continue.", workMode: "standard" },
workspace: { cwd: original, repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: sessionId, driverKind: "codex_app_server", protocolVersion: 1,
lifecyclePolicy: { mode: scenario === "warm" ? "warm" : "per_turn", idleTimeoutMs: scenario === "warm" ? 120000 : null } },
completionContract: { id: randomUUID(), sha256: "fixture", schemaVersion: "paperclip.completion-contract.v1",
contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Save work." }] } },
credentialBindings: [], interactionResponses: [],
});
if (execution.schema !== "paperclip.native-execution-input.v1") throw new Error("Unexpected fixture schema");
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date(), nativeSessionId: sessionId,
runnerInstanceId, runnerProfileJson: { nativeExecutionInput: execution, sessionCheckpoint: { sessionId }, nativeWorkspaceSync: {
schema: "paperclip.native-workspace-sync/v3", state: "finalized", descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64),
finalHostSha256: "c".repeat(64), workspaceId: local.id, leaseId: ids.lease, providerLeaseId: `allocation-${ids.lease}`,
remoteCwd: remote.remoteCwd, resourceDisposition: "destroy",
} } }).where(eq(heartbeatRuns.id, ids.run));
await db.insert(agentTaskSessions).values({ companyId: ids.company, agentId: ids.agent, adapterType: "paperclip_runner",
taskKey: ids.issue, lastRunId: ids.run });
await db.update(issues).set({ executionWorkspaceId: local.id }).where(eq(issues.id, ids.issue));
const backupRoot = path.join(root, "private-state"), current = path.join(backupRoot, "failover-backups", "current");
await fs.mkdir(path.join(current, "runner"), { recursive: true }); await fs.mkdir(path.join(current, "codex-home"));
await fs.writeFile(path.join(current, "runner", "state.json"), "unchanged opaque provider state");
await fs.writeFile(path.join(current, "manifest.json"), JSON.stringify(buildNativeHarnessBackupManifest({ backupRoot: current,
execution, runnerInstanceId, sourceProviderLeaseId: "retained-allocation", providerSessionIdentity: {
providerSessionId: "retained-thread", providerBackendSessionId: "retained-backend", providerSessionIdentity: null } })));
expect(verifyNativeHarnessBackup({ root: backupRoot, execution, runnerInstanceId })).not.toBeNull();
await migrateRemoteWorkspace({ db, storage, companyId: ids.company, workspaceId: local.id, source: { kind: "empty" }, apply: true });
const migrated = await workspace(local.id);
const receipt = await readNativeWorkspaceMigration(db, migrated);
expect(receipt?.remoteCwd).toBe(remote.remoteCwd);
const continuation = { companyId: ids.company, workspaceId: local.id, agentId: ids.agent, issueId: ids.issue, sessionId, logicalCwd: original, remoteCwd: remote.remoteCwd };
expect(await canRestoreMigratedNativeSession(db, continuation)).toBe(true);
expect(await canRestoreMigratedNativeSession(db, { ...continuation, sessionId: randomUUID() })).toBe(false);
expect(await canRestoreMigratedNativeSession(db, { ...continuation, remoteCwd: "/foreign" })).toBe(false);
expect(migratedNativeExecutionCwd(receipt, { agentId: ids.agent, issueId: ids.issue, sessionId })).toBe(original);
expect(migratedNativeExecutionCwd(receipt, { agentId: randomUUID(), issueId: ids.issue, sessionId })).toBeNull();
await fs.rm(path.join(root, "old-controller"), { recursive: true });
ids.run = randomUUID();
await db.insert(heartbeatRuns).values({ id: ids.run, companyId: ids.company, agentId: ids.agent, nativeIssueId: ids.issue, runtimeMode: "native", status: "running" });
await db.update(environmentLeases).set({ heartbeatRunId: ids.run }).where(eq(environmentLeases.id, ids.lease));
const reserved = await reserveRemoteWorkspace({ db, storage, companyId: ids.company, issueId: ids.issue, runId: ids.run,
agentId: ids.agent, environmentId: ids.environment, existingWorkspaceId: local.id, projectId: null, projectWorkspaceId: null, source: { kind: "empty" } });
const preparation = { db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: remote };
const allocationPath = ".paperclip-runtime/reusable-sandbox-lease.json";
const allocationBytes = JSON.stringify({ version: 1, token: "owned-allocation", companyId: ids.company,
environmentId: ids.environment, agentId: ids.agent, executionWorkspaceId: local.id, provider: "daytona" });
await fs.mkdir(path.join(remote.remoteCwd, ".paperclip-runtime"));
await fs.writeFile(path.join(remote.remoteCwd, allocationPath), allocationBytes);
preparation.lease.metadata = { ...preparation.lease.metadata, workspaceAllocationFiles: {
providerLeaseId: preparation.lease.providerLeaseId, remoteCwd: remote.remoteCwd,
files: [{ relativePath: allocationPath, sha256: createHash("sha256").update(allocationBytes).digest("hex") }],
} };
if (scenario === "forged-receipt") {
await db.update(executionWorkspaces).set({ metadata: { ...reserved.metadata,
nativeWorkspaceMigration: { schema: "paperclip.native-workspace-migration.v1", operationId: randomUUID() } } }).where(eq(executionWorkspaces.id, local.id));
await expect(prepareRemoteWorkspace(preparation)).rejects.toThrow("native_workspace_migration_receipt_invalid"); return;
}
if (scenario === "occupied") {
await fs.writeFile(path.join(remote.remoteCwd, "unrelated.txt"), "do not overwrite");
await expect(prepareRemoteWorkspace(preparation)).rejects.toThrow();
expect(await fs.readFile(path.join(remote.remoteCwd, "unrelated.txt"), "utf8")).toBe("do not overwrite"); return;
}
if (scenario === "foreign-allocation") {
const receipt = preparation.lease.metadata.workspaceAllocationFiles as { providerLeaseId: string };
receipt.providerLeaseId = "another-allocation";
await expect(prepareRemoteWorkspace(preparation)).rejects.toThrow("remote_workspace_allocation_receipt_invalid");
expect(await fs.readFile(path.join(remote.remoteCwd, allocationPath), "utf8")).toBe(allocationBytes); return;
}
if (scenario === "foreign-target") {
remote.remoteCwd = path.join(root, "different-provider-root");
await expect(prepareRemoteWorkspace(preparation)).rejects.toThrow("native_workspace_migration_target_changed"); return;
}
const prepared = await prepareRemoteWorkspace(preparation);
expect(prepared.remoteRoot).toBe(remote.remoteCwd);
expect(await fs.readFile(path.join(prepared.remoteRoot, allocationPath), "utf8")).toBe(allocationBytes);
expect(await fs.readFile(path.join(prepared.remoteRoot, "saved.txt"), "utf8")).toBe("retained work");
const successor = { ...execution, workspace: { ...execution.workspace,
cwd: migratedNativeExecutionCwd(receipt, { agentId: ids.agent, issueId: ids.issue, sessionId }) ?? prepared.remoteRoot } };
expect(verifyNativeHarnessBackup({ root: backupRoot, execution: successor, runnerInstanceId })).not.toBeNull();
expect(verifyNativeHarnessBackup({ root: backupRoot, execution: { ...successor, workspace: { ...successor.workspace,
cwd: prepared.remoteRoot } }, runnerInstanceId })).toBeNull();
expect(await fs.readFile(path.join(current, "runner", "state.json"), "utf8")).toBe("unchanged opaque provider state");
}, 60_000);
it.each(["path", "file-url"] as const)("cold restores an imported secondary %s repository and acquires only a new descriptor", async sourceKind => {
const origin = path.join(root, "import-origin"), original = path.join(root, "old-checkout");
await fs.mkdir(origin); await fs.mkdir(original);
+8 -2
View File
@@ -1,3 +1,5 @@
import { workspaceAllocationFiles } from "./remote-workspace-allocation.js";
import { readNativeWorkspaceMigration } from "./native-runtime/native-workspace-migration.js";
import { createHash, randomUUID } from "node:crypto";
import { existsSync } from "node:fs";
import { mkdtemp, rm, statfs } from "node:fs/promises";
@@ -352,7 +354,10 @@ export async function prepareRemoteWorkspace(input: {
const sameAllocation = initial.location?.providerLeaseId === input.lease.providerLeaseId
&& initial.location?.environmentId === input.lease.environmentId;
if (initial.location && !sameAllocation && initial.dirty) throw conflict("The prior workspace has unsaved work; recover it before replacing its allocation.");
const remoteRoot = sameAllocation ? initial.location!.remoteCwd : path.posix.join(target.remoteCwd, ".paperclip-workspaces",
const migration = input.workspace.metadata?.nativeWorkspaceMigration
? await readNativeWorkspaceMigration(input.db, input.workspace) : null;
if (migration && migration.remoteCwd !== target.remoteCwd) throw new Error("native_workspace_migration_target_changed");
const remoteRoot = sameAllocation ? initial.location!.remoteCwd : migration?.remoteCwd ?? path.posix.join(target.remoteCwd, ".paperclip-workspaces",
publishedWorkspaceDirectory(input.workspace, initial) ?? `${input.workspace.id}-${initial.epoch}-${input.lease.id}`);
const preparationIdentity = createHash("sha256").update(JSON.stringify([input.workspace.companyId, input.workspace.id,
input.runId, initial.epoch, input.lease.id, input.lease.providerLeaseId, remoteRoot, initial.source])).digest("hex");
@@ -369,7 +374,8 @@ export async function prepareRemoteWorkspace(input: {
const archivePath = path.join(scratch, "workspace.pcws");
await loadWorkspaceSnapshotArchive({ storage: input.storage ?? getStorageService(), reference: snapshot, targetPath: archivePath });
restoredExport = await restoreRemoteWorkspaceDirectory({ target, remoteCwd: remoteRoot, archivePath, manifestDigest: snapshot.manifestDigest,
restorationIdentity: preparationIdentity });
restorationIdentity: preparationIdentity, allowEmptyRoot: migration !== null,
allocationFiles: migration ? workspaceAllocationFiles(input.lease, remoteRoot) : [] });
} finally { await rm(scratch, { recursive: true, force: true }); }
} else {
try {
@@ -0,0 +1,66 @@
import { createReadStream, createWriteStream } from "node:fs";
import { promises as fs } from "node:fs";
import { randomUUID } from "node:crypto";
import os from "node:os";
import path from "node:path";
import { pipeline } from "node:stream/promises";
import { Writable } from "node:stream";
import { and, asc, eq, gt, sql } from "drizzle-orm";
import { companySkillRepositorySnapshots as snapshots, type Db, type SkillPackageStorageReceipt } from "@paperclipai/db";
import type { StorageService } from "../storage/index.js";
import { boundedHashStream } from "./skill-package-files.js";
import { skillPackageStore } from "./skill-package-store.js";
export async function migrateLocalSkillPackageObjects(db: Db, input: {
source: StorageService; destination: StorageService; writersStopped: boolean; signal: AbortSignal; after?: string;
}) {
if (!input.writersStopped || process.env.PAPERCLIP_EPHEMERAL_MIGRATION !== "true"
|| input.source.provider !== "local_disk" || input.destination.provider !== "s3") throw new Error("offline_skill_object_migration_required");
const rows = await db.select({ id: snapshots.id, companyId: snapshots.companyId, storage: snapshots.storage })
.from(snapshots).where(and(sql`${snapshots.storage}->>'provider' = 'local_disk'`,
input.after ? gt(snapshots.id, input.after) : undefined)).orderBy(asc(snapshots.id)).limit(50);
const source = skillPackageStore(db, input.source);
for (const row of rows) {
input.signal.throwIfAborted();
const original = row.storage!;
if (!await source.verifyDurable(row.companyId, row.id, input.signal)) throw new Error("local_skill_object_verification_failed");
const token = randomUUID(), prefix = `${row.companyId}/skill-packages/${token}`;
const upload = { token, legacy: true, provider: "s3" as const, archiveKey: `${prefix}/files.tar.gz`,
manifestKey: `${prefix}/manifest.json`, expiresAt: new Date(Date.now() + 10 * 60_000).toISOString() };
const unchanged = sql`${snapshots.storage} = ${JSON.stringify(original)}::jsonb`;
const [claimed] = await db.update(snapshots).set({ upload }).where(and(eq(snapshots.companyId, row.companyId), eq(snapshots.id, row.id), unchanged,
sql`(${snapshots.upload} is null or (${snapshots.upload}->>'expiresAt')::timestamptz <= now())`)).returning({ id: snapshots.id });
if (!claimed) throw new Error("skill_object_migration_owner_busy");
const directory = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-skill-object-migration-"));
try {
for (const object of [
{ source: original.archiveKey, target: upload.archiveKey, bytes: original.archiveBytes, sha256: original.archiveSha256,
filename: "files.tar.gz", contentType: "application/gzip" },
{ source: original.manifestKey, target: upload.manifestKey, bytes: original.manifestBytes, sha256: original.manifestSha256,
filename: "manifest.json", contentType: "application/json" },
]) {
const filename = path.join(directory, object.filename);
const read = await input.source.getObject(row.companyId, object.source, { signal: input.signal });
const originalDigest = boundedHashStream(object.bytes);
await pipeline(read.stream, originalDigest.stream, createWriteStream(filename, { flags: "wx", mode: 0o600 }), { signal: input.signal });
const actual = originalDigest.result();
if (actual.bytes !== object.bytes || actual.sha256 !== object.sha256) throw new Error("local_skill_object_changed");
await input.destination.putFile({ companyId: row.companyId, namespace: "skill-packages", objectKey: object.target,
originalFilename: object.filename, contentType: object.contentType, body: createReadStream(filename),
byteSize: object.bytes, sha256: object.sha256, signal: input.signal });
const saved = await input.destination.getObject(row.companyId, object.target, { signal: input.signal });
const remoteDigest = boundedHashStream(object.bytes);
await pipeline(saved.stream, remoteDigest.stream, new Writable({ write(_chunk, _encoding, done) { done(); } }), { signal: input.signal });
const verified = remoteDigest.result();
if (verified.bytes !== object.bytes || verified.sha256 !== object.sha256) throw new Error("remote_skill_object_verification_failed");
}
input.signal.throwIfAborted();
const receipt: SkillPackageStorageReceipt = { ...original, provider: "s3", archiveKey: upload.archiveKey, manifestKey: upload.manifestKey };
const [published] = await db.update(snapshots).set({ storage: receipt, upload: null, migrationAt: null })
.where(and(eq(snapshots.companyId, row.companyId), eq(snapshots.id, row.id), unchanged,
sql`${snapshots.upload}->>'token' = ${token}`)).returning({ id: snapshots.id });
if (!published) throw new Error("skill_object_migration_owner_changed");
} finally { await fs.rm(directory, { recursive: true, force: true }); }
}
return { inspected: rows.length, migrated: rows.length, after: rows.at(-1)?.id ?? null };
}
+16
View File
@@ -3492,3 +3492,19 @@ separate BROKEN gap shape and must not be used for this recovery bundle. Final
coverage must match all selected case IDs and preserve the separate gap count.
This mode prepares data only. It does not upload, start jobs, reopen launches,
change credentials, or change evaluation and evidence limits.
# Legacy workspace migration
`legacy-workspace-migration` and `legacy-workspace-migration-warm` are explicit,
paid, two-turn Daytona workflows for registered native production profiles. They
create a private reference file and ask a question on a local-disk controller.
The supervisor stops that controller, runs the packaged offline migration with
owned PostgreSQL and S3, verifies the export, starts an empty-home controller,
and deletes the original home. The browser answers the existing question. The
oracle requires the same native and provider session, workspace addresses and
file bytes, plus the exact final document. Provider session identifiers are
hashed in the added evidence. The company budget is $10 and automatic retries
are disabled. Existing fixture teardown owns sandboxes, credentials, processes,
database and storage; failed private evidence is retained under the usual policy.
This proves preservation during initial enrollment, which has an offline step.
Use the separate rolling deployment suites for uninterrupted deployment proof.
+3 -3
View File
@@ -312,11 +312,11 @@ describe("runner E2E catalog", () => {
expect(localIntegrityTasks).toHaveLength(2);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
1, 9, 9, 4, 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2,
1, 9, 9, 9, 9, 4, 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2,
]);
expect(validateRunnerCatalog()).toHaveLength(819);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(819);
expect(validateRunnerCatalog()).toHaveLength(837);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(837);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(48);
+15
View File
@@ -1272,6 +1272,21 @@ const rollingDeploymentSuites: RunnerSuiteFixture[] = [
export const runnerSuites: readonly RunnerSuiteFixture[] = [
remoteWorkspaceStorageSuite,
...rollingDeploymentSuites,
...[
{ id: "legacy-workspace-migration", environment: runnerEnvironments.find(environment => environment.id === "daytona")! },
{ id: "legacy-workspace-migration-warm", environment: daytonaControllerOutageEnvironment },
].map(({ id, environment }): RunnerSuiteFixture => ({
id, label: `Legacy native session migration (${environment.configurationKey ? "warm" : "per-turn"})`, manualOnly: true,
description: "Resume an existing native session and workspace after the production offline migration and deletion of the original controller home.",
groups: [], profiles: rollingDeploymentProfiles, environments: [environment],
tasks: [{ ...necessaryQuestionTask, minimumExpectedRunCount: 2, turnTimeoutMs: 6 * 60_000,
attemptTimeoutMs: { local: 15 * 60_000, daytona: 20 * 60_000 } }],
expectedMatrixSize: rollingDeploymentProfiles.length,
definitionMetadata: { version: 1, scheduling: "explicit-only", maximumAttemptsPerCell: 1, providerTurns: 2, budgetMonthlyCents: 1000,
lifecycleCoverage: environment.configurationKey ? "warm-reusable" : "per-turn-disposable",
migration: "packaged-offline-entry-verified-export-delete-entire-source-home", authentication: "api-key", stagingQualification: false,
grading: "same-native-and-provider-session-logical-and-remote-paths-reference-bytes-question-and-final-document" },
})),
{
id: CURSOR_CLOUD_SUITE, label: "Native Cursor Cloud", manualOnly: true, groups: [],
description: "Local Paperclip controller with hosted Cursor work: verified artifact, same-agent follow-up, real active-run Stop and semantic plan revision/approval.",
+17 -2
View File
@@ -1,4 +1,5 @@
import { gradeNecessaryQuestion } from "./necessary-question.js";
import { gradeLegacyWorkspaceMigration, migrationPrimaryOutputCheckpoints, isLegacyWorkspaceMigration, legacyWorkspaceMigrationPrompt, readMigrationPoint, type MigrationPoint } from "./legacy-workspace-migration.js";
import { necessaryQuestionContract } from "./continuation-cases.js";
import { validResumeQuestionForm, validResumeReferenceForm } from "./question-resume-scoring.js";
import { answerResumeQuestionThroughUi } from "./question-resume-ui.js";
@@ -52,6 +53,8 @@ export async function runContinuationFlow(input: {
evidence(name: string, value: unknown): Promise<void>;
}) {
const { page, api, fixtures, execution } = input;
const migration = isLegacyWorkspaceMigration(execution);
let migrationBefore: MigrationPoint | undefined, migrationAfter: MigrationPoint | undefined;
const lifecycleProbe = lifecycleLiveCase(execution.task.id);
const scenario = lifecycleProbe
? lifecycleLiveContinuation(execution.task.id, input.nonce)
@@ -90,6 +93,7 @@ export async function runContinuationFlow(input: {
const idle =
continuationAnswerCommitted(state.interactions, answeredInteractionId) &&
continuationCheckpointReady({
requireCompletedTurn: migration,
issue: { status: state.issue.status, executionRunId: state.issue.executionRunId },
runs: state.runs.map(r => ({ id: r.id, status: r.status, runtimeMode: r.runtimeMode })),
interactions: state.interactions,
@@ -256,7 +260,7 @@ export async function runContinuationFlow(input: {
issuePrefix: fixtures.company.issuePrefix!,
agentName: fixtures.agent.name,
title: execution.task.buildTitle(input.nonce),
prompt: scenario.prompt,
prompt: migration ? legacyWorkspaceMigrationPrompt(scenario.prompt) : scenario.prompt,
workMode: "standard",
});
issue = await pollUntil({
@@ -272,6 +276,12 @@ export async function runContinuationFlow(input: {
await settle(new Set(), scenario.id !== "revision-preserves-approval");
await snapshot("initial");
assertWaiting();
if (migration) {
migrationBefore = await readMigrationPoint({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id, issueId: issue.id });
await input.evidence("legacy-migration-before.json", migrationBefore);
await input.restart();
await open();
}
if (scenario.id === "untrusted-evidence") {
const parentRun = runs.find((r) => r.contextSnapshot?.issueId === issue!.id);
await seedContinuationContext({
@@ -300,14 +310,19 @@ export async function runContinuationFlow(input: {
assertWaiting();
await reply(scenario.approval);
}
if (migration) migrationAfter = await readMigrationPoint({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id, issueId: issue!.id });
await snapshot("final");
} finally {
checks = gradeContinuation({
...scenario,
checkpoints,
checkpoints: migration ? migrationPrimaryOutputCheckpoints(checkpoints, migrationBefore, migrationAfter) : checkpoints,
runtimeMode: execution.profile.expectedRuntimeMode,
});
checks.push(...gradeLifecycleBaseline(checkpoints));
if (migration) {
checks.push(...gradeLegacyWorkspaceMigration(migrationBefore, migrationAfter, checkpoints));
await input.evidence("legacy-migration.json", { before: migrationBefore, after: migrationAfter, checks: gradeLegacyWorkspaceMigration(migrationBefore, migrationAfter, checkpoints) });
}
if (scenario.id === "necessary-free-text") checks.push(...gradeNecessaryQuestion(checkpoints, scenario.answer));
if (lifecycleProbe) checks.push(gradeLifecycleNarrative({
narrative: lifecycleProbe.narrative,
@@ -51,6 +51,12 @@ describe("checkpoint finalization boundary", () => {
runs: [{ ...run, status: "running" }], interactions: [{ ...pending[0], sourceRunId: "run",
payload: { runtimeRequestId: "provider-request" } }] };
expect(continuationCheckpointReady(input)).toBe(true);
expect(continuationCheckpointReady({ ...input, requireCompletedTurn: true })).toBe(false);
expect(continuationCheckpointReady({ ...input, requireCompletedTurn: true,
runs: [run], issue: { status: "in_review", executionRunId: "run" } })).toBe(false);
expect(continuationCheckpointReady({ ...input, requireCompletedTurn: true,
runs: [run], issue: { status: "in_review", executionRunId: null } })).toBe(true);
expect(continuationCheckpointReady({ ...input, interactions: pending })).toBe(false);
expect(continuationCheckpointReady({ ...input, issue: { ...input.issue, executionRunId: "other" } })).toBe(false);
expect(continuationCheckpointReady({ ...input, runs: [{ ...input.runs[0], runtimeMode: "legacy" }] })).toBe(false);
+2 -1
View File
@@ -13,6 +13,7 @@ export function continuationAnswerCommitted(interactions: ReadonlyArray<{ id?: u
/** Run completion precedes task-lock release. Observe both before taking a
* checkpoint, even when the terminal run status is stable across many polls. */
export function continuationCheckpointReady(input: {
requireCompletedTurn?: boolean;
issue: { status: string; executionRunId?: string | null };
runs: ReadonlyArray<{ id: string; status: string; runtimeMode?: string }>;
interactions: ReadonlyArray<{ id?: unknown; kind?: unknown; status?: unknown;
@@ -25,7 +26,7 @@ export function continuationCheckpointReady(input: {
const waiting = pending.some(i => ["ask_user_questions", "request_confirmation", "request_approval"].includes(String(i.kind)));
return input.issue.executionRunId === null && (!waiting || input.issue.status === "in_review");
}
if (active.length !== 1) return false;
if (input.requireCompletedTurn || active.length !== 1) return false;
const run = active[0];
return run.status === "running" && run.runtimeMode === "native" &&
input.issue.executionRunId === run.id && ["in_progress", "in_review"].includes(input.issue.status) &&
+3 -2
View File
@@ -1,3 +1,4 @@
import { isLegacyWorkspaceMigration } from "./legacy-workspace-migration.js";
import { isRollingWebDeployment } from "./controller-rolling-deployment.js";
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
@@ -8,7 +9,7 @@ import { afterEach, beforeEach, expect, it, vi } from "vitest";
import { runnerMatrix } from "./catalog.js";
import { assertInstalledCliSelection, installedCliKeys, probeInstalledCodexStartup, verifyInstalledCli } from "./installed-cli.js";
import { buildPaperclipServerEnvironment, buildRunnerE2EProcessEnvironment } from "./harness-env.js";
const cells = runnerMatrix.filter(e => !isRollingWebDeployment(e) && e.profile.id === "runner-acpx-pi");
const cells = runnerMatrix.filter(e => !isRollingWebDeployment(e) && !isLegacyWorkspaceMigration(e) && e.profile.id === "runner-acpx-pi");
const roots: string[] = [];
// pnpm's source test wrapper sets NODE_PATH; the disposable public-install fixture must start closed.
beforeEach(() => { vi.stubEnv("NODE_PATH", undefined); vi.stubEnv("NODE_OPTIONS", undefined); });
@@ -47,7 +48,7 @@ it("admits existing Codex onboarding, useful-file revision and restart journeys
});
it("admits only the 20 agreed Copilot cells through normal installed resolution with no qualification override", async () => {
const { env, server } = await fixture();
const selected = runnerMatrix.filter(e => !isRollingWebDeployment(e) && e.profile.id === "runner-acpx-copilot");
const selected = runnerMatrix.filter(e => !isRollingWebDeployment(e) && !isLegacyWorkspaceMigration(e) && e.profile.id === "runner-acpx-copilot");
expect(selected).toHaveLength(20);
expect(selected.filter(e => e.environment.id === "local")).toHaveLength(10);
expect(selected.filter(e => e.environment.id === "daytona")).toHaveLength(10);
@@ -1,3 +1,4 @@
import { isLegacyWorkspaceMigration } from "./legacy-workspace-migration.js";
import { isRollingWebDeployment } from "./controller-rolling-deployment.js";
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
@@ -37,7 +38,7 @@ it("admits pinned published plugin and resolved SDK graph without claiming a ful
});
it("admits all ten Copilot Daytona cells only with the pinned normal plugin graph", async () => {
const f = await fixture();
const selected = runnerMatrix.filter(e => !isRollingWebDeployment(e) && e.profile.id === "runner-acpx-copilot" && e.environment.id === "daytona");
const selected = runnerMatrix.filter(e => !isRollingWebDeployment(e) && !isLegacyWorkspaceMigration(e) && e.profile.id === "runner-acpx-copilot" && e.environment.id === "daytona");
expect(selected).toHaveLength(10);
for (const selectedCell of selected) {
expect(await verifyInstalledDaytonaPlugin(f.env, [selectedCell])).toMatchObject({ packageRoot: f.authority.plugin.root, sourceFallback: false, fullGraphVerified: false, completeGraphAuthorityRequired: true });
@@ -0,0 +1,124 @@
import { spawn, type ChildProcess } from "node:child_process";
import { randomBytes, randomUUID } from "node:crypto";
import { createWriteStream } from "node:fs";
import { mkdir, readFile, rename, rm, writeFile } from "node:fs/promises";
import path from "node:path";
import { setTimeout as delay } from "node:timers/promises";
import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js";
import { makeDirectoryTreeRemovable } from "./directory-cleanup.js";
import { runnerE2EServerControlPaths } from "./harness-env.js";
import { reserveRunnerE2EServerPort } from "./ports.js";
import { startRollingDeploymentProcess } from "./rolling-deployment-process.js";
import { startRollingDeploymentProxy } from "./rolling-deployment-proxy.js";
import { readRestartRequest } from "./server-restart-request.js";
import { createRunnerE2EServerStopper } from "./server-stop.js";
import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js";
type Server = Awaited<ReturnType<typeof startRollingDeploymentProcess>>;
export async function superviseLegacyWorkspaceMigration(input: {
temporaryRoot: string; repositoryRoot: string; configPath: string; port: number;
environment: NodeJS.ProcessEnv; executionIds: string[]; cancelled: () => boolean;
}) {
const config = paperclipConfigSchema.parse(JSON.parse(await readFile(input.configPath, "utf8")));
if (input.executionIds.length !== 1 || config.database.mode !== "postgres" || !config.database.connectionString
|| input.environment.PAPERCLIP_STORAGE_PROVIDER !== "s3") throw new Error("Migration qualification requires one owned Postgres/S3 cell");
const databaseUrl = config.database.connectionString, controls = runnerE2EServerControlPaths(input.temporaryRoot);
const proxy = await startRollingDeploymentProxy(input.port), servers: Server[] = [];
const environment: NodeJS.ProcessEnv = { ...input.environment,
PAPERCLIP_DATABASE_LISTEN_URL: databaseUrl, PAPERCLIP_SECRETS_MASTER_KEY: randomBytes(32).toString("base64"),
HEARTBEAT_SCHEDULER_ENABLED: "true", PAPERCLIP_EPHEMERAL_SERVER: "true",
PAPERCLIP_DURABLE_AGENT_FILES: "true", PAPERCLIP_SHARED_UI_ASSETS: "true",
RUN_LOG_S3_BUCKET: input.environment.PAPERCLIP_STORAGE_S3_BUCKET,
RUN_LOG_S3_ENDPOINT: input.environment.PAPERCLIP_STORAGE_S3_ENDPOINT,
RUN_LOG_S3_REGION: input.environment.PAPERCLIP_STORAGE_S3_REGION,
RUN_LOG_S3_FORCE_PATH_STYLE: input.environment.PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE };
const start = async (name: string, env: NodeJS.ProcessEnv) => {
const server = await startRollingDeploymentProcess({ root: input.temporaryRoot, name, repositoryRoot: input.repositoryRoot,
databaseUrl, ingressOrigin: proxy.origin, sharedEnvironment: env, browser: {} });
servers.push(server); return server;
};
const json = async (filename: string, value: unknown) => {
await mkdir(path.dirname(filename), { recursive: true });
await writeFile(`${filename}.tmp`, JSON.stringify(value), { mode: 0o600 }); await rename(`${filename}.tmp`, filename);
};
const settings = async (enabled: boolean) => {
const response = await fetch(`${proxy.origin}/api/instance/settings/experimental`, { method: "PATCH",
headers: { "content-type": "application/json", origin: proxy.origin },
body: JSON.stringify({ remoteWorkspaceStorageEnabled: enabled, enableExperimentalFileViewer: true }), signal: AbortSignal.timeout(10_000) });
if (!response.ok) throw new Error(`Migration fixture settings returned ${response.status}`);
await response.body?.cancel();
};
let maintenance: { stop(): Promise<void> } | undefined;
try {
const keyPath = path.resolve(config.secrets.localEncrypted.keyFilePath);
if (!keyPath.startsWith(`${path.resolve(input.temporaryRoot)}${path.sep}`)) throw new Error("Migration fixture key escaped its root");
await mkdir(path.dirname(keyPath), { recursive: true, mode: 0o700 });
await writeFile(keyPath, environment.PAPERCLIP_SECRETS_MASTER_KEY!, { mode: 0o600, flag: "wx" });
const legacyEnvironment: NodeJS.ProcessEnv = { ...environment, PAPERCLIP_EPHEMERAL_SERVER: "false", PAPERCLIP_DURABLE_AGENT_FILES: "false",
PAPERCLIP_SHARED_UI_ASSETS: "false", PAPERCLIP_STORAGE_PROVIDER: "local_disk" };
for (const name of Object.keys(legacyEnvironment)) if (name.startsWith("RUN_LOG_S3_")) delete legacyEnvironment[name];
let current = await start("legacy-controller", legacyEnvironment);
await current.waitForReady(); proxy.routeTo(current.origin); await settings(false);
let migrated = false, lastRequest: string | undefined;
while (!input.cancelled()) {
if (current.child.exitCode !== null || current.child.signalCode !== null) throw new Error("Migration controller exited unexpectedly");
const request = await readRestartRequest(controls.restartRequestPath, input.executionIds);
if (!request || request.requestId === lastRequest) { await delay(100); continue; }
lastRequest = request.requestId;
try {
if (migrated || request.rollingDeployment || request.controllerOutage || request.preserveRunner || request.crashAfterResult) {
throw new Error("Legacy migration permits exactly one offline enrollment");
}
await settings(true);
const previous = current; await previous.stop({ requireGraceful: true });
const port = await reserveRunnerE2EServerPort(), token = randomBytes(32).toString("base64url"), id = randomUUID();
const log = createWriteStream(path.join(input.temporaryRoot, "legacy-maintenance.log"), { mode: 0o600 });
const errors = new WeakMap<ChildProcess, Error>();
const stopper = createRunnerE2EServerStopper({ gracefulTimeoutMs: 15_000, forcedTimeoutMs: 5_000,
hasSpawnError: child => errors.has(child), markExpectedStop() {}, log: message => log.write(message) });
const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(input.repositoryRoot,
path.join(previous.home, "app/server/dist/maintenance/ephemeral-migration.js")), {
cwd: input.repositoryRoot, detached: process.platform !== "win32", stdio: ["ignore", "pipe", "pipe"],
env: { ...environment, HOME: previous.home, PAPERCLIP_HOME: previous.home, PAPERCLIP_CONFIG: previous.configPath,
PAPERCLIP_INSTANCE_ID: "default", DATABASE_URL: databaseUrl, PORT: String(port),
PAPERCLIP_EPHEMERAL_MIGRATION: "true", PAPERCLIP_EPHEMERAL_MIGRATION_ID: id, PAPERCLIP_EPHEMERAL_MIGRATION_TOKEN: token } });
child.on("error", error => errors.set(child, error));
child.stdout!.pipe(log, { end: false }); child.stderr!.pipe(log, { end: false });
await stopper.watch(child);
maintenance = { async stop() { await stopper.stopAll(); await new Promise<void>(resolve => log.end(resolve)); } };
const endpoint = `http://127.0.0.1:${port}`;
const headers = { authorization: `Bearer ${token}` };
const deadline = Date.now() + 180_000;
let receipt: Record<string, any> | undefined;
while (!input.cancelled() && Date.now() < deadline) {
if (errors.has(child) || child.exitCode !== null || child.signalCode !== null) throw new Error("Maintenance exited before a verified export");
receipt = await fetch(`${endpoint}/status`, { headers, signal: AbortSignal.timeout(2_000) })
.then(async response => response.ok ? await response.json() as Record<string, any> : undefined).catch(() => undefined);
if (receipt?.state === "blocked") throw new Error(`Migration blocked: ${receipt.failure}`);
if (receipt?.state === "ready") break;
await delay(250);
}
if (receipt?.state !== "ready" || receipt.id !== id || receipt.eligibility?.eligible !== true) throw new Error("Maintenance did not qualify migration");
const exported = await fetch(`${endpoint}/export`, { method: "POST", headers, signal: AbortSignal.timeout(10_000) });
if (!exported.ok) throw new Error(`Migration export returned ${exported.status}`);
const data = await exported.json() as Record<string, any>;
if (data.id !== id || typeof data.decisionSigningSecret !== "string" || data.decisionSigningSecret.length < 32) throw new Error("Migration key export is incomplete");
environment.PAPERCLIP_DECISION_SIGNING_SECRET = data.decisionSigningSecret;
await json(path.join(controls.controlDirectory, "legacy-migration-receipt.json"), receipt);
await maintenance.stop(); maintenance = undefined;
current = await start("ephemeral-controller", environment); await current.waitForReady(); proxy.routeTo(current.origin);
await makeDirectoryTreeRemovable(previous.home); await rm(previous.home, { recursive: true });
migrated = true;
await json(controls.restartAcknowledgementPath, { requestId: request.requestId, status: "ready", controllerCacheDiscarded: true,
completedAt: new Date().toISOString(), legacyMigration: { ...receipt, oldHomeDeleted: true } });
} catch (error) {
await json(controls.restartAcknowledgementPath, { requestId: request.requestId, status: "failed", message: String(error) }); throw error;
}
}
} finally {
try { await maintenance?.stop(); } finally {
try { await Promise.all(servers.map(server => server.stop())); } finally { await proxy.close(); }
}
}
}
@@ -0,0 +1,48 @@
import { createHash } from "node:crypto";
import type { ContinuationCheckpoint } from "./continuation-scoring.js";
import { describe, expect, it } from "vitest";
import { runnerMatrix } from "./catalog.js";
import { gradeLegacyWorkspaceMigration, migrationPrimaryOutputCheckpoints, isLegacyWorkspaceMigration, type MigrationPoint } from "./legacy-workspace-migration.js";
import { remoteWorkspaceStorageServerEnvironment } from "./remote-workspace-storage.js";
const reference = "a".repeat(32) + "\n", referenceSha256 = createHash("sha256").update(reference).digest("hex");
const checkpoints = [{ phase: "final", attachments: [{ originalFilename: "migration-reference.txt", body: reference,
contentSha256: referenceSha256, contentVerified: true }] }] as ContinuationCheckpoint[];
const before: MigrationPoint = { workspaceId: "workspace", providerType: "isolated_worktree", migrated: false,
proofSha256: referenceSha256, runId: "first", sessionId: "session", providerSessionSha256: "provider-session",
logicalCwd: "/old-controller/workspace", remoteCwd: "/home/daytona/paperclip-workspace" };
const after: MigrationPoint = { ...before, runId: "second", migrated: true, providerType: "cloud_sandbox" };
describe("legacy workspace migration qualification", () => {
it("requires the preserved session and file, not only a successful new turn", () => {
expect(gradeLegacyWorkspaceMigration(before, after, checkpoints).every(check => check.passed)).toBe(true);
for (const patch of [{ proofSha256: "recreated" }, { sessionId: "fresh" }, { providerSessionSha256: "fresh" },
{ logicalCwd: "/new-home/workspace" }, { remoteCwd: "/new-sandbox/workspace" }, { workspaceId: "other" },
{ runId: before.runId }, { migrated: false }]) {
expect(gradeLegacyWorkspaceMigration(before, { ...after, ...patch }, checkpoints).every(check => check.passed)).toBe(false);
}
expect(gradeLegacyWorkspaceMigration(before).every(check => check.passed)).toBe(false);
});
it("excludes only the one verified preserved reference from the primary output count", () => {
expect(migrationPrimaryOutputCheckpoints(checkpoints, before, after)[0].attachments).toEqual([]);
const original = checkpoints[0].attachments[0] as Record<string, unknown>;
for (const patch of [{ body: "changed" }, { contentVerified: false }, { originalFilename: "unrequested.txt" }]) {
const changed = [{ ...checkpoints[0], attachments: [{ ...original, ...patch }] }];
expect(migrationPrimaryOutputCheckpoints(changed, before, after)).toEqual(changed);
expect(gradeLegacyWorkspaceMigration(before, after, changed).every(check => check.passed)).toBe(false);
}
const duplicated = [{ ...checkpoints[0], attachments: [original, { ...original }] }];
expect(migrationPrimaryOutputCheckpoints(duplicated, before, after)).toEqual(duplicated);
expect(gradeLegacyWorkspaceMigration(before, after, duplicated).every(check => check.passed)).toBe(false);
});
it("registers explicit warm and per-turn production profiles with isolated object storage", () => {
const cells = runnerMatrix.filter(isLegacyWorkspaceMigration);
expect(cells.length).toBeGreaterThan(0);
expect(new Set(cells.map(cell => cell.suite.id)).size).toBe(2);
expect(cells.every(cell => cell.suite.manualOnly && cell.task.flow === "continuation" && cell.task.expectedRunCount === 2)).toBe(true);
expect(remoteWorkspaceStorageServerEnvironment({ executions: [cells[0]!], instanceId: "migration-test",
env: { PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_BUCKET: "owned-migration-bucket" } })).toMatchObject({ PAPERCLIP_STORAGE_PROVIDER: "s3" });
});
});
@@ -0,0 +1,71 @@
import type { ContinuationCheckpoint } from "./continuation-scoring.js";
import { createHash } from "node:crypto";
import type { ExecutionWorkspace, WorkspaceFileContent } from "../../packages/shared/src/index.js";
import type { RunnerApi } from "./api.js";
import type { StoryCheck } from "./everyday-observations.js";
import type { MatrixExecution } from "./types.js";
export function isLegacyWorkspaceMigration(execution: MatrixExecution) {
return ["legacy-workspace-migration", "legacy-workspace-migration-warm"].includes(execution.suite.id);
}
export function legacyWorkspaceMigrationPrompt(prompt: string) {
const continuation = prompt.replace("Do not create other tasks or deliverables.", "Do not create other tasks.");
return `Work in this task's own workspace. Before asking the question, create migration-reference.txt containing a fresh random 32-character lowercase hexadecimal reference and a trailing newline. Do not deliver any files or documents before I answer. After I answer, read this same file, keep its original bytes, and attach the original migration-reference.txt as an additional deliverable alongside the confirmation document. Include a clickable Markdown link to the saved confirmation document in your final reply; its title or key alone is not a link. ${continuation}`;
}
export interface MigrationPoint {
workspaceId: string; providerType: string; migrated: boolean; proofSha256: string;
runId: string; sessionId: string; providerSessionSha256: string; logicalCwd: string; remoteCwd: string;
}
export async function readMigrationPoint(input: { api: Pick<RunnerApi, "get">; companyId: string; agentId: string; issueId: string }): Promise<MigrationPoint> {
const { workspace } = await input.api.get<{ workspace: ExecutionWorkspace | null }>(`/api/issues/${input.issueId}/workspace`);
if (!workspace || workspace.companyId !== input.companyId) throw new Error("Migration oracle has no bound workspace");
const runs = await input.api.get<Array<{ runId: string; status: string; agentId: string; finishedAt: string }>>(`/api/issues/${input.issueId}/runs`);
const latest = runs.filter(run => run.agentId === input.agentId && run.status === "succeeded")
.sort((a, b) => b.finishedAt.localeCompare(a.finishedAt))[0];
if (!latest || runs.some(run => ["queued", "running"].includes(run.status))) throw new Error("Migration requires a settled retained turn");
const run = await input.api.get<Record<string, any>>(`/api/heartbeat-runs/${latest.runId}`);
const execution = run.runnerProfileJson?.nativeExecutionInput;
const reference = run.runnerProfileJson?.nativeWorkspaceSync ?? run.runnerProfileJson?.remoteWorkspace;
if (run.companyId !== input.companyId || run.agentId !== input.agentId || run.nativeIssueId !== input.issueId
|| run.runtimeMode !== "native" || !run.nativeSessionId || typeof run.sessionIdAfter !== "string" || !run.sessionIdAfter
|| execution?.binding?.executionWorkspaceId !== workspace.id
|| !execution?.workspace?.cwd || !reference?.remoteCwd) throw new Error("Migration run identity is incomplete");
const file = await input.api.get<WorkspaceFileContent>(`/api/issues/${input.issueId}/file-resources/content?workspace=execution&path=migration-reference.txt`);
if (file.resource?.workspaceId !== workspace.id || file.content?.encoding !== "utf8"
|| !/^[a-f0-9]{32}\n$/.test(file.content.data)) throw new Error("Migration reference bytes are missing");
return { workspaceId: workspace.id, providerType: workspace.providerType,
migrated: Boolean(workspace.metadata?.nativeWorkspaceMigration),
proofSha256: createHash("sha256").update(file.content.data).digest("hex"),
runId: run.id, sessionId: run.nativeSessionId, providerSessionSha256: createHash("sha256").update(run.sessionIdAfter).digest("hex"),
logicalCwd: execution.workspace.cwd, remoteCwd: reference.remoteCwd };
}
function migrationReferenceAttachments(checkpoints: ContinuationCheckpoint[], before?: MigrationPoint, after?: MigrationPoint) {
if (!before || !after || before.proofSha256 !== after.proofSha256) return [];
return (checkpoints.find(point => point.phase === "final")?.attachments ?? []).filter(value => {
const file = value as Record<string, unknown>;
return file.originalFilename === "migration-reference.txt" && file.contentVerified === true
&& typeof file.body === "string" && file.contentSha256 === before.proofSha256
&& createHash("sha256").update(file.body).digest("hex") === before.proofSha256;
});
}
export function migrationPrimaryOutputCheckpoints(checkpoints: ContinuationCheckpoint[], before?: MigrationPoint, after?: MigrationPoint) {
const references = migrationReferenceAttachments(checkpoints, before, after);
if (references.length !== 1) return checkpoints;
return checkpoints.map(point => point.phase === "final"
? { ...point, attachments: point.attachments.filter(file => file !== references[0]) } : point);
}
export function gradeLegacyWorkspaceMigration(before?: MigrationPoint, after?: MigrationPoint, checkpoints: ContinuationCheckpoint[] = []): StoryCheck[] {
return [{ id: "legacy-workspace-migration-continuation", passed: Boolean(before && after && !before.migrated && after.migrated
&& before.providerType !== "cloud_sandbox" && after.providerType === "cloud_sandbox" && before.workspaceId === after.workspaceId
&& before.runId !== after.runId && before.sessionId === after.sessionId && before.logicalCwd === after.logicalCwd
&& before.remoteCwd === after.remoteCwd && before.proofSha256 === after.proofSha256 && before.providerSessionSha256 === after.providerSessionSha256),
detail: "A new turn retains the exact native session, logical and provider paths, workspace, and reference file bytes after offline migration and destruction of the old home." },
{ id: "legacy-workspace-migration-reference-delivered", passed: migrationReferenceAttachments(checkpoints, before, after).length === 1,
detail: "The one requested reference attachment has the exact original workspace bytes, independently downloaded and verified." }];
}
+4 -2
View File
@@ -1,4 +1,5 @@
import { isRollingWebDeployment } from "./controller-rolling-deployment.js";
import { isLegacyWorkspaceMigration } from "./legacy-workspace-migration.js";
import { createHash } from "node:crypto";
import { remoteWorkspaceAuthoritySchema, remoteWorkspaceRunReferenceSchema } from "../../packages/shared/src/remote-workspace.js";
import type { ExecutionWorkspace, WorkspaceFileContent } from "../../packages/shared/src/index.js";
@@ -134,8 +135,9 @@ export function remoteWorkspaceStorageServerEnvironment(input: {
env: NodeJS.ProcessEnv;
instanceId: string;
}): NodeJS.ProcessEnv | undefined {
if (!input.executions.some(isRemoteWorkspaceStorageExecution)) return undefined;
if (!input.executions.length || !input.executions.every(isRemoteWorkspaceStorageExecution)) {
const requiresStorage = (execution: MatrixExecution) => isRemoteWorkspaceStorageExecution(execution) || isLegacyWorkspaceMigration(execution);
if (!input.executions.some(requiresStorage)) return undefined;
if (!input.executions.length || !input.executions.every(requiresStorage)) {
throw new Error("Remote workspace storage qualification must run alone");
}
const bucket = input.env.PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_BUCKET?.trim();
+10 -1
View File
@@ -1,5 +1,7 @@
import { isRollingWebDeployment } from "./controller-rolling-deployment.js";
import { superviseRollingDeployment } from "./rolling-deployment-supervisor.js";
import { isLegacyWorkspaceMigration } from "./legacy-workspace-migration.js";
import { superviseLegacyWorkspaceMigration } from "./legacy-workspace-migration-supervisor.js";
import { readRestartRequest, type RestartRequest } from "./server-restart-request.js";
import { startCursorCloudCallbackProxy } from "./cursor-cloud-callback-proxy.js";
import { prepareControllerDatabase } from "./controller-database.js";
@@ -449,11 +451,18 @@ async function supervise() {
});
// Postgres needs the socket itself; release immediately before child spawn.
await databaseReservation?.close();
if (upgradeArtifacts || executionIds.some(id => id.endsWith(".recover-controller"))) {
const legacyMigration = executionIds.some(id => { const execution = runnerMatrix.find(row => row.id === id); return execution && isLegacyWorkspaceMigration(execution); });
if (upgradeArtifacts || legacyMigration || executionIds.some(id => id.endsWith(".recover-controller"))) {
// The wrapper owns this sibling process. Killing the serving child must
// leave Postgres available to the stopped-controller observer and successor.
controllerDatabase = await prepareControllerDatabase(configPath, temporaryRoot);
}
if (legacyMigration) {
if (upgradeArtifacts || installedRelease) throw new Error("Migration source qualification cannot claim installed-artifact evidence");
await superviseLegacyWorkspaceMigration({ temporaryRoot, repositoryRoot, configPath, port: Number(port),
environment: definedServerEnvironment, executionIds, cancelled: shutdownRequested });
return;
}
if (executionIds.some(id => { const execution = runnerMatrix.find(row => row.id === id); return execution && isRollingWebDeployment(execution); })) {
if (upgradeArtifacts || installedRelease) throw new Error("Rolling source qualification cannot claim installed-artifact evidence");
await superviseRollingDeployment({ temporaryRoot, repositoryRoot, configPath, port: Number(port),