fix(recovery): report exact retained sandbox resumes as resumed (#21259)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A controller restart must keep the same remote execution owner.
> - A retained sandbox lease stores the result of its first acquisition.
> - That result can say replacement after the same allocation resumes.
> - The Runner then rejects a valid resume as a changed sandbox.
> - This pull request reports the verified operation as resumed.
> - The saved acquisition history and identity checks stay intact.

## Linked Issues or Issue Description

Refs #18934. That change makes the retained sandbox ready before
preparation. This change corrects the acquisition receipt after that
resume.

**What happened?**
A remote native run resumed the exact retained sandbox after a
controller restart. The lease still carried a historical replacement
receipt. Runner verification rejected it with
`native_remote_recovery_lease_mismatch`, stage `runner_verification`,
field `replacement_sandbox`.

**What did you expect to happen?**
Report the verified exact-allocation operation as resumed. Preserve the
original saved history. Reject a changed, absent or expired allocation.

**Steps to reproduce**
Save an active lease with historical replacement metadata. Resume that
exact provider allocation through `acquireForRun`. Resolve its execution
target. Before the fix, the target still reports replacement and Runner
verification refuses it.

**Deployment mode**
Native Runner on a remote sandbox. The reported provider was Daytona.

## What Changed

- Return a separate operation receipt after the exact retained
allocation passes the existing resume checks.
- Pass the receipt through heartbeat preparation and apply it only to
the execution target.
- Keep the saved lease metadata and workspace unchanged.
- Test historical created, replacement and resumed receipts. Test
ordinary and remote-authority realization through the real execution
target resolver. Check every saved metadata write.
- Document the difference between saved acquisition history and the
current resume operation.

## Verification

- Head: `74d03e8ddadd1274891bea99a10455795c500557`.
- Red: three historical receipt cases fail without the resume receipt.
- Review regression red: realization writes resumed over the historical
replacement receipt. The saved metadata loses the previous allocation
and cause.
- Green: all 69 orchestrator tests pass. Ordinary and remote-authority
realization preserve history in every metadata write and return resumed
in the target.
- Green: 135 focused orchestrator and remote controller adoption cases
pass. Identity and changed-allocation refusals remain effective.
- Server typecheck passes, including its TypeScript build dependencies
and test layers.
- `git diff --check` passes.
- `pnpm verify` exits 2 because the execution-semantics document needs
broad CI. Required CI supplies that coverage.
- Linux x64, Node 25.9.0. Provider boundaries are simulated.
- The first post-merge attempt could not collect tests because the
compiled plugin SDK lacked the new taskLease schema. Shared and plugin
SDK builds passed, then the unchanged selection passed.
- Full preflight stopped because Cargo is absent. Dependency
installation completed. The later server TypeScript build and typecheck
passed.
- Live retained Runner reattachment is not verified. This workspace has
no provider credentials. No rollout or original-task recovery by this
patch is claimed.
- Required current-head CI and automated review remain pending.

## Risks

A wrong receipt could hide an actual replacement. The projection occurs
only after the existing company, task, run, agent, workspace, provider
allocation, expiry and directory checks pass. The provider must resume
the exact allocation. Runner process and identity verification still
runs. No replacement is acquired and no saved acquisition history is
changed. The heartbeat change only carries the typed receipt between the
owning orchestrator stages. Callers without a receipt keep their
existing behavior.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
execution and tool use. The exact backend snapshot and context window
are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-13 12:55:55 -06:00
1 parent 2b1d0fc825
commit 08be8be414
4 files changed
+95 -6

No files matched your search

+8
View File
@@ -6216,3 +6216,11 @@ with company/run identifiers to the process logger; it cannot append provider te
to a completed run or borrow a successor run's recorder. Canonical native output
export, source coverage, secret redaction and persistence-failure warnings retain
their existing contracts.
Exact sandbox reattachment reports the current acquisition as `resumed` only
when the recorded lease passes scope and identity checks and the provider confirms
that same allocation. The lease's original acquisition can have been `replacement`;
that historical label cannot describe the later reattachment. This changes the
operation receipt passed separately through preparation to the execution target,
not persisted lease metadata or the Runner verification guard. Missing,
expired, foreign or changed allocations retain their named recovery refusals.
@@ -984,9 +984,73 @@ describe("live remote runner lease reattachment", () => {
runnerReconnectGraceMs: REMOTE_RUNNER_RECONNECT_GRACE_MS,
});
expect(result.lease).toEqual(savedLease);
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
expect(result.leaseContext.executionWorkspaceId).toBe("ew-1");
expect(result.environment.config.reuseLease).toBe(false);
});
it.each(["created", "replacement", "resumed"] as const)(
"reports exact-lease reattachment as resumed after historical %s acquisition", async outcome => {
const savedLease = originalLease();
savedLease.metadata!.sandboxLeaseAcquisition = {
outcome, providerLeaseId: "original-sandbox", previousProviderLeaseId: "older-sandbox", reason: "resume_failed",
};
const before = structuredClone(savedLease);
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime();
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
expect(result.lease.id).toBe(savedLease.id);
expect(result.lease.metadata?.remoteCwd).toBe("/remote/workspace");
expect(savedLease).toEqual(before);
expect(runtime.resumeRunLease).toHaveBeenCalledOnce();
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
},
);
it.each([false, true])("preserves acquisition history when realization persists workspace metadata (remote authority: %s)", async remoteAuthority => {
const savedLease = originalLease();
savedLease.metadata!.sandboxLeaseAcquisition = {
outcome: "replacement", providerLeaseId: "original-sandbox",
previousProviderLeaseId: "older-sandbox", reason: "resume_failed",
};
const history = structuredClone(savedLease.metadata!.sandboxLeaseAcquisition);
mockGetLease.mockResolvedValue(savedLease);
mockUpdateLeaseMetadata.mockImplementation(async (_id, metadata) => ({ ...savedLease, metadata }));
mockBuildWorkspaceRealizationRequest.mockReturnValue({ runtimeOverlay: {} });
const { resolveEnvironmentExecutionTarget } = await vi.importActual<typeof import("../services/environment-execution-target.js")>("../services/environment-execution-target.js");
mockResolveEnvironmentExecutionTarget.mockImplementation(resolveEnvironmentExecutionTarget);
const config = await import("../services/environment-config.js");
const resolveConfig = vi.spyOn(config, "resolveEnvironmentDriverConfigForRuntime").mockResolvedValue({
driver: "sandbox", config: { provider: "daytona", reuseLease: false },
} as never);
const workspace = makePersistedExecutionWorkspace({ metadata: remoteAuthority ? {
remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, source: { kind: "empty" },
writerRunId: randomUUID(), dirty: false, snapshotId: null, location: null },
} : null });
const prepare = vi.spyOn(remoteWorkspaces, "prepareRemoteWorkspace").mockImplementation(async args => ({
workspace, lease: args.lease, remoteRoot: "/remote/workspace", reference: {} as never,
} as never));
mockGetStoredWorkspace.mockResolvedValue(workspace);
try {
const runtime = makeMockRuntime({ supportsSync: () => false });
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: runtime });
const acquisition = await orchestrator.acquireForRun(input);
const result = await orchestrator.realizeForRun({
...makeRealizeInput(), ...acquisition, adapterType: input.adapterType, persistedExecutionWorkspace: workspace,
});
expect(mockUpdateLeaseMetadata).toHaveBeenCalled();
for (const [, metadata] of mockUpdateLeaseMetadata.mock.calls) {
expect(metadata.sandboxLeaseAcquisition).toEqual(history);
}
expect(result.lease.metadata!.sandboxLeaseAcquisition).toEqual(history);
expect(result.executionTarget).toMatchObject({ kind: "remote", transport: "sandbox",
sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "original-sandbox" } });
} finally {
prepare.mockRestore();
resolveConfig.mockRestore();
mockResolveEnvironmentExecutionTarget.mockReset();
}
});
it("forwards the admitted warm idle window to the recovery resume", async () => {
const savedLease = originalLease();
mockGetLease.mockResolvedValue(savedLease);
@@ -1021,7 +1085,10 @@ describe("live remote runner lease reattachment", () => {
expect(state).toBe("pending");
expect(ready).toBe(false);
} finally { release(); }
expect((await pending).lease).toEqual(before);
const result = await pending;
expect(result.lease).toEqual(before);
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
expect(savedLease).toEqual(before);
expect(ready).toBe(true);
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
expect(runtime.execute).not.toHaveBeenCalled();
@@ -1047,7 +1114,7 @@ describe("live remote runner lease reattachment", () => {
providerLeaseId: "original-sandbox", metadata,
})) });
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
expect(result.lease).toBe(savedLease);
expect(result.lease).toMatchObject({ id: savedLease.id, providerLeaseId: savedLease.providerLeaseId });
expect(result.lease.metadata?.remoteCwd).toBe("/remote/workspace");
expect(runtime.resumeRunLease).toHaveBeenCalledOnce();
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
@@ -61,6 +61,7 @@ import {
type AdapterExecutionTarget,
type AdapterRemoteExecutionSpec,
type AdapterWorkspaceRealization,
type SandboxLeaseAcquisition,
} from "@paperclipai/adapter-utils/execution-target";
import type { DuplexObservabilityRecorder } from "@paperclipai/adapter-utils/duplex-observability";
import { buildWorkspaceRealizationRequest } from "./workspace-realization.js";
@@ -124,6 +125,7 @@ export interface EnvironmentAcquisitionResult {
leaseContext: ReturnType<typeof buildEnvironmentLeaseContext>;
executionTransport: Record<string, unknown> | null;
pluginTaskDescription?: PluginTaskDescription;
sandboxLeaseAcquisition?: SandboxLeaseAcquisition;
}
export interface EnvironmentRealizationResult {
@@ -379,6 +381,7 @@ export function environmentRunOrchestrator(
runnerReconnectGraceMs,
};
let leaseRecord: EnvironmentRuntimeLeaseRecord;
let sandboxLeaseAcquisition: SandboxLeaseAcquisition | undefined;
const expected = input.reattachRemoteLease;
if (expected) {
// Reattachment keeps the original lease, including ephemeral leases.
@@ -432,6 +435,7 @@ export function environmentRunOrchestrator(
if (resumed.metadata?.expired === true) throw remoteRecoveryLeaseMismatch("environment_resume", "lease_expired");
// Resume confirms allocation readiness. Its optional metadata may
// describe the allocation root; keep the validated lease's workspace.
sandboxLeaseAcquisition = { outcome: "resumed", providerLeaseId: expected.providerLeaseId };
}
leaseRecord = { environment, lease, leaseContext: buildEnvironmentLeaseContext(input) };
} else {
@@ -473,6 +477,7 @@ export function environmentRunOrchestrator(
lease: leaseRecord.lease,
leaseContext: leaseRecord.leaseContext,
executionTransport,
...(sandboxLeaseAcquisition ? { sandboxLeaseAcquisition } : {}),
};
}
@@ -517,6 +522,7 @@ export function environmentRunOrchestrator(
async function realizeForRun(input: {
environment: Environment;
lease: EnvironmentLease;
sandboxLeaseAcquisition?: SandboxLeaseAcquisition;
adapterType: string;
companyId: string;
issueId: string | null;
@@ -544,15 +550,22 @@ export function environmentRunOrchestrator(
effectiveExecutionWorkspaceMode,
} = input;
let { lease, persistedExecutionWorkspace } = input;
const withAcquisitionReceipt = (target: AdapterExecutionTarget | null): AdapterExecutionTarget | null => {
if (!input.sandboxLeaseAcquisition || target?.kind !== "remote" || target.transport !== "sandbox") return target;
if (lease.providerLeaseId !== input.sandboxLeaseAcquisition.providerLeaseId) {
throw remoteRecoveryLeaseMismatch("environment_resume", "provider_lease_id");
}
return { ...target, sandboxLeaseAcquisition: input.sandboxLeaseAcquisition };
};
const remoteAuthority = readRemoteWorkspaceAuthority(persistedExecutionWorkspace?.metadata);
if (remoteAuthority && persistedExecutionWorkspace) {
if (adapterType !== "paperclip_runner" || environment.driver !== "sandbox") {
throw new EnvironmentRunError("unsupported_adapter_environment", "Remote workspace storage requires a qualified native sandbox");
}
const target = await resolveEnvironmentExecutionTarget({
const target = withAcquisitionReceipt(await resolveEnvironmentExecutionTarget({
db, companyId, adapterType, environment, leaseId: lease.id, leaseMetadata: lease.metadata,
lease, environmentRuntime, duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
});
}));
if (!target || target.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.runnerTask) {
throw new EnvironmentRunError("transport_resolution_failed", "Remote workspace storage requires a sandbox file transport");
}
@@ -753,7 +766,7 @@ export function environmentRunOrchestrator(
// Step 4: Resolve execution target for the adapter
let executionTarget: AdapterExecutionTarget | null;
try {
executionTarget = await resolveEnvironmentExecutionTarget({
executionTarget = withAcquisitionReceipt(await resolveEnvironmentExecutionTarget({
db,
companyId,
adapterType,
@@ -764,7 +777,7 @@ export function environmentRunOrchestrator(
environmentRuntime,
signal: input.signal,
duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
});
}));
// The native plugin task bridge owns guest paths. Never fall back to host execution.
if (executionWorkspace.providerBinding && (executionTarget?.kind !== "remote" ||
!("runnerTask" in executionTarget) || typeof executionTarget.runnerTask !== "function")) {
@@ -1562,6 +1562,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
realizationResult = await timeWorkspace("workspace_environment_realize", () => envOrchestrator.realizeForRun({
environment: selectedEnvironment,
lease: activeEnvironmentLease.lease,
sandboxLeaseAcquisition: acquiredEnvironment.sandboxLeaseAcquisition,
adapterType: agent.adapterType,
companyId: agent.companyId,
issueId: issueId ?? null,