mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
fix(recovery): report exact retained sandbox resumes as resumed (#21259)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A controller restart must keep the same remote execution owner. > - A retained sandbox lease stores the result of its first acquisition. > - That result can say replacement after the same allocation resumes. > - The Runner then rejects a valid resume as a changed sandbox. > - This pull request reports the verified operation as resumed. > - The saved acquisition history and identity checks stay intact. ## Linked Issues or Issue Description Refs #18934. That change makes the retained sandbox ready before preparation. This change corrects the acquisition receipt after that resume. **What happened?** A remote native run resumed the exact retained sandbox after a controller restart. The lease still carried a historical replacement receipt. Runner verification rejected it with `native_remote_recovery_lease_mismatch`, stage `runner_verification`, field `replacement_sandbox`. **What did you expect to happen?** Report the verified exact-allocation operation as resumed. Preserve the original saved history. Reject a changed, absent or expired allocation. **Steps to reproduce** Save an active lease with historical replacement metadata. Resume that exact provider allocation through `acquireForRun`. Resolve its execution target. Before the fix, the target still reports replacement and Runner verification refuses it. **Deployment mode** Native Runner on a remote sandbox. The reported provider was Daytona. ## What Changed - Return a separate operation receipt after the exact retained allocation passes the existing resume checks. - Pass the receipt through heartbeat preparation and apply it only to the execution target. - Keep the saved lease metadata and workspace unchanged. - Test historical created, replacement and resumed receipts. Test ordinary and remote-authority realization through the real execution target resolver. Check every saved metadata write. - Document the difference between saved acquisition history and the current resume operation. ## Verification - Head: `74d03e8ddadd1274891bea99a10455795c500557`. - Red: three historical receipt cases fail without the resume receipt. - Review regression red: realization writes resumed over the historical replacement receipt. The saved metadata loses the previous allocation and cause. - Green: all 69 orchestrator tests pass. Ordinary and remote-authority realization preserve history in every metadata write and return resumed in the target. - Green: 135 focused orchestrator and remote controller adoption cases pass. Identity and changed-allocation refusals remain effective. - Server typecheck passes, including its TypeScript build dependencies and test layers. - `git diff --check` passes. - `pnpm verify` exits 2 because the execution-semantics document needs broad CI. Required CI supplies that coverage. - Linux x64, Node 25.9.0. Provider boundaries are simulated. - The first post-merge attempt could not collect tests because the compiled plugin SDK lacked the new taskLease schema. Shared and plugin SDK builds passed, then the unchanged selection passed. - Full preflight stopped because Cargo is absent. Dependency installation completed. The later server TypeScript build and typecheck passed. - Live retained Runner reattachment is not verified. This workspace has no provider credentials. No rollout or original-task recovery by this patch is claimed. - Required current-head CI and automated review remain pending. ## Risks A wrong receipt could hide an actual replacement. The projection occurs only after the existing company, task, run, agent, workspace, provider allocation, expiry and directory checks pass. The provider must resume the exact allocation. Runner process and identity verification still runs. No replacement is acquired and no saved acquisition history is changed. The heartbeat change only carries the typed receipt between the owning orchestrator stages. Callers without a receipt keep their existing behavior. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code execution and tool use. The exact backend snapshot and context window are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
2b1d0fc825
commit
08be8be414
4 files changed
+95
-6
No files matched your search
@@ -6216,3 +6216,11 @@ with company/run identifiers to the process logger; it cannot append provider te
|
||||
to a completed run or borrow a successor run's recorder. Canonical native output
|
||||
export, source coverage, secret redaction and persistence-failure warnings retain
|
||||
their existing contracts.
|
||||
|
||||
Exact sandbox reattachment reports the current acquisition as `resumed` only
|
||||
when the recorded lease passes scope and identity checks and the provider confirms
|
||||
that same allocation. The lease's original acquisition can have been `replacement`;
|
||||
that historical label cannot describe the later reattachment. This changes the
|
||||
operation receipt passed separately through preparation to the execution target,
|
||||
not persisted lease metadata or the Runner verification guard. Missing,
|
||||
expired, foreign or changed allocations retain their named recovery refusals.
|
||||
@@ -984,9 +984,73 @@ describe("live remote runner lease reattachment", () => {
|
||||
runnerReconnectGraceMs: REMOTE_RUNNER_RECONNECT_GRACE_MS,
|
||||
});
|
||||
expect(result.lease).toEqual(savedLease);
|
||||
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
|
||||
expect(result.leaseContext.executionWorkspaceId).toBe("ew-1");
|
||||
expect(result.environment.config.reuseLease).toBe(false);
|
||||
});
|
||||
it.each(["created", "replacement", "resumed"] as const)(
|
||||
"reports exact-lease reattachment as resumed after historical %s acquisition", async outcome => {
|
||||
const savedLease = originalLease();
|
||||
savedLease.metadata!.sandboxLeaseAcquisition = {
|
||||
outcome, providerLeaseId: "original-sandbox", previousProviderLeaseId: "older-sandbox", reason: "resume_failed",
|
||||
};
|
||||
const before = structuredClone(savedLease);
|
||||
mockGetLease.mockResolvedValue(savedLease);
|
||||
const runtime = makeMockRuntime();
|
||||
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
|
||||
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
|
||||
expect(result.lease.id).toBe(savedLease.id);
|
||||
expect(result.lease.metadata?.remoteCwd).toBe("/remote/workspace");
|
||||
expect(savedLease).toEqual(before);
|
||||
expect(runtime.resumeRunLease).toHaveBeenCalledOnce();
|
||||
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
|
||||
expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
it.each([false, true])("preserves acquisition history when realization persists workspace metadata (remote authority: %s)", async remoteAuthority => {
|
||||
const savedLease = originalLease();
|
||||
savedLease.metadata!.sandboxLeaseAcquisition = {
|
||||
outcome: "replacement", providerLeaseId: "original-sandbox",
|
||||
previousProviderLeaseId: "older-sandbox", reason: "resume_failed",
|
||||
};
|
||||
const history = structuredClone(savedLease.metadata!.sandboxLeaseAcquisition);
|
||||
mockGetLease.mockResolvedValue(savedLease);
|
||||
mockUpdateLeaseMetadata.mockImplementation(async (_id, metadata) => ({ ...savedLease, metadata }));
|
||||
mockBuildWorkspaceRealizationRequest.mockReturnValue({ runtimeOverlay: {} });
|
||||
const { resolveEnvironmentExecutionTarget } = await vi.importActual<typeof import("../services/environment-execution-target.js")>("../services/environment-execution-target.js");
|
||||
mockResolveEnvironmentExecutionTarget.mockImplementation(resolveEnvironmentExecutionTarget);
|
||||
const config = await import("../services/environment-config.js");
|
||||
const resolveConfig = vi.spyOn(config, "resolveEnvironmentDriverConfigForRuntime").mockResolvedValue({
|
||||
driver: "sandbox", config: { provider: "daytona", reuseLease: false },
|
||||
} as never);
|
||||
const workspace = makePersistedExecutionWorkspace({ metadata: remoteAuthority ? {
|
||||
remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, source: { kind: "empty" },
|
||||
writerRunId: randomUUID(), dirty: false, snapshotId: null, location: null },
|
||||
} : null });
|
||||
const prepare = vi.spyOn(remoteWorkspaces, "prepareRemoteWorkspace").mockImplementation(async args => ({
|
||||
workspace, lease: args.lease, remoteRoot: "/remote/workspace", reference: {} as never,
|
||||
} as never));
|
||||
mockGetStoredWorkspace.mockResolvedValue(workspace);
|
||||
try {
|
||||
const runtime = makeMockRuntime({ supportsSync: () => false });
|
||||
const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: runtime });
|
||||
const acquisition = await orchestrator.acquireForRun(input);
|
||||
const result = await orchestrator.realizeForRun({
|
||||
...makeRealizeInput(), ...acquisition, adapterType: input.adapterType, persistedExecutionWorkspace: workspace,
|
||||
});
|
||||
expect(mockUpdateLeaseMetadata).toHaveBeenCalled();
|
||||
for (const [, metadata] of mockUpdateLeaseMetadata.mock.calls) {
|
||||
expect(metadata.sandboxLeaseAcquisition).toEqual(history);
|
||||
}
|
||||
expect(result.lease.metadata!.sandboxLeaseAcquisition).toEqual(history);
|
||||
expect(result.executionTarget).toMatchObject({ kind: "remote", transport: "sandbox",
|
||||
sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "original-sandbox" } });
|
||||
} finally {
|
||||
prepare.mockRestore();
|
||||
resolveConfig.mockRestore();
|
||||
mockResolveEnvironmentExecutionTarget.mockReset();
|
||||
}
|
||||
});
|
||||
it("forwards the admitted warm idle window to the recovery resume", async () => {
|
||||
const savedLease = originalLease();
|
||||
mockGetLease.mockResolvedValue(savedLease);
|
||||
@@ -1021,7 +1085,10 @@ describe("live remote runner lease reattachment", () => {
|
||||
expect(state).toBe("pending");
|
||||
expect(ready).toBe(false);
|
||||
} finally { release(); }
|
||||
expect((await pending).lease).toEqual(before);
|
||||
const result = await pending;
|
||||
expect(result.lease).toEqual(before);
|
||||
expect(result.sandboxLeaseAcquisition).toEqual({ outcome: "resumed", providerLeaseId: "original-sandbox" });
|
||||
expect(savedLease).toEqual(before);
|
||||
expect(ready).toBe(true);
|
||||
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
|
||||
expect(runtime.execute).not.toHaveBeenCalled();
|
||||
@@ -1047,7 +1114,7 @@ describe("live remote runner lease reattachment", () => {
|
||||
providerLeaseId: "original-sandbox", metadata,
|
||||
})) });
|
||||
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
|
||||
expect(result.lease).toBe(savedLease);
|
||||
expect(result.lease).toMatchObject({ id: savedLease.id, providerLeaseId: savedLease.providerLeaseId });
|
||||
expect(result.lease.metadata?.remoteCwd).toBe("/remote/workspace");
|
||||
expect(runtime.resumeRunLease).toHaveBeenCalledOnce();
|
||||
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
|
||||
|
||||
@@ -61,6 +61,7 @@ import {
|
||||
type AdapterExecutionTarget,
|
||||
type AdapterRemoteExecutionSpec,
|
||||
type AdapterWorkspaceRealization,
|
||||
type SandboxLeaseAcquisition,
|
||||
} from "@paperclipai/adapter-utils/execution-target";
|
||||
import type { DuplexObservabilityRecorder } from "@paperclipai/adapter-utils/duplex-observability";
|
||||
import { buildWorkspaceRealizationRequest } from "./workspace-realization.js";
|
||||
@@ -124,6 +125,7 @@ export interface EnvironmentAcquisitionResult {
|
||||
leaseContext: ReturnType<typeof buildEnvironmentLeaseContext>;
|
||||
executionTransport: Record<string, unknown> | null;
|
||||
pluginTaskDescription?: PluginTaskDescription;
|
||||
sandboxLeaseAcquisition?: SandboxLeaseAcquisition;
|
||||
}
|
||||
|
||||
export interface EnvironmentRealizationResult {
|
||||
@@ -379,6 +381,7 @@ export function environmentRunOrchestrator(
|
||||
runnerReconnectGraceMs,
|
||||
};
|
||||
let leaseRecord: EnvironmentRuntimeLeaseRecord;
|
||||
let sandboxLeaseAcquisition: SandboxLeaseAcquisition | undefined;
|
||||
const expected = input.reattachRemoteLease;
|
||||
if (expected) {
|
||||
// Reattachment keeps the original lease, including ephemeral leases.
|
||||
@@ -432,6 +435,7 @@ export function environmentRunOrchestrator(
|
||||
if (resumed.metadata?.expired === true) throw remoteRecoveryLeaseMismatch("environment_resume", "lease_expired");
|
||||
// Resume confirms allocation readiness. Its optional metadata may
|
||||
// describe the allocation root; keep the validated lease's workspace.
|
||||
sandboxLeaseAcquisition = { outcome: "resumed", providerLeaseId: expected.providerLeaseId };
|
||||
}
|
||||
leaseRecord = { environment, lease, leaseContext: buildEnvironmentLeaseContext(input) };
|
||||
} else {
|
||||
@@ -473,6 +477,7 @@ export function environmentRunOrchestrator(
|
||||
lease: leaseRecord.lease,
|
||||
leaseContext: leaseRecord.leaseContext,
|
||||
executionTransport,
|
||||
...(sandboxLeaseAcquisition ? { sandboxLeaseAcquisition } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -517,6 +522,7 @@ export function environmentRunOrchestrator(
|
||||
async function realizeForRun(input: {
|
||||
environment: Environment;
|
||||
lease: EnvironmentLease;
|
||||
sandboxLeaseAcquisition?: SandboxLeaseAcquisition;
|
||||
adapterType: string;
|
||||
companyId: string;
|
||||
issueId: string | null;
|
||||
@@ -544,15 +550,22 @@ export function environmentRunOrchestrator(
|
||||
effectiveExecutionWorkspaceMode,
|
||||
} = input;
|
||||
let { lease, persistedExecutionWorkspace } = input;
|
||||
const withAcquisitionReceipt = (target: AdapterExecutionTarget | null): AdapterExecutionTarget | null => {
|
||||
if (!input.sandboxLeaseAcquisition || target?.kind !== "remote" || target.transport !== "sandbox") return target;
|
||||
if (lease.providerLeaseId !== input.sandboxLeaseAcquisition.providerLeaseId) {
|
||||
throw remoteRecoveryLeaseMismatch("environment_resume", "provider_lease_id");
|
||||
}
|
||||
return { ...target, sandboxLeaseAcquisition: input.sandboxLeaseAcquisition };
|
||||
};
|
||||
const remoteAuthority = readRemoteWorkspaceAuthority(persistedExecutionWorkspace?.metadata);
|
||||
if (remoteAuthority && persistedExecutionWorkspace) {
|
||||
if (adapterType !== "paperclip_runner" || environment.driver !== "sandbox") {
|
||||
throw new EnvironmentRunError("unsupported_adapter_environment", "Remote workspace storage requires a qualified native sandbox");
|
||||
}
|
||||
const target = await resolveEnvironmentExecutionTarget({
|
||||
const target = withAcquisitionReceipt(await resolveEnvironmentExecutionTarget({
|
||||
db, companyId, adapterType, environment, leaseId: lease.id, leaseMetadata: lease.metadata,
|
||||
lease, environmentRuntime, duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
|
||||
});
|
||||
}));
|
||||
if (!target || target.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.runnerTask) {
|
||||
throw new EnvironmentRunError("transport_resolution_failed", "Remote workspace storage requires a sandbox file transport");
|
||||
}
|
||||
@@ -753,7 +766,7 @@ export function environmentRunOrchestrator(
|
||||
// Step 4: Resolve execution target for the adapter
|
||||
let executionTarget: AdapterExecutionTarget | null;
|
||||
try {
|
||||
executionTarget = await resolveEnvironmentExecutionTarget({
|
||||
executionTarget = withAcquisitionReceipt(await resolveEnvironmentExecutionTarget({
|
||||
db,
|
||||
companyId,
|
||||
adapterType,
|
||||
@@ -764,7 +777,7 @@ export function environmentRunOrchestrator(
|
||||
environmentRuntime,
|
||||
signal: input.signal,
|
||||
duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
|
||||
});
|
||||
}));
|
||||
// The native plugin task bridge owns guest paths. Never fall back to host execution.
|
||||
if (executionWorkspace.providerBinding && (executionTarget?.kind !== "remote" ||
|
||||
!("runnerTask" in executionTarget) || typeof executionTarget.runnerTask !== "function")) {
|
||||
|
||||
@@ -1562,6 +1562,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
|
||||
realizationResult = await timeWorkspace("workspace_environment_realize", () => envOrchestrator.realizeForRun({
|
||||
environment: selectedEnvironment,
|
||||
lease: activeEnvironmentLease.lease,
|
||||
sandboxLeaseAcquisition: acquiredEnvironment.sandboxLeaseAcquisition,
|
||||
adapterType: agent.adapterType,
|
||||
companyId: agent.companyId,
|
||||
issueId: issueId ?? null,
|
||||
|
||||
Reference in new issue
Block a user