From b4315943e87d86d68e119c295042b00621c40402 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 2 Nov 2026 16:11:30 -0600 Subject: [PATCH] fix(heartbeat): keep remote retry workspace strategy stable (#18660) ## Thinking Path > - Paperclip preserves agent work across separate execution services. > - A remote retry can create a new isolated Git workspace. > - Later admissions compare its saved configuration with the current policy. > - Fresh preparation also creates a temporary controller-side strategy. > - Remote preparation saved that temporary strategy in its fingerprint. > - The next unchanged turn then reported a workspace strategy replacement. > - This change saves the admitted remote strategy so unchanged reuse stays stable. ## Linked Issues or Issue Description Related preparation work: #18543 and #17924. I searched open and merged workspace/session repairs and public workspace-strategy issues. None covers this fresh remote retry case. **What happened?** After Retry created an isolated remote Git workspace, each unchanged follow-up reported `workspace_config_freshness` with `action: replace` and `changedCategories: [strategy]`. It retained the same workspace. The stored fingerprint contained `project_primary`, which was only a temporary controller provisioning choice. **Expected behavior** Save the configured remote strategy. An unchanged follow-up must reuse the same workspace without a false replacement report. A real strategy, base-ref, mode or environment change must retain its replacement checks. **Steps to reproduce** 1. Prepare a fresh isolated remote Git retry with a `git_worktree` strategy. 2. Complete its runtime handoff. 3. Admit another turn with the same workspace policy. 4. Inspect the recorded workspace freshness decision. **Paperclip version or commit** Reproduced on `b9e94b85a69bf679818eeef9ecf4655554a89c6b`. The regression also fails on the unchanged preparation code in the current base. **Deployment mode** Authenticated server with remote workspace authority and a sandbox execution target. ## What Changed - Keep the admitted strategy when a fresh workspace uses remote authority. - Retain the temporary controller strategy for existing host-side seed preparation. - Extend the PostgreSQL preparation test through a fresh admission and its next ordinary admission. Assert the same workspace, current writer and stable fingerprint, with no false drift operation. - Cover a genuinely configured `project_primary` fresh remote Git retry followed by a real strategy change, preserving its replacement decision and stored fingerprint. - Document the execution rule and the separate live verification recipe. ## Verification - The final regression was run before and after the production change. Before: the next admission recorded a strategy replacement. After: unchanged reuse records no drift and retains the workspace and fingerprint. - All 312 targeted cases pass on `48b3e548afa42e740871feae2763d88fd5a00431` across workspace preparation, workspace/session policy and inherited session preparation. These include actual strategy/base-ref changes and the rule that explicit reuse must not hide real replacement-class drift. - All eight remote preparation cases also pass with the compatibility regression: a genuinely configured `project_primary` fresh Git retry creates a `git_worktree` row, and its later real strategy change retains the replacement decision and saved fingerprint. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Full recursive typecheck and `pnpm build` pass locally on this head. - The [full current-head CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41524105868) passes: typecheck/build, all general server, chat, workspace and serialized test groups, plus Runner and E2E checks. The workflow and `run-vitest-stable.mjs` partition the complete suite. - The duplicate local `pnpm test:run` passed all 1,828 chat cases and was intentionally stopped during general-server after the same complete CI matrix passed. This is not recorded as a full local-suite pass. - Greptile and Paperclip Review both assessed this head at 5/5 with no new findings. Paperclip Review accepted the compatibility boundary after the counterexample; its check conclusion is neutral. The addressed review thread is resolved. - The regression uses real PostgreSQL and preparation policy. Runtime acquisition is controlled at its external boundary. The new fix has not been deployed; live provider resumption remains unverified for this patch. ## Risks - Existing incorrect fingerprints are not rewritten. The same stored shape can come from a genuinely configured `project_primary` remote Git retry; a Git workspace row alone cannot prove the old admission policy. Rewriting it would hide a real policy change. After preserving and inspecting retained work, an explicitly admitted fresh recovery after rollout establishes a correct new fingerprint. Ordinary reuse of the old workspace does not migrate it. Normal recovery gates remain required. - Real policy changes still use the current replacement guard. No schema, credential or authority change is included. ## Model Used OpenAI GPT-6 Codex with reasoning, repository editing, shell execution, PostgreSQL tests and browser inspection. The host does not expose the exact model variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 14 +++++ maps/agent-access-map/runs-adapters.md | 12 ++++ maps/feature-map/runs-adapters.md | 12 ++++ .../heartbeat/workspace-preparation.test.ts | 60 +++++++++++++++---- .../heartbeat/workspace-preparation.ts | 2 +- 5 files changed, 87 insertions(+), 13 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 5889dc4634a..054872f8515 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -734,6 +734,20 @@ A workspace-coherent adapter path means: - the effective cwd exists or is provider-reachable, according to the workspace provider - when the adapter or workspace strategy relies on git state, the cwd is git-valid for the selected workspace: it resolves to the expected repository root, required base refs or branch metadata can be resolved, and runtime-created worktrees are still registered or explicitly recoverable +Fresh remote-workspace recovery fingerprints the admitted remote strategy. A +temporary strategy used to seed a controller directory must not become the saved +remote configuration. An unchanged follow-up reuses that configuration without +reporting strategy drift. Real strategy, source, mode and environment changes +retain their existing replacement checks. + +Older remote Git workspaces can retain the temporary `project_primary` +fingerprint. That fingerprint alone cannot distinguish the old preparation bug +from a genuinely configured `project_primary` strategy; the remote workspace row +uses `git_worktree` for either Git source. Do not rewrite those fingerprints in +place or suppress their replacement checks. After saving and inspecting the +retained work, an explicitly admitted fresh recovery creates a workspace with the +correct fingerprint. Ordinary reuse of the old workspace does not migrate it. + Adapter-backed liveness also requires control-plane reachability from the agent's actual mutation surface, not just from the host adapter process. If the agent is expected to use Bash, shell tools, runtime helpers, or in-sandbox command execution to update issues, create comments, upload artifacts, or submit review decisions, the `PAPERCLIP_API_URL` and `PAPERCLIP_API_KEY` visible to that surface must route to Paperclip successfully. For sandbox-backed local adapters, Paperclip may satisfy that contract with a run-scoped in-sandbox bridge. The host adapter keeps the real run JWT on the host side, injects only the bridge URL/token into the sandbox tool environment, and forwards allowed Paperclip API requests with the run id attached. The bridge credentials are execution plumbing, not user-facing context: they must not be written into prompts, visible comments, issue documents, restored workspace files, or durable logs. Agents and skills must use the env vars available in Bash/curl rather than assuming that the host's localhost API URL is reachable from browser or web-extraction tools inside the sandbox. diff --git a/maps/agent-access-map/runs-adapters.md b/maps/agent-access-map/runs-adapters.md index 53d9400f4de..74728827d57 100644 --- a/maps/agent-access-map/runs-adapters.md +++ b/maps/agent-access-map/runs-adapters.md @@ -320,6 +320,18 @@ session-control access. Deployed diagnostic usefulness remains unverified. ## `session` +Fresh remote Retry preparation retains the admitted strategy in its saved +fingerprint. An unchanged follow-up must not report strategy drift; actual policy +changes retain the replacement guard. This changes no retry, session-reset, +workspace or credential authority. The +[preparation regression](../../server/src/services/heartbeat/workspace-preparation.test.ts) +uses PostgreSQL and controlled runtime acquisition. Live provider resumption +remains a separate check in the [session recipe](../feature-map/runs-adapters.md#session). + +Older strategy fingerprints retain their existing guard until an explicitly +admitted fresh recovery. A remote Git row alone does not authorize rewriting its +configuration history or bypassing a real policy change. + Scoped source evidence: restart preparation transfers its lease renewal to the native executor after the executor acquires its own reference. Completion stops renewal before coordinator release, while real ownership loss retains the detach diff --git a/maps/feature-map/runs-adapters.md b/maps/feature-map/runs-adapters.md index b728b8bac04..51092b077e0 100644 --- a/maps/feature-map/runs-adapters.md +++ b/maps/feature-map/runs-adapters.md @@ -146,6 +146,18 @@ must still reset the session and preserve the stored workspace owner. The checks this with PostgreSQL and synthetic provider execution; it is not live warm-process or latency evidence. +After an isolated remote Retry, send a follow-up with unchanged workspace policy. +Verify the same workspace remains bound and no strategy-replacement operation is +recorded. Change the strategy or base ref and confirm that the normal replacement +guard still applies. The +[remote preparation regression](../../server/src/services/heartbeat/workspace-preparation.test.ts) +checks persisted configuration across both admissions; it does not prove a live +provider conversation resumed. + +For a workspace carrying an older temporary fingerprint, verify that ordinary +reuse retains the strategy warning. Use the normal fresh recovery flow after +preserving its work, then repeat the unchanged follow-up check on the new workspace. + For `runs-adapters/session`, complete a task in a disposable warm sandbox and retain its saved workspace file. Let the warm session close normally and verify that the retained lease has a complete, lease-bound harness backup before diff --git a/server/src/services/heartbeat/workspace-preparation.test.ts b/server/src/services/heartbeat/workspace-preparation.test.ts index beb43a34b0f..255d0a5806f 100644 --- a/server/src/services/heartbeat/workspace-preparation.test.ts +++ b/server/src/services/heartbeat/workspace-preparation.test.ts @@ -7,7 +7,7 @@ import { pathToFileURL } from "node:url"; import { promisify } from "node:util"; import { captureDirectorySnapshot, directorySnapshotSha256, serializeDirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge"; import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; -import { agentWakeupRequests, heartbeatRunEvents, companyObservability, issueRecoveryActions, agents, authUsers, companyMemberships, companies, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, environments, executionWorkspaces, executionWorkspaceRepositoryAccess, heartbeatRuns, issues, issueAccessGrants, nativeRunFinalizations, plugins, projects, projectWorkspaces, type Db } from "@paperclipai/db"; +import { agentWakeupRequests, heartbeatRunEvents, companyObservability, issueRecoveryActions, agents, authUsers, companyMemberships, companies, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, environments, executionWorkspaces, executionWorkspaceRepositoryAccess, heartbeatRuns, issues, issueAccessGrants, nativeRunFinalizations, plugins, projects, projectWorkspaces, workspaceOperations, type Db } from "@paperclipai/db"; import { eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; @@ -170,15 +170,16 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace }); it.each([ - { kind: "empty", repoUrl: null, branchMode: null, fresh: false, ephemeral: true }, - { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: null, fresh: false }, - { kind: "git", repoUrl: "ssh://git@github.com/example/remote.git", branchMode: null, fresh: false }, - { kind: "git", repoUrl: "git@github.com:example/remote.git", branchMode: null, fresh: false }, - { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "existing", fresh: false }, - { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: false }, - { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true }, - ] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh) without materializing a controller checkout", async (scenario) => { - const { kind, repoUrl, branchMode, fresh } = scenario; + { kind: "empty", repoUrl: null, branchMode: null, fresh: false, strategyChange: false, ephemeral: true }, + { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: null, fresh: false, strategyChange: false }, + { kind: "git", repoUrl: "ssh://git@github.com/example/remote.git", branchMode: null, fresh: false, strategyChange: false }, + { kind: "git", repoUrl: "git@github.com:example/remote.git", branchMode: null, fresh: false, strategyChange: false }, + { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "existing", fresh: false, strategyChange: false }, + { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: false, strategyChange: false }, + { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true, strategyChange: false }, + { kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true, strategyChange: true }, + ] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh, strategy change: $strategyChange) without materializing a controller checkout", async (scenario) => { + const { kind, repoUrl, branchMode, fresh, strategyChange } = scenario; const { input, source, issue, envOrchestrator } = await fixture(); const [savedEnvironment] = await db.insert(environments).values({ name: `Remote ${randomUUID()}`, driver: "sandbox", status: "active", config: { provider: "daytona" } }).returning(); @@ -194,7 +195,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace if (branchMode) { input.policy.requestedExecutionWorkspaceMode = "isolated_workspace"; input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace"; - input.config.mergedConfig = { workspaceStrategy: { type: "git_worktree", baseRef: "main", + input.config.mergedConfig = strategyChange ? { workspaceStrategy: { type: "project_primary" } } : { workspaceStrategy: { type: "git_worktree", baseRef: "main", ...(branchMode === "existing" ? { existingBranch: "feature/diverged" } : { branchTemplate: "test-{{issue.identifier}}" }) } }; } if (kind === "git") { @@ -243,9 +244,44 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace expect(workspace.mode).toBe(branchMode ? "isolated_workspace" : "shared_workspace"); expect(workspace.metadata?.remoteWorkspace).toMatchObject({ source: { kind }, writerRunId: input.run.id }); if (branchMode) expect(workspace.metadata?.remoteWorkspace).toMatchObject({ source: { - ref: "main", branchMode, branchName: branchMode === "existing" ? "feature/diverged" : "test-WSP-1", + ref: "main", branchMode, + ...(strategyChange ? {} : { branchName: branchMode === "existing" ? "feature/diverged" : "test-WSP-1" }), } }); await expect(access(workspace.cwd!)).rejects.toThrow(); + if ((fresh || strategyChange) && branchMode === "create") { + expect(workspace.strategyType).toBe("git_worktree"); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }) + .where(eq(heartbeatRuns.id, input.run.id)); + const [saved] = await db.update(executionWorkspaces).set({ metadata: { + ...workspace.metadata, + remoteWorkspace: { ...workspace.metadata!.remoteWorkspace as object, writerRunId: null, dirty: false }, + } }).where(eq(executionWorkspaces.id, workspace.id)).returning(); + const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: issue.companyId, + agentId: input.agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning(); + input.run = nextRun; + input.task.freshWorkspace = null; + input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: saved.id, executionWorkspacePreference: "reuse_existing" }; + input.reuse.existingExecutionWorkspace = toExecutionWorkspace(saved); + input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(saved); + input.reuse.requestedShouldReuseExisting = true; + input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({ + issueExecutionWorkspaceId: saved.id, issueExecutionWorkspacePreference: "reuse_existing", existingExecutionWorkspaceStatus: "active", + }); + if (strategyChange) input.config.mergedConfig = { workspaceStrategy: { type: "git_worktree" } }; + await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached remote acquisition"); + const operations = await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, nextRun.id)); + const freshness = operations.filter(operation => operation.phase === "workspace_config_freshness"); + if (strategyChange) { + expect(freshness).toHaveLength(1); + expect(freshness[0].metadata).toMatchObject({ action: "replace", changedCategories: ["strategy"] }); + } else expect(freshness).toEqual([]); + const [retained] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, saved.id)); + expect(retained.id).toBe(workspace.id); + expect(retained.metadata?.configFingerprint).toMatchObject({ + workspaceHash: (saved.metadata?.configFingerprint as { workspaceHash: string }).workspaceHash, + }); + expect(retained.metadata?.remoteWorkspace).toMatchObject({ writerRunId: nextRun.id }); + } } finally { for (const [key, value] of previous) { if (value === undefined) delete process.env[key]; else process.env[key] = value; diff --git a/server/src/services/heartbeat/workspace-preparation.ts b/server/src/services/heartbeat/workspace-preparation.ts index e4c76adb48b..824822ff1a2 100644 --- a/server/src/services/heartbeat/workspace-preparation.ts +++ b/server/src/services/heartbeat/workspace-preparation.ts @@ -543,7 +543,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace const skipHostProvision = copyOnlyProvisioning || hostProvisionDriver === "sandbox" || hostProvisionDriver === "ssh" || hostProvisionDriver === "plugin"; const hostExecutionWorkspaceConfig = isDotRun ? {} : stripHostWorkspaceProvision({ - config: fresh ? { ...mergedConfig, workspaceStrategy: { type: "project_primary" } } : mergedConfig, + config: fresh && !remoteAuthority ? { ...mergedConfig, workspaceStrategy: { type: "project_primary" } } : mergedConfig, skipHostProvision, }); const executionWorkspaceBase = {