fix(heartbeat): keep remote retry workspace strategy stable (#18660)

## Thinking Path

> - Paperclip preserves agent work across separate execution services.
> - A remote retry can create a new isolated Git workspace.
> - Later admissions compare its saved configuration with the current
policy.
> - Fresh preparation also creates a temporary controller-side strategy.
> - Remote preparation saved that temporary strategy in its fingerprint.
> - The next unchanged turn then reported a workspace strategy
replacement.
> - This change saves the admitted remote strategy so unchanged reuse
stays stable.

## Linked Issues or Issue Description

Related preparation work: #18543 and #17924. I searched open and merged
workspace/session repairs and public workspace-strategy issues. None
covers this fresh remote retry case.

**What happened?**

After Retry created an isolated remote Git workspace, each unchanged
follow-up reported `workspace_config_freshness` with `action: replace`
and `changedCategories: [strategy]`. It retained the same workspace. The
stored fingerprint contained `project_primary`, which was only a
temporary controller provisioning choice.

**Expected behavior**

Save the configured remote strategy. An unchanged follow-up must reuse
the same workspace without a false replacement report. A real strategy,
base-ref, mode or environment change must retain its replacement checks.

**Steps to reproduce**

1. Prepare a fresh isolated remote Git retry with a `git_worktree`
strategy.
2. Complete its runtime handoff.
3. Admit another turn with the same workspace policy.
4. Inspect the recorded workspace freshness decision.

**Paperclip version or commit**

Reproduced on `b9e94b85a69bf679818eeef9ecf4655554a89c6b`. The regression
also fails on the unchanged preparation code in the current base.

**Deployment mode**

Authenticated server with remote workspace authority and a sandbox
execution target.

## What Changed

- Keep the admitted strategy when a fresh workspace uses remote
authority.
- Retain the temporary controller strategy for existing host-side seed
preparation.
- Extend the PostgreSQL preparation test through a fresh admission and
its next ordinary admission. Assert the same workspace, current writer
and stable fingerprint, with no false drift operation.
- Cover a genuinely configured `project_primary` fresh remote Git retry
followed by a real strategy change, preserving its replacement decision
and stored fingerprint.
- Document the execution rule and the separate live verification recipe.

## Verification

- The final regression was run before and after the production change.
Before: the next admission recorded a strategy replacement. After:
unchanged reuse records no drift and retains the workspace and
fingerprint.
- All 312 targeted cases pass on
`48b3e548afa42e740871feae2763d88fd5a00431` across workspace preparation,
workspace/session policy and inherited session preparation. These
include actual strategy/base-ref changes and the rule that explicit
reuse must not hide real replacement-class drift.
- All eight remote preparation cases also pass with the compatibility
regression: a genuinely configured `project_primary` fresh Git retry
creates a `git_worktree` row, and its later real strategy change retains
the replacement decision and saved fingerprint.
- `node scripts/check-maps.mjs` and `git diff --check` pass.
- Full recursive typecheck and `pnpm build` pass locally on this head.
- The [full current-head CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/41524105868)
passes: typecheck/build, all general server, chat, workspace and
serialized test groups, plus Runner and E2E checks. The workflow and
`run-vitest-stable.mjs` partition the complete suite.
- The duplicate local `pnpm test:run` passed all 1,828 chat cases and
was intentionally stopped during general-server after the same complete
CI matrix passed. This is not recorded as a full local-suite pass.
- Greptile and Paperclip Review both assessed this head at 5/5 with no
new findings. Paperclip Review accepted the compatibility boundary after
the counterexample; its check conclusion is neutral. The addressed
review thread is resolved.
- The regression uses real PostgreSQL and preparation policy. Runtime
acquisition is controlled at its external boundary. The new fix has not
been deployed; live provider resumption remains unverified for this
patch.

## Risks

- Existing incorrect fingerprints are not rewritten. The same stored
shape can come from a genuinely configured `project_primary` remote Git
retry; a Git workspace row alone cannot prove the old admission policy.
Rewriting it would hide a real policy change. After preserving and
inspecting retained work, an explicitly admitted fresh recovery after
rollout establishes a correct new fingerprint. Ordinary reuse of the old
workspace does not migrate it. Normal recovery gates remain required.
- Real policy changes still use the current replacement guard. No
schema, credential or authority change is included.

## Model Used

OpenAI GPT-6 Codex with reasoning, repository editing, shell execution,
PostgreSQL tests and browser inspection. The host does not expose the
exact model variant or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-02 16:11:30 -06:00
1 parent 482706e7dd
commit b4315943e8
5 files changed
+87 -13

No files matched your search

+14
View File
@@ -734,6 +734,20 @@ A workspace-coherent adapter path means:
- the effective cwd exists or is provider-reachable, according to the workspace provider
- when the adapter or workspace strategy relies on git state, the cwd is git-valid for the selected workspace: it resolves to the expected repository root, required base refs or branch metadata can be resolved, and runtime-created worktrees are still registered or explicitly recoverable
Fresh remote-workspace recovery fingerprints the admitted remote strategy. A
temporary strategy used to seed a controller directory must not become the saved
remote configuration. An unchanged follow-up reuses that configuration without
reporting strategy drift. Real strategy, source, mode and environment changes
retain their existing replacement checks.
Older remote Git workspaces can retain the temporary `project_primary`
fingerprint. That fingerprint alone cannot distinguish the old preparation bug
from a genuinely configured `project_primary` strategy; the remote workspace row
uses `git_worktree` for either Git source. Do not rewrite those fingerprints in
place or suppress their replacement checks. After saving and inspecting the
retained work, an explicitly admitted fresh recovery creates a workspace with the
correct fingerprint. Ordinary reuse of the old workspace does not migrate it.
Adapter-backed liveness also requires control-plane reachability from the agent's actual mutation surface, not just from the host adapter process. If the agent is expected to use Bash, shell tools, runtime helpers, or in-sandbox command execution to update issues, create comments, upload artifacts, or submit review decisions, the `PAPERCLIP_API_URL` and `PAPERCLIP_API_KEY` visible to that surface must route to Paperclip successfully.
For sandbox-backed local adapters, Paperclip may satisfy that contract with a run-scoped in-sandbox bridge. The host adapter keeps the real run JWT on the host side, injects only the bridge URL/token into the sandbox tool environment, and forwards allowed Paperclip API requests with the run id attached. The bridge credentials are execution plumbing, not user-facing context: they must not be written into prompts, visible comments, issue documents, restored workspace files, or durable logs. Agents and skills must use the env vars available in Bash/curl rather than assuming that the host's localhost API URL is reachable from browser or web-extraction tools inside the sandbox.
+12
View File
@@ -320,6 +320,18 @@ session-control access. Deployed diagnostic usefulness remains unverified.
## `session`
Fresh remote Retry preparation retains the admitted strategy in its saved
fingerprint. An unchanged follow-up must not report strategy drift; actual policy
changes retain the replacement guard. This changes no retry, session-reset,
workspace or credential authority. The
[preparation regression](../../server/src/services/heartbeat/workspace-preparation.test.ts)
uses PostgreSQL and controlled runtime acquisition. Live provider resumption
remains a separate check in the [session recipe](../feature-map/runs-adapters.md#session).
Older strategy fingerprints retain their existing guard until an explicitly
admitted fresh recovery. A remote Git row alone does not authorize rewriting its
configuration history or bypassing a real policy change.
Scoped source evidence: restart preparation transfers its lease renewal to the
native executor after the executor acquires its own reference. Completion stops
renewal before coordinator release, while real ownership loss retains the detach
+12
View File
@@ -146,6 +146,18 @@ must still reset the session and preserve the stored workspace owner. The
checks this with PostgreSQL and synthetic provider execution; it is not live
warm-process or latency evidence.
After an isolated remote Retry, send a follow-up with unchanged workspace policy.
Verify the same workspace remains bound and no strategy-replacement operation is
recorded. Change the strategy or base ref and confirm that the normal replacement
guard still applies. The
[remote preparation regression](../../server/src/services/heartbeat/workspace-preparation.test.ts)
checks persisted configuration across both admissions; it does not prove a live
provider conversation resumed.
For a workspace carrying an older temporary fingerprint, verify that ordinary
reuse retains the strategy warning. Use the normal fresh recovery flow after
preserving its work, then repeat the unchanged follow-up check on the new workspace.
For `runs-adapters/session`, complete a task in a disposable warm sandbox and
retain its saved workspace file. Let the warm session close normally and verify
that the retained lease has a complete, lease-bound harness backup before
@@ -7,7 +7,7 @@ import { pathToFileURL } from "node:url";
import { promisify } from "node:util";
import { captureDirectorySnapshot, directorySnapshotSha256, serializeDirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge";
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import { agentWakeupRequests, heartbeatRunEvents, companyObservability, issueRecoveryActions, agents, authUsers, companyMemberships, companies, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, environments, executionWorkspaces, executionWorkspaceRepositoryAccess, heartbeatRuns, issues, issueAccessGrants, nativeRunFinalizations, plugins, projects, projectWorkspaces, type Db } from "@paperclipai/db";
import { agentWakeupRequests, heartbeatRunEvents, companyObservability, issueRecoveryActions, agents, authUsers, companyMemberships, companies, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, environments, executionWorkspaces, executionWorkspaceRepositoryAccess, heartbeatRuns, issues, issueAccessGrants, nativeRunFinalizations, plugins, projects, projectWorkspaces, workspaceOperations, type Db } from "@paperclipai/db";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
@@ -170,15 +170,16 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
});
it.each([
{ kind: "empty", repoUrl: null, branchMode: null, fresh: false, ephemeral: true },
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: null, fresh: false },
{ kind: "git", repoUrl: "ssh://git@github.com/example/remote.git", branchMode: null, fresh: false },
{ kind: "git", repoUrl: "git@github.com:example/remote.git", branchMode: null, fresh: false },
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "existing", fresh: false },
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: false },
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true },
] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh) without materializing a controller checkout", async (scenario) => {
const { kind, repoUrl, branchMode, fresh } = scenario;
{ kind: "empty", repoUrl: null, branchMode: null, fresh: false, strategyChange: false, ephemeral: true },
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: null, fresh: false, strategyChange: false },
{ kind: "git", repoUrl: "ssh://git@github.com/example/remote.git", branchMode: null, fresh: false, strategyChange: false },
{ kind: "git", repoUrl: "git@github.com:example/remote.git", branchMode: null, fresh: false, strategyChange: false },
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "existing", fresh: false, strategyChange: false },
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: false, strategyChange: false },
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true, strategyChange: false },
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true, strategyChange: true },
] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh, strategy change: $strategyChange) without materializing a controller checkout", async (scenario) => {
const { kind, repoUrl, branchMode, fresh, strategyChange } = scenario;
const { input, source, issue, envOrchestrator } = await fixture();
const [savedEnvironment] = await db.insert(environments).values({ name: `Remote ${randomUUID()}`,
driver: "sandbox", status: "active", config: { provider: "daytona" } }).returning();
@@ -194,7 +195,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
if (branchMode) {
input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
input.config.mergedConfig = { workspaceStrategy: { type: "git_worktree", baseRef: "main",
input.config.mergedConfig = strategyChange ? { workspaceStrategy: { type: "project_primary" } } : { workspaceStrategy: { type: "git_worktree", baseRef: "main",
...(branchMode === "existing" ? { existingBranch: "feature/diverged" } : { branchTemplate: "test-{{issue.identifier}}" }) } };
}
if (kind === "git") {
@@ -243,9 +244,44 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
expect(workspace.mode).toBe(branchMode ? "isolated_workspace" : "shared_workspace");
expect(workspace.metadata?.remoteWorkspace).toMatchObject({ source: { kind }, writerRunId: input.run.id });
if (branchMode) expect(workspace.metadata?.remoteWorkspace).toMatchObject({ source: {
ref: "main", branchMode, branchName: branchMode === "existing" ? "feature/diverged" : "test-WSP-1",
ref: "main", branchMode,
...(strategyChange ? {} : { branchName: branchMode === "existing" ? "feature/diverged" : "test-WSP-1" }),
} });
await expect(access(workspace.cwd!)).rejects.toThrow();
if ((fresh || strategyChange) && branchMode === "create") {
expect(workspace.strategyType).toBe("git_worktree");
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() })
.where(eq(heartbeatRuns.id, input.run.id));
const [saved] = await db.update(executionWorkspaces).set({ metadata: {
...workspace.metadata,
remoteWorkspace: { ...workspace.metadata!.remoteWorkspace as object, writerRunId: null, dirty: false },
} }).where(eq(executionWorkspaces.id, workspace.id)).returning();
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: issue.companyId,
agentId: input.agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
input.run = nextRun;
input.task.freshWorkspace = null;
input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: saved.id, executionWorkspacePreference: "reuse_existing" };
input.reuse.existingExecutionWorkspace = toExecutionWorkspace(saved);
input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(saved);
input.reuse.requestedShouldReuseExisting = true;
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({
issueExecutionWorkspaceId: saved.id, issueExecutionWorkspacePreference: "reuse_existing", existingExecutionWorkspaceStatus: "active",
});
if (strategyChange) input.config.mergedConfig = { workspaceStrategy: { type: "git_worktree" } };
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached remote acquisition");
const operations = await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, nextRun.id));
const freshness = operations.filter(operation => operation.phase === "workspace_config_freshness");
if (strategyChange) {
expect(freshness).toHaveLength(1);
expect(freshness[0].metadata).toMatchObject({ action: "replace", changedCategories: ["strategy"] });
} else expect(freshness).toEqual([]);
const [retained] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, saved.id));
expect(retained.id).toBe(workspace.id);
expect(retained.metadata?.configFingerprint).toMatchObject({
workspaceHash: (saved.metadata?.configFingerprint as { workspaceHash: string }).workspaceHash,
});
expect(retained.metadata?.remoteWorkspace).toMatchObject({ writerRunId: nextRun.id });
}
} finally {
for (const [key, value] of previous) {
if (value === undefined) delete process.env[key]; else process.env[key] = value;
@@ -543,7 +543,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
const skipHostProvision = copyOnlyProvisioning || hostProvisionDriver === "sandbox" || hostProvisionDriver === "ssh" || hostProvisionDriver === "plugin";
const hostExecutionWorkspaceConfig = isDotRun ? {} :
stripHostWorkspaceProvision({
config: fresh ? { ...mergedConfig, workspaceStrategy: { type: "project_primary" } } : mergedConfig,
config: fresh && !remoteAuthority ? { ...mergedConfig, workspaceStrategy: { type: "project_primary" } } : mergedConfig,
skipHostProvision,
});
const executionWorkspaceBase = {