fix(heartbeat): record environment drift once for plugin-owned workspaces (#21947)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before each run, the heartbeat compares a fingerprint of the
workspace config with the fingerprint saved on the reused execution
workspace
> - An environment change is replacement-class drift. When the task asks
to reuse its workspace, the "replace" decision does not replace the
workspace. It only keeps the old fingerprint, so the drift stays visible
> - A workspace-provider plugin owns the files of the workspaces it
prepares. The environment only decides where the runner executes, so an
environment change cannot make those files wrong
> - A managed sandbox environment can change its template id with each
release. Each existing plugin workspace then logs a "replace" freshness
notice on every later run, and nothing ever clears it
> - This pull request records environment drift as a refresh for
plugin-owned workspaces, and makes Core's fingerprint win over the
provider row's stale copy
> - The benefit is that the notice appears once after a real change and
then stops. Reuse does not change

## Linked Issues or Issue Description

Refs #19349. That PR stopped cosmetic sandbox updates from advancing the
environment revision. This PR covers real config changes on plugin-owned
workspaces.

**What happened?**

A managed plugin environment changed its `driverConfig` (a new sandbox
template id after a release). After that, every run on an existing
plugin workspace printed:

```
[paperclip] Execution workspace reuse freshness action "replace" because execution workspace configuration changed: environment.
```

The run result showed `workspaceReused: true`, with the same workspace
id before and after. The stored fingerprint stayed the same on every
run, so the notice repeated forever.

**Expected behavior**

The workspace is reused, as it is now. The new fingerprint is saved, the
drift is reported once, and later runs report no drift.

**Steps to reproduce**

1. Prepare a plugin-owned workspace with a plugin environment.
2. Change the environment's `driverConfig`.
3. Run the task twice more with workspace reuse. Both runs report
`replace` for `environment`, and the stored `configFingerprint` does not
change.

## What Changed

- `server/src/services/heartbeat/workspaces.ts`:
`resolveExecutionWorkspaceConfigFreshness` takes an optional
`providerOwnedStorage` flag. With the flag, `environment` drift does not
require replacement. Other replacement-class drift (repo, strategy,
mode, project workspace, realization) still gives `replace`.
- `server/src/services/heartbeat/workspace-preparation.ts`:
  - Pass `providerOwnedStorage: Boolean(pluginPrepared)`.
- The plugin path spread the provider row's metadata over Core's
metadata. That row still holds the fingerprint it was prepared with, so
a refresh could never save the new one. Core's `configFingerprint` now
wins. The rest of the provider metadata, including its `config`
snapshot, is unchanged.
- `doc/heartbeat-internals.md`: describe the provider-owned storage
rule.

The freshness decision never controls reuse here. With reuse requested,
restore is `requestedShouldReuseExisting` for every action, and the
plugin picks the workspace before the decision is made. This change only
alters what Core saves on the workspace row and what it reports.

## Verification

- Red first. Without the fix, both new tests failed: the decision was
`replace`, not `refresh`.
- With only the resolver change, the preparation test still failed
because the stored fingerprint did not change. This shows the metadata
merge fix is necessary.
- `npx vitest run src/__tests__/heartbeat-workspace-session.test.ts
src/services/heartbeat/workspace-preparation.test.ts` (in `server/`): 2
files, 333 tests pass.
- `npx tsc --noEmit -p tsconfig.json` (in `server/`): passes.
- New tests:
- `heartbeat-workspace-session.test.ts`: environment drift on
provider-owned storage gives `refresh` and persists the fingerprint. The
same drift on Core-owned storage still gives `replace`. Environment plus
repo drift on provider-owned storage still gives `replace`.
- `workspace-preparation.test.ts`: a new `environmentDrift` case runs
three preparations on one plugin workspace. Run 2 records one `refresh`
freshness operation for `environment` with `workspaceReused: true` and
saves a new fingerprint. Run 3 records no freshness operation.
- Not run locally: `pnpm verify` and the full suite. CI supplies that
coverage.

## Risks

- Low risk. Reuse does not change, and no schema changes.
- Behavioral shift: a plugin-owned workspace whose environment changed
now saves the new fingerprint. A later return to the old environment
config is reported as drift again, as for any other refresh.
- If a future plugin environment change does affect the files of a
provider-owned workspace, this rule would hide it. The provider owns
those files, so the provider is responsible for that check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Nicky LeachandClaude Opus 5.5 authored and GitHub committed 2026-11-17 07:18:20 -08:00
1 parent c41ca4a992
commit 79c8cf8b6b
5 files changed
+94 -5

No files matched your search

@@ -1713,6 +1713,62 @@ describe("effective run execution workspace config freshness", () => {
expect(decision.changedCategories).toContain(category);
});
describe("provider-owned storage", () => {
const pluginEnvironment = (template: string) => ({
selectedEnvironmentId: "environment-1",
driver: "plugin",
config: { pluginKey: "example.sandbox", driverKey: "tasks", driverConfig: { template } },
});
it("records environment drift as a refresh that persists the new fingerprint", () => {
const base = buildWorkspaceConfigMetadata({ environment: pluginEnvironment("template-a") });
const next = buildWorkspaceConfigMetadata({ environment: pluginEnvironment("template-b") });
const decision = resolveExecutionWorkspaceConfigFreshness({
hasExistingWorkspace: true,
existingWorkspaceMetadata: persistedWorkspaceConfigFingerprint(base),
nextMetadata: next,
providerOwnedStorage: true,
});
const policy = resolveExecutionWorkspaceReuseProvisioningPolicy({
requestedShouldReuseExisting: true,
workspaceConfigFreshness: decision,
});
expect(decision).toMatchObject({ action: "refresh", shouldReuseExisting: true, changedCategories: ["environment"] });
expect(policy).toMatchObject({ shouldRestoreExistingWorkspace: true, shouldPersistLatestWorkspaceConfigMetadata: true });
});
it("still replaces on environment drift when Core owns the workspace", () => {
const decision = resolveExecutionWorkspaceConfigFreshness({
hasExistingWorkspace: true,
existingWorkspaceMetadata: persistedWorkspaceConfigFingerprint(
buildWorkspaceConfigMetadata({ environment: pluginEnvironment("template-a") }),
),
nextMetadata: buildWorkspaceConfigMetadata({ environment: pluginEnvironment("template-b") }),
});
expect(decision.action).toBe("replace");
});
it("still replaces when another replacement-class category drifts alongside the environment", () => {
const decision = resolveExecutionWorkspaceConfigFreshness({
hasExistingWorkspace: true,
existingWorkspaceMetadata: persistedWorkspaceConfigFingerprint(
buildWorkspaceConfigMetadata({ environment: pluginEnvironment("template-a") }),
),
nextMetadata: buildWorkspaceConfigMetadata({
environment: pluginEnvironment("template-b"),
repoRef: "origin/release",
}),
providerOwnedStorage: true,
});
expect(decision.action).toBe("replace");
expect(decision.changedCategories).toEqual(expect.arrayContaining(["environment", "repo"]));
});
});
it("keeps replacement-class drift visible when explicit reuse restores the old workspace", () => {
const base = buildWorkspaceConfigMetadata();
const next = buildWorkspaceConfigMetadata({
@@ -969,10 +969,11 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
{ fromDefault: false, mode: "isolated_workspace" as const, fresh: false, portable: true, verify: true },
{ fromDefault: false, mode: "isolated_workspace" as const, fresh: false, portable: true, verify: true, refuse: true },
{ fromDefault: false, mode: "isolated_workspace" as const, fresh: false, portable: true, verify: true, restart: true },
])("prepares managed storage with the selected Launcher (fromDefault: $fromDefault, mode: $mode, fresh: $fresh, portable: $portable, replace: $replace, replaceAfterPrepare: $replaceAfterPrepare, verify: $verify, refuse: $refuse, restart: $restart)", async ({ fromDefault, mode: defaultMode, fresh, portable, replace, replaceAfterPrepare, verify, refuse, restart }) => {
{ fromDefault: false, mode: "isolated_workspace" as const, fresh: false, portable: false, environmentDrift: true },
])("prepares managed storage with the selected Launcher (fromDefault: $fromDefault, mode: $mode, fresh: $fresh, portable: $portable, replace: $replace, replaceAfterPrepare: $replaceAfterPrepare, verify: $verify, refuse: $refuse, restart: $restart, environmentDrift: $environmentDrift)", async ({ fromDefault, mode: defaultMode, fresh, portable, replace, replaceAfterPrepare, verify, refuse, restart, environmentDrift }) => {
const { input, source, issue } = await fixture(true);
if (portable) input.agent.adapterType = "paperclip_runner";
const pluginKey = `test.managed-storage-${fromDefault}-${defaultMode}-${fresh}-${portable}${replace ? "-replacement" : replaceAfterPrepare ? "-late-replacement" : refuse ? "-refuse" : restart ? "-restart" : verify ? "-verify" : ""}`;
const pluginKey = `test.managed-storage-${fromDefault}-${defaultMode}-${fresh}-${portable}${replace ? "-replacement" : replaceAfterPrepare ? "-late-replacement" : refuse ? "-refuse" : restart ? "-restart" : verify ? "-verify" : environmentDrift ? "-environment-drift" : ""}`;
const [plugin] = await db.insert(plugins).values({ pluginKey, packageName: "test-storage", version: "1.0.0", apiVersion: 1, status: "ready", categories: ["workspace"],
manifestJson: { id: pluginKey, apiVersion: 1, version: "1.0.0", displayName: "Storage", description: "Storage", author: "Test", categories: ["workspace"],
workspaceProvider: { freshWorkspaceVersion: 1, finalizationVersion: 1, workspaceVerificationVersion: 1, modes: ["isolated_workspace", "operator_branch"], strategies: ["git_worktree"], ...(fromDefault ? { defaultForProjects: { mode: defaultMode, strategy: "git_worktree" as const } } : {}) }, capabilities: ["workspaces.manage", "events.subscribe", "environment.drivers.register"],
@@ -1195,6 +1196,27 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
const [task] = await db.select().from(issues).where(eq(issues.id, issue.id));
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, task.executionWorkspaceId!));
expect(workspace).toMatchObject({ providerType: "plugin", cwd: null, mode: defaultMode, strategyType: "git_worktree", branchName: fresh ? null : fromDefault ? "WSP-1-prepare-this-workspace" : "test-WSP-1" });
if (environmentDrift) {
const fingerprintOf = (row: typeof workspace) => (row.metadata?.configFingerprint as { workspaceHash: string }).workspaceHash;
const freshnessFor = async (runId: string) => (await db.select().from(workspaceOperations)
.where(eq(workspaceOperations.heartbeatRunId, runId))).filter(operation => operation.phase === "workspace_config_freshness");
await db.update(environments).set({ config: { pluginKey, driverKey: "tasks", driverConfig: { template: "next" } } })
.where(eq(environments.id, environment.id));
input.environment.selectedEnvironmentForConfig = await environmentService(db).getById(environment.id);
await reuse(input, toExecutionWorkspace(workspace));
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached execution handoff");
const drifted = await freshnessFor(input.run.id);
expect(drifted).toHaveLength(1);
expect(drifted[0].metadata).toMatchObject({ action: "refresh", changedCategories: ["environment"], workspaceReused: true });
const [refreshed] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id));
expect(fingerprintOf(refreshed)).not.toBe(fingerprintOf(workspace));
await reuse(input, toExecutionWorkspace(refreshed));
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached execution handoff");
expect(await freshnessFor(input.run.id)).toEqual([]);
const [settled] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id));
expect(fingerprintOf(settled)).toBe(fingerprintOf(refreshed));
return;
}
if (portable) {
const finalization = { runId: input.run.id, agentId: input.agent.id };
const plan = await resolvePluginWorkspaceFinalization(db, workspace, finalization);
@@ -35,6 +35,7 @@ import {
reconcileReusedExecutionWorkspaceProjectWorkspaceId,
recordWorkspaceConfigFreshnessOperation,
prepareProjectRepositoryWorkspaces,
WORKSPACE_CONFIG_FINGERPRINT_METADATA_KEY,
} from "./workspaces.js";
import { resolvePaperclipInstanceRoot } from "../../home-paths.js";
import fs from "node:fs/promises";
@@ -863,6 +864,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
reusableExistingExecutionWorkspace?.metadata ?? null,
inferredMetadata: inferredExistingWorkspaceConfigMetadata,
nextMetadata: latestWorkspaceConfigMetadata,
providerOwnedStorage: Boolean(pluginPrepared),
},
);
const workspaceReuseProvisioningPolicy =
@@ -1182,9 +1184,13 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
);
}
}
const coreConfigFingerprint = baseExecutionWorkspaceMetadata?.[WORKSPACE_CONFIG_FINGERPRINT_METADATA_KEY];
const nextExecutionWorkspaceMetadata = {
...baseExecutionWorkspaceMetadata,
...(pluginPrepared?.metadata ?? {}),
...(pluginPrepared && coreConfigFingerprint !== undefined
? { [WORKSPACE_CONFIG_FINGERPRINT_METADATA_KEY]: coreConfigFingerprint }
: {}),
...(taskWorkspaceOwner && !pluginPrepared ? { runnerTaskWorkspaceOwner: taskWorkspaceOwner } : {}),
...(providerFresh && pluginTaskWorkspace && !pluginPrepared ? { runnerTaskFreshWorkspace: providerFresh } : {}),
...(copyOnlyProvisioning ? { [CONTROLLER_PROVISION_COPY_ONLY]: true } : {}),
+6 -3
View File
@@ -2128,7 +2128,7 @@ const PAPERCLIP_SESSION_METADATA_KEYS = new Set([
SESSION_CONFIG_CATEGORY_FINGERPRINTS_KEY,
]);
const WORKSPACE_CONFIG_FINGERPRINT_METADATA_KEY = "configFingerprint";
export const WORKSPACE_CONFIG_FINGERPRINT_METADATA_KEY = "configFingerprint";
const EFFECTIVE_RUN_SESSION_CONFIG_CATEGORIES = [
"adapter",
@@ -3034,6 +3034,7 @@ export function resolveExecutionWorkspaceConfigFreshness(input: {
existingWorkspaceMetadata: Record<string, unknown> | null | undefined;
inferredMetadata?: EffectiveRunWorkspaceConfigMetadata | null;
nextMetadata: EffectiveRunWorkspaceConfigMetadata | null;
providerOwnedStorage?: boolean;
}): ExecutionWorkspaceConfigFreshnessDecision {
if (!input.hasExistingWorkspace) {
return {
@@ -3138,8 +3139,10 @@ export function resolveExecutionWorkspaceConfigFreshness(input: {
previous: previous.categoryFingerprints,
next: input.nextMetadata.categoryFingerprints,
});
const replacementRequired = changedCategories.some((category) =>
WORKSPACE_REPLACEMENT_CONFIG_CATEGORIES.has(category),
const replacementRequired = changedCategories.some(
(category) =>
WORKSPACE_REPLACEMENT_CONFIG_CATEGORIES.has(category) &&
!(input.providerOwnedStorage && category === "environment"),
);
const action: WorkspaceConfigFreshnessDecisionAction = replacementRequired
? "replace"