mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(heartbeat): keep remote retry workspace strategy stable (#18660)
## Thinking Path > - Paperclip preserves agent work across separate execution services. > - A remote retry can create a new isolated Git workspace. > - Later admissions compare its saved configuration with the current policy. > - Fresh preparation also creates a temporary controller-side strategy. > - Remote preparation saved that temporary strategy in its fingerprint. > - The next unchanged turn then reported a workspace strategy replacement. > - This change saves the admitted remote strategy so unchanged reuse stays stable. ## Linked Issues or Issue Description Related preparation work: #18543 and #17924. I searched open and merged workspace/session repairs and public workspace-strategy issues. None covers this fresh remote retry case. **What happened?** After Retry created an isolated remote Git workspace, each unchanged follow-up reported `workspace_config_freshness` with `action: replace` and `changedCategories: [strategy]`. It retained the same workspace. The stored fingerprint contained `project_primary`, which was only a temporary controller provisioning choice. **Expected behavior** Save the configured remote strategy. An unchanged follow-up must reuse the same workspace without a false replacement report. A real strategy, base-ref, mode or environment change must retain its replacement checks. **Steps to reproduce** 1. Prepare a fresh isolated remote Git retry with a `git_worktree` strategy. 2. Complete its runtime handoff. 3. Admit another turn with the same workspace policy. 4. Inspect the recorded workspace freshness decision. **Paperclip version or commit** Reproduced on `b9e94b85a69bf679818eeef9ecf4655554a89c6b`. The regression also fails on the unchanged preparation code in the current base. **Deployment mode** Authenticated server with remote workspace authority and a sandbox execution target. ## What Changed - Keep the admitted strategy when a fresh workspace uses remote authority. - Retain the temporary controller strategy for existing host-side seed preparation. - Extend the PostgreSQL preparation test through a fresh admission and its next ordinary admission. Assert the same workspace, current writer and stable fingerprint, with no false drift operation. - Cover a genuinely configured `project_primary` fresh remote Git retry followed by a real strategy change, preserving its replacement decision and stored fingerprint. - Document the execution rule and the separate live verification recipe. ## Verification - The final regression was run before and after the production change. Before: the next admission recorded a strategy replacement. After: unchanged reuse records no drift and retains the workspace and fingerprint. - All 312 targeted cases pass on `48b3e548afa42e740871feae2763d88fd5a00431` across workspace preparation, workspace/session policy and inherited session preparation. These include actual strategy/base-ref changes and the rule that explicit reuse must not hide real replacement-class drift. - All eight remote preparation cases also pass with the compatibility regression: a genuinely configured `project_primary` fresh Git retry creates a `git_worktree` row, and its later real strategy change retains the replacement decision and saved fingerprint. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Full recursive typecheck and `pnpm build` pass locally on this head. - The [full current-head CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41524105868) passes: typecheck/build, all general server, chat, workspace and serialized test groups, plus Runner and E2E checks. The workflow and `run-vitest-stable.mjs` partition the complete suite. - The duplicate local `pnpm test:run` passed all 1,828 chat cases and was intentionally stopped during general-server after the same complete CI matrix passed. This is not recorded as a full local-suite pass. - Greptile and Paperclip Review both assessed this head at 5/5 with no new findings. Paperclip Review accepted the compatibility boundary after the counterexample; its check conclusion is neutral. The addressed review thread is resolved. - The regression uses real PostgreSQL and preparation policy. Runtime acquisition is controlled at its external boundary. The new fix has not been deployed; live provider resumption remains unverified for this patch. ## Risks - Existing incorrect fingerprints are not rewritten. The same stored shape can come from a genuinely configured `project_primary` remote Git retry; a Git workspace row alone cannot prove the old admission policy. Rewriting it would hide a real policy change. After preserving and inspecting retained work, an explicitly admitted fresh recovery after rollout establishes a correct new fingerprint. Ordinary reuse of the old workspace does not migrate it. Normal recovery gates remain required. - Real policy changes still use the current replacement guard. No schema, credential or authority change is included. ## Model Used OpenAI GPT-6 Codex with reasoning, repository editing, shell execution, PostgreSQL tests and browser inspection. The host does not expose the exact model variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
482706e7dd
commit
b4315943e8
5 files changed
+87
-13
No files matched your search
@@ -734,6 +734,20 @@ A workspace-coherent adapter path means:
|
||||
- the effective cwd exists or is provider-reachable, according to the workspace provider
|
||||
- when the adapter or workspace strategy relies on git state, the cwd is git-valid for the selected workspace: it resolves to the expected repository root, required base refs or branch metadata can be resolved, and runtime-created worktrees are still registered or explicitly recoverable
|
||||
|
||||
Fresh remote-workspace recovery fingerprints the admitted remote strategy. A
|
||||
temporary strategy used to seed a controller directory must not become the saved
|
||||
remote configuration. An unchanged follow-up reuses that configuration without
|
||||
reporting strategy drift. Real strategy, source, mode and environment changes
|
||||
retain their existing replacement checks.
|
||||
|
||||
Older remote Git workspaces can retain the temporary `project_primary`
|
||||
fingerprint. That fingerprint alone cannot distinguish the old preparation bug
|
||||
from a genuinely configured `project_primary` strategy; the remote workspace row
|
||||
uses `git_worktree` for either Git source. Do not rewrite those fingerprints in
|
||||
place or suppress their replacement checks. After saving and inspecting the
|
||||
retained work, an explicitly admitted fresh recovery creates a workspace with the
|
||||
correct fingerprint. Ordinary reuse of the old workspace does not migrate it.
|
||||
|
||||
Adapter-backed liveness also requires control-plane reachability from the agent's actual mutation surface, not just from the host adapter process. If the agent is expected to use Bash, shell tools, runtime helpers, or in-sandbox command execution to update issues, create comments, upload artifacts, or submit review decisions, the `PAPERCLIP_API_URL` and `PAPERCLIP_API_KEY` visible to that surface must route to Paperclip successfully.
|
||||
|
||||
For sandbox-backed local adapters, Paperclip may satisfy that contract with a run-scoped in-sandbox bridge. The host adapter keeps the real run JWT on the host side, injects only the bridge URL/token into the sandbox tool environment, and forwards allowed Paperclip API requests with the run id attached. The bridge credentials are execution plumbing, not user-facing context: they must not be written into prompts, visible comments, issue documents, restored workspace files, or durable logs. Agents and skills must use the env vars available in Bash/curl rather than assuming that the host's localhost API URL is reachable from browser or web-extraction tools inside the sandbox.
|
||||
|
||||
@@ -320,6 +320,18 @@ session-control access. Deployed diagnostic usefulness remains unverified.
|
||||
|
||||
## `session`
|
||||
|
||||
Fresh remote Retry preparation retains the admitted strategy in its saved
|
||||
fingerprint. An unchanged follow-up must not report strategy drift; actual policy
|
||||
changes retain the replacement guard. This changes no retry, session-reset,
|
||||
workspace or credential authority. The
|
||||
[preparation regression](../../server/src/services/heartbeat/workspace-preparation.test.ts)
|
||||
uses PostgreSQL and controlled runtime acquisition. Live provider resumption
|
||||
remains a separate check in the [session recipe](../feature-map/runs-adapters.md#session).
|
||||
|
||||
Older strategy fingerprints retain their existing guard until an explicitly
|
||||
admitted fresh recovery. A remote Git row alone does not authorize rewriting its
|
||||
configuration history or bypassing a real policy change.
|
||||
|
||||
Scoped source evidence: restart preparation transfers its lease renewal to the
|
||||
native executor after the executor acquires its own reference. Completion stops
|
||||
renewal before coordinator release, while real ownership loss retains the detach
|
||||
|
||||
@@ -146,6 +146,18 @@ must still reset the session and preserve the stored workspace owner. The
|
||||
checks this with PostgreSQL and synthetic provider execution; it is not live
|
||||
warm-process or latency evidence.
|
||||
|
||||
After an isolated remote Retry, send a follow-up with unchanged workspace policy.
|
||||
Verify the same workspace remains bound and no strategy-replacement operation is
|
||||
recorded. Change the strategy or base ref and confirm that the normal replacement
|
||||
guard still applies. The
|
||||
[remote preparation regression](../../server/src/services/heartbeat/workspace-preparation.test.ts)
|
||||
checks persisted configuration across both admissions; it does not prove a live
|
||||
provider conversation resumed.
|
||||
|
||||
For a workspace carrying an older temporary fingerprint, verify that ordinary
|
||||
reuse retains the strategy warning. Use the normal fresh recovery flow after
|
||||
preserving its work, then repeat the unchanged follow-up check on the new workspace.
|
||||
|
||||
For `runs-adapters/session`, complete a task in a disposable warm sandbox and
|
||||
retain its saved workspace file. Let the warm session close normally and verify
|
||||
that the retained lease has a complete, lease-bound harness backup before
|
||||
|
||||
@@ -7,7 +7,7 @@ import { pathToFileURL } from "node:url";
|
||||
import { promisify } from "node:util";
|
||||
import { captureDirectorySnapshot, directorySnapshotSha256, serializeDirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import { agentWakeupRequests, heartbeatRunEvents, companyObservability, issueRecoveryActions, agents, authUsers, companyMemberships, companies, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, environments, executionWorkspaces, executionWorkspaceRepositoryAccess, heartbeatRuns, issues, issueAccessGrants, nativeRunFinalizations, plugins, projects, projectWorkspaces, type Db } from "@paperclipai/db";
|
||||
import { agentWakeupRequests, heartbeatRunEvents, companyObservability, issueRecoveryActions, agents, authUsers, companyMemberships, companies, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, environments, executionWorkspaces, executionWorkspaceRepositoryAccess, heartbeatRuns, issues, issueAccessGrants, nativeRunFinalizations, plugins, projects, projectWorkspaces, workspaceOperations, type Db } from "@paperclipai/db";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
@@ -170,15 +170,16 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ kind: "empty", repoUrl: null, branchMode: null, fresh: false, ephemeral: true },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: null, fresh: false },
|
||||
{ kind: "git", repoUrl: "ssh://git@github.com/example/remote.git", branchMode: null, fresh: false },
|
||||
{ kind: "git", repoUrl: "git@github.com:example/remote.git", branchMode: null, fresh: false },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "existing", fresh: false },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: false },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true },
|
||||
] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh) without materializing a controller checkout", async (scenario) => {
|
||||
const { kind, repoUrl, branchMode, fresh } = scenario;
|
||||
{ kind: "empty", repoUrl: null, branchMode: null, fresh: false, strategyChange: false, ephemeral: true },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: null, fresh: false, strategyChange: false },
|
||||
{ kind: "git", repoUrl: "ssh://git@github.com/example/remote.git", branchMode: null, fresh: false, strategyChange: false },
|
||||
{ kind: "git", repoUrl: "git@github.com:example/remote.git", branchMode: null, fresh: false, strategyChange: false },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "existing", fresh: false, strategyChange: false },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: false, strategyChange: false },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true, strategyChange: false },
|
||||
{ kind: "git", repoUrl: "https://github.com/example/remote.git", branchMode: "create", fresh: true, strategyChange: true },
|
||||
] as const)("reserves a $kind remote workspace ($repoUrl, branch: $branchMode, fresh: $fresh, strategy change: $strategyChange) without materializing a controller checkout", async (scenario) => {
|
||||
const { kind, repoUrl, branchMode, fresh, strategyChange } = scenario;
|
||||
const { input, source, issue, envOrchestrator } = await fixture();
|
||||
const [savedEnvironment] = await db.insert(environments).values({ name: `Remote ${randomUUID()}`,
|
||||
driver: "sandbox", status: "active", config: { provider: "daytona" } }).returning();
|
||||
@@ -194,7 +195,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
|
||||
if (branchMode) {
|
||||
input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
|
||||
input.policy.effectiveExecutionWorkspaceMode = "isolated_workspace";
|
||||
input.config.mergedConfig = { workspaceStrategy: { type: "git_worktree", baseRef: "main",
|
||||
input.config.mergedConfig = strategyChange ? { workspaceStrategy: { type: "project_primary" } } : { workspaceStrategy: { type: "git_worktree", baseRef: "main",
|
||||
...(branchMode === "existing" ? { existingBranch: "feature/diverged" } : { branchTemplate: "test-{{issue.identifier}}" }) } };
|
||||
}
|
||||
if (kind === "git") {
|
||||
@@ -243,9 +244,44 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
|
||||
expect(workspace.mode).toBe(branchMode ? "isolated_workspace" : "shared_workspace");
|
||||
expect(workspace.metadata?.remoteWorkspace).toMatchObject({ source: { kind }, writerRunId: input.run.id });
|
||||
if (branchMode) expect(workspace.metadata?.remoteWorkspace).toMatchObject({ source: {
|
||||
ref: "main", branchMode, branchName: branchMode === "existing" ? "feature/diverged" : "test-WSP-1",
|
||||
ref: "main", branchMode,
|
||||
...(strategyChange ? {} : { branchName: branchMode === "existing" ? "feature/diverged" : "test-WSP-1" }),
|
||||
} });
|
||||
await expect(access(workspace.cwd!)).rejects.toThrow();
|
||||
if ((fresh || strategyChange) && branchMode === "create") {
|
||||
expect(workspace.strategyType).toBe("git_worktree");
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() })
|
||||
.where(eq(heartbeatRuns.id, input.run.id));
|
||||
const [saved] = await db.update(executionWorkspaces).set({ metadata: {
|
||||
...workspace.metadata,
|
||||
remoteWorkspace: { ...workspace.metadata!.remoteWorkspace as object, writerRunId: null, dirty: false },
|
||||
} }).where(eq(executionWorkspaces.id, workspace.id)).returning();
|
||||
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: issue.companyId,
|
||||
agentId: input.agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: "legacy" }).returning();
|
||||
input.run = nextRun;
|
||||
input.task.freshWorkspace = null;
|
||||
input.task.issueRef = { ...input.task.issueRef!, executionWorkspaceId: saved.id, executionWorkspacePreference: "reuse_existing" };
|
||||
input.reuse.existingExecutionWorkspace = toExecutionWorkspace(saved);
|
||||
input.reuse.reusableExistingExecutionWorkspace = toExecutionWorkspace(saved);
|
||||
input.reuse.requestedShouldReuseExisting = true;
|
||||
input.reuse.workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({
|
||||
issueExecutionWorkspaceId: saved.id, issueExecutionWorkspacePreference: "reuse_existing", existingExecutionWorkspaceStatus: "active",
|
||||
});
|
||||
if (strategyChange) input.config.mergedConfig = { workspaceStrategy: { type: "git_worktree" } };
|
||||
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("reached remote acquisition");
|
||||
const operations = await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, nextRun.id));
|
||||
const freshness = operations.filter(operation => operation.phase === "workspace_config_freshness");
|
||||
if (strategyChange) {
|
||||
expect(freshness).toHaveLength(1);
|
||||
expect(freshness[0].metadata).toMatchObject({ action: "replace", changedCategories: ["strategy"] });
|
||||
} else expect(freshness).toEqual([]);
|
||||
const [retained] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, saved.id));
|
||||
expect(retained.id).toBe(workspace.id);
|
||||
expect(retained.metadata?.configFingerprint).toMatchObject({
|
||||
workspaceHash: (saved.metadata?.configFingerprint as { workspaceHash: string }).workspaceHash,
|
||||
});
|
||||
expect(retained.metadata?.remoteWorkspace).toMatchObject({ writerRunId: nextRun.id });
|
||||
}
|
||||
} finally {
|
||||
for (const [key, value] of previous) {
|
||||
if (value === undefined) delete process.env[key]; else process.env[key] = value;
|
||||
|
||||
@@ -543,7 +543,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
|
||||
const skipHostProvision = copyOnlyProvisioning || hostProvisionDriver === "sandbox" || hostProvisionDriver === "ssh" || hostProvisionDriver === "plugin";
|
||||
const hostExecutionWorkspaceConfig = isDotRun ? {} :
|
||||
stripHostWorkspaceProvision({
|
||||
config: fresh ? { ...mergedConfig, workspaceStrategy: { type: "project_primary" } } : mergedConfig,
|
||||
config: fresh && !remoteAuthority ? { ...mergedConfig, workspaceStrategy: { type: "project_primary" } } : mergedConfig,
|
||||
skipHostProvision,
|
||||
});
|
||||
const executionWorkspaceBase = {
|
||||
|
||||
Reference in new issue
Block a user