Add auto-forward execution workspace branch reconciliation (#9172)

## Thinking Path

> - Paperclip manages execution workspaces for AI agent runs, each
associated with a git branch so the agent always works in a known code
state.
> - Workspace runtime reconciles an agent's checkout branch against the
workspace's recorded branch when a heartbeat resumes; without
forward-ancestry detection, any divergence fails closed and blocks the
run.
> - When the workspace branch has moved forward (e.g. after a feature
merge), the recorded branch is an ancestor of the current HEAD — a safe,
forward-only case that the previous implementation refused even though
it carries no safety risk.
> - The gap means legitimate forward-advancing deployments require
manual operator intervention to unblock agents every time, creating
operational friction and interrupting automated workflows.
> - This pull request adds a flag-gated reconcile-forward path that
detects when the current branch is a strict forward descendant of the
workspace branch and auto-reconciles, while preserving fail-closed
behavior for all non-forward or flag-off cases.
> - It also threads the active execution workspace id through restore
and finalize call sites so the reconcile verdict can be persisted
durably across heartbeats.
> - The benefit is that agents resume automatically from
forward-advancing workspace branches without operator intervention,
while adversarial and backward branch changes continue to fail closed.

## Linked Issues or Issue Description

This PR adds an auto-forward reconcile path for execution workspace
branch tracking. When a workspace's recorded branch is a strict ancestor
of the current HEAD (a forward-only advancement), the runtime now
auto-reconciles rather than hard-blocking. The feature is gated behind
an explicit runtime flag, defaults to off, and falls back to fail-closed
behavior for all non-forward or flag-off cases.

The prior implementation treated all branch divergences identically: any
mismatch between the recorded workspace branch and the current HEAD
failed closed. This prevented agents from resuming after routine forward
deployments (e.g. after a feature branch merges into the workspace
branch), requiring manual operator action to unblock every affected run.

## What Changed

- Added `reconcileForward` flag-gated path in workspace runtime
reconciliation logic that allows auto-reconciliation when the workspace
branch is a strict ancestor of the current HEAD.
- Threaded active execution workspace id through `restore` and
`finalize` call sites so reconcile verdicts are persisted durably.
- Added `plainLanguageReason` and `ancestryVerdict` evidence fields to
the reconciliation result structure for operator visibility.
- Stabilized a branch containment test that exposed a late run-linked
activity FK cleanup race during the focused Vitest rerun.
- All new paths remain fail-closed when the flag is off or when the
branch relationship is not strictly forward.

## Verification

```bash
pnpm --filter @paperclipai/shared typecheck
pnpm --filter @paperclipai/server typecheck
pnpm exec vitest run \
  server/src/__tests__/workspace-runtime.test.ts \
  server/src/__tests__/heartbeat-workspace-branch-containment.test.ts \
  server/src/__tests__/execution-workspaces-service.test.ts
git diff --check origin/master..HEAD
```

All 3 test files / 98 tests pass. Typecheck passes for both shared and
server packages.

> **Note:** This is a stacked PR on top of PR #9170 (Add execution
workspace branch reconciliation route). The diff shown targets that
branch; the combined change builds on the reconciliation route
infrastructure it provides.

## Risks

- **Flag-off default:** The reconcile-forward path is off by default. No
behavior change for existing workspaces unless the flag is explicitly
enabled by an operator.
- **Ancestry check correctness:** The forward-only guard uses git
ancestry verification; a branch that is not a strict ancestor of HEAD
remains fail-closed. Adversarial or concurrent branch resets are not
auto-reconciled.
- **FK cleanup race (stabilized):** A late run-linked activity FK
cleanup race in the containment test was exposed during the Vitest
rerun. The stabilization commit addresses the non-deterministic ordering
without changing production behavior.
- **Stacking dependency:** This PR must not be merged before PR #9170
merges, as it is built on top of the reconciliation route
infrastructure.

## Model Used

- Provider: Anthropic
- Model: Claude Sonnet 4.6 (`claude-sonnet-4-6`)
- Context: 200k token context window
- Mode: Agentic tool use with code execution and git operations

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed 2026-07-07 14:12:17 -07:00
1 parent f17202b571
commit bb87047fe6
5 files changed
+852 -19

No files matched your search

+291 -7
View File
@@ -29,7 +29,8 @@ import {
writeLocalServiceRegistryRecord,
} from "./local-service-supervisor.js";
import type { WorkspaceOperationRecorder } from "./workspace-operations.js";
import { readExecutionWorkspaceConfig } from "./execution-workspaces.js";
import { executionWorkspaceService, readExecutionWorkspaceConfig } from "./execution-workspaces.js";
import { logActivity } from "./activity-log.js";
import { readProjectWorkspaceRuntimeConfig } from "./project-workspace-runtime-config.js";
export function resolveShell(): string {
@@ -70,6 +71,7 @@ export interface RealizedExecutionWorkspace extends ExecutionWorkspaceInput {
warnings: string[];
created: boolean;
baseRefSha?: string | null;
pendingForwardBranchReconcile?: PendingForwardBranchReconcile | null;
}
export class WorkspaceRuntimeValidationFailure extends Error {
@@ -659,6 +661,19 @@ type GitWorktreeCleanliness = SharedGitWorktreeBranchIncoherenceEvidence["cleanl
type GitWorktreeBranchIncoherenceEvidence = SharedGitWorktreeBranchIncoherenceEvidence;
type GitWorktreeBranchCoherenceResult = {
branchName: string | null;
reconciledForward: boolean;
pendingForwardBranchReconcile?: PendingForwardBranchReconcile | null;
};
export type PendingForwardBranchReconcile = {
recordedBranchName: string;
adoptedBranchName: string;
prePersistenceFingerprint: string;
reason: string;
};
function formatBranchForMessage(branch: string | null | undefined) {
return branch && branch.length > 0 ? branch : "<detached>";
}
@@ -845,22 +860,183 @@ function branchIncoherenceValidationFailure(evidence: GitWorktreeBranchIncoheren
);
}
async function recordForwardBranchReconcileOperation(input: {
recorder?: WorkspaceOperationRecorder | null;
phase?: "worktree_prepare" | "workspace_finalize";
cwd: string;
repoRoot: string;
worktreePath: string;
expectedBranchName: string;
actualBranchName: string;
executionWorkspaceId: string | null;
sourceIssueId: string | null;
fingerprint: string;
expectedHeadSha: string | null;
actualHeadSha: string | null;
ancestryVerdict: GitWorktreeBranchAncestryVerdict;
mode: "record_updated" | "adopt_for_realize";
auditCommentId?: string | null;
recoveryActionId?: string | null;
}) {
if (!input.recorder) return;
await input.recorder.recordOperation({
phase: input.phase ?? "worktree_prepare",
cwd: input.cwd,
metadata: {
repoRoot: input.repoRoot,
worktreePath: input.worktreePath,
expectedBranchName: input.expectedBranchName,
actualBranchName: input.actualBranchName,
branchIncoherenceReconcileForward: true,
reconcileMode: input.mode,
fingerprint: input.fingerprint,
sourceIssueId: input.sourceIssueId,
executionWorkspaceId: input.executionWorkspaceId,
expectedHeadSha: input.expectedHeadSha,
actualHeadSha: input.actualHeadSha,
ancestryVerdict: input.ancestryVerdict,
auditCommentId: input.auditCommentId ?? null,
recoveryActionId: input.recoveryActionId ?? null,
},
run: async () => ({
status: "succeeded",
system:
input.mode === "record_updated"
? `Reconciled execution workspace branch record from ${input.expectedBranchName} to ${input.actualBranchName}; worktree left unchanged.\n`
: `Adopted live git worktree branch ${input.actualBranchName} for this execution workspace realization; worktree left unchanged.\n`,
}),
});
}
async function logForwardBranchReconcileActivity(input: {
db: Db;
companyId: string;
executionWorkspaceId: string;
sourceIssueId: string | null;
runId: string | null;
mode: "forward";
reason: string | null;
fromBranch: string;
toBranch: string;
fromSha: string | null;
toSha: string | null;
ancestryVerdict: GitWorktreeBranchAncestryVerdict;
fingerprint: string;
auditCommentId: string | null;
recoveryActionId: string | null;
}) {
await logActivity(input.db, {
companyId: input.companyId,
actorType: "system",
actorId: "workspace_runtime",
runId: input.runId,
action: "execution_workspace.branch_reconciled",
entityType: "execution_workspace",
entityId: input.executionWorkspaceId,
details: {
mode: input.mode,
reason: input.reason,
fromBranch: input.fromBranch,
toBranch: input.toBranch,
fromSha: input.fromSha,
toSha: input.toSha,
ancestryVerdict: input.ancestryVerdict,
fingerprint: input.fingerprint,
sourceIssueId: input.sourceIssueId,
auditCommentId: input.auditCommentId,
recoveryActionId: input.recoveryActionId,
actor: {
type: "system",
id: "workspace_runtime",
source: "workspace_runtime",
},
},
});
}
export async function reconcilePendingForwardBranchAfterPersistence(input: {
db: Db;
executionWorkspaceId: string;
pending: PendingForwardBranchReconcile;
heartbeatRunId?: string | null;
reconcileOperationPhase?: "worktree_prepare" | "workspace_finalize";
recorder?: WorkspaceOperationRecorder | null;
}) {
const result = await executionWorkspaceService(input.db).reconcileExecutionWorkspaceBranch(
input.executionWorkspaceId,
{
mode: "forward",
reason: input.pending.reason,
alternateRecoveryFingerprints: [input.pending.prePersistenceFingerprint],
actor: {
actorType: "system",
actorId: "workspace_runtime",
agentId: null,
runId: input.heartbeatRunId ?? null,
},
},
);
await logForwardBranchReconcileActivity({
db: input.db,
companyId: result.workspace.companyId,
executionWorkspaceId: result.workspace.id,
sourceIssueId: result.workspace.sourceIssueId,
runId: input.heartbeatRunId ?? null,
mode: "forward",
reason: input.pending.reason,
fromBranch: result.inspection.fromBranch,
toBranch: result.inspection.toBranch,
fromSha: result.inspection.fromSha,
toSha: result.inspection.toSha,
ancestryVerdict: result.inspection.ancestryVerdict,
fingerprint: result.inspection.fingerprint,
auditCommentId: result.auditCommentId,
recoveryActionId: result.recoveryAction?.id ?? null,
});
await recordForwardBranchReconcileOperation({
recorder: input.recorder,
phase: input.reconcileOperationPhase,
cwd: result.inspection.worktreePath,
repoRoot: result.inspection.repoRoot,
worktreePath: result.inspection.worktreePath,
expectedBranchName: result.inspection.fromBranch,
actualBranchName: result.inspection.toBranch,
executionWorkspaceId: result.workspace.id,
sourceIssueId: result.workspace.sourceIssueId,
fingerprint: result.inspection.fingerprint,
expectedHeadSha: result.inspection.fromSha,
actualHeadSha: result.inspection.toSha,
ancestryVerdict: result.inspection.ancestryVerdict,
mode: "adopt_for_realize",
auditCommentId: result.auditCommentId,
recoveryActionId: result.recoveryAction?.id ?? null,
});
return result;
}
export async function ensureGitWorktreeBranchCoherent(input: {
db?: Db | null;
repoRoot: string;
worktreePath: string;
expectedBranchName: string | null;
sourceIssue: ExecutionWorkspaceIssueRef | null;
executionWorkspaceId?: string | null;
actualBranchName?: string | null;
heartbeatRunId?: string | null;
enableWorkspaceBranchReconcileForward?: boolean;
reconcileOperationPhase?: "worktree_prepare" | "workspace_finalize";
recorder?: WorkspaceOperationRecorder | null;
}) {
}): Promise<GitWorktreeBranchCoherenceResult> {
const expectedBranchName = input.expectedBranchName?.trim();
if (!expectedBranchName) return;
if (!expectedBranchName) return { branchName: null, reconciledForward: false };
const currentBranch = input.actualBranchName !== undefined
? input.actualBranchName
: await runGit(["symbolic-ref", "--quiet", "--short", "HEAD"], input.worktreePath).catch(() => null);
if (currentBranch === expectedBranchName) return;
if (currentBranch === expectedBranchName) {
return { branchName: expectedBranchName, reconciledForward: false };
}
const evidence = await inspectGitWorktreeBranchIncoherence({
repoRoot: input.repoRoot,
@@ -871,6 +1047,90 @@ export async function ensureGitWorktreeBranchCoherent(input: {
executionWorkspaceId: input.executionWorkspaceId ?? null,
});
if (
input.enableWorkspaceBranchReconcileForward === true &&
evidence.provenance.ancestryVerdict === "ancestor" &&
currentBranch
) {
const reason = "Automatic forward reconciliation: recorded branch is an ancestor of the checked-out branch.";
if (input.executionWorkspaceId) {
if (!input.db) {
evidence.safeRepair.reason = "forward reconciliation requires database access to update the execution workspace record";
throw branchIncoherenceValidationFailure(evidence);
}
try {
const result = await executionWorkspaceService(input.db).reconcileExecutionWorkspaceBranch(
input.executionWorkspaceId,
{
mode: "forward",
reason,
actor: {
actorType: "system",
actorId: "workspace_runtime",
agentId: null,
runId: input.heartbeatRunId ?? null,
},
},
);
await logForwardBranchReconcileActivity({
db: input.db,
companyId: result.workspace.companyId,
executionWorkspaceId: result.workspace.id,
sourceIssueId: result.workspace.sourceIssueId ?? evidence.sourceIssueId ?? null,
runId: input.heartbeatRunId ?? null,
mode: "forward",
reason,
fromBranch: result.inspection.fromBranch,
toBranch: result.inspection.toBranch,
fromSha: result.inspection.fromSha,
toSha: result.inspection.toSha,
ancestryVerdict: result.inspection.ancestryVerdict,
fingerprint: result.inspection.fingerprint,
auditCommentId: result.auditCommentId,
recoveryActionId: result.recoveryAction?.id ?? null,
});
await recordForwardBranchReconcileOperation({
recorder: input.recorder,
phase: input.reconcileOperationPhase,
cwd: input.worktreePath,
repoRoot: result.inspection.repoRoot,
worktreePath: result.inspection.worktreePath,
expectedBranchName: result.inspection.fromBranch,
actualBranchName: result.inspection.toBranch,
executionWorkspaceId: result.workspace.id,
sourceIssueId: result.workspace.sourceIssueId ?? evidence.sourceIssueId ?? null,
fingerprint: result.inspection.fingerprint,
expectedHeadSha: result.inspection.fromSha,
actualHeadSha: result.inspection.toSha,
ancestryVerdict: result.inspection.ancestryVerdict,
mode: "record_updated",
auditCommentId: result.auditCommentId,
recoveryActionId: result.recoveryAction?.id ?? null,
});
return { branchName: result.inspection.toBranch, reconciledForward: true };
} catch (error) {
evidence.safeRepair.reason =
`forward reconciliation failed: ${error instanceof Error ? error.message : String(error)}`;
throw branchIncoherenceValidationFailure(evidence);
}
}
if (!input.db) {
evidence.safeRepair.reason = "forward reconciliation adoption requires database access to audit after workspace realization";
throw branchIncoherenceValidationFailure(evidence);
}
return {
branchName: currentBranch,
reconciledForward: true,
pendingForwardBranchReconcile: {
recordedBranchName: expectedBranchName,
adoptedBranchName: currentBranch,
prePersistenceFingerprint: evidence.fingerprint,
reason,
},
};
}
if (!evidence.safeRepair.eligible) {
throw branchIncoherenceValidationFailure(evidence);
}
@@ -910,6 +1170,7 @@ export async function ensureGitWorktreeBranchCoherent(input: {
evidence.safeRepair.succeeded = true;
evidence.safeRepair.reason = "clean worktree checked out the recorded branch";
return { branchName: expectedBranchName, reconciledForward: false };
}
// Resolve the authoritative base ref for a fresh worktree. A configured local
@@ -1603,10 +1864,13 @@ async function resolveGitRepoRootForWorkspaceCleanup(
}
export async function realizeExecutionWorkspace(input: {
db?: Db | null;
base: ExecutionWorkspaceInput;
config: Record<string, unknown>;
issue: ExecutionWorkspaceIssueRef | null;
agent: ExecutionWorkspaceAgentRef;
heartbeatRunId?: string | null;
enableWorkspaceBranchReconcileForward?: boolean;
recorder?: WorkspaceOperationRecorder | null;
}): Promise<RealizedExecutionWorkspace> {
const rawStrategy = parseObject(input.config.workspaceStrategy);
@@ -1632,12 +1896,13 @@ export async function realizeExecutionWorkspace(input: {
projectId: input.base.projectId,
repoRef: input.base.repoRef,
});
const branchName = sanitizeBranchName(renderedBranch);
let branchName = sanitizeBranchName(renderedBranch);
const configuredParentDir = asString(rawStrategy.worktreeParentDir, "");
const worktreeParentDir = configuredParentDir
? resolveConfiguredPath(configuredParentDir, repoRoot)
: path.join(repoRoot, ".paperclip", "worktrees");
const worktreePath = path.join(worktreeParentDir, branchName);
let pendingForwardBranchReconcile: PendingForwardBranchReconcile | null = null;
const configuredBaseRef = typeof rawStrategy.baseRef === "string" && rawStrategy.baseRef.length > 0
? rawStrategy.baseRef
: input.base.repoRef ?? null;
@@ -1715,6 +1980,7 @@ export async function realizeExecutionWorkspace(input: {
warnings: [...baseRefreshWarnings, ...baseDrift.warnings],
created: false,
baseRefSha: refresh.baseRefSha ?? baseDrift.branchBaseRefSha ?? baseDrift.currentBaseRefSha,
pendingForwardBranchReconcile,
};
}
@@ -1725,15 +1991,23 @@ export async function realizeExecutionWorkspace(input: {
expectedBranchName: branchName,
}).catch(() => null);
if (validation && !validation.valid && validation.reasonCode === "branch_mismatch") {
await ensureGitWorktreeBranchCoherent({
const coherence = await ensureGitWorktreeBranchCoherent({
db: input.db ?? null,
repoRoot,
worktreePath: reusablePath,
expectedBranchName: branchName,
actualBranchName: validation.actualBranchName ?? null,
sourceIssue: input.issue,
executionWorkspaceId: null,
heartbeatRunId: input.heartbeatRunId ?? null,
enableWorkspaceBranchReconcileForward: input.enableWorkspaceBranchReconcileForward === true,
reconcileOperationPhase: "worktree_prepare",
recorder: input.recorder ?? null,
});
if (coherence.reconciledForward && coherence.branchName) {
branchName = coherence.branchName;
pendingForwardBranchReconcile = coherence.pendingForwardBranchReconcile ?? null;
}
return await validateLinkedGitWorktree({
repoRoot,
worktreePath: reusablePath,
@@ -1837,6 +2111,7 @@ export async function realizeExecutionWorkspace(input: {
}
export async function ensurePersistedExecutionWorkspaceAvailable(input: {
db?: Db | null;
base: ExecutionWorkspaceInput;
workspace: {
id?: string | null;
@@ -1856,6 +2131,8 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {
};
issue: ExecutionWorkspaceIssueRef | null;
agent: ExecutionWorkspaceAgentRef;
heartbeatRunId?: string | null;
enableWorkspaceBranchReconcileForward?: boolean;
recorder?: WorkspaceOperationRecorder | null;
}): Promise<RealizedExecutionWorkspace | null> {
const cwd = asString(input.workspace.cwd ?? input.workspace.providerRef, "").trim();
@@ -1891,14 +2168,21 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {
const reuseBaseRef = input.workspace.baseRef ?? input.base.repoRef ?? null;
const reuseWorktreePath = realized.worktreePath ?? cwd;
if (await isGitCheckout(reuseWorktreePath)) {
await ensureGitWorktreeBranchCoherent({
const coherence = await ensureGitWorktreeBranchCoherent({
db: input.db ?? null,
repoRoot,
worktreePath: reuseWorktreePath,
expectedBranchName: realized.branchName,
sourceIssue: input.issue,
executionWorkspaceId: input.workspace.id ?? null,
heartbeatRunId: input.heartbeatRunId ?? null,
enableWorkspaceBranchReconcileForward: input.enableWorkspaceBranchReconcileForward === true,
reconcileOperationPhase: "worktree_prepare",
recorder: input.recorder ?? null,
});
if (coherence.reconciledForward && coherence.branchName) {
realized.branchName = coherence.branchName;
}
}
const validation = await validateLinkedGitWorktree({
repoRoot,