fix(grok): avoid false file recovery after Stop (#16427)

## Thinking Path

> - Paperclip lets people manage AI agents and their work.
> - Grok Bot uses a managed Runner and a fixed external work session.
> - Its admitted contract gives it no access to Paperclip workspace
files.
> - The Runner still has a sync descriptor to support durable recovery.
> - Cancellation treated that descriptor as proof that files needed
repair.
> - This pull request uses the admitted access contract while preserving
ownership settlement.
> - Users get the correct recovery action after Stop, with native file
safeguards intact.

## Linked Issues or Issue Description

Refs #16219 and #16422. The duplicate search found no open Grok
workspace correction.

**What happened?**

A real Grok Stop test cancelled its run, fenced its assignment, denied a
stale progress update and cleaned up the sandbox. Paperclip also created
a file-repair hold. The recovery panel told the user to recover
workspace files, although the admitted external contract has workspace
access set to none.

**Expected behavior**

Keep the stopped session fenced and require inspection before new work.
Keep failed-run ownership settlement. Do not invent a file export
obligation for a session that cannot access the workspace. Native
sessions with writable or unknown workspace access must keep their file
safeguards.

**Steps to reproduce**

1. Connect a Grok Bot and assign a task.
2. Let it accept the task and save a Paperclip checkpoint document.
3. Use Stop while it is working.
4. Inspect the task recovery panel. It asks for file repair after
sandbox cleanup succeeds.

**Paperclip version or commit**

Observed on Core 11d69d04c7 through the
actual Grok client. Reproduced against master with database-backed
regressions.

## What Changed

- Match the admitted native input to its run, company, agent, task and
external driver before treating workspace access as disabled.
- Keep coordinator ownership checks and failure settlement. Skip file
retention and the false file-repair obligation for that exact contract.
- Add inspection-only settlement for old Grok file holds through Runtime
recovery. Verify exact run ownership, fenced assignments, terminal
coordinators and managed lease stop receipts. Archive the false file
evidence and preserve no-replay.
- Keep historical Dot recovery controls. Do not claim global
external-provider termination.
- Fix the later dispatch-failure owner so it cannot retain a
no-workspace external sandbox for a false file obligation.
- Cover both Grok and Dot, mismatched access evidence, native file
recovery and the assembled heartbeat cleanup path.
- Update the Grok guide and the existing release evidence plan.

## Verification

- Four database regressions fail before the correction.
- 68 file-recovery and cancellation checks pass. Grok, Dot and the
shared recovery notice pass 98 checks. The assembled heartbeat selection
passes both cases; 404 unrelated cases were not selected in that local
command.
- The lifecycle regression caught an unsafe early-return prototype. The
final code keeps coordinator settlement for both workspace cases. A
separate assertion was corrected for the run-state projection's explicit
nulls. The failed attempts remain in the private evidence record.
- Repository typecheck and build, token gates and Storybook build pass.
The branch was rebased from ea4aa185b0
onto 7420af26fc, whose additional change
affects CI profile exports. The core recovery checks are repeated after
rebase.
- On the earlier deployed source, a real 824-second task survived an
actual same-image controller restart with its original run, assignment,
process and lease. It completed canonically and cleaned up without an
extra wake. The active Stop test denied a real stale update and
completed connection inspection and authenticated re-verification. These
results establish the observed baseline and defect; they do not qualify
this patch's deployed behavior.
- Review found two valid gaps: the old file obligation still blocked
fresh input, and dispatch failure could request retention earlier. Both
are corrected. The follow-up passes 128 focused tests and server
typecheck. The recovery test proves fresh user continuation after
inspection, authenticated re-verification, and managed Runner cleanup
while the old assignment stays fenced.
- A second review found a partial-transaction gap. Routine readiness now
commits as invalid before task holds retire. All 43 Grok routine tests
pass, including a held task lock and an unrelated active assignment.
- Repository typecheck and build pass after the first review
corrections; final server typecheck also passes. The full local suite is
still running and is not claimed as passed. The first follow-up CI timed
out at 15 seconds in one unchanged run-history case; all 91 cases pass
locally. The cause is unproven.
- The final head, 33e4361911, passes 54
checks plus one successful status context. Two optional Storybook jobs
are skipped. Greptile reviewed that exact head at 5/5 with no unresolved
threads. All source CI suites pass on this final head without retry. The
earlier runner-shutdown and timeout attempts remain recorded.
- An actual deployed historical recovery and fresh Stop repeat are still
required.

## Risks

An incomplete or mismatched contract must retain file safeguards. The
change uses stored admission evidence rather than current agent
configuration. Historical records stay auditable and do not gain
automatic replay or a generic Retry action. Operator inspection and a
fresh authenticated verification remain required before new Grok work.
Global Grok termination remains unconfirmed. No schema migration is
needed.

## Model Used

OpenAI GPT-6 in Codex, with reasoning, repository editing, shell tools
and computer use. The exact serving model ID, snapshot and context
window are not exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-15 08:52:50 -05:00
1 parent 57fa6b358d
commit 5a9e6b6315
13 files changed
+413 -30

No files matched your search

+15
View File
@@ -80,3 +80,18 @@ inspected recovery hold before admitting another assignment.
stops future dispatch. Remote Grok execution may continue. Open the Grok Bot app
to inspect it. Paperclip shows reported activity, not a complete remote transcript
or computer recording.
The admitted Grok execution contract has no access to the Runner's workspace files. Its managed Runner still keeps durable recovery state, but stopping that session does not require workspace file repair. Inspect the Bot in Grok, clear the connection's recovery hold in the agent's Runtime settings, and verify the routine again before assigning new work. This records inspection without asserting global Grok termination.
Historical stopped Grok runs can have an incorrect workspace-file hold from an
earlier release. Runtime shows Recovery needed for these runs even when the
routine is healthy. After inspecting Grok, use Clear recovery hold and verify
the connection again. Paperclip retires the false file obligation only when the
original admission denied workspace access, the assignment is fenced, the
coordinator has settled, and the exact managed Runner leases have stop receipts.
Connection readiness is invalidated before task inspection, so a partial failure
cannot retain an old verified connection while removing a task hold.
The inspection preserves the old result, action uncertainty, and no-replay hold.
Send a new task message to request fresh work. Inspection does not certify global
Grok termination or delete any previously retained allocation. Such allocations
still use the existing environment cleanup controls.
@@ -1,6 +1,6 @@
# Grok Bot production readiness
Status as of October 15: real assigned and Bot-initiated tasks reach canonical completion. The active-session wake correction is merged. A real proactive test found a delayed-admission echo; its correction is under verification. Production acceptance is incomplete. The current release goal and matrix below supersede the earlier dated campaign entries. Historical successes and failures remain evidence for the revisions actually tested.
Status as of October 15: real assigned and Bot-initiated tasks reach canonical completion. The active-session wake correction is merged. The proactive admission echo correction is merged and source-verified; image publication and a real no-echo repeat remain pending. A real controller restart and active Stop diagnostic passed their authority checks; Stop also exposed a false workspace-repair hold, now under correction. Production acceptance is incomplete. The current release goal and matrix below supersede the earlier dated campaign entries. Historical successes and failures remain evidence for the revisions actually tested.
## Current production finish line — October 14
@@ -13,7 +13,7 @@ There is no testing call, case-count, token or spending cap. The former six-case
| Ordinary onboarding works | Actual Grok UI, fixed agent consent, saved setup, wrong-key behavior, authenticated challenge, refresh, reconnect and revocation | Resumed real Grok authorization, saved setup Done, authenticated routine challenge and automatic refresh passed on the October 15 serving revision. Runtime reconnect copy fix is merged; verify its published UI and fresh production setup next. |
| Assigned work reaches a usable result | One actual managed workflow: progress, question and answer, document, downloaded file/hash, canonical completion or review and cleanup | A preliminary comparison task on serving Core ff0 completed progress → question → answer → report/CSV → canonical Done, with both cleanup receipts. The cited CSV downloads with the exact publication hash. Two distinct CSV uploads accompanied an active-session result wake; correct that wake behavior and requalify the final candidate. |
| Bot initiates governed work | Real-client task creation/claim/completion with stable receipts and no outbound echo or duplicate effects | A real idle-conversation request created one task and one managed run, saved a 100-word report, reached accepted Done and cleaned up its lease. A delayed-admission wake and two acceptance attempts were also recorded. Functional completion passed; no-echo acceptance failed. Requalify the correction. |
| Recovery and authority are reliable | Healthy work past the prior five-minute failure window; actual controller restart preserving assignment/session/process and no duplicate wake; Stop, pause, reassignment, revocation, stale calls, queues and cleanup | The old hold was inspected and cleared. The successful comparison continuation ended below five minutes. Healthy work beyond that window, an actual restart and active Stop remain unverified; protocol tests are separate evidence. |
| Recovery and authority are reliable | Healthy work past the prior five-minute failure window; actual controller restart preserving assignment/session/process and no duplicate wake; Stop, pause, reassignment, revocation, stale calls, queues and cleanup | A real managed diagnostic ran for 13 minutes 44 seconds, survived an actual same-image controller restart with the original assignment/session/process/lease, completed canonically and cleaned up without another wake. Active Stop fenced its work, rejected a real stale progress update, kept its checkpoint unchanged and cleaned up. Cancellation also created a false file-repair hold despite an admitted workspace-access-none contract; correct and requalify that recovery behavior. These diagnostics ran on Core 11d, not the final release image. |
| One identifiable reviewed candidate passes source gates | Focused protocol/auth/Runner/UI tests, full CI/typecheck/tests/build/token/Storybook evidence and exact artifacts | Active-session wake correction #16394 merged with 55 passing checks, two optional skips and review 5/5. The proactive echo correction started from master 7fc2dc8b3236e1149ae32756b6153a503623404b and was rebased onto freshly fetched master 250429942e27d568f20d3bc7dddcfc84a666480a before submission. Two regression cases reproduce the real delayed-admission wake before the fix. All 214 focused protocol/auth/question checks pass, including 80 Grok and Dot checks. Repository typecheck/build, token gates, the 2,731-story structure and Storybook build pass. Fresh CI, review, publication and the real no-echo repeat remain required. |
| Production opt-in is delivered and smoke-tested | Staging acceptance, correct served Core/distribution/Runner, normal editable opt-in controls/defaults, actual production Bot workflow and inspectable task/output links | Cloud opt-in policy and release-guide corrections are merged. Staging Cloud provenance and minimal feature values were verified, with Dot disabled and defaults unchanged. The corrected wake candidate, remaining real-client cases, ordinary production opt-in controls and production smoke are outstanding. |
@@ -33,6 +33,12 @@ The proactive run also received an authenticated delayed-admission webhook while
Normal staging automation has advanced beyond the older prepared candidate. No shared CI routing switch was changed. The earlier temporary fallback request is no longer needed for the already served successor. The dedicated production workspace still has no qualification agent grant or smoke task.
The proactive no-echo correction merged as #16422 after 55 successful checks, two optional Storybook skips and fresh review 5/5. Canonical source verification and exact-source migrator publication passed. The first image attempt was displaced by an older rerun before any build job started; its single retry was then superseded by a newer master push. The successor contains the fix. Neither cancellation is a successful image publication, and no shared CI routing flag changed.
The controlled recovery diagnostic preserved one original run for 824 seconds. The operator created the task, performed one same-image controller redeploy after checking for other work and rollouts, and sent one planned normal resume comment. The Bot read that queued comment at a safe boundary, read its checkpoint back, saved revision two and reached accepted Done. Its exact source identities and successful cleanup receipt are retained privately. The Bot corrected its own rejected document operation and reported correcting an invalid finish attachment; neither was repaired by the operator. This proves observed recovery on Core 11d, not the cause of the older failure or final-image acceptance.
The active Stop diagnostic cancelled its run, fenced the assignment, kept the checkpoint at revision one and recorded successful sandbox cleanup. One planned negative probe in the real Grok client returned “Assignment authority expired or was revoked.” No late completion was accepted. Inspection found a generic workspace-repair obligation attached to an external native input that explicitly forbids workspace access. Four database regressions reproduce that obligation and its misleading recovery guidance. The correction keeps coordinator settlement, no-replay holds and native file safeguards while omitting file retention for an exact admitted workspace-less external contract. Repository typecheck and 68 focused file-recovery/cancellation checks pass; an assembled heartbeat regression exposed and preserved the coordinator-settlement requirement. Fresh review, full checks and deployed recovery remain required.
Keep native providers and person-purpose MCP connections unchanged. No browser automation belongs in normal operation. Defer localhost relays, internal Grok APIs, marketplace distribution and Paperclip-connected third-party tools. Completion of a Paperclip work session does not assert remote Grok termination.
## Candidate and release contract
@@ -255,3 +261,31 @@ assignment authority. Disconnect completes while storage is pending, and the
late upload returns 403 with no attachment or publication receipt. Person write
checks remain in place. The full repository command was interrupted when this
new review finding arrived; its partial output is not a passing test result.
## October 15 — Stop review follow-up
Review found that hiding the historical file-repair card alone left the stored
file obligation blocking fresh task input. Runtime inspection now finds these
holds even after routine recovery. It retires only the false file fields after
exact run, assignment, coordinator and managed-lease cleanup checks, archives
the previous evidence, and preserves no-replay. A real authenticated readiness
exchange and new user input are still required. No global provider-stop claim is
created. Dot historical recovery controls remain available.
The later dispatch-failure owner also now honors the validated no-workspace
contract before asking for sandbox retention. Both external providers and the
writable native path pass the runtime boundary regression. The review follow-up
passes 128 focused protocol, recovery, cancellation and runtime checks. An initial
negative assertion saw the earlier readiness gate instead of the revoked-session
gate; the corrected test checks both before and after re-verification. Server
typecheck passes after narrowing the native workspace union. Final current-head
CI, review, deployed historical recovery and fresh Stop are still release gates.
A second review caught a partial-transaction ordering gap. Routine readiness now
commits as invalid before the separate task inspection transactions. New tests
hold the task lock to prove readiness is already gone and verification remains
blocked; an unrelated active assignment cannot cause a task hold to disappear.
All 43 Grok routine tests pass. The first follow-up CI attempt also timed out in
one unchanged run-history case at 15 seconds; all 91 tests in that file pass
locally. The timeout remains a recorded failure with cause unproven. The new
source head requires fresh CI and review.
+142 -1
View File
@@ -8,7 +8,7 @@ import { createHash, randomBytes, randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { beforeAll, afterAll, afterEach, describe, expect, it, vi } from "vitest";
import { agents, authUsers, companies, companyMemberships, createDb, dotAgentBindings, grokBotDispatches,
grokBotRoutines, mcpOauthGrants, activityLog, issues, heartbeatRuns, dotRunnerAssignments, dotRunnerOperations, dotMailboxItems, nativeRunFinalizations } from "@paperclipai/db";
grokBotRoutines, mcpOauthGrants, activityLog, issues, heartbeatRuns, dotRunnerAssignments, dotRunnerOperations, dotMailboxItems, nativeRunFinalizations, environmentLeases, environments, issueRecoveryActions, issueComments } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { accessService } from "../services/access.js";
import { instanceSettingsService } from "../services/instance-settings.js";
@@ -20,6 +20,10 @@ import * as secretModule from "../services/secrets.js";
import * as externalAuthority from "../services/external-bot-authority.js";
import { type GrokRoutineWake } from "../services/grok-bot/webhook.js";
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
import { admitExplicitNativeContinuation } from "../services/explicit-native-continuation.js";
import { hasRequiredWorkspaceRecovery } from "../services/workspace-restore-recovery-state.js";
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
@@ -479,3 +483,140 @@ describe("durable Grok routine qualification", () => {
});
});
// Exercise the public recovery service and the actual native admission boundary,
// including a stale file obligation persisted before this fix.
describe("Grok inspection of historical workspace-less Stop", () => {
async function historicalHold() {
const f = await fixture();
const { run, task, work } = await admittedWork(f, "accepted");
const finishedAt = new Date(Date.now() - 1000);
const fileHold = { workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: {
schema: "paperclip.workspace-restore-recovery.v1", runId: run.id, leaseIds: [] as string[],
} };
const [environment] = await db.insert(environments).values({ name: `Grok ${run.id}`, driver: "sandbox" }).returning();
const [lease] = await db.insert(environmentLeases).values({ companyId: f.company.id, issueId: task.id,
heartbeatRunId: run.id, environmentId: environment!.id, provider: "daytona", providerLeaseId: randomUUID(),
status: "released", releasedAt: new Date(), cleanupStatus: "success" }).returning();
await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination:
remoteTerminationReceipt(lease!, { providerLeaseId: lease!.providerLeaseId, state: "destroyed" }) } }).where(eq(environmentLeases.id, lease!.id));
fileHold.workspaceRestoreRecovery.leaseIds.push(lease!.id);
await db.update(heartbeatRuns).set({ status: "cancelled", driverKind: "grok_bot_mcp", finishedAt,
nativeIssueId: task.id, issueId: task.id, contextSnapshot: { issueId: task.id },
runnerProfileJson: { nativeExecutionInput: { schema: "paperclip.native-execution-input.v6",
binding: { runId: run.id, companyId: f.company.id, agentId: f.agent.id, issueId: task.id },
provider: { kind: "grok_bot" }, workspace: { access: "none", cwd: null } } },
resultJson: { ...fileHold, cancellation: { source: "user", reason: "Paperclip work stopped" } },
}).where(eq(heartbeatRuns.id, run.id));
await db.update(issues).set({ status: "blocked", executionRunId: null, checkoutRunId: null }).where(eq(issues.id, task.id));
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", leaseOwner: null,
leaseExpiresAt: null, recoveryState: "blocked" }).where(eq(nativeRunFinalizations.runId, run.id));
await db.update(dotRunnerAssignments).set({ status: "fenced" }).where(eq(dotRunnerAssignments.id, work.id));
const [action] = await db.insert(issueRecoveryActions).values({ companyId: f.company.id, sourceIssueId: task.id,
kind: "active_run_watchdog", cause: "native_continuation_requires_reconciliation", fingerprint: run.id,
status: "resolved", outcome: "blocked", nextAction: "Recover files.",
evidence: { ...fileHold, runId: run.id, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } },
}).returning();
const [comment] = await db.insert(issueComments).values({ companyId: f.company.id, issueId: task.id,
authorType: "user", authorUserId: f.principal.grant.userId, body: "Start a fresh task turn after inspection." }).returning();
const continueWork = (dryRun = true) => db.transaction(async tx => {
await tx.select().from(issues).where(eq(issues.id, task.id)).for("update");
return admitExplicitNativeContinuation({ db: tx as unknown as typeof db, companyId: f.company.id,
issueId: task.id, agentId: f.agent.id, actorType: "user", actorId: f.principal.grant.userId,
reason: "issue_commented", commentId: comment!.id, successorRunId: randomUUID(), dryRun });
});
return { ...f, run, task, work, lease: lease!, action: action!, continueWork };
}
it("offers inspection for an old task hold and admits only new explicit input after inspection and Runner cleanup", async () => {
const f = await historicalHold();
f.delivery.mockClear();
expect(await f.service.recoveryNeeded(f.company.id, f.agent.id)).toBe(true);
expect(await f.continueWork()).toBeNull();
expect(await f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId)).toMatchObject({
requiresVerification: true, remoteTerminationConfirmed: false,
});
expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: false });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(false);
expect(run).toMatchObject({ status: "cancelled", resultJson: { externalWorkspaceInspection: {
inspectedBy: f.principal.grant.userId, remoteTerminationConfirmed: false,
previousWorkspaceRestoreFailure: "restore_failed", runnerLeaseIds: [f.lease.id],
} } });
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id));
expect(action).toMatchObject({ status: "resolved", outcome: "blocked", evidence: {
automaticRecovery: { replay: "blocked", actionOutcome: "unknown" }, externalWorkspaceInspection: { workspaceAccess: "none" },
} });
expect(action!.evidence).not.toHaveProperty("workspaceRestoreRecovery");
expect(f.delivery).not.toHaveBeenCalled();
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.company.id))).toHaveLength(1);
await expect(dotRunnerBroker(db).operation(f.principal, f.work.id, randomUUID(), "progress", { text: "late effect" })).rejects.toThrow("wake verification");
// Connection recovery still requires a real authenticated challenge exchange.
const row = await verification(f); await f.service.send(f.company.id, f.agent.id, row.id);
await f.service.acknowledge(f.principal, acknowledgement(row));
await expect(dotRunnerBroker(db).operation(f.principal, f.work.id, randomUUID(), "progress", { text: "late effect" })).rejects.toThrow("revoked");
expect(await f.continueWork(false)).toMatchObject({ previousRunId: f.run.id });
const [settled] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id));
expect(settled!.evidence.automaticRecovery).toMatchObject({ replay: "explicit_user_continuation", actionOutcome: "unknown" });
expect((await db.select().from(dotRunnerAssignments).where(eq(dotRunnerAssignments.id, f.work.id)))[0]!.status).toBe("fenced");
await f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.company.id), eq(activityLog.action, "grok.workspace_obligation_inspected")))).toHaveLength(1);
});
it("does not retire task holds if another assignment prevents recovery", async () => {
const f = await historicalHold();
const [other] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id,
status: "running", invocationSource: "assignment", triggerDetail: "system" }).returning();
await db.insert(dotRunnerAssignments).values({ ...f.work, id: randomUUID(), runId: other!.id,
normalizedSessionId: randomUUID(), turnId: randomUUID(), status: "accepted" });
await expect(f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId)).rejects.toThrow("current Paperclip work session");
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(true);
expect(await f.service.recoveryNeeded(f.company.id, f.agent.id)).toBe(true);
});
it("commits verification invalidation before waiting for historical task inspection", async () => {
const f = await historicalHold();
let recovery: Promise<unknown> | undefined;
try {
await db.transaction(async tx => {
await tx.select().from(issues).where(eq(issues.id, f.task.id)).for("update");
recovery = f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId);
void recovery.catch(() => {});
await vi.waitFor(async () => {
expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: true });
});
await expect(verification(f)).rejects.toThrow("recover");
const [binding] = await db.select().from(dotAgentBindings).where(eq(dotAgentBindings.id, f.pairing.bindingId));
expect(binding).toMatchObject({ status: "connected", readyAt: null });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(true);
});
} finally { await recovery; }
expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: false });
});
it.each(["cleanup_pending", "receipt_mismatch", "controller_owned", "assignment_active", "running", "operator_revoked", "foreign_company"])("does not retire an old obligation when %s", async condition => {
const f = await historicalHold();
if (condition === "cleanup_pending") await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, f.lease.id));
if (condition === "receipt_mismatch") await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", runId: randomUUID() } } }).where(eq(environmentLeases.id, f.lease.id));
if (condition === "controller_owned") await db.update(nativeRunFinalizations).set({ leaseOwner: "other-owner" }).where(eq(nativeRunFinalizations.runId, f.run.id));
if (condition === "assignment_active") await db.update(dotRunnerAssignments).set({ status: "accepted" }).where(eq(dotRunnerAssignments.id, f.work.id));
if (condition === "running") await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, f.run.id));
if (condition === "operator_revoked") await accessService(db).setPrincipalPermission(f.company.id, "user", f.principal.grant.userId, "agents:create", false, f.principal.grant.userId);
if (condition === "foreign_company") {
const foreign = await fixture();
await f.service.recover(foreign.company.id, foreign.agent.id, foreign.principal.grant.userId);
} else if (condition === "running") {
// Not an eligible historical source. Its active assignment is fenced in
// this deliberately malformed fixture, but no file state may be removed.
await f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId);
} else await expect(f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId)).rejects.toThrow();
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(true);
if (["cleanup_pending", "receipt_mismatch", "controller_owned"].includes(condition)) {
expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: true });
}
expect(await f.continueWork()).toBeNull();
});
});
@@ -3521,7 +3521,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
}
}
it("retains the source when a saved native descriptor fails inside runtime selection", async () => {
it.each([false, true])("settles a failed saved native descriptor without inventing files when workspace access is disabled: %s", async withoutWorkspace => {
await withTempPaperclipHome(async () => {
const f = await seedQueuedIssueRunFixture();
await db.update(agents).set({ adapterType: "paperclip_runner",
@@ -3547,7 +3547,12 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null,
workspaceId: randomUUID(), leaseId: lease!.id, providerLeaseId: lease!.providerLeaseId!, remoteCwd: "/work", resourceDisposition: "destroy" as const };
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issueId, nativePhase: "observed",
runnerProfileJson: { nativeWorkspaceSync: reference },
driverKind: withoutWorkspace ? "grok_bot_mcp" : null,
runnerProfileJson: { nativeWorkspaceSync: reference, ...(withoutWorkspace ? { nativeExecutionInput: {
schema: "paperclip.native-execution-input.v6", provider: { kind: "grok_bot" },
binding: { companyId: f.companyId, agentId: f.agentId, runId, issueId: f.issueId },
workspace: { access: "none", cwd: null, repoUrl: null, repoRef: null, branchName: null },
} } : {}) },
}).where(eq(heartbeatRuns.id, runId));
await db.insert(nativeRunFinalizations).values({ runId, companyId: f.companyId, issueId: f.issueId, phase: "observed" });
await nativeWorkspaceSyncInternals.readDescriptor({ companyId: f.companyId, runId, workspaceId: reference.workspaceId, reference });
@@ -3556,15 +3561,21 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
await heartbeat.resumeQueuedRuns();
await heartbeat.drainActiveRunExecutions();
expect(leaseId).toBeDefined();
expect(await heartbeat.getRun(f.runId, { includeExecutionEvidence: true })).toMatchObject({ status: "failed",
runtimeMode: "native", nativePhase: "terminal_failure", resultJson: {
workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [leaseId] },
},
});
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId!)))
.toMatchObject([{ status: "pending_cleanup", leasePolicy: "retain_on_failure" }]);
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ workspaceRepairRequired: true, canRetry: false });
expect(release).toHaveBeenCalledWith(expect.objectContaining({ providerResourceDisposition: "stop_and_retain" }));
const failed = await heartbeat.getRun(f.runId, { includeExecutionEvidence: true });
expect(failed).toMatchObject({ status: "failed", runtimeMode: "native", nativePhase: "terminal_failure" });
if (withoutWorkspace) {
expect(failed?.resultJson?.workspaceRestoreFailure ?? null).toBeNull();
expect(failed?.resultJson?.workspaceRestoreRecovery ?? null).toBeNull();
expect((await getExecutionBlocker(db, f.companyId, f.issueId))?.workspaceRepairRequired).not.toBe(true);
expect(release).toHaveBeenCalled();
expect(release).not.toHaveBeenCalledWith(expect.objectContaining({ providerResourceDisposition: "stop_and_retain" }));
} else {
expect(failed?.resultJson).toMatchObject({ workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [leaseId] } });
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId!)))
.toMatchObject([{ status: "pending_cleanup", leasePolicy: "retain_on_failure" }]);
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ workspaceRepairRequired: true, canRetry: false });
expect(release).toHaveBeenCalledWith(expect.objectContaining({ providerResourceDisposition: "stop_and_retain" }));
}
expect(provider).not.toHaveBeenCalled();
expect(mockAdapterExecute).not.toHaveBeenCalled();
} finally { factory.mockRestore(); }
+14 -2
View File
@@ -7,7 +7,7 @@ import { canContinueCancelledRun, readRunCancellation } from "./run-cancellation
import { canRetryStoppedRun } from "./cancelled-native-startup.js";
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
import { EXECUTION_RECONCILIATION_CAUSES, type ExecutionBlocker } from "@paperclipai/shared";
import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA } from "./workspace-restore-recovery-state.js";
import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA, nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
/** Resolved recovery bookkeeping can still carry an effective no-replay hold. */
export function executionBlockerPredicate() {
@@ -65,6 +65,8 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
const [run] = runId ? await db.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId),
)).limit(1) : [];
const withoutWorkspace = Boolean(run?.driverKind === "grok_bot_mcp" && nativeRunHasNoWorkspaceAccess(run));
const workspaceRepairRequired = Boolean(restoreHold && !withoutWorkspace);
const queue = await db.select({ payload: agentWakeupRequests.payload }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
@@ -118,6 +120,16 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
nextAction += " Inspect the run before sending a new message to request continuation.";
}
if (withoutWorkspace && (restoreHold || run?.status === "cancelled") && !run?.resultJson?.externalWorkspaceInspection) {
nextAction = run?.driverKind === "grok_bot_mcp"
? "Inspect this Bot in Grok Bot, then clear the recovery hold and verify the connection in the agent's Runtime settings before assigning more work."
: "Inspect the stopped work in the external agent and its connection before requesting continuation.";
}
if (run?.resultJson?.externalWorkspaceInspection && !restoreHold) {
nextAction = "Verify the Grok connection, then send a new message to continue. The stopped work will not be replayed.";
}
return {
recoveryActionId: action.id,
runId,
@@ -129,7 +141,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
runError: runError?.slice(0, 1024) ?? null,
canContinue: eligibleContinuation && !chatBinding,
canRetry: canRetry && !chatBinding,
...(restoreHold ? { workspaceRepairRequired: true, canRecordWorkspaceRepair: hasRequiredWorkspaceRecovery(action.evidence) } : {}),
...(restoreHold ? { workspaceRepairRequired, canRecordWorkspaceRepair: workspaceRepairRequired && hasRequiredWorkspaceRecovery(action.evidence) } : {}),
savedMessageCount: saved.length,
};
}
+7 -1
View File
@@ -10,6 +10,8 @@ import { scheduleAgentLifecycle } from "../agent-lifecycle.js";
import { logActivity } from "../activity-log.js";
import { deliverGrokRoutineWake, validateGrokRoutineCredential, type GrokRoutineDelivery, type GrokRoutineWake } from "./webhook.js";
import { grokWorkspaceInspectionCandidates, inspectGrokWorkspaceObligations } from "./workspace-inspection.js";
const digest = (value: string) => createHash("sha256").update(value).digest("hex");
const fail = (message: string) => new McpOAuthError("access_denied", message, 409);
export const grokWakeAcknowledgementSchema = z.object({ correlationId: z.uuid(), bindingId: z.uuid(),
@@ -44,7 +46,7 @@ export function grokRoutineService(db: Db, deliver: Delivery = deliverGrokRoutin
const [uncertain] = await db.select({ id: dispatches.id }).from(dispatches).innerJoin(routines, eq(routines.id, dispatches.routineId))
.where(and(eq(dispatches.companyId, companyId), eq(routines.agentId, agentId),
or(eq(dispatches.status, "unknown"), and(eq(dispatches.status, "sending"), lte(dispatches.updatedAt, new Date(Date.now() - 30_000))), and(eq(dispatches.status, "accepted"), lte(dispatches.expiresAt, new Date()))))).limit(1);
return !!held || !!uncertain;
return !!held || !!uncertain || (await grokWorkspaceInspectionCandidates(db, companyId, agentId)).length > 0;
}
async function recover(companyId: string, agentId: string, operatorId: string) {
@@ -65,6 +67,10 @@ export function grokRoutineService(db: Db, deliver: Delivery = deliverGrokRoutin
await logActivity(tx as unknown as Db, { companyId, actorType: "user", actorId: operatorId, action: "grok.recovery_inspected",
entityType: "agent", entityId: agentId, details: { inspectedRemoteWork: true, remoteTerminationConfirmed: false, requiresVerification: true } });
});
// Commit loss of readiness before retiring task holds. If inspection fails
// or waits for a task lock, the old verification cannot admit new work.
// The remaining task hold also prevents verification during this gap.
await inspectGrokWorkspaceObligations(db, companyId, agentId, operatorId);
return { recovered: true, requiresVerification: true, remoteTerminationConfirmed: false };
}
@@ -0,0 +1,87 @@
import { and, eq, inArray, isNotNull, or, sql } from "drizzle-orm";
import { dotRunnerAssignments, environmentLeases, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { authorizeExternalBotOperator } from "../external-bot-authority.js";
import { logActivity } from "../activity-log.js";
import { McpOAuthError } from "../public-mcp/oauth.js";
import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
import { hasRequiredWorkspaceRecovery, nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js";
/** Old Runner synchronization descriptors could incorrectly require a file
* export for a Bot explicitly admitted without workspace access. Keep the old
* execution fenced and its no-replay hold; inspection only retires that false
* file obligation. Native Runner cleanup never certifies global Grok exit. */
export async function grokWorkspaceInspectionCandidates(db: Db, companyId: string, agentId: string) {
const runs = await db.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.agentId, agentId),
eq(heartbeatRuns.driverKind, "grok_bot_mcp"), isNotNull(heartbeatRuns.finishedAt),
inArray(heartbeatRuns.status, ["cancelled", "failed", "timed_out", "interrupted"]),
sql`${heartbeatRuns.resultJson}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'`,
));
return runs.filter(run => nativeRunHasNoWorkspaceAccess(run) && hasRequiredWorkspaceRecovery(run.resultJson));
}
export async function inspectGrokWorkspaceObligations(db: Db, companyId: string, agentId: string, operatorId: string) {
for (const selected of await grokWorkspaceInspectionCandidates(db, companyId, agentId)) {
// Independent task transactions keep the native task/coordinator/run lock
// order out of the routine writer's binding/routine lock order. A partial
// inspection grants no authority and never schedules work.
await db.transaction(async tx => {
const database = tx as unknown as Db;
await authorizeExternalBotOperator(database, companyId, operatorId);
const issueId = selected.nativeIssueId ?? selected.issueId;
if (!issueId) return;
const [task] = await tx.select().from(issues).where(and(
eq(issues.companyId, companyId), eq(issues.id, issueId),
)).for("update");
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, selected.id),
eq(nativeRunFinalizations.issueId, issueId),
)).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.id, selected.id),
)).for("update");
if (!run || !hasRequiredWorkspaceRecovery(run.resultJson)) return;
const fail = () => new McpOAuthError("access_denied", "The stopped task still needs verified Runner cleanup before inspection can clear its file obligation.", 409);
if (!task || !nativeRunHasNoWorkspaceAccess(run) || run.driverKind !== "grok_bot_mcp"
|| !run.finishedAt || !["cancelled", "failed", "timed_out", "interrupted"].includes(run.status)
|| !coordinator || coordinator.phase !== "terminal_failure" || coordinator.leaseOwner
|| coordinator.failureDetail?.successorRunId) throw fail();
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["running", "queued", "scheduled_retry"]),
or(eq(heartbeatRuns.nativeIssueId, issueId), eq(heartbeatRuns.issueId, issueId)),
)).limit(1);
const work = await tx.select().from(dotRunnerAssignments).where(and(
eq(dotRunnerAssignments.companyId, companyId), eq(dotRunnerAssignments.agentId, agentId),
eq(dotRunnerAssignments.runId, run.id),
));
const leases = await tx.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, run.id),
));
if (active || !work.length || work.some(row => row.status !== "fenced") || !leases.length
|| !leases.every(lease => lease.issueId === issueId && hasRemoteTerminationReceipt(lease))) throw fail();
const actions = await tx.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
or(sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
sql`${issueRecoveryActions.evidence}->>'sourceRunId' = ${run.id}`),
)).for("update");
const inspection = { schema: "paperclip.grok-workspace-inspection.v1", runId: run.id,
inspectedBy: operatorId, inspectedAt: new Date().toISOString(), workspaceAccess: "none",
remoteTerminationConfirmed: false, runnerLeaseIds: leases.map(lease => lease.id),
previousWorkspaceRestoreFailure: run.resultJson?.workspaceRestoreFailure,
previousWorkspaceRestoreRecovery: run.resultJson?.workspaceRestoreRecovery };
const retireFileObligation = (value: Record<string, unknown>) => {
const { workspaceRestoreFailure: _failure, workspaceRestoreRecovery: _recovery, ...rest } = value;
return { ...rest, externalWorkspaceInspection: inspection };
};
await tx.update(heartbeatRuns).set({ resultJson: retireFileObligation(run.resultJson!) }).where(eq(heartbeatRuns.id, run.id));
for (const action of actions) {
await tx.update(issueRecoveryActions).set({ evidence: retireFileObligation(action.evidence), updatedAt: new Date(),
nextAction: "Inspect Grok and verify its connection before sending a new task message. The old work will not be replayed.",
}).where(eq(issueRecoveryActions.id, action.id));
}
await logActivity(database, { companyId, actorType: "user", actorId: operatorId,
action: "grok.workspace_obligation_inspected", entityType: "issue", entityId: issueId,
agentId, runId: run.id, details: inspection });
});
}
}
+2 -1
View File
@@ -10,6 +10,7 @@ import { selectHeartbeatEnvironment } from "./heartbeat/environment-selection.js
import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js";
import { executeHeartbeatRuntime, NativeSessionResumeScheduledError, NativeWorkspaceFinalizeScheduledError } from "./heartbeat/runtime-execution.js";
import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js";
import { nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
import { selectHeartbeatRuntime } from "./heartbeat/runtime-selection.js";
import { createHeartbeatRunCompletion } from "./heartbeat/run-completion.js";
export {
@@ -1948,7 +1949,7 @@ export function heartbeatService(
const failureRun = await getRun(run.id);
if (failureRun?.runtimeMode === "native"
&& parseObject(failureRun.runnerProfileJson?.nativeWorkspaceSync).state === "prepared") {
providerResourceDispositionForRun = "stop_and_retain";
if (!nativeRunHasNoWorkspaceAccess(failureRun)) providerResourceDispositionForRun = "stop_and_retain";
try {
await preserveNativeFailedWorkspace(db, failureRun, {
allowPreterminal: true,
@@ -59,9 +59,9 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () =
vi.mocked(executePaperclipNativeSession).mockResolvedValue({ exitCode: 0, signal: null, timedOut: false });
});
async function fixture(native = false) {
async function fixture(native = false, provider: "codex" | "grok_bot" | "openai_dot" = "codex") {
const [company] = await db.insert(companies).values({ name: "Runtime execution", issuePrefix: "EXE" }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Executing agent", status: "running", adapterType: native ? "paperclip_runner" : "process", adapterConfig: { provider: "codex" } }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Executing agent", status: "running", adapterType: native ? "paperclip_runner" : "process", adapterConfig: provider === "codex" ? { provider } : { provider, lifecycleMode: "per_turn", allowUnmeteredProvider: true, [provider === "grok_bot" ? "externalBindingId" : "dotBindingId"]: randomUUID() } }).returning();
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Complete this task", assigneeAgentId: agent.id }).returning();
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: native ? "native" : "legacy" }).returning();
const state = createHeartbeatRunState(db);
@@ -71,14 +71,15 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () =
const nativeExecution = native ? buildNativeExecutionInput({
companyId: company.id, agentId: agent.id, runId: run.id, issue,
taskPrompt: issue.title, workspace: { id: run.id, cwd: home, repoUrl: null, repoRef: null, branchName: null },
normalizedSessionId: null, provider: "codex", runtimeContext: nativeRuntimeContextFixture(),
normalizedSessionId: null, provider, runtimeContext: nativeRuntimeContextFixture(),
dotBinding: { bindingId: randomUUID(), bindingGeneration: 1, companyId: company.id, agentId: agent.id, acceptByUnixMs: Date.now() + 60_000, expiresAtUnixMs: Date.now() + 120_000 },
completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", contract: { revision: "1", objective: issue.title, criteria: [{ id: "output", requirement: issue.title }] } },
}) : null;
const input: HeartbeatRuntimeExecutionInput = {
run, agent, options: {},
task: { issueRef: issue, issueContext: null, context: {}, issueId: issue.id, taskKey: issue.id, executionContinuation: null },
runtime: { nativeExecution, nativeRunnerInstanceId: native ? randomUUID() : null, getNativeFreshSessionHandoff: undefined,
nativeRuntimeResolution: resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: native, runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: null, workspaceId: null }),
nativeRuntimeResolution: resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: native, grokEnabled: true, dotEnabled: true, runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: null, workspaceId: null }),
adapter, runtimeForAdapter: { taskKey: issue.id, sessionId: null, sessionParams: null, sessionDisplayId: null }, getFreshSessionHandoff: undefined, runOptions: {},
},
workspace: { persistedExecutionWorkspace: null, executionWorkspace: { baseCwd: home, cwd: home, source: "agent_home", projectId: null, workspaceId: null, repoUrl: null, repoRef: null, strategy: "project_primary", branchName: null, worktreePath: null, warnings: [], created: false, branchCreatedByRuntime: false },
@@ -259,8 +260,8 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () =
}
});
it("retains an unexported native source before provider-error teardown", async () => {
const { input, run } = await fixture(true);
it.each(["codex", "grok_bot", "openai_dot"] as const)("retains user files only when %s can access the workspace after a dispatch failure", async provider => {
const { input, run } = await fixture(true, provider);
input.workspace.nativeWorkspaceSync = {
mode: "host_current",
reference: { schema: "paperclip.native-workspace-sync/v3", state: "prepared",
@@ -271,7 +272,8 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () =
const error = new Error("runner_remote_process_identity_unavailable");
vi.mocked(executePaperclipNativeSession).mockRejectedValueOnce(error);
input.instructions.releaseInstructionCopy = vi.fn(async () => {
expect(input.effects.onProviderResourceDisposition).toHaveBeenCalledWith("stop_and_retain");
if (provider === "codex") expect(input.effects.onProviderResourceDisposition).toHaveBeenCalledWith("stop_and_retain");
else expect(input.effects.onProviderResourceDisposition).not.toHaveBeenCalledWith("stop_and_retain");
});
await expect(executeHeartbeatRuntime(db, input)).rejects.toBe(error);
expect(input.workspace.nativeWorkspaceSync.restoreWorkspace).not.toHaveBeenCalled();
@@ -995,7 +995,8 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
input.effects.onNativeOwnershipHeld();
throw adapterErr;
}
if (nativeRuntimeResolution.kind === "native" && nativeWorkspaceSync?.reference.state === "prepared") {
if (nativeRuntimeResolution.kind === "native" && nativeWorkspaceSync?.reference.state === "prepared"
&& !(nativeExecution && "access" in nativeExecution.workspace && nativeExecution.workspace.access === "none")) {
input.effects.onProviderResourceDisposition("stop_and_retain");
}
await db
@@ -76,6 +76,58 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
eq(activityLog.action, "issue.workspace_recovery_required")))).toHaveLength(1);
});
async function bindExternalInput(f: Awaited<ReturnType<typeof seed>>, provider: "grok_bot" | "openai_dot", change?: (input: { schema: string; binding: { runId: string; companyId: string; agentId: string }; workspace: { access: string } }) => void) {
const run = await readRun(f.runId);
const input = { schema: "paperclip.native-execution-input.v6",
binding: { runId: f.runId, companyId: f.companyId, agentId: f.agentId, issueId: f.issueId },
provider: { kind: provider }, workspace: { access: "none", cwd: null, repoUrl: null, repoRef: null, branchName: null } };
change?.(input);
const [updated] = await db.update(heartbeatRuns).set({ driverKind: provider === "grok_bot" ? "grok_bot_mcp" : "openai_dot_mcp",
runnerProfileJson: { ...run.runnerProfileJson, nativeExecutionInput: input } }).where(eq(heartbeatRuns.id, f.runId)).returning();
return updated;
}
it.each(["grok_bot", "openai_dot"] as const)("does not invent file recovery for %s admitted without workspace access", async provider => {
const f = await seed(), run = await bindExternalInput(f, provider);
await preserveNativeFailedWorkspace(db, run);
expect((await readRun(f.runId)).resultJson).not.toHaveProperty("workspaceRestoreFailure");
expect(await readLease(f.lease.id)).toMatchObject({ status: "active", leasePolicy: "ephemeral" });
expect(await actions(f.issueId)).toHaveLength(0);
});
it.each(["run", "company", "agent", "schema", "write_access"])("preserves file safeguards when no-workspace evidence has an invalid %s", async mismatch => {
const f = await seed();
const run = await bindExternalInput(f, "grok_bot", input => {
if (mismatch === "run") input.binding.runId = randomUUID();
if (mismatch === "company") input.binding.companyId = randomUUID();
if (mismatch === "agent") input.binding.agentId = randomUUID();
if (mismatch === "schema") input.schema = "unknown";
if (mismatch === "write_access") input.workspace.access = "read_write";
});
await preserveNativeFailedWorkspace(db, run);
expect((await readRun(f.runId)).resultJson).toHaveProperty("workspaceRestoreFailure", "restore_failed");
expect(await readLease(f.lease.id)).toMatchObject({ leasePolicy: "retain_on_failure" });
});
it.each(["grok_bot", "openai_dot"] as const)("keeps historical %s inspection holds with an available recovery path", async provider => {
const f = await seed();
// Reproduce a hold persisted by an older cancellation path, then attach its admitted contract.
await preserveNativeFailedWorkspace(db, f.run);
const lease = await readLease(f.lease.id);
await settleStopOnlyCleanup(db, lease, { attemptId: String(lease.metadata?.pendingCleanupAttemptId),
receipt: { providerLeaseId: lease.providerLeaseId, state: "stopped" } });
await bindExternalInput(f, provider);
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.runId));
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
expect(blocker).toMatchObject({ runId: f.runId, canRetry: false, canContinue: false,
workspaceRepairRequired: provider !== "grok_bot", canRecordWorkspaceRepair: provider !== "grok_bot" });
if (provider === "grok_bot") {
expect(blocker?.nextAction).not.toContain("files");
expect(blocker?.nextAction).toContain("Runtime");
}
expect(await actions(f.issueId)).toHaveLength(1);
});
it("records verified file repair without retrying the native run or changing its task", async () => {
const f = await seed();
await db.update(issues).set({ status: "done", assigneeAgentId: null }).where(eq(issues.id, f.issueId));
@@ -1,7 +1,7 @@
import { and, eq, gt, isNull, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { recordRequiredWorkspaceRestoreFailure } from "../legacy-execution-recovery.js";
import { hasRequiredWorkspaceRecovery } from "../workspace-restore-recovery-state.js";
import { hasRequiredWorkspaceRecovery, nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js";
import { logActivity } from "../activity-log.js";
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
import type { NativeRestartRecoveryClaim } from "./native-restart-recovery.js";
@@ -45,9 +45,12 @@ export async function preserveNativeFailedWorkspace(db: Db, selected: typeof hea
)).limit(1);
if (!source || source.issueId !== issueId || source.metadata?.remoteCwd !== currentReference.remoteCwd
|| (source.executionWorkspaceId && source.executionWorkspaceId !== currentReference.workspaceId)) return;
// Ownership settlement still applies to workspace-less external sessions.
// Their Runner descriptor is not evidence that user files need exporting.
const retainFiles = !nativeRunHasNoWorkspaceAccess(run);
const alreadyRetained = hasRequiredWorkspaceRecovery(run.resultJson);
let retained = alreadyRetained ? run : await recordRequiredWorkspaceRestoreFailure(tx as unknown as Db, run, { workspaceRestoreFailure: "restore_failed" }, source);
if (!retained || !hasRequiredWorkspaceRecovery(retained.resultJson)) return;
let retained = !retainFiles || alreadyRetained ? run : await recordRequiredWorkspaceRestoreFailure(tx as unknown as Db, run, { workspaceRestoreFailure: "restore_failed" }, source);
if (!retained || retainFiles && !hasRequiredWorkspaceRecovery(retained.resultJson)) return;
if (options.allowPreterminal && coordinator.phase === "observed") {
const [settled] = await tx.update(nativeRunFinalizations).set({ phase: "terminal_failure",
leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null, nextAttemptAt: null,
@@ -62,7 +65,7 @@ export async function preserveNativeFailedWorkspace(db: Db, selected: typeof hea
[retained] = await tx.update(heartbeatRuns).set({ nativePhase: "terminal_failure", nativePhaseUpdatedAt: new Date() })
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId))).returning();
}
if (!alreadyRetained) await logActivity(tx as unknown as Db, {
if (retainFiles && !alreadyRetained) await logActivity(tx as unknown as Db, {
companyId: run.companyId, actorType: "system", actorId: "native-workspace-recovery",
action: "issue.workspace_recovery_required", entityType: "issue",
entityId: issueId,
@@ -1,8 +1,26 @@
import type { heartbeatRuns } from "@paperclipai/db";
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
export const LEGACY_WORKSPACE_RECOVERY_KEY = "workspaceRestoreRecovery";
export const LEGACY_WORKSPACE_RECOVERY_SCHEMA = "paperclip.workspace-restore-recovery.v1";
/** The admitted external contract can prohibit workspace access even though its
* managed Runner has a synchronization descriptor for durable recovery. That
* descriptor alone does not create a user-file export obligation. */
export function nativeRunHasNoWorkspaceAccess(run: Pick<typeof heartbeatRuns.$inferSelect,
"id" | "companyId" | "agentId" | "issueId" | "nativeIssueId" | "runtimeMode" | "driverKind" | "runnerProfileJson">): boolean {
const input = run.runnerProfileJson?.nativeExecutionInput as Record<string, unknown> | undefined;
if (run.runtimeMode !== "native" || input?.schema !== "paperclip.native-execution-input.v6") return false;
const binding = input.binding as Record<string, unknown> | undefined;
const workspace = input.workspace as Record<string, unknown> | undefined;
const provider = input.provider as Record<string, unknown> | undefined;
const driver = provider?.kind === "grok_bot" ? "grok_bot_mcp"
: provider?.kind === "openai_dot" ? "openai_dot_mcp" : null;
return driver !== null && run.driverKind === driver && workspace?.access === "none" && workspace.cwd === null
&& binding?.runId === run.id && binding.companyId === run.companyId && binding.agentId === run.agentId
&& binding.issueId === (run.nativeIssueId ?? run.issueId);
}
export function hasRequiredWorkspaceRecovery(result: Record<string, unknown> | null | undefined): boolean {
const value = result?.[LEGACY_WORKSPACE_RECOVERY_KEY];
return hasWorkspaceRestoreFailure(result) && !!value && typeof value === "object" && !Array.isArray(value)