mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 21:07:40 +01:00
fix(grok): avoid false file recovery after Stop (#16427)
## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Grok Bot uses a managed Runner and a fixed external work session. > - Its admitted contract gives it no access to Paperclip workspace files. > - The Runner still has a sync descriptor to support durable recovery. > - Cancellation treated that descriptor as proof that files needed repair. > - This pull request uses the admitted access contract while preserving ownership settlement. > - Users get the correct recovery action after Stop, with native file safeguards intact. ## Linked Issues or Issue Description Refs #16219 and #16422. The duplicate search found no open Grok workspace correction. **What happened?** A real Grok Stop test cancelled its run, fenced its assignment, denied a stale progress update and cleaned up the sandbox. Paperclip also created a file-repair hold. The recovery panel told the user to recover workspace files, although the admitted external contract has workspace access set to none. **Expected behavior** Keep the stopped session fenced and require inspection before new work. Keep failed-run ownership settlement. Do not invent a file export obligation for a session that cannot access the workspace. Native sessions with writable or unknown workspace access must keep their file safeguards. **Steps to reproduce** 1. Connect a Grok Bot and assign a task. 2. Let it accept the task and save a Paperclip checkpoint document. 3. Use Stop while it is working. 4. Inspect the task recovery panel. It asks for file repair after sandbox cleanup succeeds. **Paperclip version or commit** Observed on Core11d69d04c7through the actual Grok client. Reproduced against master with database-backed regressions. ## What Changed - Match the admitted native input to its run, company, agent, task and external driver before treating workspace access as disabled. - Keep coordinator ownership checks and failure settlement. Skip file retention and the false file-repair obligation for that exact contract. - Add inspection-only settlement for old Grok file holds through Runtime recovery. Verify exact run ownership, fenced assignments, terminal coordinators and managed lease stop receipts. Archive the false file evidence and preserve no-replay. - Keep historical Dot recovery controls. Do not claim global external-provider termination. - Fix the later dispatch-failure owner so it cannot retain a no-workspace external sandbox for a false file obligation. - Cover both Grok and Dot, mismatched access evidence, native file recovery and the assembled heartbeat cleanup path. - Update the Grok guide and the existing release evidence plan. ## Verification - Four database regressions fail before the correction. - 68 file-recovery and cancellation checks pass. Grok, Dot and the shared recovery notice pass 98 checks. The assembled heartbeat selection passes both cases; 404 unrelated cases were not selected in that local command. - The lifecycle regression caught an unsafe early-return prototype. The final code keeps coordinator settlement for both workspace cases. A separate assertion was corrected for the run-state projection's explicit nulls. The failed attempts remain in the private evidence record. - Repository typecheck and build, token gates and Storybook build pass. The branch was rebased fromea4aa185b0onto7420af26fc, whose additional change affects CI profile exports. The core recovery checks are repeated after rebase. - On the earlier deployed source, a real 824-second task survived an actual same-image controller restart with its original run, assignment, process and lease. It completed canonically and cleaned up without an extra wake. The active Stop test denied a real stale update and completed connection inspection and authenticated re-verification. These results establish the observed baseline and defect; they do not qualify this patch's deployed behavior. - Review found two valid gaps: the old file obligation still blocked fresh input, and dispatch failure could request retention earlier. Both are corrected. The follow-up passes 128 focused tests and server typecheck. The recovery test proves fresh user continuation after inspection, authenticated re-verification, and managed Runner cleanup while the old assignment stays fenced. - A second review found a partial-transaction gap. Routine readiness now commits as invalid before task holds retire. All 43 Grok routine tests pass, including a held task lock and an unrelated active assignment. - Repository typecheck and build pass after the first review corrections; final server typecheck also passes. The full local suite is still running and is not claimed as passed. The first follow-up CI timed out at 15 seconds in one unchanged run-history case; all 91 cases pass locally. The cause is unproven. - The final head,33e4361911, passes 54 checks plus one successful status context. Two optional Storybook jobs are skipped. Greptile reviewed that exact head at 5/5 with no unresolved threads. All source CI suites pass on this final head without retry. The earlier runner-shutdown and timeout attempts remain recorded. - An actual deployed historical recovery and fresh Stop repeat are still required. ## Risks An incomplete or mismatched contract must retain file safeguards. The change uses stored admission evidence rather than current agent configuration. Historical records stay auditable and do not gain automatic replay or a generic Retry action. Operator inspection and a fresh authenticated verification remain required before new Grok work. Global Grok termination remains unconfirmed. No schema migration is needed. ## Model Used OpenAI GPT-6 in Codex, with reasoning, repository editing, shell tools and computer use. The exact serving model ID, snapshot and context window are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
57fa6b358d
commit
5a9e6b6315
13 files changed
+413
-30
No files matched your search
@@ -80,3 +80,18 @@ inspected recovery hold before admitting another assignment.
|
||||
stops future dispatch. Remote Grok execution may continue. Open the Grok Bot app
|
||||
to inspect it. Paperclip shows reported activity, not a complete remote transcript
|
||||
or computer recording.
|
||||
|
||||
The admitted Grok execution contract has no access to the Runner's workspace files. Its managed Runner still keeps durable recovery state, but stopping that session does not require workspace file repair. Inspect the Bot in Grok, clear the connection's recovery hold in the agent's Runtime settings, and verify the routine again before assigning new work. This records inspection without asserting global Grok termination.
|
||||
|
||||
Historical stopped Grok runs can have an incorrect workspace-file hold from an
|
||||
earlier release. Runtime shows Recovery needed for these runs even when the
|
||||
routine is healthy. After inspecting Grok, use Clear recovery hold and verify
|
||||
the connection again. Paperclip retires the false file obligation only when the
|
||||
original admission denied workspace access, the assignment is fenced, the
|
||||
coordinator has settled, and the exact managed Runner leases have stop receipts.
|
||||
Connection readiness is invalidated before task inspection, so a partial failure
|
||||
cannot retain an old verified connection while removing a task hold.
|
||||
The inspection preserves the old result, action uncertainty, and no-replay hold.
|
||||
Send a new task message to request fresh work. Inspection does not certify global
|
||||
Grok termination or delete any previously retained allocation. Such allocations
|
||||
still use the existing environment cleanup controls.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Grok Bot production readiness
|
||||
|
||||
Status as of October 15: real assigned and Bot-initiated tasks reach canonical completion. The active-session wake correction is merged. A real proactive test found a delayed-admission echo; its correction is under verification. Production acceptance is incomplete. The current release goal and matrix below supersede the earlier dated campaign entries. Historical successes and failures remain evidence for the revisions actually tested.
|
||||
Status as of October 15: real assigned and Bot-initiated tasks reach canonical completion. The active-session wake correction is merged. The proactive admission echo correction is merged and source-verified; image publication and a real no-echo repeat remain pending. A real controller restart and active Stop diagnostic passed their authority checks; Stop also exposed a false workspace-repair hold, now under correction. Production acceptance is incomplete. The current release goal and matrix below supersede the earlier dated campaign entries. Historical successes and failures remain evidence for the revisions actually tested.
|
||||
|
||||
## Current production finish line — October 14
|
||||
|
||||
@@ -13,7 +13,7 @@ There is no testing call, case-count, token or spending cap. The former six-case
|
||||
| Ordinary onboarding works | Actual Grok UI, fixed agent consent, saved setup, wrong-key behavior, authenticated challenge, refresh, reconnect and revocation | Resumed real Grok authorization, saved setup Done, authenticated routine challenge and automatic refresh passed on the October 15 serving revision. Runtime reconnect copy fix is merged; verify its published UI and fresh production setup next. |
|
||||
| Assigned work reaches a usable result | One actual managed workflow: progress, question and answer, document, downloaded file/hash, canonical completion or review and cleanup | A preliminary comparison task on serving Core ff0 completed progress → question → answer → report/CSV → canonical Done, with both cleanup receipts. The cited CSV downloads with the exact publication hash. Two distinct CSV uploads accompanied an active-session result wake; correct that wake behavior and requalify the final candidate. |
|
||||
| Bot initiates governed work | Real-client task creation/claim/completion with stable receipts and no outbound echo or duplicate effects | A real idle-conversation request created one task and one managed run, saved a 100-word report, reached accepted Done and cleaned up its lease. A delayed-admission wake and two acceptance attempts were also recorded. Functional completion passed; no-echo acceptance failed. Requalify the correction. |
|
||||
| Recovery and authority are reliable | Healthy work past the prior five-minute failure window; actual controller restart preserving assignment/session/process and no duplicate wake; Stop, pause, reassignment, revocation, stale calls, queues and cleanup | The old hold was inspected and cleared. The successful comparison continuation ended below five minutes. Healthy work beyond that window, an actual restart and active Stop remain unverified; protocol tests are separate evidence. |
|
||||
| Recovery and authority are reliable | Healthy work past the prior five-minute failure window; actual controller restart preserving assignment/session/process and no duplicate wake; Stop, pause, reassignment, revocation, stale calls, queues and cleanup | A real managed diagnostic ran for 13 minutes 44 seconds, survived an actual same-image controller restart with the original assignment/session/process/lease, completed canonically and cleaned up without another wake. Active Stop fenced its work, rejected a real stale progress update, kept its checkpoint unchanged and cleaned up. Cancellation also created a false file-repair hold despite an admitted workspace-access-none contract; correct and requalify that recovery behavior. These diagnostics ran on Core 11d, not the final release image. |
|
||||
| One identifiable reviewed candidate passes source gates | Focused protocol/auth/Runner/UI tests, full CI/typecheck/tests/build/token/Storybook evidence and exact artifacts | Active-session wake correction #16394 merged with 55 passing checks, two optional skips and review 5/5. The proactive echo correction started from master 7fc2dc8b3236e1149ae32756b6153a503623404b and was rebased onto freshly fetched master 250429942e27d568f20d3bc7dddcfc84a666480a before submission. Two regression cases reproduce the real delayed-admission wake before the fix. All 214 focused protocol/auth/question checks pass, including 80 Grok and Dot checks. Repository typecheck/build, token gates, the 2,731-story structure and Storybook build pass. Fresh CI, review, publication and the real no-echo repeat remain required. |
|
||||
| Production opt-in is delivered and smoke-tested | Staging acceptance, correct served Core/distribution/Runner, normal editable opt-in controls/defaults, actual production Bot workflow and inspectable task/output links | Cloud opt-in policy and release-guide corrections are merged. Staging Cloud provenance and minimal feature values were verified, with Dot disabled and defaults unchanged. The corrected wake candidate, remaining real-client cases, ordinary production opt-in controls and production smoke are outstanding. |
|
||||
|
||||
@@ -33,6 +33,12 @@ The proactive run also received an authenticated delayed-admission webhook while
|
||||
|
||||
Normal staging automation has advanced beyond the older prepared candidate. No shared CI routing switch was changed. The earlier temporary fallback request is no longer needed for the already served successor. The dedicated production workspace still has no qualification agent grant or smoke task.
|
||||
|
||||
The proactive no-echo correction merged as #16422 after 55 successful checks, two optional Storybook skips and fresh review 5/5. Canonical source verification and exact-source migrator publication passed. The first image attempt was displaced by an older rerun before any build job started; its single retry was then superseded by a newer master push. The successor contains the fix. Neither cancellation is a successful image publication, and no shared CI routing flag changed.
|
||||
|
||||
The controlled recovery diagnostic preserved one original run for 824 seconds. The operator created the task, performed one same-image controller redeploy after checking for other work and rollouts, and sent one planned normal resume comment. The Bot read that queued comment at a safe boundary, read its checkpoint back, saved revision two and reached accepted Done. Its exact source identities and successful cleanup receipt are retained privately. The Bot corrected its own rejected document operation and reported correcting an invalid finish attachment; neither was repaired by the operator. This proves observed recovery on Core 11d, not the cause of the older failure or final-image acceptance.
|
||||
|
||||
The active Stop diagnostic cancelled its run, fenced the assignment, kept the checkpoint at revision one and recorded successful sandbox cleanup. One planned negative probe in the real Grok client returned “Assignment authority expired or was revoked.” No late completion was accepted. Inspection found a generic workspace-repair obligation attached to an external native input that explicitly forbids workspace access. Four database regressions reproduce that obligation and its misleading recovery guidance. The correction keeps coordinator settlement, no-replay holds and native file safeguards while omitting file retention for an exact admitted workspace-less external contract. Repository typecheck and 68 focused file-recovery/cancellation checks pass; an assembled heartbeat regression exposed and preserved the coordinator-settlement requirement. Fresh review, full checks and deployed recovery remain required.
|
||||
|
||||
Keep native providers and person-purpose MCP connections unchanged. No browser automation belongs in normal operation. Defer localhost relays, internal Grok APIs, marketplace distribution and Paperclip-connected third-party tools. Completion of a Paperclip work session does not assert remote Grok termination.
|
||||
|
||||
## Candidate and release contract
|
||||
@@ -255,3 +261,31 @@ assignment authority. Disconnect completes while storage is pending, and the
|
||||
late upload returns 403 with no attachment or publication receipt. Person write
|
||||
checks remain in place. The full repository command was interrupted when this
|
||||
new review finding arrived; its partial output is not a passing test result.
|
||||
|
||||
## October 15 — Stop review follow-up
|
||||
|
||||
Review found that hiding the historical file-repair card alone left the stored
|
||||
file obligation blocking fresh task input. Runtime inspection now finds these
|
||||
holds even after routine recovery. It retires only the false file fields after
|
||||
exact run, assignment, coordinator and managed-lease cleanup checks, archives
|
||||
the previous evidence, and preserves no-replay. A real authenticated readiness
|
||||
exchange and new user input are still required. No global provider-stop claim is
|
||||
created. Dot historical recovery controls remain available.
|
||||
|
||||
The later dispatch-failure owner also now honors the validated no-workspace
|
||||
contract before asking for sandbox retention. Both external providers and the
|
||||
writable native path pass the runtime boundary regression. The review follow-up
|
||||
passes 128 focused protocol, recovery, cancellation and runtime checks. An initial
|
||||
negative assertion saw the earlier readiness gate instead of the revoked-session
|
||||
gate; the corrected test checks both before and after re-verification. Server
|
||||
typecheck passes after narrowing the native workspace union. Final current-head
|
||||
CI, review, deployed historical recovery and fresh Stop are still release gates.
|
||||
|
||||
A second review caught a partial-transaction ordering gap. Routine readiness now
|
||||
commits as invalid before the separate task inspection transactions. New tests
|
||||
hold the task lock to prove readiness is already gone and verification remains
|
||||
blocked; an unrelated active assignment cannot cause a task hold to disappear.
|
||||
All 43 Grok routine tests pass. The first follow-up CI attempt also timed out in
|
||||
one unchanged run-history case at 15 seconds; all 91 tests in that file pass
|
||||
locally. The timeout remains a recorded failure with cause unproven. The new
|
||||
source head requires fresh CI and review.
|
||||
@@ -8,7 +8,7 @@ import { createHash, randomBytes, randomUUID } from "node:crypto";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { beforeAll, afterAll, afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { agents, authUsers, companies, companyMemberships, createDb, dotAgentBindings, grokBotDispatches,
|
||||
grokBotRoutines, mcpOauthGrants, activityLog, issues, heartbeatRuns, dotRunnerAssignments, dotRunnerOperations, dotMailboxItems, nativeRunFinalizations } from "@paperclipai/db";
|
||||
grokBotRoutines, mcpOauthGrants, activityLog, issues, heartbeatRuns, dotRunnerAssignments, dotRunnerOperations, dotMailboxItems, nativeRunFinalizations, environmentLeases, environments, issueRecoveryActions, issueComments } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { accessService } from "../services/access.js";
|
||||
import { instanceSettingsService } from "../services/instance-settings.js";
|
||||
@@ -20,6 +20,10 @@ import * as secretModule from "../services/secrets.js";
|
||||
import * as externalAuthority from "../services/external-bot-authority.js";
|
||||
import { type GrokRoutineWake } from "../services/grok-bot/webhook.js";
|
||||
|
||||
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
|
||||
import { admitExplicitNativeContinuation } from "../services/explicit-native-continuation.js";
|
||||
import { hasRequiredWorkspaceRecovery } from "../services/workspace-restore-recovery-state.js";
|
||||
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
beforeAll(async () => {
|
||||
@@ -479,3 +483,140 @@ describe("durable Grok routine qualification", () => {
|
||||
});
|
||||
|
||||
});
|
||||
|
||||
// Exercise the public recovery service and the actual native admission boundary,
|
||||
// including a stale file obligation persisted before this fix.
|
||||
describe("Grok inspection of historical workspace-less Stop", () => {
|
||||
async function historicalHold() {
|
||||
const f = await fixture();
|
||||
const { run, task, work } = await admittedWork(f, "accepted");
|
||||
const finishedAt = new Date(Date.now() - 1000);
|
||||
const fileHold = { workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: {
|
||||
schema: "paperclip.workspace-restore-recovery.v1", runId: run.id, leaseIds: [] as string[],
|
||||
} };
|
||||
const [environment] = await db.insert(environments).values({ name: `Grok ${run.id}`, driver: "sandbox" }).returning();
|
||||
const [lease] = await db.insert(environmentLeases).values({ companyId: f.company.id, issueId: task.id,
|
||||
heartbeatRunId: run.id, environmentId: environment!.id, provider: "daytona", providerLeaseId: randomUUID(),
|
||||
status: "released", releasedAt: new Date(), cleanupStatus: "success" }).returning();
|
||||
await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination:
|
||||
remoteTerminationReceipt(lease!, { providerLeaseId: lease!.providerLeaseId, state: "destroyed" }) } }).where(eq(environmentLeases.id, lease!.id));
|
||||
fileHold.workspaceRestoreRecovery.leaseIds.push(lease!.id);
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled", driverKind: "grok_bot_mcp", finishedAt,
|
||||
nativeIssueId: task.id, issueId: task.id, contextSnapshot: { issueId: task.id },
|
||||
runnerProfileJson: { nativeExecutionInput: { schema: "paperclip.native-execution-input.v6",
|
||||
binding: { runId: run.id, companyId: f.company.id, agentId: f.agent.id, issueId: task.id },
|
||||
provider: { kind: "grok_bot" }, workspace: { access: "none", cwd: null } } },
|
||||
resultJson: { ...fileHold, cancellation: { source: "user", reason: "Paperclip work stopped" } },
|
||||
}).where(eq(heartbeatRuns.id, run.id));
|
||||
await db.update(issues).set({ status: "blocked", executionRunId: null, checkoutRunId: null }).where(eq(issues.id, task.id));
|
||||
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", leaseOwner: null,
|
||||
leaseExpiresAt: null, recoveryState: "blocked" }).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
await db.update(dotRunnerAssignments).set({ status: "fenced" }).where(eq(dotRunnerAssignments.id, work.id));
|
||||
const [action] = await db.insert(issueRecoveryActions).values({ companyId: f.company.id, sourceIssueId: task.id,
|
||||
kind: "active_run_watchdog", cause: "native_continuation_requires_reconciliation", fingerprint: run.id,
|
||||
status: "resolved", outcome: "blocked", nextAction: "Recover files.",
|
||||
evidence: { ...fileHold, runId: run.id, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } },
|
||||
}).returning();
|
||||
const [comment] = await db.insert(issueComments).values({ companyId: f.company.id, issueId: task.id,
|
||||
authorType: "user", authorUserId: f.principal.grant.userId, body: "Start a fresh task turn after inspection." }).returning();
|
||||
const continueWork = (dryRun = true) => db.transaction(async tx => {
|
||||
await tx.select().from(issues).where(eq(issues.id, task.id)).for("update");
|
||||
return admitExplicitNativeContinuation({ db: tx as unknown as typeof db, companyId: f.company.id,
|
||||
issueId: task.id, agentId: f.agent.id, actorType: "user", actorId: f.principal.grant.userId,
|
||||
reason: "issue_commented", commentId: comment!.id, successorRunId: randomUUID(), dryRun });
|
||||
});
|
||||
return { ...f, run, task, work, lease: lease!, action: action!, continueWork };
|
||||
}
|
||||
|
||||
it("offers inspection for an old task hold and admits only new explicit input after inspection and Runner cleanup", async () => {
|
||||
const f = await historicalHold();
|
||||
f.delivery.mockClear();
|
||||
expect(await f.service.recoveryNeeded(f.company.id, f.agent.id)).toBe(true);
|
||||
expect(await f.continueWork()).toBeNull();
|
||||
expect(await f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId)).toMatchObject({
|
||||
requiresVerification: true, remoteTerminationConfirmed: false,
|
||||
});
|
||||
expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: false });
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
|
||||
expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(false);
|
||||
expect(run).toMatchObject({ status: "cancelled", resultJson: { externalWorkspaceInspection: {
|
||||
inspectedBy: f.principal.grant.userId, remoteTerminationConfirmed: false,
|
||||
previousWorkspaceRestoreFailure: "restore_failed", runnerLeaseIds: [f.lease.id],
|
||||
} } });
|
||||
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id));
|
||||
expect(action).toMatchObject({ status: "resolved", outcome: "blocked", evidence: {
|
||||
automaticRecovery: { replay: "blocked", actionOutcome: "unknown" }, externalWorkspaceInspection: { workspaceAccess: "none" },
|
||||
} });
|
||||
expect(action!.evidence).not.toHaveProperty("workspaceRestoreRecovery");
|
||||
expect(f.delivery).not.toHaveBeenCalled();
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.company.id))).toHaveLength(1);
|
||||
await expect(dotRunnerBroker(db).operation(f.principal, f.work.id, randomUUID(), "progress", { text: "late effect" })).rejects.toThrow("wake verification");
|
||||
// Connection recovery still requires a real authenticated challenge exchange.
|
||||
const row = await verification(f); await f.service.send(f.company.id, f.agent.id, row.id);
|
||||
await f.service.acknowledge(f.principal, acknowledgement(row));
|
||||
await expect(dotRunnerBroker(db).operation(f.principal, f.work.id, randomUUID(), "progress", { text: "late effect" })).rejects.toThrow("revoked");
|
||||
expect(await f.continueWork(false)).toMatchObject({ previousRunId: f.run.id });
|
||||
const [settled] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id));
|
||||
expect(settled!.evidence.automaticRecovery).toMatchObject({ replay: "explicit_user_continuation", actionOutcome: "unknown" });
|
||||
expect((await db.select().from(dotRunnerAssignments).where(eq(dotRunnerAssignments.id, f.work.id)))[0]!.status).toBe("fenced");
|
||||
await f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId);
|
||||
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.company.id), eq(activityLog.action, "grok.workspace_obligation_inspected")))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("does not retire task holds if another assignment prevents recovery", async () => {
|
||||
const f = await historicalHold();
|
||||
const [other] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id,
|
||||
status: "running", invocationSource: "assignment", triggerDetail: "system" }).returning();
|
||||
await db.insert(dotRunnerAssignments).values({ ...f.work, id: randomUUID(), runId: other!.id,
|
||||
normalizedSessionId: randomUUID(), turnId: randomUUID(), status: "accepted" });
|
||||
await expect(f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId)).rejects.toThrow("current Paperclip work session");
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
|
||||
expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(true);
|
||||
expect(await f.service.recoveryNeeded(f.company.id, f.agent.id)).toBe(true);
|
||||
});
|
||||
|
||||
it("commits verification invalidation before waiting for historical task inspection", async () => {
|
||||
const f = await historicalHold();
|
||||
let recovery: Promise<unknown> | undefined;
|
||||
try {
|
||||
await db.transaction(async tx => {
|
||||
await tx.select().from(issues).where(eq(issues.id, f.task.id)).for("update");
|
||||
recovery = f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId);
|
||||
void recovery.catch(() => {});
|
||||
await vi.waitFor(async () => {
|
||||
expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: true });
|
||||
});
|
||||
await expect(verification(f)).rejects.toThrow("recover");
|
||||
const [binding] = await db.select().from(dotAgentBindings).where(eq(dotAgentBindings.id, f.pairing.bindingId));
|
||||
expect(binding).toMatchObject({ status: "connected", readyAt: null });
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
|
||||
expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(true);
|
||||
});
|
||||
} finally { await recovery; }
|
||||
expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: false });
|
||||
});
|
||||
|
||||
it.each(["cleanup_pending", "receipt_mismatch", "controller_owned", "assignment_active", "running", "operator_revoked", "foreign_company"])("does not retire an old obligation when %s", async condition => {
|
||||
const f = await historicalHold();
|
||||
if (condition === "cleanup_pending") await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, f.lease.id));
|
||||
if (condition === "receipt_mismatch") await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", runId: randomUUID() } } }).where(eq(environmentLeases.id, f.lease.id));
|
||||
if (condition === "controller_owned") await db.update(nativeRunFinalizations).set({ leaseOwner: "other-owner" }).where(eq(nativeRunFinalizations.runId, f.run.id));
|
||||
if (condition === "assignment_active") await db.update(dotRunnerAssignments).set({ status: "accepted" }).where(eq(dotRunnerAssignments.id, f.work.id));
|
||||
if (condition === "running") await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, f.run.id));
|
||||
if (condition === "operator_revoked") await accessService(db).setPrincipalPermission(f.company.id, "user", f.principal.grant.userId, "agents:create", false, f.principal.grant.userId);
|
||||
if (condition === "foreign_company") {
|
||||
const foreign = await fixture();
|
||||
await f.service.recover(foreign.company.id, foreign.agent.id, foreign.principal.grant.userId);
|
||||
} else if (condition === "running") {
|
||||
// Not an eligible historical source. Its active assignment is fenced in
|
||||
// this deliberately malformed fixture, but no file state may be removed.
|
||||
await f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId);
|
||||
} else await expect(f.service.recover(f.company.id, f.agent.id, f.principal.grant.userId)).rejects.toThrow();
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
|
||||
expect(hasRequiredWorkspaceRecovery(run!.resultJson)).toBe(true);
|
||||
if (["cleanup_pending", "receipt_mismatch", "controller_owned"].includes(condition)) {
|
||||
expect(await f.service.state(f.company.id, f.agent.id, f.pairing.bindingId)).toMatchObject({ verified: false, recoveryNeeded: true });
|
||||
}
|
||||
expect(await f.continueWork()).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -3521,7 +3521,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
}
|
||||
}
|
||||
|
||||
it("retains the source when a saved native descriptor fails inside runtime selection", async () => {
|
||||
it.each([false, true])("settles a failed saved native descriptor without inventing files when workspace access is disabled: %s", async withoutWorkspace => {
|
||||
await withTempPaperclipHome(async () => {
|
||||
const f = await seedQueuedIssueRunFixture();
|
||||
await db.update(agents).set({ adapterType: "paperclip_runner",
|
||||
@@ -3547,7 +3547,12 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null,
|
||||
workspaceId: randomUUID(), leaseId: lease!.id, providerLeaseId: lease!.providerLeaseId!, remoteCwd: "/work", resourceDisposition: "destroy" as const };
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issueId, nativePhase: "observed",
|
||||
runnerProfileJson: { nativeWorkspaceSync: reference },
|
||||
driverKind: withoutWorkspace ? "grok_bot_mcp" : null,
|
||||
runnerProfileJson: { nativeWorkspaceSync: reference, ...(withoutWorkspace ? { nativeExecutionInput: {
|
||||
schema: "paperclip.native-execution-input.v6", provider: { kind: "grok_bot" },
|
||||
binding: { companyId: f.companyId, agentId: f.agentId, runId, issueId: f.issueId },
|
||||
workspace: { access: "none", cwd: null, repoUrl: null, repoRef: null, branchName: null },
|
||||
} } : {}) },
|
||||
}).where(eq(heartbeatRuns.id, runId));
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId: f.companyId, issueId: f.issueId, phase: "observed" });
|
||||
await nativeWorkspaceSyncInternals.readDescriptor({ companyId: f.companyId, runId, workspaceId: reference.workspaceId, reference });
|
||||
@@ -3556,15 +3561,21 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
expect(leaseId).toBeDefined();
|
||||
expect(await heartbeat.getRun(f.runId, { includeExecutionEvidence: true })).toMatchObject({ status: "failed",
|
||||
runtimeMode: "native", nativePhase: "terminal_failure", resultJson: {
|
||||
workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [leaseId] },
|
||||
},
|
||||
});
|
||||
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId!)))
|
||||
.toMatchObject([{ status: "pending_cleanup", leasePolicy: "retain_on_failure" }]);
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ workspaceRepairRequired: true, canRetry: false });
|
||||
expect(release).toHaveBeenCalledWith(expect.objectContaining({ providerResourceDisposition: "stop_and_retain" }));
|
||||
const failed = await heartbeat.getRun(f.runId, { includeExecutionEvidence: true });
|
||||
expect(failed).toMatchObject({ status: "failed", runtimeMode: "native", nativePhase: "terminal_failure" });
|
||||
if (withoutWorkspace) {
|
||||
expect(failed?.resultJson?.workspaceRestoreFailure ?? null).toBeNull();
|
||||
expect(failed?.resultJson?.workspaceRestoreRecovery ?? null).toBeNull();
|
||||
expect((await getExecutionBlocker(db, f.companyId, f.issueId))?.workspaceRepairRequired).not.toBe(true);
|
||||
expect(release).toHaveBeenCalled();
|
||||
expect(release).not.toHaveBeenCalledWith(expect.objectContaining({ providerResourceDisposition: "stop_and_retain" }));
|
||||
} else {
|
||||
expect(failed?.resultJson).toMatchObject({ workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [leaseId] } });
|
||||
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId!)))
|
||||
.toMatchObject([{ status: "pending_cleanup", leasePolicy: "retain_on_failure" }]);
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ workspaceRepairRequired: true, canRetry: false });
|
||||
expect(release).toHaveBeenCalledWith(expect.objectContaining({ providerResourceDisposition: "stop_and_retain" }));
|
||||
}
|
||||
expect(provider).not.toHaveBeenCalled();
|
||||
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
||||
} finally { factory.mockRestore(); }
|
||||
|
||||
@@ -7,7 +7,7 @@ import { canContinueCancelledRun, readRunCancellation } from "./run-cancellation
|
||||
import { canRetryStoppedRun } from "./cancelled-native-startup.js";
|
||||
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
|
||||
import { EXECUTION_RECONCILIATION_CAUSES, type ExecutionBlocker } from "@paperclipai/shared";
|
||||
import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA } from "./workspace-restore-recovery-state.js";
|
||||
import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA, nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
|
||||
|
||||
/** Resolved recovery bookkeeping can still carry an effective no-replay hold. */
|
||||
export function executionBlockerPredicate() {
|
||||
@@ -65,6 +65,8 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
|
||||
const [run] = runId ? await db.select().from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId),
|
||||
)).limit(1) : [];
|
||||
const withoutWorkspace = Boolean(run?.driverKind === "grok_bot_mcp" && nativeRunHasNoWorkspaceAccess(run));
|
||||
const workspaceRepairRequired = Boolean(restoreHold && !withoutWorkspace);
|
||||
const queue = await db.select({ payload: agentWakeupRequests.payload }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
||||
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
||||
@@ -118,6 +120,16 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
|
||||
nextAction += " Inspect the run before sending a new message to request continuation.";
|
||||
}
|
||||
|
||||
if (withoutWorkspace && (restoreHold || run?.status === "cancelled") && !run?.resultJson?.externalWorkspaceInspection) {
|
||||
nextAction = run?.driverKind === "grok_bot_mcp"
|
||||
? "Inspect this Bot in Grok Bot, then clear the recovery hold and verify the connection in the agent's Runtime settings before assigning more work."
|
||||
: "Inspect the stopped work in the external agent and its connection before requesting continuation.";
|
||||
}
|
||||
|
||||
if (run?.resultJson?.externalWorkspaceInspection && !restoreHold) {
|
||||
nextAction = "Verify the Grok connection, then send a new message to continue. The stopped work will not be replayed.";
|
||||
}
|
||||
|
||||
return {
|
||||
recoveryActionId: action.id,
|
||||
runId,
|
||||
@@ -129,7 +141,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
|
||||
runError: runError?.slice(0, 1024) ?? null,
|
||||
canContinue: eligibleContinuation && !chatBinding,
|
||||
canRetry: canRetry && !chatBinding,
|
||||
...(restoreHold ? { workspaceRepairRequired: true, canRecordWorkspaceRepair: hasRequiredWorkspaceRecovery(action.evidence) } : {}),
|
||||
...(restoreHold ? { workspaceRepairRequired, canRecordWorkspaceRepair: workspaceRepairRequired && hasRequiredWorkspaceRecovery(action.evidence) } : {}),
|
||||
savedMessageCount: saved.length,
|
||||
};
|
||||
}
|
||||
@@ -10,6 +10,8 @@ import { scheduleAgentLifecycle } from "../agent-lifecycle.js";
|
||||
import { logActivity } from "../activity-log.js";
|
||||
import { deliverGrokRoutineWake, validateGrokRoutineCredential, type GrokRoutineDelivery, type GrokRoutineWake } from "./webhook.js";
|
||||
|
||||
import { grokWorkspaceInspectionCandidates, inspectGrokWorkspaceObligations } from "./workspace-inspection.js";
|
||||
|
||||
const digest = (value: string) => createHash("sha256").update(value).digest("hex");
|
||||
const fail = (message: string) => new McpOAuthError("access_denied", message, 409);
|
||||
export const grokWakeAcknowledgementSchema = z.object({ correlationId: z.uuid(), bindingId: z.uuid(),
|
||||
@@ -44,7 +46,7 @@ export function grokRoutineService(db: Db, deliver: Delivery = deliverGrokRoutin
|
||||
const [uncertain] = await db.select({ id: dispatches.id }).from(dispatches).innerJoin(routines, eq(routines.id, dispatches.routineId))
|
||||
.where(and(eq(dispatches.companyId, companyId), eq(routines.agentId, agentId),
|
||||
or(eq(dispatches.status, "unknown"), and(eq(dispatches.status, "sending"), lte(dispatches.updatedAt, new Date(Date.now() - 30_000))), and(eq(dispatches.status, "accepted"), lte(dispatches.expiresAt, new Date()))))).limit(1);
|
||||
return !!held || !!uncertain;
|
||||
return !!held || !!uncertain || (await grokWorkspaceInspectionCandidates(db, companyId, agentId)).length > 0;
|
||||
}
|
||||
|
||||
async function recover(companyId: string, agentId: string, operatorId: string) {
|
||||
@@ -65,6 +67,10 @@ export function grokRoutineService(db: Db, deliver: Delivery = deliverGrokRoutin
|
||||
await logActivity(tx as unknown as Db, { companyId, actorType: "user", actorId: operatorId, action: "grok.recovery_inspected",
|
||||
entityType: "agent", entityId: agentId, details: { inspectedRemoteWork: true, remoteTerminationConfirmed: false, requiresVerification: true } });
|
||||
});
|
||||
// Commit loss of readiness before retiring task holds. If inspection fails
|
||||
// or waits for a task lock, the old verification cannot admit new work.
|
||||
// The remaining task hold also prevents verification during this gap.
|
||||
await inspectGrokWorkspaceObligations(db, companyId, agentId, operatorId);
|
||||
return { recovered: true, requiresVerification: true, remoteTerminationConfirmed: false };
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
import { and, eq, inArray, isNotNull, or, sql } from "drizzle-orm";
|
||||
import { dotRunnerAssignments, environmentLeases, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import { authorizeExternalBotOperator } from "../external-bot-authority.js";
|
||||
import { logActivity } from "../activity-log.js";
|
||||
import { McpOAuthError } from "../public-mcp/oauth.js";
|
||||
import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
|
||||
import { hasRequiredWorkspaceRecovery, nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js";
|
||||
|
||||
/** Old Runner synchronization descriptors could incorrectly require a file
|
||||
* export for a Bot explicitly admitted without workspace access. Keep the old
|
||||
* execution fenced and its no-replay hold; inspection only retires that false
|
||||
* file obligation. Native Runner cleanup never certifies global Grok exit. */
|
||||
export async function grokWorkspaceInspectionCandidates(db: Db, companyId: string, agentId: string) {
|
||||
const runs = await db.select().from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.agentId, agentId),
|
||||
eq(heartbeatRuns.driverKind, "grok_bot_mcp"), isNotNull(heartbeatRuns.finishedAt),
|
||||
inArray(heartbeatRuns.status, ["cancelled", "failed", "timed_out", "interrupted"]),
|
||||
sql`${heartbeatRuns.resultJson}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'`,
|
||||
));
|
||||
return runs.filter(run => nativeRunHasNoWorkspaceAccess(run) && hasRequiredWorkspaceRecovery(run.resultJson));
|
||||
}
|
||||
|
||||
export async function inspectGrokWorkspaceObligations(db: Db, companyId: string, agentId: string, operatorId: string) {
|
||||
for (const selected of await grokWorkspaceInspectionCandidates(db, companyId, agentId)) {
|
||||
// Independent task transactions keep the native task/coordinator/run lock
|
||||
// order out of the routine writer's binding/routine lock order. A partial
|
||||
// inspection grants no authority and never schedules work.
|
||||
await db.transaction(async tx => {
|
||||
const database = tx as unknown as Db;
|
||||
await authorizeExternalBotOperator(database, companyId, operatorId);
|
||||
const issueId = selected.nativeIssueId ?? selected.issueId;
|
||||
if (!issueId) return;
|
||||
const [task] = await tx.select().from(issues).where(and(
|
||||
eq(issues.companyId, companyId), eq(issues.id, issueId),
|
||||
)).for("update");
|
||||
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(
|
||||
eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, selected.id),
|
||||
eq(nativeRunFinalizations.issueId, issueId),
|
||||
)).for("update");
|
||||
const [run] = await tx.select().from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.id, selected.id),
|
||||
)).for("update");
|
||||
if (!run || !hasRequiredWorkspaceRecovery(run.resultJson)) return;
|
||||
const fail = () => new McpOAuthError("access_denied", "The stopped task still needs verified Runner cleanup before inspection can clear its file obligation.", 409);
|
||||
if (!task || !nativeRunHasNoWorkspaceAccess(run) || run.driverKind !== "grok_bot_mcp"
|
||||
|| !run.finishedAt || !["cancelled", "failed", "timed_out", "interrupted"].includes(run.status)
|
||||
|| !coordinator || coordinator.phase !== "terminal_failure" || coordinator.leaseOwner
|
||||
|| coordinator.failureDetail?.successorRunId) throw fail();
|
||||
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["running", "queued", "scheduled_retry"]),
|
||||
or(eq(heartbeatRuns.nativeIssueId, issueId), eq(heartbeatRuns.issueId, issueId)),
|
||||
)).limit(1);
|
||||
const work = await tx.select().from(dotRunnerAssignments).where(and(
|
||||
eq(dotRunnerAssignments.companyId, companyId), eq(dotRunnerAssignments.agentId, agentId),
|
||||
eq(dotRunnerAssignments.runId, run.id),
|
||||
));
|
||||
const leases = await tx.select().from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, run.id),
|
||||
));
|
||||
if (active || !work.length || work.some(row => row.status !== "fenced") || !leases.length
|
||||
|| !leases.every(lease => lease.issueId === issueId && hasRemoteTerminationReceipt(lease))) throw fail();
|
||||
const actions = await tx.select().from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
||||
or(sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
|
||||
sql`${issueRecoveryActions.evidence}->>'sourceRunId' = ${run.id}`),
|
||||
)).for("update");
|
||||
const inspection = { schema: "paperclip.grok-workspace-inspection.v1", runId: run.id,
|
||||
inspectedBy: operatorId, inspectedAt: new Date().toISOString(), workspaceAccess: "none",
|
||||
remoteTerminationConfirmed: false, runnerLeaseIds: leases.map(lease => lease.id),
|
||||
previousWorkspaceRestoreFailure: run.resultJson?.workspaceRestoreFailure,
|
||||
previousWorkspaceRestoreRecovery: run.resultJson?.workspaceRestoreRecovery };
|
||||
const retireFileObligation = (value: Record<string, unknown>) => {
|
||||
const { workspaceRestoreFailure: _failure, workspaceRestoreRecovery: _recovery, ...rest } = value;
|
||||
return { ...rest, externalWorkspaceInspection: inspection };
|
||||
};
|
||||
await tx.update(heartbeatRuns).set({ resultJson: retireFileObligation(run.resultJson!) }).where(eq(heartbeatRuns.id, run.id));
|
||||
for (const action of actions) {
|
||||
await tx.update(issueRecoveryActions).set({ evidence: retireFileObligation(action.evidence), updatedAt: new Date(),
|
||||
nextAction: "Inspect Grok and verify its connection before sending a new task message. The old work will not be replayed.",
|
||||
}).where(eq(issueRecoveryActions.id, action.id));
|
||||
}
|
||||
await logActivity(database, { companyId, actorType: "user", actorId: operatorId,
|
||||
action: "grok.workspace_obligation_inspected", entityType: "issue", entityId: issueId,
|
||||
agentId, runId: run.id, details: inspection });
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,7 @@ import { selectHeartbeatEnvironment } from "./heartbeat/environment-selection.js
|
||||
import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js";
|
||||
import { executeHeartbeatRuntime, NativeSessionResumeScheduledError, NativeWorkspaceFinalizeScheduledError } from "./heartbeat/runtime-execution.js";
|
||||
import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js";
|
||||
import { nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
|
||||
import { selectHeartbeatRuntime } from "./heartbeat/runtime-selection.js";
|
||||
import { createHeartbeatRunCompletion } from "./heartbeat/run-completion.js";
|
||||
export {
|
||||
@@ -1948,7 +1949,7 @@ export function heartbeatService(
|
||||
const failureRun = await getRun(run.id);
|
||||
if (failureRun?.runtimeMode === "native"
|
||||
&& parseObject(failureRun.runnerProfileJson?.nativeWorkspaceSync).state === "prepared") {
|
||||
providerResourceDispositionForRun = "stop_and_retain";
|
||||
if (!nativeRunHasNoWorkspaceAccess(failureRun)) providerResourceDispositionForRun = "stop_and_retain";
|
||||
try {
|
||||
await preserveNativeFailedWorkspace(db, failureRun, {
|
||||
allowPreterminal: true,
|
||||
|
||||
@@ -59,9 +59,9 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () =
|
||||
vi.mocked(executePaperclipNativeSession).mockResolvedValue({ exitCode: 0, signal: null, timedOut: false });
|
||||
});
|
||||
|
||||
async function fixture(native = false) {
|
||||
async function fixture(native = false, provider: "codex" | "grok_bot" | "openai_dot" = "codex") {
|
||||
const [company] = await db.insert(companies).values({ name: "Runtime execution", issuePrefix: "EXE" }).returning();
|
||||
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Executing agent", status: "running", adapterType: native ? "paperclip_runner" : "process", adapterConfig: { provider: "codex" } }).returning();
|
||||
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Executing agent", status: "running", adapterType: native ? "paperclip_runner" : "process", adapterConfig: provider === "codex" ? { provider } : { provider, lifecycleMode: "per_turn", allowUnmeteredProvider: true, [provider === "grok_bot" ? "externalBindingId" : "dotBindingId"]: randomUUID() } }).returning();
|
||||
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Complete this task", assigneeAgentId: agent.id }).returning();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, issueId: issue.id, scopeKind: "issue", status: "running", runtimeMode: native ? "native" : "legacy" }).returning();
|
||||
const state = createHeartbeatRunState(db);
|
||||
@@ -71,14 +71,15 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () =
|
||||
const nativeExecution = native ? buildNativeExecutionInput({
|
||||
companyId: company.id, agentId: agent.id, runId: run.id, issue,
|
||||
taskPrompt: issue.title, workspace: { id: run.id, cwd: home, repoUrl: null, repoRef: null, branchName: null },
|
||||
normalizedSessionId: null, provider: "codex", runtimeContext: nativeRuntimeContextFixture(),
|
||||
normalizedSessionId: null, provider, runtimeContext: nativeRuntimeContextFixture(),
|
||||
dotBinding: { bindingId: randomUUID(), bindingGeneration: 1, companyId: company.id, agentId: agent.id, acceptByUnixMs: Date.now() + 60_000, expiresAtUnixMs: Date.now() + 120_000 },
|
||||
completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", contract: { revision: "1", objective: issue.title, criteria: [{ id: "output", requirement: issue.title }] } },
|
||||
}) : null;
|
||||
const input: HeartbeatRuntimeExecutionInput = {
|
||||
run, agent, options: {},
|
||||
task: { issueRef: issue, issueContext: null, context: {}, issueId: issue.id, taskKey: issue.id, executionContinuation: null },
|
||||
runtime: { nativeExecution, nativeRunnerInstanceId: native ? randomUUID() : null, getNativeFreshSessionHandoff: undefined,
|
||||
nativeRuntimeResolution: resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: native, runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: null, workspaceId: null }),
|
||||
nativeRuntimeResolution: resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: native, grokEnabled: true, dotEnabled: true, runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: null, workspaceId: null }),
|
||||
adapter, runtimeForAdapter: { taskKey: issue.id, sessionId: null, sessionParams: null, sessionDisplayId: null }, getFreshSessionHandoff: undefined, runOptions: {},
|
||||
},
|
||||
workspace: { persistedExecutionWorkspace: null, executionWorkspace: { baseCwd: home, cwd: home, source: "agent_home", projectId: null, workspaceId: null, repoUrl: null, repoRef: null, strategy: "project_primary", branchName: null, worktreePath: null, warnings: [], created: false, branchCreatedByRuntime: false },
|
||||
@@ -259,8 +260,8 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () =
|
||||
}
|
||||
});
|
||||
|
||||
it("retains an unexported native source before provider-error teardown", async () => {
|
||||
const { input, run } = await fixture(true);
|
||||
it.each(["codex", "grok_bot", "openai_dot"] as const)("retains user files only when %s can access the workspace after a dispatch failure", async provider => {
|
||||
const { input, run } = await fixture(true, provider);
|
||||
input.workspace.nativeWorkspaceSync = {
|
||||
mode: "host_current",
|
||||
reference: { schema: "paperclip.native-workspace-sync/v3", state: "prepared",
|
||||
@@ -271,7 +272,8 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () =
|
||||
const error = new Error("runner_remote_process_identity_unavailable");
|
||||
vi.mocked(executePaperclipNativeSession).mockRejectedValueOnce(error);
|
||||
input.instructions.releaseInstructionCopy = vi.fn(async () => {
|
||||
expect(input.effects.onProviderResourceDisposition).toHaveBeenCalledWith("stop_and_retain");
|
||||
if (provider === "codex") expect(input.effects.onProviderResourceDisposition).toHaveBeenCalledWith("stop_and_retain");
|
||||
else expect(input.effects.onProviderResourceDisposition).not.toHaveBeenCalledWith("stop_and_retain");
|
||||
});
|
||||
await expect(executeHeartbeatRuntime(db, input)).rejects.toBe(error);
|
||||
expect(input.workspace.nativeWorkspaceSync.restoreWorkspace).not.toHaveBeenCalled();
|
||||
|
||||
@@ -995,7 +995,8 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
|
||||
input.effects.onNativeOwnershipHeld();
|
||||
throw adapterErr;
|
||||
}
|
||||
if (nativeRuntimeResolution.kind === "native" && nativeWorkspaceSync?.reference.state === "prepared") {
|
||||
if (nativeRuntimeResolution.kind === "native" && nativeWorkspaceSync?.reference.state === "prepared"
|
||||
&& !(nativeExecution && "access" in nativeExecution.workspace && nativeExecution.workspace.access === "none")) {
|
||||
input.effects.onProviderResourceDisposition("stop_and_retain");
|
||||
}
|
||||
await db
|
||||
|
||||
@@ -76,6 +76,58 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
|
||||
eq(activityLog.action, "issue.workspace_recovery_required")))).toHaveLength(1);
|
||||
});
|
||||
|
||||
async function bindExternalInput(f: Awaited<ReturnType<typeof seed>>, provider: "grok_bot" | "openai_dot", change?: (input: { schema: string; binding: { runId: string; companyId: string; agentId: string }; workspace: { access: string } }) => void) {
|
||||
const run = await readRun(f.runId);
|
||||
const input = { schema: "paperclip.native-execution-input.v6",
|
||||
binding: { runId: f.runId, companyId: f.companyId, agentId: f.agentId, issueId: f.issueId },
|
||||
provider: { kind: provider }, workspace: { access: "none", cwd: null, repoUrl: null, repoRef: null, branchName: null } };
|
||||
change?.(input);
|
||||
const [updated] = await db.update(heartbeatRuns).set({ driverKind: provider === "grok_bot" ? "grok_bot_mcp" : "openai_dot_mcp",
|
||||
runnerProfileJson: { ...run.runnerProfileJson, nativeExecutionInput: input } }).where(eq(heartbeatRuns.id, f.runId)).returning();
|
||||
return updated;
|
||||
}
|
||||
|
||||
it.each(["grok_bot", "openai_dot"] as const)("does not invent file recovery for %s admitted without workspace access", async provider => {
|
||||
const f = await seed(), run = await bindExternalInput(f, provider);
|
||||
await preserveNativeFailedWorkspace(db, run);
|
||||
expect((await readRun(f.runId)).resultJson).not.toHaveProperty("workspaceRestoreFailure");
|
||||
expect(await readLease(f.lease.id)).toMatchObject({ status: "active", leasePolicy: "ephemeral" });
|
||||
expect(await actions(f.issueId)).toHaveLength(0);
|
||||
});
|
||||
|
||||
it.each(["run", "company", "agent", "schema", "write_access"])("preserves file safeguards when no-workspace evidence has an invalid %s", async mismatch => {
|
||||
const f = await seed();
|
||||
const run = await bindExternalInput(f, "grok_bot", input => {
|
||||
if (mismatch === "run") input.binding.runId = randomUUID();
|
||||
if (mismatch === "company") input.binding.companyId = randomUUID();
|
||||
if (mismatch === "agent") input.binding.agentId = randomUUID();
|
||||
if (mismatch === "schema") input.schema = "unknown";
|
||||
if (mismatch === "write_access") input.workspace.access = "read_write";
|
||||
});
|
||||
await preserveNativeFailedWorkspace(db, run);
|
||||
expect((await readRun(f.runId)).resultJson).toHaveProperty("workspaceRestoreFailure", "restore_failed");
|
||||
expect(await readLease(f.lease.id)).toMatchObject({ leasePolicy: "retain_on_failure" });
|
||||
});
|
||||
|
||||
it.each(["grok_bot", "openai_dot"] as const)("keeps historical %s inspection holds with an available recovery path", async provider => {
|
||||
const f = await seed();
|
||||
// Reproduce a hold persisted by an older cancellation path, then attach its admitted contract.
|
||||
await preserveNativeFailedWorkspace(db, f.run);
|
||||
const lease = await readLease(f.lease.id);
|
||||
await settleStopOnlyCleanup(db, lease, { attemptId: String(lease.metadata?.pendingCleanupAttemptId),
|
||||
receipt: { providerLeaseId: lease.providerLeaseId, state: "stopped" } });
|
||||
await bindExternalInput(f, provider);
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.runId));
|
||||
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
|
||||
expect(blocker).toMatchObject({ runId: f.runId, canRetry: false, canContinue: false,
|
||||
workspaceRepairRequired: provider !== "grok_bot", canRecordWorkspaceRepair: provider !== "grok_bot" });
|
||||
if (provider === "grok_bot") {
|
||||
expect(blocker?.nextAction).not.toContain("files");
|
||||
expect(blocker?.nextAction).toContain("Runtime");
|
||||
}
|
||||
expect(await actions(f.issueId)).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("records verified file repair without retrying the native run or changing its task", async () => {
|
||||
const f = await seed();
|
||||
await db.update(issues).set({ status: "done", assigneeAgentId: null }).where(eq(issues.id, f.issueId));
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { and, eq, gt, isNull, sql } from "drizzle-orm";
|
||||
import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import { recordRequiredWorkspaceRestoreFailure } from "../legacy-execution-recovery.js";
|
||||
import { hasRequiredWorkspaceRecovery } from "../workspace-restore-recovery-state.js";
|
||||
import { hasRequiredWorkspaceRecovery, nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js";
|
||||
import { logActivity } from "../activity-log.js";
|
||||
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
|
||||
import type { NativeRestartRecoveryClaim } from "./native-restart-recovery.js";
|
||||
@@ -45,9 +45,12 @@ export async function preserveNativeFailedWorkspace(db: Db, selected: typeof hea
|
||||
)).limit(1);
|
||||
if (!source || source.issueId !== issueId || source.metadata?.remoteCwd !== currentReference.remoteCwd
|
||||
|| (source.executionWorkspaceId && source.executionWorkspaceId !== currentReference.workspaceId)) return;
|
||||
// Ownership settlement still applies to workspace-less external sessions.
|
||||
// Their Runner descriptor is not evidence that user files need exporting.
|
||||
const retainFiles = !nativeRunHasNoWorkspaceAccess(run);
|
||||
const alreadyRetained = hasRequiredWorkspaceRecovery(run.resultJson);
|
||||
let retained = alreadyRetained ? run : await recordRequiredWorkspaceRestoreFailure(tx as unknown as Db, run, { workspaceRestoreFailure: "restore_failed" }, source);
|
||||
if (!retained || !hasRequiredWorkspaceRecovery(retained.resultJson)) return;
|
||||
let retained = !retainFiles || alreadyRetained ? run : await recordRequiredWorkspaceRestoreFailure(tx as unknown as Db, run, { workspaceRestoreFailure: "restore_failed" }, source);
|
||||
if (!retained || retainFiles && !hasRequiredWorkspaceRecovery(retained.resultJson)) return;
|
||||
if (options.allowPreterminal && coordinator.phase === "observed") {
|
||||
const [settled] = await tx.update(nativeRunFinalizations).set({ phase: "terminal_failure",
|
||||
leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null, nextAttemptAt: null,
|
||||
@@ -62,7 +65,7 @@ export async function preserveNativeFailedWorkspace(db: Db, selected: typeof hea
|
||||
[retained] = await tx.update(heartbeatRuns).set({ nativePhase: "terminal_failure", nativePhaseUpdatedAt: new Date() })
|
||||
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId))).returning();
|
||||
}
|
||||
if (!alreadyRetained) await logActivity(tx as unknown as Db, {
|
||||
if (retainFiles && !alreadyRetained) await logActivity(tx as unknown as Db, {
|
||||
companyId: run.companyId, actorType: "system", actorId: "native-workspace-recovery",
|
||||
action: "issue.workspace_recovery_required", entityType: "issue",
|
||||
entityId: issueId,
|
||||
|
||||
@@ -1,8 +1,26 @@
|
||||
import type { heartbeatRuns } from "@paperclipai/db";
|
||||
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
|
||||
|
||||
export const LEGACY_WORKSPACE_RECOVERY_KEY = "workspaceRestoreRecovery";
|
||||
export const LEGACY_WORKSPACE_RECOVERY_SCHEMA = "paperclip.workspace-restore-recovery.v1";
|
||||
|
||||
/** The admitted external contract can prohibit workspace access even though its
|
||||
* managed Runner has a synchronization descriptor for durable recovery. That
|
||||
* descriptor alone does not create a user-file export obligation. */
|
||||
export function nativeRunHasNoWorkspaceAccess(run: Pick<typeof heartbeatRuns.$inferSelect,
|
||||
"id" | "companyId" | "agentId" | "issueId" | "nativeIssueId" | "runtimeMode" | "driverKind" | "runnerProfileJson">): boolean {
|
||||
const input = run.runnerProfileJson?.nativeExecutionInput as Record<string, unknown> | undefined;
|
||||
if (run.runtimeMode !== "native" || input?.schema !== "paperclip.native-execution-input.v6") return false;
|
||||
const binding = input.binding as Record<string, unknown> | undefined;
|
||||
const workspace = input.workspace as Record<string, unknown> | undefined;
|
||||
const provider = input.provider as Record<string, unknown> | undefined;
|
||||
const driver = provider?.kind === "grok_bot" ? "grok_bot_mcp"
|
||||
: provider?.kind === "openai_dot" ? "openai_dot_mcp" : null;
|
||||
return driver !== null && run.driverKind === driver && workspace?.access === "none" && workspace.cwd === null
|
||||
&& binding?.runId === run.id && binding.companyId === run.companyId && binding.agentId === run.agentId
|
||||
&& binding.issueId === (run.nativeIssueId ?? run.issueId);
|
||||
}
|
||||
|
||||
export function hasRequiredWorkspaceRecovery(result: Record<string, unknown> | null | undefined): boolean {
|
||||
const value = result?.[LEGACY_WORKSPACE_RECOVERY_KEY];
|
||||
return hasWorkspaceRestoreFailure(result) && !!value && typeof value === "object" && !Array.isArray(value)
|
||||
|
||||
Reference in new issue
Block a user