mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 20:46:28 +01:00
fix: carry AI connections through hiring and unblock task execution (#13438)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents hire other agents and assign tasks to them. > - Managed AI connections must follow those hires across legacy and native runners. > - Missing accounts should pause task execution and let the user connect from the task. > - Subscription contention must wait without asking for new credentials. > - This pull request fixes these paths and the native tool and Daytona staging failures found during live tests. > - The result is a working hire, subtask, and connection setup flow on local and remote runners. ## Linked Issues or Issue Description **What happened?** A managed Claude or Codex agent could hire a teammate without a usable AI binding. Cross-provider hiring could fail before the user had a chance to connect the new provider. First-time task setup did not show the existing AI credential form inline. A busy subscription could request a new connection. Native API replies could stop the parent after a hire had already committed. Fresh Daytona sandboxes could fail to extract read-only skill directories created on macOS. **Expected behavior** Compatible hires inherit the managed connection choice. A hire for another provider uses the responsible user's default. If that account is missing, the hire succeeds and the task asks for a connection. Completing setup in the task resumes work automatically. Explicit child auth settings and existing unmanaged login paths keep precedence. Shared-account access checks remain in force. **Steps to reproduce** 1. Connect a Claude or Codex parent with a managed AI account. 2. Ask it to hire one agent of each provider and create a self-assigned subtask. 3. Assign work to both hires without connecting the second provider first. 4. Connect the missing provider from its task card. 5. Check that all tasks finish and same-provider work uses the original account. 6. Repeat with native runners and fresh Daytona sandboxes. The opt-in browser suite in `tests/hiring-ai-connections/README.md` performs these steps. **Paperclip version or commit** The live failures were reproduced from `f2c5e54dc`. The branch is rebased onto `5282cabde`. **Deployment mode** Isolated local development instance. Legacy CLI and native runners. Local execution and ephemeral Daytona sandboxes. Related work: Refs #13247 for managed AI connections. Refs #13268 for legacy credential-reference inheritance, which this branch preserves. Refs #13432 for a concurrent managed-inheritance fix. This PR also covers cross-provider task setup, subscription waits, native API replies, and Daytona extraction. It permits missing responsible-user defaults at hire time; restricted shared selections still fail. ## What Changed - Apply managed connection defaults to both agent creation routes. Preserve explicit auth choices and legacy credential-reference inheritance. - Allow hires before their responsible user connects the provider. Keep approval gates, company boundaries, and shared-account access checks. - Reuse the production AI credential form inside the pending task card. Resume the task after setup. - Retry subscription lease contention without consuming the provider-failure allowance or creating a connection request. - Require task execution-lock ownership when scheduling, promoting, and dispatching subscription retries. Recheck ownership under the issue row lock. - Rename the HTTP operation identity at the native tool boundary so it cannot override the runner's operation identity. - Delay directory permission restoration during Daytona extraction. Preserve the final read-only modes. - Add database-backed regressions, real browser acceptance tests, and Storybook states. Document setup and run-log behavior. ## Verification - Six real browser scenarios passed: both parent providers on legacy local and legacy Daytona; native Codex locally; native Claude on Daytona. Each scenario hires both providers, completes a self-subtask and assigned work, and connects the missing provider inline with automatic continuation. - Successful runs verify the account, responsible user, runner mode, and Daytona lease. All 18 test sandboxes were deleted. - Live authentication used API keys. Subscription inheritance, lease contention, and retry have integration coverage. Fresh subscription OAuth sign-in was not automated. - Red/green tests reproduced missing bindings, missing inline forms, subscription contention, native API reply failure, and GNU tar permission failure. - Seven Storybook browser checks passed. They cover both providers, method selection, narrow layout, completion, cancellation, and invalid credentials. - Full local suite coverage completed before rebase. Initial timing and fixture startup failures passed unchanged on isolated reruns. The first full command did not exit cleanly; the remaining workspace and serialized groups were completed separately. - After rebase, 107 hiring/auth/retry tests and 59 native API, task-card, and Daytona tests passed. The full workspace typecheck, production build, and token gates passed again. Storybook build passed before rebase. - Review fixes: 169 hiring/retry/dispatch tests, 37 adjacent tests, and four explicit cancellation-race cases passed. Eight cross-provider cases cover stale auth keys on both creation routes and both runner types. Server typecheck and build passed. - Final CI on `ee4890837a8a4913e07453392b9a75969580dae1`: 32 checks passed. Two optional Storybook jobs were skipped. The full server, workspace, browser, native runner, build, typecheck, and release checks passed. - Three unchanged tests initially failed on a busy port, a chat row-lock race, and preview-server readiness. Each affected job passed after one CI rerun. Isolated local checks also passed: 41 credential tests, the chat-concurrency case, and 25 preview-runtime tests. - Greptile reviewed the final commit at 5/5. Both review threads are resolved. GitHub reports no merge conflicts. ## Risks - A missing personal account now defers authentication to the first task. Explicit incompatible bindings and restricted shared accounts still fail at hire time. - An inherited personal default uses the responsible user's existing authorization to install access for the new agent. It never copies credentials or another user's identity. - Subscription contention retries after a delay and rechecks task eligibility. It does not consume the provider-failure budget. - Native hiring uses the existing managed API-tools opt-in. Remote native runners require a matching Linux binary and provider pack, as documented in the acceptance README. - No schema changes. Live tests make paid provider calls and remain opt-in. ## Model Used OpenAI GPT-6 (`gpt-6-astra`) in Codex, with reasoning, repository inspection, code execution, browser automation, and API tools. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
5282cabde8
commit
f912ecaacf
26 files changed
+1001
-26
No files matched your search
@@ -0,0 +1,242 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, companyMemberships, createDb, heartbeatRuns, issues, issueThreadInteractions, principalPermissionGrants, toolConnectionInstalls } from "@paperclipai/db";
|
||||
import { type AiConnectionBinding } from "@paperclipai/shared";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { agentRoutes } from "../routes/agents.js";
|
||||
import { errorHandler } from "../middleware/index.js";
|
||||
import { aiConnectionService } from "../services/ai-connections.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { getServerAdapter, registerServerAdapter, unregisterServerAdapter } from "../adapters/index.js";
|
||||
import { prepareManagedAiRuntime } from "../services/ai-connection-runtime.js";
|
||||
import { secretService } from "../services/secrets.js";
|
||||
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let home: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
home = await mkdtemp(path.join(os.tmpdir(), "paperclip-hire-ai-"));
|
||||
vi.stubEnv("PAPERCLIP_HOME", home);
|
||||
vi.stubEnv("PAPERCLIP_INSTANCE_ID", "hire-ai");
|
||||
database = await startEmbeddedPostgresTestDatabase("paperclip-hire-ai-db-");
|
||||
db = createDb(database.connectionString);
|
||||
}, 90_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await database?.cleanup();
|
||||
vi.unstubAllEnvs();
|
||||
if (home) await rm(home, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function fixture(provider: "anthropic" | "openai", method: "api_key" | "subscription" = "api_key") {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
const userId = `owner-${companyId}`;
|
||||
const binding = { provider, method, mode: "responsible_user" } as const;
|
||||
const adapterType = provider === "anthropic" ? "claude_local" : "codex_local";
|
||||
await db.insert(companies).values({ id: companyId, name: "Hiring connection test", issuePrefix: `H${companyId.slice(0, 7)}`, defaultResponsibleUserId: userId });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "owner", status: "active" });
|
||||
await db.insert(principalPermissionGrants).values({ companyId, principalType: "user", principalId: userId, permissionKey: "agents:create" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Manager", role: "ceo", adapterType, runtimeConfig: { aiConnection: binding } });
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running", responsibleUserId: userId }).returning();
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
req.actor = { type: "agent", agentId, companyId, runId: run!.id, source: "agent_jwt", onBehalfOfUserId: userId, onBehalfOfMemberships: [{ companyId, membershipRole: "owner", status: "active" }] };
|
||||
next();
|
||||
});
|
||||
app.use("/api", agentRoutes(db));
|
||||
app.use(errorHandler);
|
||||
const credential = method === "api_key" ? "fixture-api-key" : provider === "anthropic" ? "fixture-subscription-token" : JSON.stringify({ tokens: { access_token: "fixture-access", refresh_token: "fixture-refresh", id_token: "fixture-id", account_id: "fixture-account" } });
|
||||
const account = await aiConnectionService(db).save(companyId, userId, {
|
||||
provider, method, name: "Manager's connection", ownership: "personal", agentIds: [agentId], allAgents: false,
|
||||
...(method === "api_key" ? { apiKey: credential } : { loginSessionId: "fixture" }),
|
||||
}, credential);
|
||||
return { app, companyId, agentId, userId, binding, adapterType, account, runId: run!.id };
|
||||
}
|
||||
|
||||
function hired(response: request.Response) {
|
||||
expect(response.status, JSON.stringify(response.body)).toBe(201);
|
||||
return response.body.agent ?? response.body;
|
||||
}
|
||||
|
||||
describe("agent-created hires use managed AI connections", () => {
|
||||
for (const endpoint of ["agent-hires", "agents"]) {
|
||||
it.each([
|
||||
["anthropic", "api_key"], ["anthropic", "subscription"],
|
||||
["openai", "api_key"], ["openai", "subscription"],
|
||||
] as const)(`${endpoint}: %s hires its own provider using the same %s binding`, async (provider, method) => {
|
||||
const f = await fixture(provider, method);
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/${endpoint}`).send({ name: "Teammate", role: "engineer", adapterType: f.adapterType, reportsTo: f.agentId }));
|
||||
expect(agent.runtimeConfig.aiConnection).toEqual(f.binding);
|
||||
const runtime = await prepareManagedAiRuntime(db, { companyId: f.companyId, agentId: agent.id, responsibleUserId: f.userId, adapterType: agent.adapterType, binding: agent.runtimeConfig.aiConnection, config: agent.adapterConfig });
|
||||
try {
|
||||
expect(runtime.attribution).toMatchObject({ connectionId: f.account.connectionId, grantId: f.account.grantId, method, responsibleUserId: f.userId });
|
||||
} finally { await runtime.cleanup(); }
|
||||
});
|
||||
}
|
||||
|
||||
for (const endpoint of ["agent-hires", "agents"]) {
|
||||
it.each([
|
||||
["ANTHROPIC_API_KEY", "child-key"],
|
||||
["ANTHROPIC_API_KEY", ""],
|
||||
["CLAUDE_CONFIG_DIR", "/tmp/child-claude-home"],
|
||||
["ANTHROPIC_BASE_URL", "https://example.invalid"],
|
||||
])(`${endpoint}: preserves an explicit child auth setting %s=%s`, async (key, value) => {
|
||||
const f = await fixture("anthropic");
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/${endpoint}`).send({
|
||||
name: "Explicit auth", role: "engineer", adapterType: f.adapterType,
|
||||
adapterConfig: { env: { [key]: value } },
|
||||
}));
|
||||
expect(agent.runtimeConfig.aiConnection).toBeUndefined();
|
||||
const [saved] = await db.select().from(agents).where(eq(agents.id, agent.id));
|
||||
expect((saved.adapterConfig.env as Record<string, unknown>)[key]).toEqual({ type: "plain", value });
|
||||
});
|
||||
}
|
||||
|
||||
it.each([true, false])("managed bindings do not inherit legacy credentials (managed parent: %s)", async (managedParent) => {
|
||||
const f = await fixture("openai");
|
||||
const secret = await secretService(db).create(f.companyId, {
|
||||
name: "Legacy parent key", provider: "local_encrypted", value: "fixture-legacy-key",
|
||||
});
|
||||
await db.update(agents).set({
|
||||
runtimeConfig: managedParent ? { aiConnection: f.binding } : {},
|
||||
adapterConfig: { env: { OPENAI_API_KEY: { type: "secret_ref", secretId: secret.id } } },
|
||||
}).where(eq(agents.id, f.agentId));
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({
|
||||
name: "Managed child", role: "engineer", adapterType: f.adapterType,
|
||||
...(managedParent ? {} : { runtimeConfig: { aiConnection: f.binding } }),
|
||||
}));
|
||||
expect(agent.runtimeConfig.aiConnection).toEqual(f.binding);
|
||||
expect(agent.adapterConfig.env?.OPENAI_API_KEY).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps unmanaged parent hires on their existing authentication path", async () => {
|
||||
const f = await fixture("openai");
|
||||
await db.update(agents).set({ runtimeConfig: {} }).where(eq(agents.id, f.agentId));
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({
|
||||
name: "Legacy authentication", role: "engineer", adapterType: f.adapterType,
|
||||
}));
|
||||
expect(agent.runtimeConfig.aiConnection).toBeUndefined();
|
||||
});
|
||||
|
||||
it.each(["anthropic", "openai"] as const)("%s can hire the other provider before that user connects it", async (provider) => {
|
||||
const f = await fixture(provider);
|
||||
const otherProvider = provider === "anthropic" ? "openai" : "anthropic";
|
||||
const adapterType = otherProvider === "anthropic" ? "claude_local" : "codex_local";
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Other provider", role: "engineer", adapterType }));
|
||||
expect(agent.runtimeConfig.aiConnection).toMatchObject({ provider: otherProvider, mode: "responsible_user" });
|
||||
await expect(prepareManagedAiRuntime(db, { companyId: f.companyId, agentId: agent.id, responsibleUserId: f.userId, adapterType, binding: agent.runtimeConfig.aiConnection, config: agent.adapterConfig })).rejects.toMatchObject({ details: { code: "ai_connection_default_missing" } });
|
||||
expect(agent.status).toBe("idle");
|
||||
});
|
||||
|
||||
for (const endpoint of ["agent-hires", "agents"]) {
|
||||
it.each([
|
||||
["anthropic", "codex_local", {}, "ANTHROPIC_API_KEY"],
|
||||
["openai", "claude_local", {}, "OPENAI_API_KEY"],
|
||||
["anthropic", "paperclip_runner", { provider: "codex" }, "ANTHROPIC_API_KEY"],
|
||||
["openai", "paperclip_runner", { provider: "acpx", acpxAgent: "claude" }, "OPENAI_API_KEY"],
|
||||
] as const)(`${endpoint}: ignores the %s auth key for a different provider in %s`, async (provider, adapterType, config, key) => {
|
||||
const f = await fixture(provider);
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/${endpoint}`).send({
|
||||
name: "Cross-provider config", role: "engineer", adapterType,
|
||||
adapterConfig: { ...config, env: { [key]: "leftover-parent-setting" } },
|
||||
}));
|
||||
expect(agent.runtimeConfig.aiConnection).toMatchObject({
|
||||
provider: provider === "anthropic" ? "openai" : "anthropic", mode: "responsible_user",
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
it("accepts an explicit personal default before authentication, including approval-gated hires", async () => {
|
||||
const f = await fixture("anthropic");
|
||||
await db.update(companies).set({ requireBoardApprovalForNewAgents: true }).where(eq(companies.id, f.companyId));
|
||||
const binding: AiConnectionBinding = { provider: "openai", method: "subscription", mode: "responsible_user" };
|
||||
const response = await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Future Codex", role: "engineer", adapterType: "codex_local", runtimeConfig: { aiConnection: binding } });
|
||||
const agent = hired(response);
|
||||
expect(agent.runtimeConfig.aiConnection).toEqual(binding);
|
||||
expect(agent.status).toBe("pending_approval");
|
||||
expect(response.body.approval.payload.runtimeConfig.aiConnection).toEqual(binding);
|
||||
expect(await db.select().from(toolConnectionInstalls).where(eq(toolConnectionInstalls.targetId, agent.id))).toEqual([]);
|
||||
});
|
||||
|
||||
it.each(["anthropic", "openai"] as const)("inherits %s when the hire uses the native runner", async (provider) => {
|
||||
const f = await fixture(provider, "subscription");
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Native teammate", role: "engineer", adapterType: "paperclip_runner", adapterConfig: provider === "anthropic" ? { provider: "acpx", acpxAgent: "claude" } : { provider: "codex" } }));
|
||||
expect(agent.runtimeConfig.aiConnection).toEqual(f.binding);
|
||||
const runtime = await prepareManagedAiRuntime(db, { companyId: f.companyId, agentId: agent.id, responsibleUserId: f.userId, adapterType: agent.adapterType, binding: agent.runtimeConfig.aiConnection, config: agent.adapterConfig });
|
||||
try { expect(runtime.attribution.connectionId).toBe(f.account.connectionId); } finally { await runtime.cleanup(); }
|
||||
});
|
||||
|
||||
it.each([true, false])("preserves shared connection access boundaries (company access: %s)", async (allAgents) => {
|
||||
const f = await fixture("anthropic");
|
||||
const account = await aiConnectionService(db).save(f.companyId, f.userId, { provider: "anthropic", method: "api_key", name: "Shared Claude", ownership: "shared", apiKey: "fixture", agentIds: [f.agentId], allAgents }, "fixture");
|
||||
const binding = { provider: "anthropic", method: "api_key", mode: "shared", ...account };
|
||||
await db.update(agents).set({ runtimeConfig: { aiConnection: binding } }).where(eq(agents.id, f.agentId));
|
||||
const response = await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Shared teammate", role: "engineer", adapterType: f.adapterType });
|
||||
if (!allAgents) {
|
||||
expect(response.status).toBe(403);
|
||||
expect(await db.select().from(agents).where(eq(agents.companyId, f.companyId))).toHaveLength(1);
|
||||
} else {
|
||||
const agent = hired(response);
|
||||
expect(agent.runtimeConfig.aiConnection).toEqual(binding);
|
||||
const runtime = await prepareManagedAiRuntime(db, { companyId: f.companyId, agentId: agent.id, responsibleUserId: f.userId, adapterType: agent.adapterType, binding: agent.runtimeConfig.aiConnection, config: agent.adapterConfig });
|
||||
try { expect(runtime.attribution.connectionId).toBe(account.connectionId); } finally { await runtime.cleanup(); }
|
||||
}
|
||||
});
|
||||
|
||||
it("still rejects an explicitly incompatible provider without creating a hire", async () => {
|
||||
const f = await fixture("anthropic");
|
||||
const response = await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Wrong provider", role: "engineer", adapterType: "codex_local", runtimeConfig: { aiConnection: f.binding } });
|
||||
expect(response.status).toBe(422);
|
||||
expect(response.body.details.code).toBe("ai_connection_incompatible");
|
||||
expect(await db.select().from(agents).where(eq(agents.companyId, f.companyId))).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("hired agents sharing a subscription", () => {
|
||||
it.each(["anthropic", "openai"] as const)("waits for the %s parent and resumes without asking for a new connection", async (provider) => {
|
||||
const f = await fixture(provider, "subscription");
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Waiting teammate", role: "engineer", adapterType: f.adapterType, reportsTo: f.agentId, adapterConfig: { cwd: home, engine: "cli" }, runtimeConfig: { heartbeat: { enabled: false } } }));
|
||||
const [issue] = await db.insert(issues).values({ companyId: f.companyId, title: "Subscription child task", status: "todo", assigneeAgentId: agent.id, responsibleUserId: f.userId, createdByUserId: f.userId }).returning();
|
||||
const parentRuntime = await prepareManagedAiRuntime(db, { companyId: f.companyId, agentId: f.agentId, responsibleUserId: f.userId, adapterType: f.adapterType, binding: f.binding, config: { cwd: home } });
|
||||
const execute = vi.fn(async () => {
|
||||
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, issue.id));
|
||||
return { exitCode: 0, signal: null, timedOut: false, resultJson: {} };
|
||||
});
|
||||
registerServerAdapter({ ...getServerAdapter(f.adapterType), execute });
|
||||
const heartbeat = heartbeatService(db);
|
||||
let parentReleased = false;
|
||||
try {
|
||||
const run = await heartbeat.invoke(agent.id, "assignment", { issueId: issue.id, wakeReason: "issue_assigned", responsibleUserId: f.userId }, "system");
|
||||
expect(run).not.toBeNull();
|
||||
await expect.poll(async () => (await heartbeat.getRun(run!.id))?.status, { timeout: 20_000 }).not.toBe("running");
|
||||
expect(await heartbeat.getRun(run!.id)).toMatchObject({ status: "cancelled", errorCode: "ai_connection_busy" });
|
||||
let retry: typeof heartbeatRuns.$inferSelect | undefined;
|
||||
await expect.poll(async () => { [retry] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run!.id)); return retry?.status; }, { timeout: 10_000 }).toBe("scheduled_retry");
|
||||
expect(execute).not.toHaveBeenCalled();
|
||||
expect(await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, issue.id))).toHaveLength(0);
|
||||
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0].executionRunId).toBe(retry!.id);
|
||||
parentReleased = true;
|
||||
await parentRuntime.cleanup();
|
||||
const promoted = await heartbeat.promoteDueScheduledRetries(new Date(retry!.scheduledRetryAt!.getTime() + 1000));
|
||||
expect(promoted.runIds).toContain(retry!.id);
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
await expect.poll(async () => (await heartbeat.getRun(retry!.id))?.status, { timeout: 20_000 }).toBe("succeeded");
|
||||
expect(execute).toHaveBeenCalledTimes(1);
|
||||
expect((await heartbeat.getRun(retry!.id))?.contextSnapshot?.aiConnection).toMatchObject({ connectionId: f.account.connectionId, responsibleUserId: f.userId, method: "subscription" });
|
||||
} finally {
|
||||
if (!parentReleased) await parentRuntime.cleanup();
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
unregisterServerAdapter(f.adapterType);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -1374,6 +1374,55 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
expect(issue?.executionRunId).toBe(retryRuns[0]?.id);
|
||||
});
|
||||
|
||||
it.each(["schedule", "transaction", "promote", "dispatch"] as const)(
|
||||
"suppresses a busy-subscription retry whose task lock was cleared before %s",
|
||||
async (phase) => {
|
||||
const { companyId, issueId, runId, now } = await seedMaxTurnFixture();
|
||||
await db.update(heartbeatRuns).set({
|
||||
status: "cancelled", errorCode: "ai_connection_busy",
|
||||
resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false } },
|
||||
}).where(eq(heartbeatRuns.id, runId));
|
||||
const clearLock = () => db.update(issues).set({ executionRunId: null }).where(eq(issues.id, issueId));
|
||||
if (phase === "schedule") await clearLock();
|
||||
const transaction = db.transaction.bind(db);
|
||||
let raceApplied = false;
|
||||
const transactionSpy = phase === "transaction"
|
||||
? vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => {
|
||||
await clearLock();
|
||||
raceApplied = true;
|
||||
return transaction(callback, config);
|
||||
})
|
||||
: null;
|
||||
const scheduled = await heartbeat.scheduleBoundedRetry(runId, {
|
||||
now, retryReason: "ai_connection_busy", wakeReason: "ai_connection_busy_retry",
|
||||
maxAttempts: 1, delayMs: 1_000,
|
||||
}).finally(() => transactionSpy?.mockRestore());
|
||||
if (phase === "transaction") expect(raceApplied).toBe(true);
|
||||
if (phase === "schedule" || phase === "transaction") {
|
||||
expect(scheduled).toMatchObject({ outcome: "not_scheduled", errorCode: "issue_execution_lock_changed" });
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
|
||||
return;
|
||||
}
|
||||
expect(scheduled.outcome).toBe("scheduled");
|
||||
if (scheduled.outcome !== "scheduled") return;
|
||||
if (phase === "promote") await clearLock();
|
||||
const promotion = await heartbeat.promoteDueScheduledRetries(scheduled.dueAt);
|
||||
if (phase === "promote") {
|
||||
expect(promotion).toEqual({ promoted: 0, runIds: [] });
|
||||
} else {
|
||||
expect(promotion.runIds).toContain(scheduled.run.id);
|
||||
await clearLock();
|
||||
const adapter = createPostgresRunDispatchAdapter(db);
|
||||
expect(await adapter.cancelStaleQueuedRun({
|
||||
companyId, runId: scheduled.run.id, expectedStatus: "queued", now: scheduled.dueAt,
|
||||
})).toMatchObject({ outcome: "cancelled", errorCode: "issue_execution_lock_changed" });
|
||||
}
|
||||
expect(await heartbeat.getRun(scheduled.run.id)).toMatchObject({
|
||||
status: "cancelled", errorCode: "issue_execution_lock_changed",
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("does not promote a duplicate max-turn continuation that does not own the issue lock", async () => {
|
||||
const { companyId, agentId, issueId, runId, now } = await seedMaxTurnFixture();
|
||||
|
||||
|
||||
Reference in new issue
Block a user