mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 21:07:40 +01:00
fix: carry AI connections through hiring and unblock task execution (#13438)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents hire other agents and assign tasks to them. > - Managed AI connections must follow those hires across legacy and native runners. > - Missing accounts should pause task execution and let the user connect from the task. > - Subscription contention must wait without asking for new credentials. > - This pull request fixes these paths and the native tool and Daytona staging failures found during live tests. > - The result is a working hire, subtask, and connection setup flow on local and remote runners. ## Linked Issues or Issue Description **What happened?** A managed Claude or Codex agent could hire a teammate without a usable AI binding. Cross-provider hiring could fail before the user had a chance to connect the new provider. First-time task setup did not show the existing AI credential form inline. A busy subscription could request a new connection. Native API replies could stop the parent after a hire had already committed. Fresh Daytona sandboxes could fail to extract read-only skill directories created on macOS. **Expected behavior** Compatible hires inherit the managed connection choice. A hire for another provider uses the responsible user's default. If that account is missing, the hire succeeds and the task asks for a connection. Completing setup in the task resumes work automatically. Explicit child auth settings and existing unmanaged login paths keep precedence. Shared-account access checks remain in force. **Steps to reproduce** 1. Connect a Claude or Codex parent with a managed AI account. 2. Ask it to hire one agent of each provider and create a self-assigned subtask. 3. Assign work to both hires without connecting the second provider first. 4. Connect the missing provider from its task card. 5. Check that all tasks finish and same-provider work uses the original account. 6. Repeat with native runners and fresh Daytona sandboxes. The opt-in browser suite in `tests/hiring-ai-connections/README.md` performs these steps. **Paperclip version or commit** The live failures were reproduced from `f2c5e54dc`. The branch is rebased onto `5282cabde`. **Deployment mode** Isolated local development instance. Legacy CLI and native runners. Local execution and ephemeral Daytona sandboxes. Related work: Refs #13247 for managed AI connections. Refs #13268 for legacy credential-reference inheritance, which this branch preserves. Refs #13432 for a concurrent managed-inheritance fix. This PR also covers cross-provider task setup, subscription waits, native API replies, and Daytona extraction. It permits missing responsible-user defaults at hire time; restricted shared selections still fail. ## What Changed - Apply managed connection defaults to both agent creation routes. Preserve explicit auth choices and legacy credential-reference inheritance. - Allow hires before their responsible user connects the provider. Keep approval gates, company boundaries, and shared-account access checks. - Reuse the production AI credential form inside the pending task card. Resume the task after setup. - Retry subscription lease contention without consuming the provider-failure allowance or creating a connection request. - Require task execution-lock ownership when scheduling, promoting, and dispatching subscription retries. Recheck ownership under the issue row lock. - Rename the HTTP operation identity at the native tool boundary so it cannot override the runner's operation identity. - Delay directory permission restoration during Daytona extraction. Preserve the final read-only modes. - Add database-backed regressions, real browser acceptance tests, and Storybook states. Document setup and run-log behavior. ## Verification - Six real browser scenarios passed: both parent providers on legacy local and legacy Daytona; native Codex locally; native Claude on Daytona. Each scenario hires both providers, completes a self-subtask and assigned work, and connects the missing provider inline with automatic continuation. - Successful runs verify the account, responsible user, runner mode, and Daytona lease. All 18 test sandboxes were deleted. - Live authentication used API keys. Subscription inheritance, lease contention, and retry have integration coverage. Fresh subscription OAuth sign-in was not automated. - Red/green tests reproduced missing bindings, missing inline forms, subscription contention, native API reply failure, and GNU tar permission failure. - Seven Storybook browser checks passed. They cover both providers, method selection, narrow layout, completion, cancellation, and invalid credentials. - Full local suite coverage completed before rebase. Initial timing and fixture startup failures passed unchanged on isolated reruns. The first full command did not exit cleanly; the remaining workspace and serialized groups were completed separately. - After rebase, 107 hiring/auth/retry tests and 59 native API, task-card, and Daytona tests passed. The full workspace typecheck, production build, and token gates passed again. Storybook build passed before rebase. - Review fixes: 169 hiring/retry/dispatch tests, 37 adjacent tests, and four explicit cancellation-race cases passed. Eight cross-provider cases cover stale auth keys on both creation routes and both runner types. Server typecheck and build passed. - Final CI on `ee4890837a8a4913e07453392b9a75969580dae1`: 32 checks passed. Two optional Storybook jobs were skipped. The full server, workspace, browser, native runner, build, typecheck, and release checks passed. - Three unchanged tests initially failed on a busy port, a chat row-lock race, and preview-server readiness. Each affected job passed after one CI rerun. Isolated local checks also passed: 41 credential tests, the chat-concurrency case, and 25 preview-runtime tests. - Greptile reviewed the final commit at 5/5. Both review threads are resolved. GitHub reports no merge conflicts. ## Risks - A missing personal account now defers authentication to the first task. Explicit incompatible bindings and restricted shared accounts still fail at hire time. - An inherited personal default uses the responsible user's existing authorization to install access for the new agent. It never copies credentials or another user's identity. - Subscription contention retries after a delay and rechecks task eligibility. It does not consume the provider-failure budget. - Native hiring uses the existing managed API-tools opt-in. Remote native runners require a matching Linux binary and provider pack, as documented in the acceptance README. - No schema changes. Live tests make paid provider calls and remain opt-in. ## Model Used OpenAI GPT-6 (`gpt-6-astra`) in Codex, with reasoning, repository inspection, code execution, browser automation, and API tools. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
5282cabde8
commit
f912ecaacf
26 files changed
+1001
-26
No files matched your search
@@ -0,0 +1,242 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, companyMemberships, createDb, heartbeatRuns, issues, issueThreadInteractions, principalPermissionGrants, toolConnectionInstalls } from "@paperclipai/db";
|
||||
import { type AiConnectionBinding } from "@paperclipai/shared";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { agentRoutes } from "../routes/agents.js";
|
||||
import { errorHandler } from "../middleware/index.js";
|
||||
import { aiConnectionService } from "../services/ai-connections.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { getServerAdapter, registerServerAdapter, unregisterServerAdapter } from "../adapters/index.js";
|
||||
import { prepareManagedAiRuntime } from "../services/ai-connection-runtime.js";
|
||||
import { secretService } from "../services/secrets.js";
|
||||
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let home: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
home = await mkdtemp(path.join(os.tmpdir(), "paperclip-hire-ai-"));
|
||||
vi.stubEnv("PAPERCLIP_HOME", home);
|
||||
vi.stubEnv("PAPERCLIP_INSTANCE_ID", "hire-ai");
|
||||
database = await startEmbeddedPostgresTestDatabase("paperclip-hire-ai-db-");
|
||||
db = createDb(database.connectionString);
|
||||
}, 90_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await database?.cleanup();
|
||||
vi.unstubAllEnvs();
|
||||
if (home) await rm(home, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function fixture(provider: "anthropic" | "openai", method: "api_key" | "subscription" = "api_key") {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
const userId = `owner-${companyId}`;
|
||||
const binding = { provider, method, mode: "responsible_user" } as const;
|
||||
const adapterType = provider === "anthropic" ? "claude_local" : "codex_local";
|
||||
await db.insert(companies).values({ id: companyId, name: "Hiring connection test", issuePrefix: `H${companyId.slice(0, 7)}`, defaultResponsibleUserId: userId });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "owner", status: "active" });
|
||||
await db.insert(principalPermissionGrants).values({ companyId, principalType: "user", principalId: userId, permissionKey: "agents:create" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Manager", role: "ceo", adapterType, runtimeConfig: { aiConnection: binding } });
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running", responsibleUserId: userId }).returning();
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
req.actor = { type: "agent", agentId, companyId, runId: run!.id, source: "agent_jwt", onBehalfOfUserId: userId, onBehalfOfMemberships: [{ companyId, membershipRole: "owner", status: "active" }] };
|
||||
next();
|
||||
});
|
||||
app.use("/api", agentRoutes(db));
|
||||
app.use(errorHandler);
|
||||
const credential = method === "api_key" ? "fixture-api-key" : provider === "anthropic" ? "fixture-subscription-token" : JSON.stringify({ tokens: { access_token: "fixture-access", refresh_token: "fixture-refresh", id_token: "fixture-id", account_id: "fixture-account" } });
|
||||
const account = await aiConnectionService(db).save(companyId, userId, {
|
||||
provider, method, name: "Manager's connection", ownership: "personal", agentIds: [agentId], allAgents: false,
|
||||
...(method === "api_key" ? { apiKey: credential } : { loginSessionId: "fixture" }),
|
||||
}, credential);
|
||||
return { app, companyId, agentId, userId, binding, adapterType, account, runId: run!.id };
|
||||
}
|
||||
|
||||
function hired(response: request.Response) {
|
||||
expect(response.status, JSON.stringify(response.body)).toBe(201);
|
||||
return response.body.agent ?? response.body;
|
||||
}
|
||||
|
||||
describe("agent-created hires use managed AI connections", () => {
|
||||
for (const endpoint of ["agent-hires", "agents"]) {
|
||||
it.each([
|
||||
["anthropic", "api_key"], ["anthropic", "subscription"],
|
||||
["openai", "api_key"], ["openai", "subscription"],
|
||||
] as const)(`${endpoint}: %s hires its own provider using the same %s binding`, async (provider, method) => {
|
||||
const f = await fixture(provider, method);
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/${endpoint}`).send({ name: "Teammate", role: "engineer", adapterType: f.adapterType, reportsTo: f.agentId }));
|
||||
expect(agent.runtimeConfig.aiConnection).toEqual(f.binding);
|
||||
const runtime = await prepareManagedAiRuntime(db, { companyId: f.companyId, agentId: agent.id, responsibleUserId: f.userId, adapterType: agent.adapterType, binding: agent.runtimeConfig.aiConnection, config: agent.adapterConfig });
|
||||
try {
|
||||
expect(runtime.attribution).toMatchObject({ connectionId: f.account.connectionId, grantId: f.account.grantId, method, responsibleUserId: f.userId });
|
||||
} finally { await runtime.cleanup(); }
|
||||
});
|
||||
}
|
||||
|
||||
for (const endpoint of ["agent-hires", "agents"]) {
|
||||
it.each([
|
||||
["ANTHROPIC_API_KEY", "child-key"],
|
||||
["ANTHROPIC_API_KEY", ""],
|
||||
["CLAUDE_CONFIG_DIR", "/tmp/child-claude-home"],
|
||||
["ANTHROPIC_BASE_URL", "https://example.invalid"],
|
||||
])(`${endpoint}: preserves an explicit child auth setting %s=%s`, async (key, value) => {
|
||||
const f = await fixture("anthropic");
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/${endpoint}`).send({
|
||||
name: "Explicit auth", role: "engineer", adapterType: f.adapterType,
|
||||
adapterConfig: { env: { [key]: value } },
|
||||
}));
|
||||
expect(agent.runtimeConfig.aiConnection).toBeUndefined();
|
||||
const [saved] = await db.select().from(agents).where(eq(agents.id, agent.id));
|
||||
expect((saved.adapterConfig.env as Record<string, unknown>)[key]).toEqual({ type: "plain", value });
|
||||
});
|
||||
}
|
||||
|
||||
it.each([true, false])("managed bindings do not inherit legacy credentials (managed parent: %s)", async (managedParent) => {
|
||||
const f = await fixture("openai");
|
||||
const secret = await secretService(db).create(f.companyId, {
|
||||
name: "Legacy parent key", provider: "local_encrypted", value: "fixture-legacy-key",
|
||||
});
|
||||
await db.update(agents).set({
|
||||
runtimeConfig: managedParent ? { aiConnection: f.binding } : {},
|
||||
adapterConfig: { env: { OPENAI_API_KEY: { type: "secret_ref", secretId: secret.id } } },
|
||||
}).where(eq(agents.id, f.agentId));
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({
|
||||
name: "Managed child", role: "engineer", adapterType: f.adapterType,
|
||||
...(managedParent ? {} : { runtimeConfig: { aiConnection: f.binding } }),
|
||||
}));
|
||||
expect(agent.runtimeConfig.aiConnection).toEqual(f.binding);
|
||||
expect(agent.adapterConfig.env?.OPENAI_API_KEY).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps unmanaged parent hires on their existing authentication path", async () => {
|
||||
const f = await fixture("openai");
|
||||
await db.update(agents).set({ runtimeConfig: {} }).where(eq(agents.id, f.agentId));
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({
|
||||
name: "Legacy authentication", role: "engineer", adapterType: f.adapterType,
|
||||
}));
|
||||
expect(agent.runtimeConfig.aiConnection).toBeUndefined();
|
||||
});
|
||||
|
||||
it.each(["anthropic", "openai"] as const)("%s can hire the other provider before that user connects it", async (provider) => {
|
||||
const f = await fixture(provider);
|
||||
const otherProvider = provider === "anthropic" ? "openai" : "anthropic";
|
||||
const adapterType = otherProvider === "anthropic" ? "claude_local" : "codex_local";
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Other provider", role: "engineer", adapterType }));
|
||||
expect(agent.runtimeConfig.aiConnection).toMatchObject({ provider: otherProvider, mode: "responsible_user" });
|
||||
await expect(prepareManagedAiRuntime(db, { companyId: f.companyId, agentId: agent.id, responsibleUserId: f.userId, adapterType, binding: agent.runtimeConfig.aiConnection, config: agent.adapterConfig })).rejects.toMatchObject({ details: { code: "ai_connection_default_missing" } });
|
||||
expect(agent.status).toBe("idle");
|
||||
});
|
||||
|
||||
for (const endpoint of ["agent-hires", "agents"]) {
|
||||
it.each([
|
||||
["anthropic", "codex_local", {}, "ANTHROPIC_API_KEY"],
|
||||
["openai", "claude_local", {}, "OPENAI_API_KEY"],
|
||||
["anthropic", "paperclip_runner", { provider: "codex" }, "ANTHROPIC_API_KEY"],
|
||||
["openai", "paperclip_runner", { provider: "acpx", acpxAgent: "claude" }, "OPENAI_API_KEY"],
|
||||
] as const)(`${endpoint}: ignores the %s auth key for a different provider in %s`, async (provider, adapterType, config, key) => {
|
||||
const f = await fixture(provider);
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/${endpoint}`).send({
|
||||
name: "Cross-provider config", role: "engineer", adapterType,
|
||||
adapterConfig: { ...config, env: { [key]: "leftover-parent-setting" } },
|
||||
}));
|
||||
expect(agent.runtimeConfig.aiConnection).toMatchObject({
|
||||
provider: provider === "anthropic" ? "openai" : "anthropic", mode: "responsible_user",
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
it("accepts an explicit personal default before authentication, including approval-gated hires", async () => {
|
||||
const f = await fixture("anthropic");
|
||||
await db.update(companies).set({ requireBoardApprovalForNewAgents: true }).where(eq(companies.id, f.companyId));
|
||||
const binding: AiConnectionBinding = { provider: "openai", method: "subscription", mode: "responsible_user" };
|
||||
const response = await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Future Codex", role: "engineer", adapterType: "codex_local", runtimeConfig: { aiConnection: binding } });
|
||||
const agent = hired(response);
|
||||
expect(agent.runtimeConfig.aiConnection).toEqual(binding);
|
||||
expect(agent.status).toBe("pending_approval");
|
||||
expect(response.body.approval.payload.runtimeConfig.aiConnection).toEqual(binding);
|
||||
expect(await db.select().from(toolConnectionInstalls).where(eq(toolConnectionInstalls.targetId, agent.id))).toEqual([]);
|
||||
});
|
||||
|
||||
it.each(["anthropic", "openai"] as const)("inherits %s when the hire uses the native runner", async (provider) => {
|
||||
const f = await fixture(provider, "subscription");
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Native teammate", role: "engineer", adapterType: "paperclip_runner", adapterConfig: provider === "anthropic" ? { provider: "acpx", acpxAgent: "claude" } : { provider: "codex" } }));
|
||||
expect(agent.runtimeConfig.aiConnection).toEqual(f.binding);
|
||||
const runtime = await prepareManagedAiRuntime(db, { companyId: f.companyId, agentId: agent.id, responsibleUserId: f.userId, adapterType: agent.adapterType, binding: agent.runtimeConfig.aiConnection, config: agent.adapterConfig });
|
||||
try { expect(runtime.attribution.connectionId).toBe(f.account.connectionId); } finally { await runtime.cleanup(); }
|
||||
});
|
||||
|
||||
it.each([true, false])("preserves shared connection access boundaries (company access: %s)", async (allAgents) => {
|
||||
const f = await fixture("anthropic");
|
||||
const account = await aiConnectionService(db).save(f.companyId, f.userId, { provider: "anthropic", method: "api_key", name: "Shared Claude", ownership: "shared", apiKey: "fixture", agentIds: [f.agentId], allAgents }, "fixture");
|
||||
const binding = { provider: "anthropic", method: "api_key", mode: "shared", ...account };
|
||||
await db.update(agents).set({ runtimeConfig: { aiConnection: binding } }).where(eq(agents.id, f.agentId));
|
||||
const response = await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Shared teammate", role: "engineer", adapterType: f.adapterType });
|
||||
if (!allAgents) {
|
||||
expect(response.status).toBe(403);
|
||||
expect(await db.select().from(agents).where(eq(agents.companyId, f.companyId))).toHaveLength(1);
|
||||
} else {
|
||||
const agent = hired(response);
|
||||
expect(agent.runtimeConfig.aiConnection).toEqual(binding);
|
||||
const runtime = await prepareManagedAiRuntime(db, { companyId: f.companyId, agentId: agent.id, responsibleUserId: f.userId, adapterType: agent.adapterType, binding: agent.runtimeConfig.aiConnection, config: agent.adapterConfig });
|
||||
try { expect(runtime.attribution.connectionId).toBe(account.connectionId); } finally { await runtime.cleanup(); }
|
||||
}
|
||||
});
|
||||
|
||||
it("still rejects an explicitly incompatible provider without creating a hire", async () => {
|
||||
const f = await fixture("anthropic");
|
||||
const response = await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Wrong provider", role: "engineer", adapterType: "codex_local", runtimeConfig: { aiConnection: f.binding } });
|
||||
expect(response.status).toBe(422);
|
||||
expect(response.body.details.code).toBe("ai_connection_incompatible");
|
||||
expect(await db.select().from(agents).where(eq(agents.companyId, f.companyId))).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("hired agents sharing a subscription", () => {
|
||||
it.each(["anthropic", "openai"] as const)("waits for the %s parent and resumes without asking for a new connection", async (provider) => {
|
||||
const f = await fixture(provider, "subscription");
|
||||
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Waiting teammate", role: "engineer", adapterType: f.adapterType, reportsTo: f.agentId, adapterConfig: { cwd: home, engine: "cli" }, runtimeConfig: { heartbeat: { enabled: false } } }));
|
||||
const [issue] = await db.insert(issues).values({ companyId: f.companyId, title: "Subscription child task", status: "todo", assigneeAgentId: agent.id, responsibleUserId: f.userId, createdByUserId: f.userId }).returning();
|
||||
const parentRuntime = await prepareManagedAiRuntime(db, { companyId: f.companyId, agentId: f.agentId, responsibleUserId: f.userId, adapterType: f.adapterType, binding: f.binding, config: { cwd: home } });
|
||||
const execute = vi.fn(async () => {
|
||||
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, issue.id));
|
||||
return { exitCode: 0, signal: null, timedOut: false, resultJson: {} };
|
||||
});
|
||||
registerServerAdapter({ ...getServerAdapter(f.adapterType), execute });
|
||||
const heartbeat = heartbeatService(db);
|
||||
let parentReleased = false;
|
||||
try {
|
||||
const run = await heartbeat.invoke(agent.id, "assignment", { issueId: issue.id, wakeReason: "issue_assigned", responsibleUserId: f.userId }, "system");
|
||||
expect(run).not.toBeNull();
|
||||
await expect.poll(async () => (await heartbeat.getRun(run!.id))?.status, { timeout: 20_000 }).not.toBe("running");
|
||||
expect(await heartbeat.getRun(run!.id)).toMatchObject({ status: "cancelled", errorCode: "ai_connection_busy" });
|
||||
let retry: typeof heartbeatRuns.$inferSelect | undefined;
|
||||
await expect.poll(async () => { [retry] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run!.id)); return retry?.status; }, { timeout: 10_000 }).toBe("scheduled_retry");
|
||||
expect(execute).not.toHaveBeenCalled();
|
||||
expect(await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, issue.id))).toHaveLength(0);
|
||||
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0].executionRunId).toBe(retry!.id);
|
||||
parentReleased = true;
|
||||
await parentRuntime.cleanup();
|
||||
const promoted = await heartbeat.promoteDueScheduledRetries(new Date(retry!.scheduledRetryAt!.getTime() + 1000));
|
||||
expect(promoted.runIds).toContain(retry!.id);
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
await expect.poll(async () => (await heartbeat.getRun(retry!.id))?.status, { timeout: 20_000 }).toBe("succeeded");
|
||||
expect(execute).toHaveBeenCalledTimes(1);
|
||||
expect((await heartbeat.getRun(retry!.id))?.contextSnapshot?.aiConnection).toMatchObject({ connectionId: f.account.connectionId, responsibleUserId: f.userId, method: "subscription" });
|
||||
} finally {
|
||||
if (!parentReleased) await parentRuntime.cleanup();
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
unregisterServerAdapter(f.adapterType);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -1374,6 +1374,55 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
expect(issue?.executionRunId).toBe(retryRuns[0]?.id);
|
||||
});
|
||||
|
||||
it.each(["schedule", "transaction", "promote", "dispatch"] as const)(
|
||||
"suppresses a busy-subscription retry whose task lock was cleared before %s",
|
||||
async (phase) => {
|
||||
const { companyId, issueId, runId, now } = await seedMaxTurnFixture();
|
||||
await db.update(heartbeatRuns).set({
|
||||
status: "cancelled", errorCode: "ai_connection_busy",
|
||||
resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false } },
|
||||
}).where(eq(heartbeatRuns.id, runId));
|
||||
const clearLock = () => db.update(issues).set({ executionRunId: null }).where(eq(issues.id, issueId));
|
||||
if (phase === "schedule") await clearLock();
|
||||
const transaction = db.transaction.bind(db);
|
||||
let raceApplied = false;
|
||||
const transactionSpy = phase === "transaction"
|
||||
? vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => {
|
||||
await clearLock();
|
||||
raceApplied = true;
|
||||
return transaction(callback, config);
|
||||
})
|
||||
: null;
|
||||
const scheduled = await heartbeat.scheduleBoundedRetry(runId, {
|
||||
now, retryReason: "ai_connection_busy", wakeReason: "ai_connection_busy_retry",
|
||||
maxAttempts: 1, delayMs: 1_000,
|
||||
}).finally(() => transactionSpy?.mockRestore());
|
||||
if (phase === "transaction") expect(raceApplied).toBe(true);
|
||||
if (phase === "schedule" || phase === "transaction") {
|
||||
expect(scheduled).toMatchObject({ outcome: "not_scheduled", errorCode: "issue_execution_lock_changed" });
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
|
||||
return;
|
||||
}
|
||||
expect(scheduled.outcome).toBe("scheduled");
|
||||
if (scheduled.outcome !== "scheduled") return;
|
||||
if (phase === "promote") await clearLock();
|
||||
const promotion = await heartbeat.promoteDueScheduledRetries(scheduled.dueAt);
|
||||
if (phase === "promote") {
|
||||
expect(promotion).toEqual({ promoted: 0, runIds: [] });
|
||||
} else {
|
||||
expect(promotion.runIds).toContain(scheduled.run.id);
|
||||
await clearLock();
|
||||
const adapter = createPostgresRunDispatchAdapter(db);
|
||||
expect(await adapter.cancelStaleQueuedRun({
|
||||
companyId, runId: scheduled.run.id, expectedStatus: "queued", now: scheduled.dueAt,
|
||||
})).toMatchObject({ outcome: "cancelled", errorCode: "issue_execution_lock_changed" });
|
||||
}
|
||||
expect(await heartbeat.getRun(scheduled.run.id)).toMatchObject({
|
||||
status: "cancelled", errorCode: "issue_execution_lock_changed",
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("does not promote a duplicate max-turn continuation that does not own the issue lock", async () => {
|
||||
const { companyId, agentId, issueId, runId, now } = await seedMaxTurnFixture();
|
||||
|
||||
|
||||
@@ -113,6 +113,7 @@ function readNonEmptyString(value: unknown): string | null {
|
||||
function classifyRetryReasonKind(retryReason: string | null): RetryReasonKind {
|
||||
if (retryReason === MAX_TURN_CONTINUATION_RETRY_REASON) return "max_turn_continuation";
|
||||
if (retryReason === ISSUE_DISPOSITION_REPAIR_RETRY_REASON) return "disposition_repair";
|
||||
if (retryReason === "ai_connection_busy") return "ai_connection_wait";
|
||||
if (retryReason === "native_safe_replacement") return "native_safe_replacement";
|
||||
return "other";
|
||||
}
|
||||
@@ -274,7 +275,7 @@ export function createPostgresRunDispatchAdapter(
|
||||
runAgentId: input.agentId,
|
||||
issueId,
|
||||
retryReasonKind,
|
||||
enforceIssueExecutionLock: retryReasonKind === "max_turn_continuation",
|
||||
enforceIssueExecutionLock: retryReasonKind === "max_turn_continuation" || retryReasonKind === "ai_connection_wait",
|
||||
isNonAssigneeWorkspaceBusyRetry: isNonAssigneeWorkspaceBusyRetry(retryReason, input.contextSnapshot),
|
||||
budgetBlock: null,
|
||||
agentInvokable: true,
|
||||
|
||||
@@ -5,11 +5,12 @@
|
||||
// then packs the result into a facts object. This file only branches on
|
||||
// that facts object; it never queries a database or reads the clock.
|
||||
|
||||
/** The retry reason a run carries, reduced to the three kinds a gate cares about. */
|
||||
/** The retry reason a run carries, reduced to the kinds a gate cares about. */
|
||||
export type RetryReasonKind =
|
||||
| "max_turn_continuation"
|
||||
| "disposition_repair"
|
||||
| "native_safe_replacement"
|
||||
| "ai_connection_wait"
|
||||
| "other";
|
||||
|
||||
export type BudgetBlockFacts = {
|
||||
@@ -403,7 +404,7 @@ export function decideScheduledRetryGate(
|
||||
}
|
||||
|
||||
const lockOutcome = decideExecutionLock({
|
||||
requiresExecutionLock: requiresInProgress && facts.enforceIssueExecutionLock,
|
||||
requiresExecutionLock: (requiresInProgress || facts.retryReasonKind === "ai_connection_wait") && facts.enforceIssueExecutionLock,
|
||||
runId: facts.runId,
|
||||
issueExecutionRunId: facts.issueExecutionRunId,
|
||||
});
|
||||
@@ -411,7 +412,7 @@ export function decideScheduledRetryGate(
|
||||
return {
|
||||
allowed: false,
|
||||
reason:
|
||||
"Scheduled max-turn continuation suppressed because the issue execution lock belongs to a different run",
|
||||
"Scheduled retry suppressed because the issue execution lock belongs to a different run",
|
||||
errorCode: "issue_execution_lock_changed",
|
||||
issueId: facts.issueId,
|
||||
details: {
|
||||
@@ -629,7 +630,7 @@ export function decideQueuedRunStaleness(
|
||||
}
|
||||
|
||||
const lockOutcome = decideExecutionLock({
|
||||
requiresExecutionLock: requiresInProgress,
|
||||
requiresExecutionLock: requiresInProgress || facts.retryReasonKind === "ai_connection_wait",
|
||||
runId: facts.runId,
|
||||
issueExecutionRunId: facts.issueExecutionRunId,
|
||||
});
|
||||
@@ -638,7 +639,7 @@ export function decideQueuedRunStaleness(
|
||||
stale: true,
|
||||
errorCode: "issue_execution_lock_changed",
|
||||
reason:
|
||||
"Cancelled because max-turn continuation no longer owns the issue execution lock before the queued run could start",
|
||||
"Cancelled because the retry no longer owns the issue execution lock before the queued run could start",
|
||||
details: {
|
||||
issueId: facts.issueId,
|
||||
expectedExecutionRunId: facts.runId,
|
||||
|
||||
@@ -3,6 +3,7 @@ import { prepareManagedAiRuntime, assertManagedAiProjectAuth, stripAiAuthBinding
|
||||
import { ADAPTER_AUTH_MISSING_CHECK_CODE, AI_CONNECTION_CAPABILITIES, aiConnectionBindingSchema, type AiConnectionBinding } from "@paperclipai/shared";
|
||||
import { toolConnections } from "@paperclipai/db";
|
||||
import { aiConnectionService } from "../services/ai-connections.js";
|
||||
import { defaultAiConnectionForHire } from "../services/agent-ai-connection-default.js";
|
||||
import { assertAiConnectionCreateAccess, canInstallSharedAiConnectionForNewAgent, responsibleUserForAiRequest, validateAiApiKey } from "./ai-connections.js";
|
||||
import { isAiConnectionCompatible } from "@paperclipai/shared";
|
||||
import { applyConnectorSkills, resolveConnectorAssignments, annotateConnectorSkills, isConnectorSkill } from "../services/connector-runtime.js";
|
||||
@@ -2412,6 +2413,22 @@ export function agentRoutes(
|
||||
return normalizedRuntimeConfig;
|
||||
}
|
||||
|
||||
async function normalizeCreatedAgentRuntimeConfig(
|
||||
req: Request,
|
||||
companyId: string,
|
||||
adapterType: string,
|
||||
adapterConfig: Record<string, unknown>,
|
||||
runtimeConfig: unknown,
|
||||
) {
|
||||
const normalized = normalizeNewAgentRuntimeConfig(runtimeConfig);
|
||||
if (req.actor.type !== "agent" || normalized.aiConnection) return normalized;
|
||||
const manager = req.actor.agentId ? await svc.getById(req.actor.agentId) : null;
|
||||
if (!manager || manager.companyId !== companyId) throw forbidden("Hiring agent is unavailable");
|
||||
const binding = defaultAiConnectionForHire(adapterType, adapterConfig, manager.runtimeConfig?.aiConnection);
|
||||
if (binding) normalized.aiConnection = binding;
|
||||
return normalized;
|
||||
}
|
||||
|
||||
async function normalizeMediatedAdapterConfigForPersistence(input: {
|
||||
companyId: string;
|
||||
adapterType: string | null | undefined;
|
||||
@@ -2528,14 +2545,18 @@ export function agentRoutes(
|
||||
companyId: string,
|
||||
adapterType: string | null | undefined,
|
||||
adapterConfig: Record<string, unknown>,
|
||||
runtimeConfig: unknown,
|
||||
): Promise<{ adapterConfig: Record<string, unknown>; inheritedFixedClaudeOAuthBinding: boolean }> {
|
||||
const noInheritance = { adapterConfig, inheritedFixedClaudeOAuthBinding: false };
|
||||
if (asRecord(runtimeConfig)?.aiConnection) return noInheritance;
|
||||
if (req.actor.type !== "agent" || !req.actor.agentId) return noInheritance;
|
||||
const credentialKeys = adapterType ? INHERITABLE_AGENT_CREDENTIAL_ENV_KEYS[adapterType] : undefined;
|
||||
if (!credentialKeys) return noInheritance;
|
||||
|
||||
const parent = await svc.getById(req.actor.agentId);
|
||||
if (!parent || parent.companyId !== companyId || parent.adapterType !== adapterType) return noInheritance;
|
||||
// Managed parents must not pass stale legacy credential references to hires.
|
||||
if (aiConnectionBindingSchema.safeParse(parent.runtimeConfig.aiConnection).success) return noInheritance;
|
||||
const parentEnv = asRecord(asRecord(parent.adapterConfig)?.env);
|
||||
if (!parentEnv) return noInheritance;
|
||||
|
||||
@@ -3297,7 +3318,17 @@ export function agentRoutes(
|
||||
async function validateManagedAgentBinding(req: Request, companyId: string, agentId: string, adapterType: string, config: Record<string, unknown>, binding: AiConnectionBinding, environmentId: string | null | undefined, test: boolean, newAgent = false) {
|
||||
const userId = responsibleUserForAiRequest(req);
|
||||
const allowUninstalledShared = newAgent && await canInstallSharedAiConnectionForNewAgent(db, req, companyId, binding);
|
||||
const selection = await aiConnectionService(db).select({ companyId, agentId, userId, adapterType, model: config.model, runnerProvider: config.provider, acpxAgent: config.acpxAgent, binding, allowUninstalledPersonal: newAgent, allowUninstalledShared, allowLegacyValidation: test });
|
||||
const selection = await aiConnectionService(db).select({ companyId, agentId, userId, adapterType, model: config.model, runnerProvider: config.provider, acpxAgent: config.acpxAgent, binding, allowUninstalledPersonal: newAgent, allowUninstalledShared, allowLegacyValidation: test }).catch((error: unknown) => {
|
||||
// Hiring is allowed before the responsible user has connected this
|
||||
// provider. Execution still resolves credentials and creates the normal
|
||||
// task connection request; compatibility and access denials stay errors.
|
||||
if (newAgent && !test && binding.mode === "responsible_user" && error instanceof HttpError
|
||||
&& ["ai_connection_default_missing", "ai_connection_missing", "ai_connection_unavailable", "ai_connection_responsible_user_missing"].includes(String(asRecord(error.details)?.code))) {
|
||||
return null;
|
||||
}
|
||||
throw error;
|
||||
});
|
||||
if (!selection) return undefined;
|
||||
if (test) {
|
||||
if (environmentId) await assertAdapterTestEnvironmentForCompany(companyId, environmentId);
|
||||
const target = await resolveAdapterTestExecutionContext({ companyId, adapterType, environmentId: environmentId ?? null });
|
||||
@@ -4399,6 +4430,7 @@ export function agentRoutes(
|
||||
hireInput.adapterType,
|
||||
rawHireAdapterConfig,
|
||||
),
|
||||
hireInput.runtimeConfig,
|
||||
);
|
||||
const requestedAdapterConfig = applyCodexLocalKeyIsolation(
|
||||
companyId,
|
||||
@@ -4431,7 +4463,7 @@ export function agentRoutes(
|
||||
adapterType: hireInput.adapterType,
|
||||
adapterConfig: desiredSkillAssignment.adapterConfig,
|
||||
});
|
||||
const normalizedRuntimeConfig = normalizeNewAgentRuntimeConfig(hireInput.runtimeConfig);
|
||||
const normalizedRuntimeConfig = await normalizeCreatedAgentRuntimeConfig(req, companyId, hireInput.adapterType, normalizedAdapterConfig, hireInput.runtimeConfig);
|
||||
const normalizedHireInput = {
|
||||
...hireInput,
|
||||
adapterConfig: normalizedAdapterConfig,
|
||||
@@ -4726,7 +4758,7 @@ export function agentRoutes(
|
||||
adapterType: createInput.adapterType,
|
||||
adapterConfig: desiredSkillAssignment.adapterConfig,
|
||||
});
|
||||
const normalizedRuntimeConfig = normalizeNewAgentRuntimeConfig(createInput.runtimeConfig);
|
||||
const normalizedRuntimeConfig = await normalizeCreatedAgentRuntimeConfig(req, companyId, createInput.adapterType, normalizedAdapterConfig, createInput.runtimeConfig);
|
||||
await assertAgentEnvironmentSelection(companyId, createInput.adapterType, createInput.defaultEnvironmentId);
|
||||
await assertAgentDefaultEnvironmentSelection(companyId, createInput.defaultEnvironmentId, {
|
||||
allowedDrivers: allowedEnvironmentDriversForAgent(createInput.adapterType),
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
import {
|
||||
AI_PROVIDERS,
|
||||
aiConnectionBindingSchema,
|
||||
isAiConnectionCompatible,
|
||||
type AiConnectionBinding,
|
||||
type AiProvider,
|
||||
} from "@paperclipai/shared";
|
||||
|
||||
// Only keys read by the child's provider express a child auth override.
|
||||
// A config copied from another provider can retain unrelated keys.
|
||||
const PROVIDER_AUTH_ENV_KEYS: Record<AiProvider, readonly string[]> = {
|
||||
anthropic: ["ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN", "CLAUDE_CODE_OAUTH_TOKEN", "CLAUDE_CONFIG_DIR", "ANTHROPIC_BASE_URL", "CLAUDE_CODE_USE_BEDROCK", "CLAUDE_CODE_USE_VERTEX", "CLAUDE_CODE_USE_FOUNDRY"],
|
||||
openai: ["OPENAI_API_KEY", "CODEX_API_KEY", "CODEX_HOME", "OPENAI_BASE_URL"],
|
||||
openrouter: ["OPENROUTER_API_KEY", "OPENCODE_AUTH_JSON", "OPENCODE_CONFIG_CONTENT", "OPENCODE_CONFIG", "OPENCODE_CONFIG_DIR", "PAPERCLIP_OPENCODE_PROVIDERS"],
|
||||
xai: ["XAI_API_KEY", "GROK_API_KEY", "GROK_HOME", "XAI_BASE_URL"],
|
||||
};
|
||||
|
||||
/** A hire inherits a connection choice, never its manager's credentials or identity. */
|
||||
export function defaultAiConnectionForHire(
|
||||
adapterType: string,
|
||||
config: Record<string, unknown>,
|
||||
managerBinding: unknown,
|
||||
): AiConnectionBinding | undefined {
|
||||
const compatible = (binding: AiConnectionBinding) =>
|
||||
isAiConnectionCompatible(binding, adapterType, config.model, config.provider, config.acpxAgent);
|
||||
const inherited = aiConnectionBindingSchema.safeParse(managerBinding);
|
||||
// Unmanaged parents keep their existing login and credential-reference paths.
|
||||
if (!inherited.success) return undefined;
|
||||
const env = config.env && typeof config.env === "object" ? config.env as Record<string, unknown> : {};
|
||||
const withChildAuthPrecedence = (binding: AiConnectionBinding) =>
|
||||
PROVIDER_AUTH_ENV_KEYS[binding.provider].some((key) => env[key] !== undefined) ? undefined : binding;
|
||||
if (inherited.data.mode !== "delegated" && compatible(inherited.data)) {
|
||||
return withChildAuthPrecedence(inherited.data);
|
||||
}
|
||||
// The selected provider's personal default supplies the actual sign-in method
|
||||
// at run time. A provider without an account can be connected on the first task.
|
||||
for (const provider of AI_PROVIDERS) {
|
||||
const binding = { provider, method: "api_key", mode: "responsible_user" } as const;
|
||||
if (compatible(binding)) return withChildAuthPrecedence(binding);
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
@@ -2,6 +2,12 @@ import { describe, expect, it } from "vitest";
|
||||
import { executionFailureRetryCount } from "./execution-recovery-attempt.js";
|
||||
|
||||
describe("failure attempts across resource waits", () => {
|
||||
it("preserves prior failures through repeated AI subscription waits", () => {
|
||||
expect(executionFailureRetryCount({ scheduledRetryReason: "ai_connection_busy", scheduledRetryAttempt: 12,
|
||||
contextSnapshot: { failureRetriesBeforeAiConnectionWait: 1 } })).toBe(1);
|
||||
expect(executionFailureRetryCount({ scheduledRetryReason: "transient_failure", scheduledRetryAttempt: 2,
|
||||
contextSnapshot: { failureRetriesBeforeAiConnectionWait: 0 } })).toBe(2);
|
||||
});
|
||||
it("preserves prior failures through repeated workspace waits", () => {
|
||||
expect(executionFailureRetryCount({ scheduledRetryReason: "workspace_busy", scheduledRetryAttempt: 12,
|
||||
contextSnapshot: { failureRetriesBeforeWorkspaceWait: 1 } })).toBe(1);
|
||||
|
||||
@@ -5,6 +5,10 @@ export function executionFailureRetryCount(run: {
|
||||
contextSnapshot?: Record<string, unknown> | null;
|
||||
}): number {
|
||||
if (run.scheduledRetryReason === "max_turns_continuation") return 0;
|
||||
if (run.scheduledRetryReason === "ai_connection_busy") {
|
||||
const count = run.contextSnapshot?.failureRetriesBeforeAiConnectionWait;
|
||||
if (typeof count === "number" && Number.isInteger(count) && count >= 0) return count;
|
||||
}
|
||||
if (run.scheduledRetryReason === "workspace_busy") {
|
||||
// Only a server-created workspace retry can consume this field. Its
|
||||
// scheduler overwrites caller context with the predecessor's durable count.
|
||||
|
||||
@@ -842,6 +842,10 @@ const MAX_TURN_CONTINUATION_LIVE_RUN_STATUSES = [
|
||||
export { WORKSPACE_BUSY_RETRY_REASON };
|
||||
export const WORKSPACE_BUSY_RETRY_WAKE_REASON = "workspace_busy_retry";
|
||||
export const WORKSPACE_BUSY_ERROR_CODE = "workspace_busy";
|
||||
const AI_CONNECTION_BUSY_RETRY_REASON = "ai_connection_busy";
|
||||
function isAiConnectionBusy(error: unknown): error is HttpError {
|
||||
return error instanceof HttpError && parseObject(error.details).code === AI_CONNECTION_BUSY_RETRY_REASON;
|
||||
}
|
||||
export const WORKSPACE_BUSY_RETRY_BASE_DELAY_MS = 60 * 1000;
|
||||
export const WORKSPACE_BUSY_RETRY_JITTER_MS = 60 * 1000;
|
||||
// A running run stops counting as a shared-workspace holder once it has been
|
||||
@@ -15018,6 +15022,7 @@ export function heartbeatService(
|
||||
);
|
||||
const nextAttempt =
|
||||
(retryReason === WORKSPACE_BUSY_RETRY_REASON ||
|
||||
retryReason === AI_CONNECTION_BUSY_RETRY_REASON ||
|
||||
retryReason === MAX_TURN_CONTINUATION_RETRY_REASON
|
||||
? (run.scheduledRetryAttempt ?? 0)
|
||||
: executionFailureRetryCount(run)) + 1;
|
||||
@@ -15144,6 +15149,7 @@ export function heartbeatService(
|
||||
|
||||
const requiresIssueGate =
|
||||
hasConversationContinuationPolicy(run.resultJson) ||
|
||||
retryReason === AI_CONNECTION_BUSY_RETRY_REASON ||
|
||||
retryReason === MAX_TURN_CONTINUATION_RETRY_REASON ||
|
||||
retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON;
|
||||
if (requiresIssueGate) {
|
||||
@@ -15212,6 +15218,9 @@ export function heartbeatService(
|
||||
executionFailureRetryCount(run),
|
||||
}
|
||||
: {}),
|
||||
...(retryReason === AI_CONNECTION_BUSY_RETRY_REASON
|
||||
? { failureRetriesBeforeAiConnectionWait: executionFailureRetryCount(run) }
|
||||
: {}),
|
||||
...(shouldQuarantineWorkspaceForRetry
|
||||
? {
|
||||
workspaceValidationRecovery: {
|
||||
@@ -15534,6 +15543,20 @@ export function heartbeatService(
|
||||
}
|
||||
}
|
||||
|
||||
if (retryReason === AI_CONNECTION_BUSY_RETRY_REASON && issueId) {
|
||||
// The issue row is locked above. Recheck after the preflight gate so
|
||||
// cancellation or recovery cannot leave a successor without its lock.
|
||||
const [lockedIssue] = await tx.select({ executionRunId: issues.executionRunId })
|
||||
.from(issues).where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId)));
|
||||
if (lockedIssue?.executionRunId !== run.id) {
|
||||
return {
|
||||
outcome: "not_scheduled", issueId, errorCode: "issue_execution_lock_changed",
|
||||
reason: "Subscription retry suppressed because the task execution lock changed",
|
||||
details: { issueId, expectedExecutionRunId: run.id, currentExecutionRunId: lockedIssue?.executionRunId ?? null },
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
const wakeupRequest = await tx
|
||||
.insert(agentWakeupRequests)
|
||||
.values({
|
||||
@@ -15916,6 +15939,41 @@ export function heartbeatService(
|
||||
.then((rows) => rows[0] ?? null);
|
||||
}
|
||||
|
||||
// Subscription homes are serialized so refresh tokens cannot be overwritten
|
||||
// by another run. Contention is a resource wait, not broken authentication.
|
||||
// The database lease is released on completion/disconnect; retrying remains
|
||||
// safe across processes and each attempt revalidates the selected account.
|
||||
async function finalizeAiConnectionBusyDeferral(run: typeof heartbeatRuns.$inferSelect, error: HttpError) {
|
||||
const now = new Date();
|
||||
const cancelled = await setRunStatusIfRunning(run.id, "cancelled", {
|
||||
error: error.message, errorCode: AI_CONNECTION_BUSY_RETRY_REASON, finishedAt: now,
|
||||
resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false } },
|
||||
});
|
||||
if (!cancelled.updated) return;
|
||||
await setWakeupStatus(run.wakeupRequestId, "cancelled", { finishedAt: now, error: error.message }).catch(() => undefined);
|
||||
const cancelledRun = cancelled.run ?? await getRun(run.id);
|
||||
const agent = await getAgent(run.agentId);
|
||||
let scheduled = false;
|
||||
try {
|
||||
if (cancelledRun && agent) {
|
||||
const retry = await scheduleBoundedRetryForRun(cancelledRun, agent, {
|
||||
now, retryReason: AI_CONNECTION_BUSY_RETRY_REASON, wakeReason: "ai_connection_busy_retry",
|
||||
maxAttempts: (cancelledRun.scheduledRetryAttempt ?? 0) + 1,
|
||||
delayMs: computeWorkspaceBusyRetryDelayMs(),
|
||||
});
|
||||
scheduled = retry.outcome === "scheduled";
|
||||
await appendRunEvent(cancelledRun, {
|
||||
eventType: "lifecycle", stream: "system", level: "info",
|
||||
message: scheduled ? "Waiting for the shared AI subscription. This task will retry automatically." : "The AI subscription is busy; this task can no longer retry automatically.",
|
||||
payload: { retryScheduled: scheduled },
|
||||
});
|
||||
}
|
||||
} finally {
|
||||
if (cancelledRun && !scheduled) await releaseIssueExecutionAndPromote(cancelledRun);
|
||||
await finalizeAgentStatus(run.agentId, "cancelled", null, { wasFirstHeartbeat: timerClaimWasFirstHeartbeat(run) });
|
||||
}
|
||||
}
|
||||
|
||||
// Terminal handling for a WorkspaceBusyDeferral thrown by the pre-dispatch
|
||||
// gate: cancel the run (contention is not a failure), schedule a
|
||||
// workspace_busy retry, and leave the agent idle. The issue execution lock
|
||||
@@ -20711,6 +20769,7 @@ export function heartbeatService(
|
||||
try {
|
||||
managedAiRuntime = await prepareManagedAiRuntime(db, { companyId: agent.companyId, agentId: agent.id, responsibleUserId, adapterType: agent.adapterType, binding: aiBinding, config: resolvedConfig });
|
||||
} catch (error) {
|
||||
if (isAiConnectionBusy(error)) throw error;
|
||||
if (responsibleUserId && issueId && aiBinding.mode === "responsible_user") {
|
||||
await connectionIntentService(db).request({ sub: agent.id, company_id: agent.companyId, run_id: run.id, responsible_user_id: responsibleUserId }, aiBinding.provider, { purpose: "ai" }).catch(() => {
|
||||
logger.warn({ runId: run.id, agentId: agent.id }, "Could not attach AI connection request; runtime configuration action remains available");
|
||||
@@ -25018,6 +25077,10 @@ export function heartbeatService(
|
||||
? outerErr.reason
|
||||
: "adopted_runner_authentication_timeout",
|
||||
}).catch(() => undefined);
|
||||
} else if (isAiConnectionBusy(outerErr)) {
|
||||
await finalizeAiConnectionBusyDeferral(run, outerErr).catch((error) => {
|
||||
logger.error({ err: error, runId }, "failed to schedule a retry for the busy AI subscription");
|
||||
});
|
||||
} else if (isWorkspaceBusyDeferral(outerErr)) {
|
||||
// Expected contention on a shared project workspace, not a
|
||||
// failure: park the run as a bounded scheduled retry and leave the
|
||||
|
||||
@@ -38,3 +38,11 @@ it("retains the hold until the provider actually acknowledges cancellation", ()
|
||||
})).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("retries a busy AI subscription only when no provider work started", () => {
|
||||
const waiting = { runtimeMode: "legacy", status: "cancelled", errorCode: "ai_connection_busy", scheduledRetryAttempt: 10,
|
||||
resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false } } };
|
||||
expect(legacyExecutionNeedsReconciliation(waiting)).toBe(false);
|
||||
expect(legacyExecutionNeedsReconciliation({ ...waiting, resultJson: {} })).toBe(true);
|
||||
expect(legacyExecutionNeedsReconciliation({ ...waiting, resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: true } } })).toBe(true);
|
||||
});
|
||||
@@ -31,8 +31,10 @@ export function legacyExecutionNeedsReconciliation(
|
||||
&& evidence.providerStopped === true && evidence.sessionPreserved === true
|
||||
&& evidence.actionOutcomes === "settled"
|
||||
&& (run.resultJson?.executionCancellation as Record<string, unknown> | undefined)?.state === "acknowledged") return false;
|
||||
// Waiting for a live workspace holder precedes provider execution. It is a
|
||||
// resource wait, not a failed provider attempt or permission to replay work.
|
||||
// Waiting for a subscription or workspace precedes provider execution. It is
|
||||
// a resource wait, not a failed provider attempt or permission to replay work.
|
||||
if (run.status === "cancelled" && run.errorCode === "ai_connection_busy" &&
|
||||
evidence?.kind === "ai_connection_wait" && evidence.providerWorkStarted === false) return false;
|
||||
if (run.status === "cancelled" && run.errorCode === "workspace_busy" &&
|
||||
evidence?.kind === "workspace_wait" && evidence.providerWorkStarted === false) return false;
|
||||
if (executionFailureRetryCount(run) >= 2) return true;
|
||||
|
||||
@@ -320,7 +320,13 @@ export class PaperclipRunnerToolAuthority {
|
||||
conversation: Boolean(context.issue.conversationAgentId) });
|
||||
}
|
||||
case "search_api": return searchRunnerApi(call.arguments);
|
||||
case "call_api": return this.#callApi(call.callId, call.arguments);
|
||||
case "call_api": {
|
||||
// PRP reserves operationId/callId for semantic result identity. The
|
||||
// HTTP operation is metadata, including in previously saved receipts;
|
||||
// exposing it as operationId makes the runner reject a valid response.
|
||||
const { operationId, ...response } = record(await this.#callApi(call.callId, call.arguments));
|
||||
return { ...response, apiOperationId: operationId };
|
||||
}
|
||||
case "get_task_context": return {
|
||||
company: { id: this.binding.companyId },
|
||||
actor: redactedActor(context.actor),
|
||||
|
||||
@@ -34,7 +34,7 @@ describe("runner API against real HTTP routes", () => {
|
||||
import { createInterface } from 'node:readline';
|
||||
const send = value => process.stdout.write(JSON.stringify(value)+'\\n');
|
||||
let step = 0;
|
||||
const calls = [{tool:'search_api',arguments:{query:'list projects'}},{tool:'call_api',arguments:{operationId:'GET /api/companies/{companyId}/projects'}}];
|
||||
const calls = [{tool:'search_api',arguments:{query:'list projects'}},{tool:'call_api',arguments:{operationId:'GET /api/companies/{companyId}/projects'}},{tool:'call_api',arguments:{operationId:'GET /api/projects/{id}',pathParams:{id:'${fixture.foreignProjectId}'}}}];
|
||||
const next = () => { const c=calls[step++]; if(c) send({id:'call-'+step,method:'item/tool/call',params:{threadId:'api-thread',turnId:'api-turn',itemId:'api-item-'+step,callId:'api-call-'+step,...c}}); else send({method:'turn/completed',params:{turn:{id:'api-turn',status:'completed'}}}); };
|
||||
for await (const line of createInterface({input:process.stdin})) {
|
||||
const m=JSON.parse(line);
|
||||
@@ -57,15 +57,24 @@ else if(m.id!==undefined) send({id:m.id,result:{}});
|
||||
const params = request.params as any;
|
||||
const result = await fixture.authority.execute({ tool: params.tool, arguments: params.arguments, callId: params.callId });
|
||||
results.push(result);
|
||||
return { success: true, contentItems: [{ type: "inputText", text: JSON.stringify({ ok: true, result }) }] };
|
||||
// Match production's dynamicToolResponse: an extra test-only envelope
|
||||
// hides collisions between API response fields and PRP tool identity.
|
||||
return { success: true, contentItems: [{ type: "inputText", text: JSON.stringify(result) }] };
|
||||
});
|
||||
try {
|
||||
await bundle.transport.request("initialize", {});
|
||||
await bundle.transport.request("thread/start", { cwd: fixture.workspace, dynamicTools: await fixture.authority.definitions() });
|
||||
await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Find the project" }] });
|
||||
for await (const notification of bundle.transport.notifications()) if (notification.method === "turn/completed") break;
|
||||
expect(results).toHaveLength(2);
|
||||
expect(results[1]).toMatchObject({ status: 200, data: [{ id: fixture.projectId, name: "Aurora" }] });
|
||||
let timeout: ReturnType<typeof setTimeout> | undefined;
|
||||
try {
|
||||
await Promise.race([
|
||||
(async () => { for await (const notification of bundle.transport.notifications()) if (notification.method === "turn/completed") return; })(),
|
||||
new Promise<never>((_, reject) => { timeout = setTimeout(() => reject(new Error("Native provider did not continue after the API tool result")), 10_000); }),
|
||||
]);
|
||||
} finally { clearTimeout(timeout); }
|
||||
expect(results).toHaveLength(3);
|
||||
expect(results[1]).toMatchObject({ status: 200, apiOperationId: "GET /api/companies/{companyId}/projects", data: [{ id: fixture.projectId, name: "Aurora" }] });
|
||||
expect(results[2]).toMatchObject({ ok: false, status: 404 });
|
||||
expect(bundle.evidence().diagnostics).toContain("runnerd authenticated to the durable PRP control plane");
|
||||
} finally { await bundle.transport.close(); }
|
||||
}, 30_000);
|
||||
|
||||
Reference in new issue
Block a user