mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 08:22:20 +02:00
fix(chat): keep permission failures scoped to the rejected operation (#16847)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External channels deliver agent replies through a durable publication queue. > - A provider can reject one operation while other operations remain allowed. > - The shared error classifier treated generic 403 and permission errors as failures of the whole connection. > - Failure handlers then changed the user's enabled setting and removed all assigned channel tools. > - This pull request limits permission failures to the rejected operation and preserves connection intent during authentication failures. > - Unrelated work can continue, while confirmed authentication failures still stop endpoint traffic. ## Linked Issues or Issue Description Refs #16137. Follows #16138, which distinguishes local authorization errors from provider errors. This change also narrows provider permission failures. Related #16806 handles a separate task-continuation tool-binding defect. A provider 403 or missing-scope response currently disables a connection that may still support other operations. The expected behavior is a visible operation failure with unrelated work preserved. Confirmed authentication failures must stop sends without changing the user's enabled setting. ## What Changed - Classify generic 403, adapter permission errors, and Slack missing-scope/no-permission responses as failed operations. - Preserve rate-limit retries, known destination quarantine, and confirmed authentication failures. - Preserve connection status and enabled intent when publication, provider-reply, or form-open authentication fails. The endpoint still enters attention and its runtime stops. - Add GitHub regressions for retained discovery and successful later publication after 403, and no further sends after 401. - Mark authentication health failed and fence persistent receiver startup, background reconciliation, and receiver callbacks. Keep signed GitHub lifecycle events available during an authentication stop. Verify Discord reconciliation and Photon restart preserve the stop, while transient Discord disconnects still recover. - Update provider, publication, and modal regression expectations and the execution contract and verification maps. ## Verification - Red: five new classifier cases returned endpoint attention before the repair. The signed-installation-removal regression also reproduced a blocked lifecycle event after a 401 before narrowing the receiver guard. - Green: all 35 classifier tests pass. - Green: 19 PostgreSQL-backed cases pass through the production channel services, including GitHub discovery/publication, Slack provider replies and question modals, Telegram credential/destination failures, rate limiting, and Discord/Photon authentication-stop persistence. - Map structural checks and `git diff --check` pass. - `pnpm -r typecheck` and `pnpm build` pass. All CI gates are green on `7e379bcd98aa8363e42efe7f8a4b2589b7111ae6`; Greptile is 5/5 on that exact commit with no unresolved findings. - The monolithic local `pnpm test:run` was started, then stopped after its revision passed the full CI matrix and was superseded by the authentication-stop repair. Final full-suite verification passed in CI shards; all 19 final local provider regressions pass. - The tests run from this source checkout on macOS arm64 with isolated PostgreSQL and controlled provider failures. They do not prove live GitHub recovery or deployment. No live connection settings were changed. ## Risks - Permission failures now leave unrelated work available. The rejected operation remains failed and is not automatically replayed. Provider authorization still applies to every request. - Keeping a connection enabled must not bypass an authentication stop. GitHub verifies blocked tool discovery and later sends; Discord and Photon verify background reconciliation and restart cannot clear the stop. Failed connection health is an execution gate while the endpoint requires attention. - Explicit disable, disconnect, signed installation revocation, and existing destination-specific denial handling remain unchanged. No migration or API schema change is required. - Previously disabled connections are not automatically re-enabled because an upgrade cannot safely infer the user's intent. ## Model Used - OpenAI Codex, GPT-6 family. The runtime does not expose the exact deployment model ID or context-window size. Used reasoning, shell tools, repository edits, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
016e40d532
commit
9966f31bee
13 files changed
+182
-49
No files matched your search
@@ -1926,6 +1926,19 @@ when the run fails, before generic recovery retries. Reconnect preserves account
|
|||||||
identity and permissions. Compatible legacy agents may explicitly adopt a
|
identity and permissions. Compatible legacy agents may explicitly adopt a
|
||||||
validated connection inline; late failures must not invalidate newer credentials.
|
validated connection inline; late failures must not invalidate newer credentials.
|
||||||
|
|
||||||
|
### External channel failure scope
|
||||||
|
|
||||||
|
Publication, provider-reply, and form-open permission failures fail the affected
|
||||||
|
operation. An unclassified provider 403 does not establish that the whole
|
||||||
|
connection is unusable. Known destination failures quarantine only that resource;
|
||||||
|
provider rate limits retain their retry deadlines. Confirmed authentication
|
||||||
|
failures put the endpoint in attention and mark connection health failed without
|
||||||
|
changing the connection's enabled setting. Admission and background reconciliation
|
||||||
|
block sends, assigned channel tools, and background receiver restart until
|
||||||
|
credentials are repaired. Signed provider lifecycle events remain admissible so
|
||||||
|
installation revocation still takes effect during an authentication stop.
|
||||||
|
Explicit disable and disconnect retain their existing behavior.
|
||||||
|
|
||||||
### Task-bound email
|
### Task-bound email
|
||||||
|
|
||||||
AgentMail is a default connection and does not require the experimental chat
|
AgentMail is a default connection and does not require the experimental chat
|
||||||
|
|||||||
@@ -651,6 +651,13 @@ Operators must release the effective task or ancestor pause before sending a new
|
|||||||
message. The draft stays intact. This applies to both task interfaces and to
|
message. The draft stays intact. This applies to both task interfaces and to
|
||||||
board comment API requests; an agent may still report interrupted work.
|
board comment API requests; an agent may still report interrupted work.
|
||||||
|
|
||||||
|
### External channel failure scope
|
||||||
|
|
||||||
|
An operation-level permission denial must not disable unrelated channel work.
|
||||||
|
Authentication failures pause runtime use while preserving the user's connection
|
||||||
|
setting. Explicit disconnect and provider installation revocation remain enforced.
|
||||||
|
See the implementation spec for publication, provider-reply, and form-open rules.
|
||||||
|
|
||||||
### Experimental iMessage Photon channel
|
### Experimental iMessage Photon channel
|
||||||
|
|
||||||
A Photon Cloud project can represent one agent through the existing
|
A Photon Cloud project can represent one agent through the existing
|
||||||
|
|||||||
@@ -87,6 +87,15 @@ remaining operations before qualifying the whole capability.
|
|||||||
Gaps: Normal agent replies work through server delivery. Explicit endpoint publication
|
Gaps: Normal agent replies work through server delivery. Explicit endpoint publication
|
||||||
APIs are board management paths.
|
APIs are board management paths.
|
||||||
|
|
||||||
|
Publication, provider-reply, and form-open 403/permission errors fail the rejected
|
||||||
|
operation without removing unrelated assigned channel tools. Confirmed
|
||||||
|
authentication failures still put the endpoint in attention, which blocks tool
|
||||||
|
admission and provider sends; connection enabled intent is retained. Known
|
||||||
|
resource denials and explicit revocation keep their narrower or installation-wide
|
||||||
|
stops. PostgreSQL provider fixtures verify discovery after 403 and rejection after
|
||||||
|
401, including signed installation removal after a 401; they do not establish
|
||||||
|
live GitHub credential recovery.
|
||||||
|
|
||||||
GitHub publication recovery (2026-10-14): assigned-agent `submit_review`, `comment`,
|
GitHub publication recovery (2026-10-14): assigned-agent `submit_review`, `comment`,
|
||||||
and `update_comment` calls reuse their run-, task-, endpoint-, and argument-bound
|
and `update_comment` calls reuse their run-, task-, endpoint-, and argument-bound
|
||||||
publication on an exact retry. Fresh gateway policy and provider authority remain
|
publication on an exact retry. Fresh gateway policy and provider authority remain
|
||||||
|
|||||||
@@ -81,6 +81,15 @@ Automated: [channel integration](../../server/src/__tests__/chat-channels.integr
|
|||||||
|
|
||||||
Manual: Inspect the bound agent and identity, recent activity, and delivery failure details. Repair a disposable credential, retry a safe message when offered, and verify exactly one external reply. Disable the endpoint and confirm new traffic is not silently processed.
|
Manual: Inspect the bound agent and identity, recent activity, and delivery failure details. Repair a disposable credential, retry a safe message when offered, and verify exactly one external reply. Disable the endpoint and confirm new traffic is not silently processed.
|
||||||
|
|
||||||
|
For publication failures, reject one operation with a provider 403 and verify
|
||||||
|
that its failure remains visible while unrelated tools and publications continue.
|
||||||
|
Repeat with an authentication failure: the endpoint must require attention and
|
||||||
|
stop provider traffic, while the connection's enabled setting remains unchanged.
|
||||||
|
Reconcile persistent receivers and restart the service; authentication failures
|
||||||
|
must remain stopped while transient disconnections can still recover.
|
||||||
|
The PostgreSQL suite uses controlled provider failures for these cases; live
|
||||||
|
provider credential repair remains a separate acceptance check.
|
||||||
|
|
||||||
For GitHub, submit a disposable PR and verify the summary heading uses the bot's
|
For GitHub, submit a disposable PR and verify the summary heading uses the bot's
|
||||||
display name and the check remains **Paperclip Review**. For two separate bots on one PR, verify
|
display name and the check remains **Paperclip Review**. For two separate bots on one PR, verify
|
||||||
their App-scoped checks and comments remain independent.
|
their App-scoped checks and comments remain independent.
|
||||||
|
|||||||
@@ -2786,6 +2786,49 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
|||||||
await expect(gateway.executeTool({ sessionToken: session.token, tool: tool.name, parameters: assessment, idempotencyKey })).rejects.toThrow();
|
await expect(gateway.executeTool({ sessionToken: session.token, tool: tool.name, parameters: assessment, idempotencyKey })).rejects.toThrow();
|
||||||
expect(summaryAttempts).toBe(2);
|
expect(summaryAttempts).toBe(2);
|
||||||
}, 25_000);
|
}, 25_000);
|
||||||
|
it.each([403, 401])("preserves GitHub connection intent after publication HTTP %s", async status => {
|
||||||
|
const f = await toolReplyFixture();
|
||||||
|
const initialEndpoint = await f.service.get(f.endpoint.id);
|
||||||
|
const toolsBefore = await githubBotToolsForSession(db, f.session);
|
||||||
|
expect(toolsBefore).toHaveLength(7);
|
||||||
|
const runtime = f.runtime.endpoints.get(f.endpoint.id)!;
|
||||||
|
runtime.postError = Object.assign(new Error("Provider rejected publication"), { status });
|
||||||
|
const comment = await issueService(db).addComment(f.conversation.issueId!, "Rejected response",
|
||||||
|
{ userId: "owner-user" }, { authorType: "user" });
|
||||||
|
await f.service.publishComment(f.endpoint.id, f.conversation.id, comment.id);
|
||||||
|
expect(await readChatPublicationsByCommentId(comment.id)).toEqual([
|
||||||
|
expect.objectContaining({ state: "failed", attempts: 1, nextAttemptAt: null }),
|
||||||
|
]);
|
||||||
|
expect((await readToolConnectionsById(f.endpoint.connectionId))[0]).toMatchObject({
|
||||||
|
status: "active", enabled: true, healthStatus: status === 401 ? "failed" : "healthy",
|
||||||
|
});
|
||||||
|
expect(await f.service.get(f.endpoint.id)).toMatchObject({ status: status === 401 ? "attention" : initialEndpoint.status });
|
||||||
|
expect(await githubBotToolsForSession(db, f.session)).toEqual(status === 401 ? [] : toolsBefore);
|
||||||
|
runtime.postError = null;
|
||||||
|
const next = await issueService(db).addComment(f.conversation.issueId!, "Independent response",
|
||||||
|
{ userId: "owner-user" }, { authorType: "user" });
|
||||||
|
const writesBefore = runtime.posts.length;
|
||||||
|
await f.service.publishComment(f.endpoint.id, f.conversation.id, next.id);
|
||||||
|
if (status === 403) {
|
||||||
|
expect((await readChatPublicationsByCommentId(next.id))[0]).toMatchObject({ state: "published" });
|
||||||
|
expect(runtime.posts.length).toBe(writesBefore + 1);
|
||||||
|
} else {
|
||||||
|
expect(runtime.posts).toHaveLength(writesBefore);
|
||||||
|
expect(f.runtime.endpoints.has(f.endpoint.id)).toBe(false);
|
||||||
|
f.setInstallationAvailable(false);
|
||||||
|
const deleted = await f.service.handleWebhook(f.endpoint.publicId, "github", signedGitHubWebhookRequest({
|
||||||
|
event: "installation", delivery: randomUUID(), webhookSecret: f.webhookSecret,
|
||||||
|
payload: { action: "deleted", installation: { id: 2468 } },
|
||||||
|
}));
|
||||||
|
expect(deleted.status).toBe(202);
|
||||||
|
expect(await f.service.get(f.endpoint.id)).toMatchObject({ status: "revoked" });
|
||||||
|
expect((await readToolConnectionsById(f.endpoint.connectionId))[0]).toMatchObject({ status: "disabled", enabled: false });
|
||||||
|
expect(await f.service.listResources(f.endpoint.id)).toEqual(expect.arrayContaining([
|
||||||
|
expect.objectContaining({ availability: "unavailable" }),
|
||||||
|
]));
|
||||||
|
expect((await readChatConversationsById(f.conversation.id))[0]).toMatchObject({ state: "unavailable" });
|
||||||
|
}
|
||||||
|
});
|
||||||
it("lists each GitHub bot tool once when an inline thread shares the task", async () => {
|
it("lists each GitHub bot tool once when an inline thread shares the task", async () => {
|
||||||
const f = await toolReplyFixture("github:paperclipai/paperclip:418");
|
const f = await toolReplyFixture("github:paperclipai/paperclip:418");
|
||||||
const { id: _id, ...conversation } = f.conversation;
|
const { id: _id, ...conversation } = f.conversation;
|
||||||
@@ -26551,6 +26594,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
|||||||
.from(chatEndpoints)
|
.from(chatEndpoints)
|
||||||
.where(eq(chatEndpoints.id, endpoint.id));
|
.where(eq(chatEndpoints.id, endpoint.id));
|
||||||
expect(currentEndpoint?.status).toBe("attention");
|
expect(currentEndpoint?.status).toBe("attention");
|
||||||
|
expect((await readToolConnectionsById(endpoint.connectionId))[0]).toMatchObject({
|
||||||
|
status: "active", enabled: true, healthStatus: "failed",
|
||||||
|
});
|
||||||
} else {
|
} else {
|
||||||
const [resource] = await db
|
const [resource] = await db
|
||||||
.select({ availability: chatEndpointResources.availability })
|
.select({ availability: chatEndpointResources.availability })
|
||||||
@@ -27857,7 +27903,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
|||||||
}),
|
}),
|
||||||
expectedAttempts: 1,
|
expectedAttempts: 1,
|
||||||
expectedDiagnostic:
|
expectedDiagnostic:
|
||||||
/Receipt reaction failed after 1 attempt \(endpoint_attention\): GitHub reaction forbidden/,
|
/Receipt reaction failed after 1 attempt \(failed\): GitHub reaction forbidden/,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
label: "retries a GitHub 429 receipt reaction with bounded backoff",
|
label: "retries a GitHub 429 receipt reaction with bounded backoff",
|
||||||
@@ -39564,6 +39610,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
|||||||
expectedResourceAvailability: "available",
|
expectedResourceAvailability: "available",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
provider: "slack",
|
||||||
label: "authentication failures as endpoint attention",
|
label: "authentication failures as endpoint attention",
|
||||||
error: Object.assign(new Error("provider token expired"), {
|
error: Object.assign(new Error("provider token expired"), {
|
||||||
status: 401,
|
status: 401,
|
||||||
@@ -39573,6 +39620,15 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
|||||||
expectedConversationState: "active",
|
expectedConversationState: "active",
|
||||||
expectedResourceAvailability: "available",
|
expectedResourceAvailability: "available",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
provider: "discord",
|
||||||
|
label: "Discord authentication failures as endpoint attention",
|
||||||
|
error: Object.assign(new Error("provider token expired"), { status: 401 }),
|
||||||
|
expectedPublicationState: "failed",
|
||||||
|
expectedEndpointStatus: "attention",
|
||||||
|
expectedConversationState: "active",
|
||||||
|
expectedResourceAvailability: "available",
|
||||||
|
},
|
||||||
{
|
{
|
||||||
label: "missing destinations as resource unavailable",
|
label: "missing destinations as resource unavailable",
|
||||||
error: Object.assign(new Error("provider destination missing"), {
|
error: Object.assign(new Error("provider destination missing"), {
|
||||||
@@ -39629,6 +39685,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
|||||||
])(
|
])(
|
||||||
"classifies publication $label",
|
"classifies publication $label",
|
||||||
async ({
|
async ({
|
||||||
|
provider = "slack",
|
||||||
error,
|
error,
|
||||||
expectedConversationState,
|
expectedConversationState,
|
||||||
expectedEndpointStatus,
|
expectedEndpointStatus,
|
||||||
@@ -39637,20 +39694,21 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
|||||||
}) => {
|
}) => {
|
||||||
const fixture = await seedCompany();
|
const fixture = await seedCompany();
|
||||||
const { callbacks, endpoint, runtime, service } =
|
const { callbacks, endpoint, runtime, service } =
|
||||||
await configuredSlackEndpoint(fixture);
|
await (provider === "discord" ? configuredDiscordEndpoint(fixture) : configuredSlackEndpoint(fixture));
|
||||||
try {
|
try {
|
||||||
const channelId = `C-PUBLICATION-ERROR-${randomUUID().slice(0, 8)}`;
|
const channelId = provider === "discord" ? "333333333333333329" : `C-PUBLICATION-ERROR-${randomUUID().slice(0, 8)}`;
|
||||||
const channel = makeThread({
|
const channel = makeThread({
|
||||||
channelId,
|
channelId,
|
||||||
id: `slack:${channelId}:4200.1`,
|
id: provider === "discord" ? `discord:1457808928258658549:${channelId}:555555555555555596` : `slack:${channelId}:4200.1`,
|
||||||
name: "publication-errors",
|
name: "publication-errors",
|
||||||
});
|
});
|
||||||
await deliverMessage({
|
await deliverMessage({
|
||||||
callbacks,
|
callbacks,
|
||||||
endpointId: endpoint.id,
|
endpointId: endpoint.id,
|
||||||
|
provider: provider as "slack" | "discord",
|
||||||
thread: channel.thread,
|
thread: channel.thread,
|
||||||
message: makeMessage({
|
message: makeMessage({
|
||||||
id: "4200.1",
|
id: provider === "discord" ? "555555555555555596" : "4200.1",
|
||||||
text: "@maya exercise provider failure handling",
|
text: "@maya exercise provider failure handling",
|
||||||
mentioned: true,
|
mentioned: true,
|
||||||
}),
|
}),
|
||||||
@@ -39698,10 +39756,13 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
|||||||
expect(runtime.endpoints.has(endpoint.id)).toBe(false);
|
expect(runtime.endpoints.has(endpoint.id)).toBe(false);
|
||||||
const [connection] = await readToolConnectionsById(endpoint.connectionId);
|
const [connection] = await readToolConnectionsById(endpoint.connectionId);
|
||||||
expect(connection).toMatchObject({
|
expect(connection).toMatchObject({
|
||||||
status: "disabled",
|
status: "active",
|
||||||
enabled: false,
|
enabled: true,
|
||||||
healthStatus: "degraded",
|
healthStatus: "failed",
|
||||||
});
|
});
|
||||||
|
await service.reconcileProviderRuntimes();
|
||||||
|
expect(runtime.endpoints.has(endpoint.id)).toBe(false);
|
||||||
|
expect((await service.get(endpoint.id)).status).toBe("attention");
|
||||||
}
|
}
|
||||||
} finally {
|
} finally {
|
||||||
await retirePublicationFixture(service, endpoint.id);
|
await retirePublicationFixture(service, endpoint.id);
|
||||||
@@ -39743,9 +39804,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
|||||||
expectedEndpointStatus: "attention",
|
expectedEndpointStatus: "attention",
|
||||||
expectedConversationState: "active",
|
expectedConversationState: "active",
|
||||||
expectedConnection: {
|
expectedConnection: {
|
||||||
status: "disabled",
|
status: "active",
|
||||||
enabled: false,
|
enabled: true,
|
||||||
healthStatus: "degraded",
|
healthStatus: "failed",
|
||||||
},
|
},
|
||||||
expectedResourceAvailability: "available",
|
expectedResourceAvailability: "available",
|
||||||
runtimeRetained: false,
|
runtimeRetained: false,
|
||||||
@@ -41821,7 +41882,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
|||||||
failedOpen.event.openModal = vi.fn(async () => {
|
failedOpen.event.openModal = vi.fn(async () => {
|
||||||
throw Object.assign(new Error("Slack Web API rejected modal"), {
|
throw Object.assign(new Error("Slack Web API rejected modal"), {
|
||||||
code: "slack_webapi_platform_error",
|
code: "slack_webapi_platform_error",
|
||||||
data: { error: "expired_trigger_id", ok: false },
|
data: { error: "missing_scope", ok: false },
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
await callbacks.onAction(
|
await callbacks.onAction(
|
||||||
|
|||||||
@@ -45,6 +45,7 @@ import {
|
|||||||
chatEndpointResources,
|
chatEndpointResources,
|
||||||
issueAttachments,
|
issueAttachments,
|
||||||
assets,
|
assets,
|
||||||
|
toolConnections,
|
||||||
} from "@paperclipai/db";
|
} from "@paperclipai/db";
|
||||||
import { startEmbeddedPostgresTestDatabase } from "../helpers/embedded-postgres.js";
|
import { startEmbeddedPostgresTestDatabase } from "../helpers/embedded-postgres.js";
|
||||||
import {
|
import {
|
||||||
@@ -363,6 +364,29 @@ describe("iMessage Photon channel control plane", () => {
|
|||||||
},
|
},
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
it("keeps publication authentication failures stopped across receiver reconciliation and restart", async () => {
|
||||||
|
const t = await setup();
|
||||||
|
const conversation = await t.start();
|
||||||
|
await t.qualify();
|
||||||
|
const callbacks = t.callbacks();
|
||||||
|
vi.spyOn(PhotonChatAdapter.prototype, "publish").mockRejectedValueOnce(new PhotonError("credentials", "Rejected credentials"));
|
||||||
|
const comment = await issueService(db).addComment(conversation.issueId!, "Response with rejected credentials", { userId: t.userId }, { authorType: "user" });
|
||||||
|
await t.service.publishComment(t.endpoint.id, conversation.id, comment.id);
|
||||||
|
expect(await db.select().from(chatPublications).where(eq(chatPublications.commentId, comment.id))).toEqual([
|
||||||
|
expect.objectContaining({ state: "failed", attempts: 1, nextAttemptAt: null }),
|
||||||
|
]);
|
||||||
|
expect(await t.service.get(t.endpoint.id)).toMatchObject({ status: "attention" });
|
||||||
|
expect(await db.select().from(toolConnections).where(eq(toolConnections.id, t.endpoint.connectionId))).toEqual([
|
||||||
|
expect.objectContaining({ status: "active", enabled: true, healthStatus: "failed" }),
|
||||||
|
]);
|
||||||
|
await expect(callbacks.onPhotonAssertOwned!()).rejects.toThrow();
|
||||||
|
await t.service.reconcileProviderRuntimes();
|
||||||
|
expect(t.service.runtime.get(t.endpoint.id)).toBeFalsy();
|
||||||
|
const restarted = await t.restart();
|
||||||
|
expect(restarted.runtime.get(t.endpoint.id)).toBeFalsy();
|
||||||
|
expect(await restarted.get(t.endpoint.id)).toMatchObject({ status: "attention" });
|
||||||
|
});
|
||||||
|
|
||||||
it("supports shared project DMs while rejecting groups, duplicate ownership and allocation changes", async () => {
|
it("supports shared project DMs while rejecting groups, duplicate ownership and allocation changes", async () => {
|
||||||
const t = await setup(true);
|
const t = await setup(true);
|
||||||
const snapshot = await t.service.get(t.endpoint.id);
|
const snapshot = await t.service.get(t.endpoint.id);
|
||||||
|
|||||||
@@ -576,7 +576,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function runtimeContextForRecord(
|
function runtimeContextForRecord(
|
||||||
record: NonNullable<Awaited<ReturnType<typeof endpointRecord>>>,
|
record: Pick<NonNullable<Awaited<ReturnType<typeof endpointRecord>>>, "endpoint" | "credentialSecretRefs">,
|
||||||
): RuntimeContext {
|
): RuntimeContext {
|
||||||
const generation = runtimeGeneration(record.endpoint.setup);
|
const generation = runtimeGeneration(record.endpoint.setup);
|
||||||
const refsFingerprint = credentialFingerprint(record.credentialSecretRefs);
|
const refsFingerprint = credentialFingerprint(record.credentialSecretRefs);
|
||||||
@@ -665,6 +665,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
|||||||
endpoint: chatEndpoints,
|
endpoint: chatEndpoints,
|
||||||
assignedAgentName: agents.name,
|
assignedAgentName: agents.name,
|
||||||
connectionName: toolConnections.name,
|
connectionName: toolConnections.name,
|
||||||
|
connectionHealthStatus: toolConnections.healthStatus,
|
||||||
applicationId: toolConnections.applicationId,
|
applicationId: toolConnections.applicationId,
|
||||||
credentialSecretRefs: toolConnections.credentialSecretRefs,
|
credentialSecretRefs: toolConnections.credentialSecretRefs,
|
||||||
})
|
})
|
||||||
@@ -775,8 +776,8 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
|||||||
runtimeGeneration(endpoint.setup) !== context.generation
|
runtimeGeneration(endpoint.setup) !== context.generation
|
||||||
)
|
)
|
||||||
return null;
|
return null;
|
||||||
const refs = await tx
|
const connection = await tx
|
||||||
.select({ refs: toolConnections.credentialSecretRefs })
|
.select({ refs: toolConnections.credentialSecretRefs, healthStatus: toolConnections.healthStatus })
|
||||||
.from(toolConnections)
|
.from(toolConnections)
|
||||||
.where(
|
.where(
|
||||||
and(
|
and(
|
||||||
@@ -784,8 +785,11 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
|||||||
eq(toolConnections.id, endpoint.connectionId),
|
eq(toolConnections.id, endpoint.connectionId),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
.then((rows) => rows[0]?.refs ?? []);
|
.then((rows) => rows[0]);
|
||||||
if (credentialFingerprint(refs) !== context.credentialFingerprint) {
|
if (
|
||||||
|
(leasedChatProvider(endpoint.provider) && endpoint.status === "attention" && connection?.healthStatus === "failed") ||
|
||||||
|
credentialFingerprint(connection?.refs ?? []) !== context.credentialFingerprint
|
||||||
|
) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
const runtimeContext = context as RuntimeContext;
|
const runtimeContext = context as RuntimeContext;
|
||||||
@@ -813,6 +817,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
return record &&
|
return record &&
|
||||||
|
!(leasedChatProvider(record.endpoint.provider) && record.endpoint.status === "attention" && record.connectionHealthStatus === "failed") &&
|
||||||
allowedStatuses.includes(record.endpoint.status) &&
|
allowedStatuses.includes(record.endpoint.status) &&
|
||||||
runtimeGeneration(record.endpoint.setup) === context.generation &&
|
runtimeGeneration(record.endpoint.setup) === context.generation &&
|
||||||
credentialFingerprint(record.credentialSecretRefs) ===
|
credentialFingerprint(record.credentialSecretRefs) ===
|
||||||
@@ -898,7 +903,8 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
|||||||
if (
|
if (
|
||||||
record.endpoint.status === "archived" ||
|
record.endpoint.status === "archived" ||
|
||||||
record.endpoint.status === "paused" ||
|
record.endpoint.status === "paused" ||
|
||||||
record.endpoint.status === "draft"
|
record.endpoint.status === "draft" ||
|
||||||
|
(leasedChatProvider(record.endpoint.provider) && record.endpoint.status === "attention" && record.connectionHealthStatus === "failed")
|
||||||
) {
|
) {
|
||||||
throw conflict("Chat endpoint runtime is not available in this state", {
|
throw conflict("Chat endpoint runtime is not available in this state", {
|
||||||
code: "chat_endpoint_runtime_unavailable",
|
code: "chat_endpoint_runtime_unavailable",
|
||||||
@@ -1140,6 +1146,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
|||||||
inArray(chatEndpoints.status, ["verifying", "active", "attention"]),
|
inArray(chatEndpoints.status, ["verifying", "active", "attention"]),
|
||||||
eq(toolConnections.status, "active"),
|
eq(toolConnections.status, "active"),
|
||||||
eq(toolConnections.enabled, true),
|
eq(toolConnections.enabled, true),
|
||||||
|
or(ne(chatEndpoints.status, "attention"), ne(toolConnections.healthStatus, "failed")),
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
const eligibleIds = new Set(eligibleRows.map((row) => row.endpointId));
|
const eligibleIds = new Set(eligibleRows.map((row) => row.endpointId));
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ export function createPhotonRuntimeCallbacks(db: Db, options: PhotonRuntimeOptio
|
|||||||
const current = await runtimeCallbackEndpoint(tx, endpointId, context, ["attention"]);
|
const current = await runtimeCallbackEndpoint(tx, endpointId, context, ["attention"]);
|
||||||
if (!current) return;
|
if (!current) return;
|
||||||
const connection = await tx.select().from(toolConnections).where(eq(toolConnections.id, current.connectionId)).then((rows) => rows[0]);
|
const connection = await tx.select().from(toolConnections).where(eq(toolConnections.id, current.connectionId)).then((rows) => rows[0]);
|
||||||
if (!connection?.enabled) throw new Error("Photon endpoint requires operator recovery");
|
if (!connection?.enabled || connection.healthStatus === "failed") throw new Error("Photon endpoint requires operator recovery");
|
||||||
await notifyChatEndpointWork(tx, connectionId);
|
await notifyChatEndpointWork(tx, connectionId);
|
||||||
await tx.update(chatEndpoints).set({ status: current.setup.step === "complete" ? "active" : "verifying", healthMessage: "Photon receiver connected", lastError: null, updatedAt: new Date() }).where(eq(chatEndpoints.id, endpointId));
|
await tx.update(chatEndpoints).set({ status: current.setup.step === "complete" ? "active" : "verifying", healthMessage: "Photon receiver connected", lastError: null, updatedAt: new Date() }).where(eq(chatEndpoints.id, endpointId));
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -423,9 +423,7 @@ export function createChatInteractionCallbacks(db: Db, options: InteractionCallb
|
|||||||
await tx
|
await tx
|
||||||
.update(toolConnections)
|
.update(toolConnections)
|
||||||
.set({
|
.set({
|
||||||
status: "disabled",
|
healthStatus: "failed",
|
||||||
enabled: false,
|
|
||||||
healthStatus: "degraded",
|
|
||||||
healthMessage:
|
healthMessage:
|
||||||
"Provider credentials or permissions need attention",
|
"Provider credentials or permissions need attention",
|
||||||
lastError: failure,
|
lastError: failure,
|
||||||
|
|||||||
@@ -328,9 +328,7 @@ export function createProviderEffects(db: Db, options: ProviderEffectsOptions) {
|
|||||||
await tx
|
await tx
|
||||||
.update(toolConnections)
|
.update(toolConnections)
|
||||||
.set({
|
.set({
|
||||||
status: "disabled",
|
healthStatus: "failed",
|
||||||
enabled: false,
|
|
||||||
healthStatus: "degraded",
|
|
||||||
healthMessage: "Provider credentials or permissions need attention",
|
healthMessage: "Provider credentials or permissions need attention",
|
||||||
lastError: failure,
|
lastError: failure,
|
||||||
healthCheckedAt: new Date(),
|
healthCheckedAt: new Date(),
|
||||||
|
|||||||
@@ -200,9 +200,7 @@ export function createChatPublicationFailures(db: Db, options: PublicationFailur
|
|||||||
await tx
|
await tx
|
||||||
.update(toolConnections)
|
.update(toolConnections)
|
||||||
.set({
|
.set({
|
||||||
status: "disabled",
|
healthStatus: "failed",
|
||||||
enabled: false,
|
|
||||||
healthStatus: "degraded",
|
|
||||||
healthMessage: "Provider credentials or permissions need attention",
|
healthMessage: "Provider credentials or permissions need attention",
|
||||||
lastError: failure,
|
lastError: failure,
|
||||||
healthCheckedAt: new Date(),
|
healthCheckedAt: new Date(),
|
||||||
|
|||||||
@@ -143,7 +143,17 @@ describe("chat publication error classification", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
it("does not mistake an ordinary GitHub permission denial for rate limiting", () => {
|
it.each([
|
||||||
|
Object.assign(new Error("Operation denied"), { status: 403 }),
|
||||||
|
new HttpError(403, "Operation denied", { providerStatus: 403 }),
|
||||||
|
providerError("PermissionError", "PERMISSION_DENIED"),
|
||||||
|
Object.assign(new Error("Missing scope"), { data: { error: "missing_scope" } }),
|
||||||
|
Object.assign(new Error("No permission"), { data: { error: "no_permission" } }),
|
||||||
|
])("keeps permission rejection scoped to its operation: %s", (error) => {
|
||||||
|
expect(classifyChatPublicationError(error, 1)).toMatchObject({ kind: "failed" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("fails an ordinary GitHub permission denial without pausing the endpoint", () => {
|
||||||
expect(
|
expect(
|
||||||
classifyChatPublicationError(
|
classifyChatPublicationError(
|
||||||
Object.assign(new Error("Resource not accessible by integration"), {
|
Object.assign(new Error("Resource not accessible by integration"), {
|
||||||
@@ -153,7 +163,7 @@ describe("chat publication error classification", () => {
|
|||||||
}),
|
}),
|
||||||
1,
|
1,
|
||||||
),
|
),
|
||||||
).toMatchObject({ kind: "endpoint_attention" });
|
).toMatchObject({ kind: "failed" });
|
||||||
});
|
});
|
||||||
|
|
||||||
it("keeps Paperclip's own authorization failures scoped to the publication", () => {
|
it("keeps Paperclip's own authorization failures scoped to the publication", () => {
|
||||||
@@ -174,7 +184,7 @@ describe("chat publication error classification", () => {
|
|||||||
}),
|
}),
|
||||||
1,
|
1,
|
||||||
),
|
),
|
||||||
).toMatchObject({ kind: "endpoint_attention" });
|
).toMatchObject({ kind: "failed" });
|
||||||
});
|
});
|
||||||
|
|
||||||
it("does not call explicit GitHub 4xx rejections ambiguous delivery", () => {
|
it("does not call explicit GitHub 4xx rejections ambiguous delivery", () => {
|
||||||
@@ -216,13 +226,13 @@ describe("chat publication error classification", () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it.each([
|
it.each([
|
||||||
["AuthenticationError", "AUTH_FAILED"],
|
["AuthenticationError", "AUTH_FAILED", "endpoint_attention"],
|
||||||
["PermissionError", "PERMISSION_DENIED"],
|
["PermissionError", "PERMISSION_DENIED", "failed"],
|
||||||
])("moves definite %s rejections to repair", (name, code) => {
|
])("scopes definite %s rejections", (name, code, kind) => {
|
||||||
expect(
|
expect(
|
||||||
classifyChatPublicationError(providerError(name, code), 1),
|
classifyChatPublicationError(providerError(name, code), 1),
|
||||||
).toMatchObject({
|
).toMatchObject({
|
||||||
kind: "endpoint_attention",
|
kind,
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -275,7 +285,7 @@ describe("chat publication error classification", () => {
|
|||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
it("keeps Teams authentication and generic permission failures endpoint-scoped", () => {
|
it("distinguishes Teams authentication from operation permission failures", () => {
|
||||||
expect(
|
expect(
|
||||||
classifyChatPublicationError(
|
classifyChatPublicationError(
|
||||||
providerError("AuthenticationError", "AUTH_FAILED", {
|
providerError("AuthenticationError", "AUTH_FAILED", {
|
||||||
@@ -295,7 +305,7 @@ describe("chat publication error classification", () => {
|
|||||||
}),
|
}),
|
||||||
1,
|
1,
|
||||||
),
|
),
|
||||||
).toMatchObject({ kind: "endpoint_attention" });
|
).toMatchObject({ kind: "failed" });
|
||||||
});
|
});
|
||||||
|
|
||||||
it.each([50001, 50013])(
|
it.each([50001, 50013])(
|
||||||
@@ -324,7 +334,7 @@ describe("chat publication error classification", () => {
|
|||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
it("keeps Discord token and generic app permission failures endpoint-scoped", () => {
|
it("distinguishes Discord token and generic app permission failures", () => {
|
||||||
for (const { status, providerCode } of [
|
for (const { status, providerCode } of [
|
||||||
{ status: 401, providerCode: 0 },
|
{ status: 401, providerCode: 0 },
|
||||||
{ status: 403, providerCode: 20012 },
|
{ status: 403, providerCode: 20012 },
|
||||||
@@ -345,7 +355,7 @@ describe("chat publication error classification", () => {
|
|||||||
}),
|
}),
|
||||||
1,
|
1,
|
||||||
),
|
),
|
||||||
).toMatchObject({ kind: "endpoint_attention" });
|
).toMatchObject({ kind: status === 401 ? "endpoint_attention" : "failed" });
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -276,11 +276,6 @@ export function classifyChatPublicationError(
|
|||||||
}
|
}
|
||||||
|
|
||||||
if (
|
if (
|
||||||
// Discord distinguishes a destination the bot cannot access (50001) or
|
|
||||||
// cannot write to (50013) from invalid credentials and app-wide failures.
|
|
||||||
// The pinned adapter preserves the bounded numeric API code on its nested
|
|
||||||
// DiscordApiError. Quarantine only the affected channel/conversation;
|
|
||||||
// generic Discord 403s and every 401 remain endpoint-scoped below.
|
|
||||||
adapters.includes("discord") &&
|
adapters.includes("discord") &&
|
||||||
status === 403 &&
|
status === 403 &&
|
||||||
(discordProviderCode === 50001 || discordProviderCode === 50013)
|
(discordProviderCode === 50001 || discordProviderCode === 50013)
|
||||||
@@ -330,23 +325,27 @@ export function classifyChatPublicationError(
|
|||||||
|
|
||||||
if (
|
if (
|
||||||
names.includes("AuthenticationError") ||
|
names.includes("AuthenticationError") ||
|
||||||
names.includes("PermissionError") ||
|
|
||||||
codes.includes("AUTH_FAILED") ||
|
codes.includes("AUTH_FAILED") ||
|
||||||
codes.includes("PERMISSION_DENIED") ||
|
|
||||||
status === 401 ||
|
status === 401 ||
|
||||||
status === 403 ||
|
|
||||||
[
|
[
|
||||||
"account_inactive",
|
"account_inactive",
|
||||||
"invalid_auth",
|
"invalid_auth",
|
||||||
"missing_scope",
|
|
||||||
"not_authed",
|
"not_authed",
|
||||||
"no_permission",
|
|
||||||
"token_revoked",
|
"token_revoked",
|
||||||
].some((platformCode) => platformCodes.includes(platformCode))
|
].some((platformCode) => platformCodes.includes(platformCode))
|
||||||
) {
|
) {
|
||||||
return { kind: "endpoint_attention", reason };
|
return { kind: "endpoint_attention", reason };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (
|
||||||
|
names.includes("PermissionError") ||
|
||||||
|
codes.includes("PERMISSION_DENIED") ||
|
||||||
|
status === 403 ||
|
||||||
|
["missing_scope", "no_permission"].some((platformCode) => platformCodes.includes(platformCode))
|
||||||
|
) {
|
||||||
|
return { kind: "failed", reason };
|
||||||
|
}
|
||||||
|
|
||||||
if (
|
if (
|
||||||
names.includes("ResourceNotFoundError") ||
|
names.includes("ResourceNotFoundError") ||
|
||||||
codes.includes("NOT_FOUND") ||
|
codes.includes("NOT_FOUND") ||
|
||||||
|
|||||||
Reference in new issue
Block a user