Recover Muse verification after contact, restart and access restoration (#19909)

## Thinking Path

> - Paperclip manages AI agents and their work under company access
rules.
> - Muse readiness uses a saved connection, receiver contact and an
authenticated challenge reply.
> - A failed challenge write could leave a saved contact with no later
recovery trigger.
> - A server restart or restored access must recover that contact under
current authority.
> - This pull request uses the existing delivery coordinator and saved
Muse state for recovery.
> - It keeps the normal receiver schedule, authorization checks and real
deadlines.
> - It lets eligible verification recover and lets idle verification
stop work.

## Linked Issues or Issue Description

**What happened?**

The first Muse receiver contact could persist while readiness challenge
creation failed. The contact then had no durable retry or startup
recovery path. Later membership, lifecycle or feature restoration could
also leave pending verification without a notice.

**Expected behavior**

Retry a temporary failure through the existing worker. Recover pending
contact after restart. Recheck the current company, agent, operator,
credential and binding generation before issuing a challenge. Stop
verification work while it is ineligible or complete. Do not revive
revoked credentials.

**Steps to reproduce**

1. Use the normal invitation and OAuth registration, PKCE and token
flow.
2. Send one receiver contact and let its normal persistence schedule
run.
3. Fail the first readiness challenge operation before it commits.
4. Observe whether the existing worker retries without another contact
or a manual Verify action.
5. Restart the App with a new database owner. Check that it creates one
usable challenge from the saved contact.

**Paperclip version or commit**

The causal baseline test used the unchanged production implementation at
`90148317b2a447794ff29a4acae5648ad11a9770` plus its normal fixture. The
accepted positive test used `9d383041028776ad8469ac4af3fe14ba6c369425`.
The initial published head was
`fe518c9b72504daf6374225987f24d45bc7bcf75`. The lease/auth correction
was locally qualified at `cfd515a207416097fe8b586277d70c4fd4a679fa`. The
current candidate is `6efe722e7eb1e22acd466125c7b4a627611d0273`; its
added Cloud actor fixture correction passed the full normal owning
program and all 35 tests. Its current normal build, lifecycle owner
controls, healthy-wait and restoration measurements are described below.
Earlier results below are bound to their stated revisions.

Related public changes: Refs #19819 for original-principal
authentication. Refs #19750 for the transaction-owned external feature
read. These are related foundations, not duplicate fixes. The current
Muse title and head-branch searches found no duplicate
contact-verification recovery PR. Related maintenance work is #17000 and
terminal assignment settlement is #17793.

## What Changed

- Add a typed, exact-binding verification notice through the existing
Muse delivery queue.
- Register the first-contact notice before its durable write in the same
transaction.
- Recover pending contact from saved state at startup. Retry temporary
failures through the existing scheduler.
- Recheck current feature, lifecycle, credential and access state after
commit. Park unavailable authority without recurring verification scans.
- Add narrowly scoped notices to the existing access, lifecycle,
configuration, company, budget and settings write owners.
- Save a healthy lifecycle verification wait in the existing operation
JSONB. Keep generic unresolved-work and error retries unchanged.
- Commit authenticated confirmation and its lifecycle recheck in one
transaction. Retain a raced recheck under the active lease, and recover
saved waits once at startup.
- Retain a committed lifecycle recheck while an expired attempt still
owns the row, so its late wait cannot park accepted confirmation.
Replacement claims keep their existing owner fence and fresh fact read.
- Keep the receiver gap, persistence and per-segment contact limits
unchanged.
- Add real App OAuth, failure, retry, restart, challenge and quiet
evidence. Add two normal-route restoration tests.
- Document recovery, reconnect requirements and the existing
connected-receiver sleep restriction.

## Verification

- Causal baseline: the real invitation and OAuth flow saved one contact.
The first challenge operation failed. The normal test failed because it
observed one verification attempt instead of a retry. No setup or
cleanup failure was counted as this regression.
- At `9d383041`: normal own Shared/SDK, DB with migration 0441 assets,
Runner TypeScript core and surfaces, module boundaries, and all seven
normal App programs passed. One new redundant role comparison first
failed typecheck; that diagnostic remains recorded and the corrected
full App gate passed.
- At `9d383041`: all 17 maintenance component tests passed with zero
skips. All original deadline, rollback, drain, terminal settlement and
startup controls remain.
- At `9d383041`: the one normal App case passed. Real OAuth authorized
one receiver signal. A retry occurred about 5.01 seconds after the first
failure. A new Db/App owner created one challenge, advanced revision 2
to 3, and accepted authenticated confirmation and mailbox ACK. Query
instrumentation had an active positive control. All measured query,
callback, session and verifier counters stayed equal across the
unchanged scheduled 15-second quiet window. Token revocation and
old-token/signal rejection passed.
- Known App, worker, HTTP and database owners closed without cleanup
errors. The tests used bounded local PostgreSQL and HTTP fixtures. They
did not call a live Muse model.
- Current integrated semantics: the full normal App driver completed all
programs. Only a new test callback had two undefined-value diagnostics.
An explicit defined-value guard corrected that test. The full 416-root
owning program then passed with empty diagnostics. A separate collector
incorrectly required an explicit child `noEmit` field; the normal
inherited setting was verified without repeating the compiler. The
original failures remain recorded.
- The first restoration run failed both cases. Permission parking
recorded 120 main-pool queries without more verifier calls. The
lifecycle counter recorded two denials without their exact scope. These
observations remain unexplained at the individual-query/call level.
Source separately shows a lifecycle readiness retry every two seconds
while the real external reply is absent.
- At `fe518c9b`: normal Shared/SDK and DB producers, module policy, all
seven full App programs, and all 33 pure diagnostic controls passed. The
lifecycle owner file passed all 19 cases, including both
confirmation/park orders, rollback, duplicate notices, startup recovery
and authority controls.
- At `fe518c9b`: the normal OAuth failure/retry/new-Db reconstruction
case passed. Its new healthy-wait window measured 15,002 ms with
unchanged total activity before genuine confirmation; the existing ready
quiet and revocation checks also passed.
- At `fe518c9b`: both normal restoration tests passed in Vitest.
Ordinary viewer-to-operator restoration and Pause/Resume recovered the
same binding, grant and single contact; each committed one challenge and
advanced revision 2 to 3. All six parked, restored and revoked/disabled
quiet windows measured at least 15 seconds with active positive read
controls and unchanged total queries, callbacks, sessions and verifier
calls. Both cleanups passed.
- The restoration wrapper still recorded failure: it required lifetime
lifecycle denial count zero, whereas the measured count was one before
the parked quiet window. The reviewed fixture explicitly permits a
current-authority denial from an earlier observation and requires no
committed challenge before restoration. Precise denial timing relative
to Pause was not serialized. The original wrapper failure is retained;
no rerun or retrospective explanation of the earlier failures is
claimed.
- Review exposed an expired-lease race. The added real-store baseline
test failed after the late defer restored `waitingForSignal`; no setup
or timeout was counted as the regression. On `cfd515a`, the whole
lifecycle owner file passed all 20 tests, including that case. The full
normal production declaration program (1067 roots), verification program
(662), and auth owning program (405) passed; all 1942 declarations
remained byte-identical.
- The original remote auth-session shard failed three tests because
their partial query adapters lacked the new binding query's
join/order/limit chain. A fixture-only correction preserved the real
notifier and existing authorization assertions. All 12 auth-session
tests passed locally.
- Complete remote inspection also found 33 failures in the 35-test Cloud
actor fixture with the same missing `innerJoin` method and downstream
incomplete-auth assertions. Its original full log and exact synthetic
Source comparison are retained. At `6efe722e`, the three-line fixture
adapter repair passed the complete normal 662-root test program and the
whole 35-test file with zero failures or skips. All existing lock,
authorization, settings, debounce and unfinished-auth assertions remain.
- The earlier healthy/restoration quiet measurements remain bound to
`fe518c9b`. Those steady 15-second windows did not exercise a
still-owned attempt passing its 120-second lease. The targeted
expired-owner regression qualifies that new boundary; it is not a fresh
measurement of the earlier windows.
- The initial `fe518c9b` remote gates failed the auth fixtures and
reported a Greptile 4/5 P1. The aggregate failure reported the
underlying shard. Those actual causes were fixed and qualified above.
- Current `6efe722e` remote gates completed with **57 successful checks
and three explicit skips**. The full latest edited Greptile summary is
**5/5**, explicitly reviews this exact head, and has no new finding. The
original P1 thread is resolved. All general, inline and nested review
pages were read.
- One current-head CI attempt timed out after 30 seconds in the Runner
evidence/privacy file, with 6,864 passes and one failure. That file and
its transitive inputs are byte-identical to the CI base. A separate
local diagnostic preserved every assertion and the original timeout, ran
the full normal Runner test project and the entire 41-case file, and
passed all 41 with zero skips. Its private bounded timing records
measured the target body at 74 ms. The diagnostic was then removed,
restoring the exact original test. This is local recovery evidence; the
original CI stalled await was not recorded and its cause remains
**unproven**.
- One retry of only the failed jobs on the unchanged head passed the
Runner support suite and dependent aggregate verification. The complete
second attempt and original failure logs are preserved. No test,
timeout, privacy guard or product code was weakened. This result does
not turn the original timeout into an explained failure or a diagnosed
flake.
- Full recursive typecheck, full `test:run` and a full release build
were not repeated locally. The normal full App programs, focused
producers and tests above passed, and the required current-head remote
gates passed. Final review is against the exact unchanged head; local
quiet remains separate from deployed suspension.

Reviewer commands use the normal project entry points: `node
server/scripts/typecheck-layers.mjs`, the whole
`server/src/services/muse-maintenance.test.ts` file, and both normal App
PostgreSQL files. PostgreSQL fixtures require explicit owned
fixture/admin URLs and normal migrations before imports. Run the files
serially with one worker. The private controller retains exact Source,
output, worker and owner-closure receipts.

## Risks

- Missing or early producer notices could lose a restoration trigger.
Notices stay in the transaction that commits the write. The worker
checks current authority after commit.
- A stale notice must not grant access. Binding generation, current
membership, company, agent and credential checks remain authoritative.
Revoked credentials require reconnecting.
- The change adds no table, migration, alternate task queue or periodic
verification scan. It adds two optional fields to existing lifecycle
operation JSONB. Generic pending/error deadlines, setup holds and manual
repair remain obligations.
- Confirmation can race with an active verification attempt. The park
compare-and-set must retain the durable recheck. Startup recovery uses a
fixed cutoff and keyset pages; it must not become another recurring
scan. Wrong company, stale version, pause and revoked credentials must
stay fenced.
- The connected receiver still has its documented tenant-sleep
restriction. Worker quiet is not proof of deployed suspension or
external wake.
- The normal challenge reply is sent by an authenticated protocol
fixture. Live Muse background replies, cost/usage, full Muse lifecycle
and full hosted Dot behavior remain unqualified. Partial Dot profile ACK
and unsigned-device cutover stay withheld.
- Restoration coverage for the additional company, budget, configuration
and feature producers remains a separate qualification obligation. This
pull request does not mark the whole hosted area complete.

## Model Used

OpenAI `gpt-6.1-sol` authored the implementation and tests with local
code and tool use. A GPT-6-based Codex coordinator reviewed the changes
and owns integration. Its exact underlying model ID is not exposed.
Context window sizes are unavailable. No context size is inferred.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Dotta authored and GitHub committed 2026-11-07 16:41:15 -06:00
1 parent adb7633d7b
commit eb7c7b083a
32 files changed
+1320 -72

No files matched your search

@@ -23,6 +23,17 @@ function fixture() {
afterEach(() => { vi.restoreAllMocks(); vi.useRealTimers(); });
describe("lifecycle readiness handoff", () => {
it("defers event-owned host verification without marking host or readiness complete", async () => {
const { store, driver, worker } = fixture();
driver.runHost.mockResolvedValue("waiting");
await worker.process(agent.id);
expect(store.defer).toHaveBeenCalledWith(agent, expect.any(String), null, expect.any(Date), { waitForSignal: true });
expect(store.completeHost).not.toHaveBeenCalled();
expect(store.change).not.toHaveBeenCalled();
expect(driver.onReady).not.toHaveBeenCalled();
await worker.stop();
});
it("notifies the scheduler only after host, plugins, and readiness have committed", async () => {
const { store, driver, worker } = fixture();
let commit!: (ready: LifecycleAgent) => void;
@@ -26,7 +26,7 @@ import { createAgentLifecycle, configureAgentLifecycle, startAgentLifecycle } fr
import { projectCompanyHolds } from "../adapters/company-policy.js";
import { createLifecycleStore as createStore } from "../adapters/postgres.js";
import { transition } from "../domain/policy.js";
import type { LifecycleAgent } from "../application/ports.js";
import type { LifecycleAgent, LifecycleDriver } from "../application/ports.js";
import { budgetService } from "../../../services/budgets.js";
import { agentConfigurationRevision } from "../../../lib/agent-configuration-revision.js";
@@ -53,7 +53,7 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
companyId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Lifecycle test", issuePrefix: `L${companyId.slice(0, 7)}` });
});
function worker(runHost: (agent: LifecycleAgent) => Promise<"complete" | "pending"> = async () => "complete") {
function worker(runHost: LifecycleDriver["runHost"] = async () => "complete") {
const value = configureAgentLifecycle(db, { requiredPluginIds: async () => [], runPlugin: async () => "complete", runHost });
workers.push(value); return value;
}
@@ -11,7 +11,7 @@ import { createLifecycleDriver } from "../../../services/agent-lifecycle-driver.
import { agentRoutes } from "../../../routes/agents.js";
import { errorHandler } from "../../../middleware/error-handler.js";
import { createLifecycleStore as createStore } from "../adapters/postgres.js";
import type { LifecycleAgent } from "../application/ports.js";
import type { LifecycleAgent, LifecycleDriver } from "../application/ports.js";
const createLifecycleStore = (db: Db) => createStore(db, createAgentLifecycleEffects());
const support = await getEmbeddedPostgresTestSupport();
@@ -27,7 +27,7 @@ const support = await getEmbeddedPostgresTestSupport();
companyId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Setup test", issuePrefix: `S${companyId.slice(0, 7)}` });
});
function worker(runHost: (agent: LifecycleAgent) => Promise<"complete" | "pending"> = async () => "complete") {
function worker(runHost: LifecycleDriver["runHost"] = async () => "complete") {
const work = configureAgentLifecycle(db, { requiredPluginIds: async () => [], runPlugin: async () => "complete", runHost });
workers.push(work); return work;
}
@@ -5,10 +5,11 @@ import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi }
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js";
import { createAgentLifecycle, configureAgentLifecycle } from "../../../services/agent-lifecycle.js";
import { createAgentLifecycleEffects } from "../../../services/agent-lifecycle.js";
import { startAgentLifecycle } from "../index.js";
import { requestAgentLifecycleRecheckInTransaction, startAgentLifecycle } from "../index.js";
import { createLifecycleStore } from "../adapters/postgres.js";
import { createLifecycleWorker } from "../application/worker.js";
import type { LifecycleAgent } from "../application/ports.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../../services/delivery-work-notifications.js";
import { resolveExternalTestDatabaseUrl } from "../../../__tests__/helpers/external-test-database.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_LIFECYCLE_VERIFICATION_RETRY_TEST_DATABASE_URL");
@@ -19,8 +20,8 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
let companyId: string;
const workers: Array<{ stop(): Promise<void> }> = [];
beforeAll(async () => {
if (external) { await applyPendingMigrations(external); db = createDb(external, { maxConnections: 6 }); }
else { database = await startEmbeddedPostgresTestDatabase("paperclip-verification-retry-"); db = createDb(database.connectionString, { maxConnections: 6 }); }
if (external) { await applyPendingMigrations(external); db = createDb(external, { maxConnections: 4 }); }
else { database = await startEmbeddedPostgresTestDatabase("paperclip-verification-retry-"); db = createDb(database.connectionString, { maxConnections: 4 }); }
}, 90_000);
afterAll(async () => { if (external) await closeRegisteredClients(external); else await database?.cleanup(); });
beforeEach(async () => {
@@ -208,4 +209,150 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
expect((await current(agent.id)).lifecycleState).toBe("terminated");
expect(runHost.mock.calls.map(([a]) => a.lifecycleState)).toContain("cleaning_up");
});
async function eventWaitFixture() {
let complete = false;
const runHost = vi.fn(async (agent: LifecycleAgent) => agent.lifecycleState !== "verifying" || complete
? "complete" as const : "waiting" as const);
const driver = { requiredPluginIds: async () => [], runPlugin: async () => "complete" as const, runHost };
const work = configureAgentLifecycle(db, driver); workers.push(work);
const agent = await createAgentLifecycle(db).requestHire(companyId, { name: "External reply wait", adapterType: "process" });
await work.process(agent.id);
return { agent: await current(agent.id), work, driver, runHost, complete: () => { complete = true; } };
}
function recheck(tx: Pick<Db, "select" | "update">, id: string, scopeCompanyId = companyId, version?: number) {
return requestAgentLifecycleRecheckInTransaction(tx,
{ companyId: scopeCompanyId, agentId: id, expectedVersion: version },
() => notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle));
}
it("parks healthy external verification without a retry deadline, claim or readiness bypass", async () => {
const { agent, work, runHost } = await eventWaitFixture();
expect(agent).toMatchObject({ lifecycleState: "verifying", status: "paused", lifecycleError: null,
lifecycleOperation: { waitingForSignal: true, hostComplete: false } });
expect(agent.lifecycleOperation).not.toHaveProperty("retryAt");
expect(agent.lifecycleOperation).not.toHaveProperty("retryRequired");
expect(agent.lifecycleOperation).not.toHaveProperty("leaseOwner");
const store = createLifecycleStore(db, createAgentLifecycleEffects()), calls = runHost.mock.calls.length;
const later = new Date(Date.now() + 86_400_000);
expect(await store.pending(100, later)).not.toContain(agent.id);
expect(await store.claim(agent.id, "unnotified", later)).toBeNull();
await work.process(agent.id);
expect(runHost).toHaveBeenCalledTimes(calls);
expect((await current(agent.id)).lifecycleState).toBe("verifying");
});
it.each(["during", "after"] as const)("retains a committed recheck %s the active attempt parks", async ordering => {
let entered!: () => void, release!: () => void, ready = false;
const inside = new Promise<void>(resolve => { entered = resolve; });
const hold = new Promise<void>(resolve => { release = resolve; });
const runHost = vi.fn(async (agent: LifecycleAgent) => {
if (agent.lifecycleState !== "verifying" || ready) return "complete" as const;
entered(); await hold; return "waiting" as const;
});
const work = configureAgentLifecycle(db, { requiredPluginIds: async () => [], runPlugin: async () => "complete", runHost });
workers.push(work);
const agent = await createAgentLifecycle(db).requestHire(companyId, { name: "Commit race", adapterType: "process" });
const processing = work.process(agent.id);
await inside;
try {
if (ordering === "after") { release(); await processing; }
await db.transaction(async tx => {
expect(await recheck(tx, agent.id)).toBe(true);
});
ready = true;
} finally { release(); await processing; }
const signalled = await current(agent.id);
expect(signalled.lifecycleOperation?.waitingForSignal).not.toBe(true);
expect(signalled.lifecycleOperation).not.toHaveProperty("retryAt");
await work.process(agent.id);
expect(await current(agent.id)).toMatchObject({ lifecycleState: "ready", status: "idle" });
expect(runHost.mock.calls.filter(([a]) => a.lifecycleState === "verifying")).toHaveLength(2);
});
it("retains a committed recheck when an expired attempt returns its earlier wait", async () => {
const { agent, work, complete, runHost } = await eventWaitFixture();
await db.transaction(async tx => { expect(await recheck(tx, agent.id)).toBe(true); });
const store = createLifecycleStore(db, createAgentLifecycleEffects());
const owner = "expired-verification-owner";
// Use the store's normal clock input: the real claim is already expired when
// confirmation commits, without changing timers or waiting two minutes.
const claimed = await store.claim(agent.id, owner, new Date(Date.now() - 121_000));
if (!claimed) throw new Error("Expected a real verification claim");
expect(Date.parse(claimed.lifecycleOperation!.leaseUntil!)).toBeLessThan(Date.now());
await db.transaction(async tx => { expect(await recheck(tx, agent.id)).toBe(true); });
const afterConfirmation = await current(agent.id);
complete();
expect(await store.defer(claimed, owner, null, new Date(), { waitForSignal: true })).toBe(true);
const signalled = await current(agent.id);
expect(signalled.lifecycleOperation).not.toHaveProperty("waitingForSignal");
expect(afterConfirmation.lifecycleOperation).toMatchObject({ leaseOwner: owner, recheckRequested: true });
expect(signalled.lifecycleOperation).not.toHaveProperty("leaseOwner");
expect(signalled.lifecycleOperation).not.toHaveProperty("retryAt");
expect(await store.pending(100, new Date())).toContain(agent.id);
await work.process(agent.id);
expect(await current(agent.id)).toMatchObject({ lifecycleState: "ready", status: "idle" });
expect(runHost.mock.calls.filter(([a]) => a.lifecycleState === "verifying")).toHaveLength(2);
});
it("rolls back the recheck with its producer and coalesces redundant committed signals", async () => {
const { agent, work, complete, runHost } = await eventWaitFixture();
await expect(db.transaction(async tx => {
expect(await recheck(tx, agent.id)).toBe(true);
throw new Error("Rollback this producer");
})).rejects.toThrow("Rollback this producer");
expect((await current(agent.id)).lifecycleOperation?.waitingForSignal).toBe(true);
await db.transaction(async tx => {
expect(await recheck(tx, agent.id)).toBe(true);
expect(await recheck(tx, agent.id)).toBe(true);
});
complete(); await work.process(agent.id);
const calls = runHost.mock.calls.length;
await work.process(agent.id);
expect(runHost).toHaveBeenCalledTimes(calls);
expect((await current(agent.id)).lifecycleState).toBe("ready");
});
it("finishes a committed recheck on a new owner without a post-commit scheduling call", async () => {
const { agent, work, driver, complete } = await eventWaitFixture();
await work.stop();
await db.transaction(async tx => { expect(await recheck(tx, agent.id)).toBe(true); });
complete();
// No process/Retry/wake callback follows the committed producer here.
const restarted = startAgentLifecycle(db, createAgentLifecycleEffects(), driver, () => true); workers.push(restarted);
await restarted.sweep();
expect(await current(agent.id)).toMatchObject({ lifecycleState: "ready", status: "idle" });
expect(await restarted.hasPending()).toBe(false);
});
it("rechecks a saved wait once at startup and remains parked without recurring startup scans", async () => {
const { agent, work, driver, runHost } = await eventWaitFixture();
await work.stop();
const restarted = startAgentLifecycle(db, createAgentLifecycleEffects(), driver, () => true); workers.push(restarted);
const calls = runHost.mock.calls.length;
expect(await restarted.hasPending()).toBe(true);
await restarted.sweep();
expect(runHost).toHaveBeenCalledTimes(calls + 1);
expect((await current(agent.id)).lifecycleOperation?.waitingForSignal).toBe(true);
expect(await restarted.hasPending()).toBe(false);
expect(restarted.nextRunAt()).toBeNull();
await restarted.sweep();
expect(runHost).toHaveBeenCalledTimes(calls + 1);
});
it("does not rearm a parked wait after a company or version mismatch, pause or required manual repair", async () => {
const { agent, work } = await eventWaitFixture();
await db.transaction(async tx => {
expect(await recheck(tx, agent.id, randomUUID())).toBe(false);
expect(await recheck(tx, agent.id, companyId, agent.lifecycleVersion - 1)).toBe(false);
});
expect((await current(agent.id)).lifecycleOperation).toEqual(agent.lifecycleOperation);
await createAgentLifecycle(db).pauseAgent(agent.id);
await work.process(agent.id);
await db.transaction(async tx => { expect(await recheck(tx, agent.id)).toBe(false); });
expect(await current(agent.id)).toMatchObject({ lifecycleState: "paused", status: "paused" });
const manual = await failed();
await db.transaction(async tx => { expect(await recheck(tx, manual.agent.id)).toBe(false); });
expect((await current(manual.agent.id)).lifecycleOperation?.retryRequired).toBe(true);
});
});
@@ -6,7 +6,7 @@ import { eq } from "drizzle-orm";
import { agents, type Db } from "@paperclipai/db";
import { agentRecordQueries, AGENT_CONFIGURATION_FIELDS, jsonEqual, type AgentConfigurationPatch, type UpdateAgentOptions } from "../../../lib/agent-records.js";
import type { ActivityPublication } from "../../../types/activity-publication.js";
import { AgentLifecycleConflict } from "../domain/policy.js";
import { AgentLifecycleConflict, canConfigureAgentConnection } from "../domain/policy.js";
import type { LifecycleEffects } from "./effects.js";
export async function updateAgentConfiguration(tx: Db, effects: LifecycleEffects, id: string, data: AgentConfigurationPatch, options?: UpdateAgentOptions, publications: ActivityPublication[] = []) {
@@ -40,6 +40,7 @@ export async function updateAgentConfiguration(tx: Db, effects: LifecycleEffects
{ adapterType: patch.adapterType ?? current.adapterType,
adapterConfig: Object.prototype.hasOwnProperty.call(patch, "adapterConfig") ? patch.adapterConfig : current.adapterConfig });
const invalidate = changed && (["preparing", "verifying", "resuming"].includes(current.lifecycleState) || prepareSetup);
if(prepareSetup&&!canConfigureAgentConnection(current))await effects.beforeConnectionEligibilityRestore?.(tx,current.companyId,current.id);
if (invalidate) await effects.notifyWork(tx);
const [updated] = await tx.update(agents).set({ ...patch, updatedAt: new Date(), ...(invalidate ? {
lifecycleVersion: current.lifecycleVersion + 1, lifecycleError: null,
@@ -48,7 +49,8 @@ export async function updateAgentConfiguration(tx: Db, effects: LifecycleEffects
? { lifecycleState: "pausing" as const } : {}),
lifecycleOperation: { ...current.lifecycleOperation!, id: randomUUID(), hostComplete: false,
completedPluginIds: [], attempts: 0, leaseOwner: undefined, leaseUntil: undefined,
retryAt: undefined, retryRequired: undefined, verificationFailures: undefined },
retryAt: undefined, retryRequired: undefined, verificationFailures: undefined,
waitingForSignal: undefined, recheckRequested: undefined },
} : {}) }).where(eq(agents.id, id)).returning();
await effects.completeConfiguration(tx, current, updated, patch, options, publications);
return agentRecordQueries(tx).getById(id);
@@ -11,6 +11,7 @@ export interface LifecycleEffects {
environmentRequiresNewPlugin(db: Db, environmentId: string | null | undefined, selectedPluginIds: string[] | null): Promise<boolean>;
notifyWork(db: Db): Promise<void>;
beforeStatusChange(db: Db, companyId: string, agentId: string, before: string, after: string): Promise<void>;
beforeConnectionEligibilityRestore?(db: Db, companyId: string, agentId: string): Promise<void>;
reportFailure(context: LifecycleFailure, error: unknown): void;
deleteDependencies(db: Db, companyId: string, agentId: string): Promise<void>;
prepareConfiguration(db: Db, current: AgentConfigurationRecord, data: AgentConfigurationPatch, options?: UpdateAgentOptions): Promise<AgentConfigurationPatch>;
@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
import { and, asc, eq, inArray, sql } from "drizzle-orm";
import { agents, agentApiKeys, companies, budgetPolicies, type Db } from "@paperclipai/db";
import type { AgentLifecycleOperation } from "@paperclipai/shared";
import { AgentLifecycleConflict, compatibilityStatus, isLifecycleWorkPending, transition, type LifecycleCommand } from "../domain/policy.js";
import { AgentLifecycleConflict, canConfigureAgentConnection, compatibilityStatus, isLifecycleWorkPending, transition, type LifecycleCommand } from "../domain/policy.js";
import type { LifecycleEffects } from "./effects.js";
import type { LifecycleStore } from "../application/ports.js";
@@ -18,8 +18,43 @@ export const pendingLifecycleWork = and(
inArray(agents.lifecycleState, ["preparing", "verifying", "pausing", "resuming", "terminating", "cleaning_up"]),
sql`not (${agents.lifecycleState} = 'verifying' and ${agents.lifecycleHolds} @> '["setup"]'::jsonb)`,
sql`not (${agents.lifecycleState} = 'verifying' and coalesce(${agents.lifecycleOperation}->>'retryRequired', 'false') = 'true')`,
sql`not (${agents.lifecycleState} = 'verifying' and coalesce(${agents.lifecycleOperation}->>'waitingForSignal', 'false') = 'true')`,
);
/** A bounded startup pass rechecks this saved wait once, without a timer. */
export const lifecycleSignalWait = and(
eq(agents.lifecycleState, "verifying"),
sql`coalesce(${agents.lifecycleOperation}->>'waitingForSignal', 'false') = 'true'`,
sql`coalesce(${agents.lifecycleOperation}->>'retryRequired', 'false') <> 'true'`,
);
/** The producer owns this outer transaction. Register its intent before either
* the recheck write or the corresponding external fact commits. */
export async function requestAgentLifecycleRecheckInTransaction(
tx: Pick<Db, "select" | "update">,
input: { companyId: string; agentId: string; expectedVersion?: number; onlyIfWaiting?: boolean },
notify: () => Promise<void>,
): Promise<boolean> {
if (!("nestedIndex" in tx)) throw new AgentLifecycleConflict("Lifecycle rechecks require the producer transaction");
const [agent] = await tx.select().from(agents).where(and(
eq(agents.id, input.agentId), eq(agents.companyId, input.companyId),
)).for("update");
if (!agent || !["preparing", "verifying"].includes(agent.lifecycleState)
|| agent.lifecycleOperation?.retryRequired
|| input.expectedVersion !== undefined && agent.lifecycleVersion !== input.expectedVersion
|| input.onlyIfWaiting && !agent.lifecycleOperation?.waitingForSignal) return false;
await notify();
// Expiry permits a replacement claim; only an owner change fences a late park.
const owned = Boolean(agent.lifecycleOperation?.leaseOwner);
await tx.update(agents).set({ lifecycleError: null,
lifecycleOperation: owned
? sql`${agents.lifecycleOperation} || '{"recheckRequested":true}'::jsonb`
: sql`${agents.lifecycleOperation} - 'waitingForSignal' - 'recheckRequested' - 'retryAt'`,
updatedAt: new Date(),
}).where(and(eq(agents.id, agent.id), eq(agents.companyId, input.companyId), eq(agents.lifecycleVersion, agent.lifecycleVersion)));
return true;
}
/** The caller owns the accounting transaction and has locked this agent. */
async function changeLockedAgent(tx: Db, effects: LifecycleEffects, agent: LifecycleRow,
command: LifecycleCommand, options: ChangeOptions = {}, policyHolds?: string[], restartPause = false) {
@@ -31,10 +66,16 @@ async function changeLockedAgent(tx: Db, effects: LifecycleEffects, agent: Lifec
agent.lifecycleRequiredPluginIds.some(id => !agent.lifecycleOperation!.completedPluginIds.includes(id)))) return null;
if (command === "retry") {
transition(agent.lifecycleState, command);
if (agent.lifecycleOperation?.leaseUntil && Date.parse(agent.lifecycleOperation.leaseUntil) > Date.now()) return agent;
if (agent.lifecycleOperation?.leaseUntil && Date.parse(agent.lifecycleOperation.leaseUntil) > Date.now()) {
await effects.notifyWork(tx);
const [requested] = await tx.update(agents).set({ lifecycleOperation:
sql`${agents.lifecycleOperation} || '{"recheckRequested":true}'::jsonb`, updatedAt: new Date(),
}).where(eq(agents.id, agent.id)).returning();
return requested;
}
await effects.notifyWork(tx);
const [updated] = await tx.update(agents).set({ lifecycleError: null,
lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt' - 'retryRequired' - 'verificationFailures' - 'leaseOwner' - 'leaseUntil'`, updatedAt: new Date() }).where(eq(agents.id, agent.id)).returning();
lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt' - 'retryRequired' - 'verificationFailures' - 'leaseOwner' - 'leaseUntil' - 'waitingForSignal' - 'recheckRequested'`, updatedAt: new Date() }).where(eq(agents.id, agent.id)).returning();
return updated;
}
if (command === "resume" && !["ready", "paused", "pausing", "resuming"].includes(agent.lifecycleState)) transition(agent.lifecycleState, command);
@@ -81,6 +122,7 @@ async function changeLockedAgent(tx: Db, effects: LifecycleEffects, agent: Lifec
};
if (next === "ready") operation.resumeState = "ready";
const status = compatibilityStatus(next, agent.status);
if(!canConfigureAgentConnection(agent)&&canConfigureAgentConnection({lifecycleState:next,status}))await effects.beforeConnectionEligibilityRestore?.(tx,agent.companyId,agent.id);
if (status !== agent.status) await effects.beforeStatusChange(tx, agent.companyId, agent.id, agent.status, status);
if (isLifecycleWorkPending(next)) await effects.notifyWork(tx);
const [updated] = await tx.update(agents).set({
@@ -139,7 +181,7 @@ export function createLifecycleStore(db: Db, effects: LifecycleEffects): Lifecyc
},
async claim(id, owner, now) {
const [agent] = await db.update(agents).set({
lifecycleOperation: sql`coalesce(${agents.lifecycleOperation}, '{"id":"", "hostComplete":false, "completedPluginIds":[], "attempts":0}'::jsonb)
lifecycleOperation: sql`(coalesce(${agents.lifecycleOperation}, '{"id":"", "hostComplete":false, "completedPluginIds":[], "attempts":0}'::jsonb) - 'recheckRequested')
|| jsonb_build_object('leaseOwner', ${owner}::text, 'leaseUntil', ${(new Date(now.getTime() + 120_000)).toISOString()}::text,
'attempts', coalesce((${agents.lifecycleOperation}->>'attempts')::integer, 0) + 1)`,
}).where(and(eq(agents.id, id), pendingLifecycleWork,
@@ -180,6 +222,16 @@ export function createLifecycleStore(db: Db, effects: LifecycleEffects): Lifecyc
return rows.length > 0;
},
async defer(agent, owner, error, now, options) {
if (options?.waitForSignal && agent.lifecycleState === "verifying") {
const rows = await db.update(agents).set({ lifecycleError: null,
lifecycleOperation: sql`(${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil' - 'retryAt' - 'waitingForSignal' - 'recheckRequested')
|| case when coalesce(${agents.lifecycleOperation}->>'recheckRequested', 'false') = 'true'
then '{}'::jsonb else '{"waitingForSignal":true}'::jsonb end`,
updatedAt: now,
}).where(and(eq(agents.id, agent.id), eq(agents.lifecycleVersion, agent.lifecycleVersion),
sql`${agents.lifecycleOperation}->>'leaseOwner' = ${owner}`)).returning({ id: agents.id });
return rows.length > 0;
}
const retry: Partial<AgentLifecycleOperation> = options?.retryRequired && agent.lifecycleState === "verifying"
? { retryRequired: true }
: { retryAt: new Date(now.getTime() + (error ? 60_000 : 2_000)).toISOString() };
@@ -188,7 +240,7 @@ export function createLifecycleStore(db: Db, effects: LifecycleEffects): Lifecyc
}
const rows = await db.update(agents).set({
lifecycleError: error,
lifecycleOperation: sql`(${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil' - 'retryAt' - 'retryRequired')
lifecycleOperation: sql`(${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil' - 'retryAt' - 'retryRequired' - 'waitingForSignal' - 'recheckRequested')
|| ${JSON.stringify(retry)}::jsonb`,
updatedAt: now,
}).where(and(eq(agents.id, agent.id), eq(agents.lifecycleVersion, agent.lifecycleVersion),
@@ -4,7 +4,7 @@ import { and, eq, sql } from "drizzle-orm";
import { agents, agentApiKeys, companies, type Db } from "@paperclipai/db";
import { conflict, notFound } from "../../../errors.js";
import { agentRecordQueries, type CreateAgentOptions, type CreateAgentData } from "../../../lib/agent-records.js";
import { isAgentSetupDraft } from "../domain/policy.js";
import { isAgentSetupDraft, canConfigureAgentConnection } from "../domain/policy.js";
export function agentRecords(db: Db, effects: LifecycleEffects) {
const { getById } = agentRecordQueries(db);
@@ -40,6 +40,7 @@ export function agentRecords(db: Db, effects: LifecycleEffects) {
const lifecycleState = status === "pending_approval" ? "pending_approval" : status === "terminated" ? "terminated"
: prepareSetup ? "preparing" : status === "paused" ? "paused" : companyHolds.length ? "pausing" : "preparing";
if (lifecycleState === "preparing" || lifecycleState === "pausing") await effects.notifyWork(tx);
const createdStatus=status === "pending_approval" ? "pending_approval" : status === "terminated" ? "terminated" : "paused";
const values: typeof agents.$inferInsert = {
...prepared,
createdByUserId: setup?.createdByUserId ?? options?.createdByUserId ?? null,
@@ -49,9 +50,10 @@ export function agentRecords(db: Db, effects: LifecycleEffects) {
pauseReason: holds[0] ?? null,
pausedAt: holds.length ? new Date() : null,
lifecycleOperation: { id: randomUUID(), hostComplete: false, completedPluginIds: [], attempts: 0, resumeState: "preparing" as const, responsibleUserId: options?.responsibleUserId ?? options?.createdByUserId },
status: status === "pending_approval" ? "pending_approval" : status === "terminated" ? "terminated" : "paused",
status: createdStatus,
companyId,
};
if(setup&&!canConfigureAgentConnection(setup)&&canConfigureAgentConnection({lifecycleState,status:createdStatus}))await effects.beforeConnectionEligibilityRestore?.(txDb,companyId,setup.id);
const created = await (options?.setupAgentId
? tx.update(agents).set({ ...values, updatedAt: new Date() }).where(eq(agents.id, options.setupAgentId))
: tx.insert(agents).values(values))
@@ -123,6 +125,7 @@ export function agentRecords(db: Db, effects: LifecycleEffects) {
const holds = existing.lifecycleHolds.filter(hold => hold !== "company_paused" && hold !== "company_archived");
if (company?.status === "paused") holds.push("company_paused");
if (company?.status === "archived") holds.push("company_archived");
if(!holds.length)await effects.beforeConnectionEligibilityRestore?.(txDb,existing.companyId,existing.id);
await effects.notifyWork(txDb);
const updated = await tx
.update(agents)
@@ -20,13 +20,14 @@ export interface LifecycleStore {
setRequiredPlugins(agent: LifecycleAgent, owner: string, pluginIds: string[]): Promise<boolean>;
completeHost(agent: LifecycleAgent, owner: string): Promise<boolean>;
completePlugin(agent: LifecycleAgent, owner: string, pluginId: string): Promise<boolean>;
defer(agent: LifecycleAgent, owner: string, error: string | null, now: Date, options?: { retryRequired: boolean; verificationFailures?: number }): Promise<boolean>;
defer(agent: LifecycleAgent, owner: string, error: string | null, now: Date, options?: { retryRequired?: boolean; verificationFailures?: number; waitForSignal?: boolean }): Promise<boolean>;
pending(limit: number, now: Date): Promise<string[]>;
}
export interface LifecycleDriver {
requiredPluginIds(agent: LifecycleAgent): Promise<string[]>;
runHost(agent: LifecycleAgent): Promise<"complete" | "pending">;
/** waiting is only a healthy verifying phase owned by an external event. */
runHost(agent: LifecycleAgent): Promise<"complete" | "pending" | "waiting">;
runPlugin(agent: LifecycleAgent, pluginId: string): Promise<"complete" | "pending">;
/** Notify the scheduler only after readiness has committed. */
onReady?(agent: LifecycleAgent): Promise<void>;
@@ -45,7 +45,13 @@ export function createLifecycleWorker(store: LifecycleStore, driver: LifecycleDr
}
if (!operation.hostComplete) {
if (!await runStep("renew", () => store.renew(agent, owner, new Date()))) continue;
if (await runStep("host", () => driver.runHost(agent)) === "pending") {
const host = await runStep("host", () => driver.runHost(agent));
if (host === "waiting") {
if (agent.lifecycleState !== "verifying") throw new Error("Only host verification may await an external event");
if (!await runStep("defer", () => store.defer(agent, owner, null, new Date(), { waitForSignal: true }))) continue;
return;
}
if (host === "pending") {
if (!await runStep("defer", () => store.defer(agent, owner, null, new Date()))) continue;
return;
}
+39 -4
View File
@@ -1,6 +1,6 @@
import { hireApprovalService as approvalRecords, type HireDecisionTarget } from "./adapters/approvals.js";
import { agents } from "@paperclipai/db";
import { inArray, sql } from "drizzle-orm";
import { agents, companies } from "@paperclipai/db";
import { and, asc, desc, eq, gt, inArray, lte, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import type { LifecycleEffects } from "./adapters/effects.js";
export type { LifecycleEffects } from "./adapters/effects.js";
@@ -8,12 +8,12 @@ import { updateAgentConfiguration } from "./adapters/configuration.js";
import type { AgentConfigurationPatch, UpdateAgentOptions } from "../../lib/agent-records.js";
import { purgeAgent } from "./adapters/deletion.js";
import { agentRecords } from "./adapters/records.js";
import { assertRootDatabase, createLifecycleStore, pendingLifecycleWork } from "./adapters/postgres.js";
import { assertRootDatabase, createLifecycleStore, pendingLifecycleWork, lifecycleSignalWait, requestAgentLifecycleRecheckInTransaction } from "./adapters/postgres.js";
import { hasCompanyHoldMismatch, projectCompanyHolds } from "./adapters/company-policy.js";
import { createLifecycleWorker } from "./application/worker.js";
import type { LifecycleDriver } from "./application/ports.js";
export { applyAgentPolicyHoldsInTransaction } from "./adapters/postgres.js";
export { applyAgentPolicyHoldsInTransaction, requestAgentLifecycleRecheckInTransaction } from "./adapters/postgres.js";
export { companyHoldMismatch, fenceActiveCompanyAgentStopsInTransaction } from "./adapters/company-policy.js";
export { AgentLifecycleConflict, canConfigureAgentConnection, isAgentAwaitingSetup, isAgentSetupDraft } from "./domain/policy.js";
@@ -105,6 +105,35 @@ export function startAgentLifecycle(db: Db, effects: LifecycleEffects, driver: L
let running: Promise<void> | undefined;
let stopped = false;
let nextDeadline: number | null = null;
const recoveryStartedAt = new Date();
let recoveryInitialized = false, recoveryDone = false;
let recoveryThrough: string | undefined, recoveryAfter: string | undefined;
let recoveryInitialization: Promise<void> | undefined;
const recoveryCondition = and(lifecycleSignalWait, lte(agents.createdAt, recoveryStartedAt));
async function initializeRecovery() {
if (recoveryInitialized) return;
return recoveryInitialization ??= (async () => {
const [last] = await db.select({ id: agents.id }).from(agents).where(recoveryCondition)
.orderBy(desc(agents.id)).limit(1);
recoveryThrough = last?.id; recoveryDone = !last; recoveryInitialized = true;
})().catch(error => { recoveryInitialization = undefined; throw error; });
}
async function recoverPage() {
await initializeRecovery();
if (recoveryDone || !recoveryThrough) return;
const page = await db.select({ id: agents.id, companyId: agents.companyId, version: agents.lifecycleVersion })
.from(agents).innerJoin(companies, eq(companies.id, agents.companyId))
.where(and(recoveryCondition, eq(companies.status, "active"), lte(agents.id, recoveryThrough),
recoveryAfter ? gt(agents.id, recoveryAfter) : undefined)).orderBy(asc(agents.id)).limit(100);
for (const agent of page) {
if (stopped || !canRun()) return;
await effects.transaction(db, agent.companyId, tx => requestAgentLifecycleRecheckInTransaction(tx,
{ companyId: agent.companyId, agentId: agent.id, expectedVersion: agent.version, onlyIfWaiting: true },
() => effects.notifyWork(tx)));
recoveryAfter = agent.id;
}
if (page.length < 100) recoveryDone = true;
}
// The coordinator supplies the wake after construction. No direct work escapes it.
wakes.set(db, () => {});
return {
@@ -112,6 +141,10 @@ export function startAgentLifecycle(db: Db, effects: LifecycleEffects, driver: L
nextRunAt: () => nextDeadline,
async hasPending() {
if (stopped) { nextDeadline = null; return false; }
if (canRun()) {
await initializeRecovery();
if (!recoveryDone) return true;
}
const [pending] = await db.select({ due: sql<number | null>`min(greatest(
coalesce(extract(epoch from (${agents.lifecycleOperation}->>'leaseUntil')::timestamptz) * 1000, 0),
coalesce(extract(epoch from (${agents.lifecycleOperation}->>'retryAt')::timestamptz) * 1000, 0)))::double precision` })
@@ -128,6 +161,8 @@ export function startAgentLifecycle(db: Db, effects: LifecycleEffects, driver: L
return running ??= (async () => {
try { await projectCompanyHolds(db, effects); }
catch (error) { effects.reportFailure({ stage: "policy_scan" }, error); throw error; }
try { await recoverPage(); }
catch (error) { effects.reportFailure({ stage: "scan" }, error); throw error; }
await worker.sweep();
})().finally(() => { running = undefined; });
},
@@ -19,3 +19,5 @@ export function createDotPrincipalHttp(oauth: Parameters<typeof dotPrincipalCont
} {
return { bodyParser: dotPrincipalControlBodyParser(), matchesPost: isDotPrincipalControlPost, router: dotPrincipalControlRoutes(oauth) };
}
export { notifyMuseContactVerification, pendingMuseContactVerifications, parseMuseVerificationHint, type MuseVerificationScope } from "./muse-verification-work.js";
@@ -0,0 +1,81 @@
import { and, asc, eq, gt, isNotNull, isNull } from "drizzle-orm";
import { agents, museAgentBindings as bindings, type Db } from "@paperclipai/db";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../services/delivery-work-notifications.js";
export type MuseVerificationScope = {
companyId?: string;
bindingId?: string;
generation?: number;
operatorId?: string;
agentId?: string;
};
export type MuseVerificationCandidate = {
binding: typeof bindings.$inferSelect;
agent: { status: string; lifecycleState: string };
};
export type MuseVerificationHint = {
kind: "muse-contact-verification";
companyId: string;
bindingId: string;
generation: number;
};
export async function pendingMuseContactVerifications(
db: Pick<Db, "select">,
scope?: MuseVerificationScope,
after?: string,
contactWillPersist = false,
): Promise<MuseVerificationCandidate[]> {
return db.select({ binding: bindings, agent: { status: agents.status, lifecycleState: agents.lifecycleState } })
.from(bindings)
.innerJoin(agents, and(eq(agents.id, bindings.agentId), eq(agents.companyId, bindings.companyId)))
.where(and(
isNull(bindings.revokedAt), isNotNull(bindings.pairedAt),
contactWillPersist ? undefined : isNotNull(bindings.receiverContactAt),
isNull(bindings.verifiedReplyAt), isNull(bindings.challengeHash),
scope?.companyId ? eq(bindings.companyId, scope.companyId) : undefined,
scope?.bindingId ? eq(bindings.id, scope.bindingId) : undefined,
scope?.generation !== undefined ? eq(bindings.generation, scope.generation) : undefined,
scope?.operatorId ? eq(bindings.operatorId, scope.operatorId) : undefined,
scope?.agentId ? eq(bindings.agentId, scope.agentId) : undefined,
after ? gt(bindings.id, after) : undefined,
)).orderBy(asc(bindings.id)).limit(100);
}
export async function notifyMuseContactVerification(
tx: Pick<Db, "select">,
scope?: MuseVerificationScope,
trigger: "authority" | "contact-persist" = "authority",
): Promise<number> {
if (trigger === "contact-persist" && (!scope?.bindingId || scope.generation === undefined)) {
throw new Error("Muse contact intent requires its exact binding generation");
}
let after: string | undefined;
let count = 0;
while (true) {
const rows = await pendingMuseContactVerifications(tx, scope, after, trigger === "contact-persist");
for (const { binding } of rows) {
const hint: MuseVerificationHint = {
kind: "muse-contact-verification", companyId: binding.companyId,
bindingId: binding.id, generation: binding.generation,
};
await notifyDeliveryWork(tx, DELIVERY_QUEUES.museMaintenance, JSON.stringify(hint));
count++;
}
if (rows.length < 100) return count;
after = rows.at(-1)!.binding.id;
}
}
export function parseMuseVerificationHint(raw: string): MuseVerificationHint | undefined {
let value: unknown;
try { value = JSON.parse(raw); } catch { return; }
if (!value || typeof value !== "object" || Array.isArray(value)
|| !("kind" in value) || value.kind !== "muse-contact-verification"
|| !("companyId" in value) || typeof value.companyId !== "string"
|| !("bindingId" in value) || typeof value.bindingId !== "string"
|| !("generation" in value) || typeof value.generation !== "number"
|| !Number.isSafeInteger(value.generation) || value.generation < 1) return;
return { kind: "muse-contact-verification", companyId: value.companyId,
bindingId: value.bindingId, generation: value.generation };
}
+7 -14
View File
@@ -15,8 +15,7 @@ import { conflict, forbidden, notFound, unprocessable } from "../../errors.js";
import { museIdentity, museCredentialHash, assertMuseCurrentAuthority, assertMuseSubjectCredential, inspectMuseConnectionCredential, assertMuseCleanupCredential, museCleanupBoundaryIds, assertMuseDisabledStopControlCredential, museStopControlBoundaryIds } from "../../services/muse-identity.js";
import { instanceSettingsService } from "../../services/instance-settings.js";
import { updateAgentConfigurationInTransaction } from "../../services/agent-configuration-transaction.js";
import { AgentLifecycleConflict, canConfigureAgentConnection } from "../agent-lifecycle/index.js";
import { createAgentLifecycle } from "../../services/agent-lifecycle.js";
import { canConfigureAgentConnection, requestAgentLifecycleRecheckInTransaction } from "../agent-lifecycle/index.js";
import { authorizationService, issueReadSqlCondition, type AuthorizationActor } from "../../services/authorization.js";
import { logActivity, publishActivity, type ActivityPublication } from "../../services/activity-log.js";
import { issueService } from "../../services/issues.js";
@@ -387,23 +386,17 @@ function createMuseBroker(db:Db, publicOrigin?:string) {
if(!b.receiverContactAt)throw conflict("No persisted receiver contact yet.");
const publications: ActivityPublication[] = [];
await db.transaction(async tx => {
// Match reconnect's agent-before-binding lock order. Both readiness
// and its durable recheck commit together, before any callback runs.
await requestAgentLifecycleRecheckInTransaction(tx,
{ companyId: b.companyId, agentId: b.agentId },
() => notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle));
await assertMuseCurrentAuthority(tx, b, true);
const [ready]=await tx.update(bindings).set({status:"ready",verifiedReplyAt:new Date(),challengeHash:null,challengeExpiresAt:null,revision:b.revision+1,updatedAt:new Date()}).where(and(eq(bindings.id,b.id),eq(bindings.generation,b.generation),eq(bindings.challengeHash,museCredentialHash(c.nonce)),gt(bindings.challengeExpiresAt,new Date()),isNull(bindings.revokedAt))).returning();
if(!ready)throw conflict("Challenge expired or does not match.");
await logActivity(tx as unknown as Db,{companyId:b.companyId,actorType:"agent",actorId:b.agentId,agentId:b.agentId,responsibleUserIdOverride:b.operatorId,action:"muse.background_reply_verified",entityType:"agent",entityId:b.agentId,details:{bindingId:b.id,generation:b.generation,authorizingUserId:b.operatorId,requestId:c.requestId}},publications);
});
publications.forEach(publishActivity);
// The completed challenge changes this hire's verification facts. Clear
// its deferred retry durably; a wake alone cannot bypass that deadline.
const lifecycle=createAgentLifecycle(db),agent=await lifecycle.get(b.agentId);
if(agent&&["preparing","verifying"].includes(agent.lifecycleState)) {
try { await lifecycle.retry(b.agentId); }
catch(error) {
// A concurrent lifecycle completion or Stop owns the newer phase.
// Confirmation remains accepted; it must not restart that phase.
const current=error instanceof AgentLifecycleConflict?await lifecycle.get(b.agentId):null;
if(!current||["preparing","verifying"].includes(current.lifecycleState))throw error;
}
}
return {status:"ready",bindingId:b.id,generation:b.generation};
}
if(c.command==="task.create"||c.command==="task.comment")return idleMutation(s,c);