mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 12:04:41 +01:00
fix(server): recover approved tool actions from committed changes (#17454)
## Thinking Path > - Paperclip manages agent work and its governed tool calls. > - A human approval can persist before the approving process dispatches the tool. > - Recovery currently scans approved requests on every maintenance tick. > - Completed and deliberately inert approvals do not owe another execution. > - Test-tab approval also needs a shared execution claim before recovery can safely race the foreground path. > - This PR wakes recovery from committed changes and retries only unresolved approvals. > - Idle history becomes quiet while existing approval and execution checks remain with the gateway. ## Linked Issues or Issue Description Refs #17453. This is the fifth slice of the recurring-server sleep migration, stacked on `codex/sleep-04-tool-review-expiry`. **Problem** Approved tool requests are scanned periodically even when no executable work remains. The Test-tab foreground path lacks the request claim needed to admit concurrent recovery safely. **Proposed Solution** Use the existing transaction-aware review notification boundary and shared deadline worker. Keep signed-payload interpretation, current policy and target checks, and provider dispatch in the gateway. Share a claim between Test-tab foreground and recovery, then persist both authoritative outcome rows atomically. **Alternatives Considered** A longer interval still scans idle history. A new durable queue would duplicate existing request/invocation state. Dispatching Test-tab approvals without a claim risks duplicate external calls. ## What Changed - Recheck both principals for delegated Test-tab approvals. Save their rate receipts in the same claim transaction, share only the same bucket, and roll back new charges on denial. - Preserve delegated gateway authorization and both issuer/target rate receipts inside the existing claimed-invocation transaction; retain the durable actor during approved replay. - Expire stale managed connector payloads before consuming approved-tool admission. - Persist current rate-policy charges and their invocation receipts atomically before dispatch, preserving exact approval claim fencing and read-only preflight. - Recover approved requests at startup and after committed review changes; retry only unresolved approvals. - Leave legacy agent approvals without execute-on-approve intent inert. Repair completed Test-tab history without another provider call. - Fence Test-tab execution and outcome settlement with an atomic claim shared by both callers. Revalidate the approved target and current policy before dispatch. - Keep provider calls outside database transactions. Retain quarantine for interrupted execution and unknown outcomes. - Add a concurrent, idempotent partial index for approved-request pagination. Reuse the parent's company-scoped review signals and shared scoped deadline worker. - Remove approved scanning from the aggregate tick. Preserve the public combined sweep and scheduler-disabled startup behavior. - Document producer ownership, quiet states, lifecycle behavior, and remaining recovery duties. ## Verification Latest published head: `23868c1751c100a27f58087f7ce16b93c52d4eba`, targeting master. The review-found delegated gateway pool deadlock is repaired by passing the admission transaction through token and issuer-authority reads. Foreground and recovery each reproduced the deadlock with a one-connection pool; all three constrained-pool cases, including revoked authority, now pass. All 81 HTTP integration cases pass. The historical foreground rate-slot regression also reproduced before its repair; verified preview hashes are accepted only by read-only preflight, while consuming admission still requires the complete identity under the exact claim. Five full suites passed 329 cases; the gateway service then passed all 141 cases after two additional mixed historical hash controls. Production types/build passed before the final normal integration of master `35820cf398` (including independent Grok/Runner changes); integration validation is running. Initial fixture/assertion errors and the prior E2E restoration/cancellation deadlock remain recorded in [the owning audit](doc/idle-approved-tool-recovery.md#gateway-admission-uses-its-transaction-connection). Fresh-head CI, Greptile 5/5, and passing Ponytail are still required before merge. - Master landing reconciliation at `8dfe3958db6e4e8a0b7499f3f5be0a396ca77361` after #17453 merged: production and server test files are byte-identical to the accepted 320-case repair. The landed migration journal prefix is unchanged, and #17928's independent eval reporting changes are preserved. Fresh CI and reviews on this exact head are required before merge. - Historical-identity repair published at `096b7c7198c3ca864aca9377e261cbaa0a8aad6f` (production/test source identical to integrated acceptance `81231824d773a79fc50c835eccf7d9b3977e79fc`): all 320 accepted gateway, policy, startup, preparation-timing and authenticated HTTP management cases pass. Server types and the production server build pass. Four real historical sensitive-argument regressions failed before repair; invalid-signature, foreign-hash and stale-claim controls remain denied without dispatch or identity changes. Both agent and Test-tab calls deliver the original signed arguments once, and new rate policies still apply. - The first integrated combined run passed 319 cases and hit a Slack bot ownership collision left in a reused HTTP test database. The guard correctly returned 409. The full 78-case HTTP suite passed on a fresh dedicated fixture without product or assertion changes. Earlier fixture-construction errors are distinguished from the four real regressions in the audit document. Fresh remote CI and reviews remain required. - Parent reconciliation at `736e7f56097055e21ce13b4e45d9033391483af8`: 299 cases pass (221 full gateway/policy/startup cases and 78 assembled-server connection management cases), with no skipped cases. Server types pass. This includes real delegated Test-tab approval admission for newly added shared and caller rate rules. Fresh remote CI and Greptile review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/bcca5b006976a471f4afd1f70c4031aa40a2b3f7/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Current head: `bcca5b006976a471f4afd1f70c4031aa40a2b3f7`. This remains a single-duty PR stacked on #17453. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/a021467f04d22be42ee5b0c87b6583b2300a62a9/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Notifications cover the existing database-owner, single-active-writer contract. Independent SQL writers and other replicas need an explicit wake path. - Unknown provider outcomes cannot be replayed safely. Failed result persistence retains an executing claim and the existing stale-execution quarantine. - Interrupted execution and terminal interaction projection repair remain periodic until their separate slices. This PR changes only approved recovery. - A pending-only review write can trigger one empty approved-recovery pass. Settled idle work schedules no further scan. - The partial index uses the existing top-level concurrent migration executor. Snapshot pruning follows repository policy. - Cloud sleep eligibility and external alarm coverage remain unchanged. Worker quiet is not proof of deployed instance suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
35820cf398
commit
77295faebc
23 files changed
+2313
-506
No files matched your search
@@ -0,0 +1,381 @@
|
||||
# Approved tool-action recovery
|
||||
|
||||
Approved tool actions recover at startup and after committed review changes.
|
||||
The aggregate maintenance tick no longer scans approved requests. The existing
|
||||
request and invocation rows remain authoritative; this change adds no queue table.
|
||||
|
||||
## Work and quiet states
|
||||
|
||||
The gateway owns the approved-work predicate. A signed agent action with
|
||||
`executionOnApprove: true` owes execution through the normal approval path.
|
||||
A legacy signed action without that flag remains deliberately inert. A Test-tab
|
||||
approval can owe execution even when no agent run exists. Recovery uses the saved
|
||||
origin and current authorization/target checks; the scheduler sees only whether
|
||||
another attempt is needed.
|
||||
|
||||
Startup discovers companies with saved approved requests. Committed review
|
||||
signals carry the affected company ID; subsequent discovery and recovery reads
|
||||
filter by that company. The shared scoped worker coalesces changes and dispatches
|
||||
one company per pass with a one-second minimum spacing. A company writing reviews
|
||||
does not scan or lock another company's approved backlog.
|
||||
|
||||
Recovery pages that company's approved requests in ID order using a partial index. It does not
|
||||
let a full batch of unavailable approvals starve later work. A failed read or
|
||||
unresolved executable approval retains a 30-second retry. New writes preserve
|
||||
that company's retry deadline, while other companies can run independently.
|
||||
A company pass still drains all its pages; this change does not cap provider
|
||||
calls within that pass or alter the provider's existing execution timeout. Once a pass leaves no
|
||||
such obligation, there is no recurring approved-recovery query. An empty system,
|
||||
configured idle tools, completed history, and deliberately inert approvals are
|
||||
quiet after reconciliation.
|
||||
|
||||
## Producer and transaction ownership
|
||||
|
||||
The existing tool-review notification boundary now signals both pending expiry
|
||||
and approved recovery. This covers human review transactions, gateway approval
|
||||
and accepted-interaction inference, administrative interaction resolution, review
|
||||
queue invalidation, and the gateway's approval claims and managed-argument
|
||||
invalidation. The notification is awaited before SQL in the existing transaction;
|
||||
caller-owned outer transactions retain commit/rollback ownership.
|
||||
|
||||
A pending-only change can cause one empty company-scoped discovery read. It does not
|
||||
create a periodic scan. No per-route recovery bookkeeping or database listener
|
||||
is added. Signals use the existing database owner's single-active-writer model.
|
||||
Independent SQL writers or another process need a separate notification contract.
|
||||
|
||||
## Single execution and outcome ownership
|
||||
|
||||
Agent actions retain their existing approved-to-executing claim. Test-tab
|
||||
foreground approval and recovery now share an equivalent claim before provider
|
||||
dispatch. The invocation then starts only if the request still holds that claim.
|
||||
Provider calls and target discovery remain outside SQL transactions. Current rate-policy admission and its receipt commit together in a short database transaction.
|
||||
|
||||
The Test-tab invocation outcome and terminal request status commit together.
|
||||
Claim timestamps fence late settlement. A later audit failure cannot overwrite
|
||||
an already committed outcome. Recovery repairs legacy approved Test-tab requests
|
||||
from a saved terminal invocation without decoding old arguments or calling the
|
||||
provider. A legacy executing invocation moves to the existing interrupted-work
|
||||
path; it is not dispatched again.
|
||||
|
||||
If the provider may have run but its result cannot be persisted, the executing
|
||||
claim remains authoritative. The existing stale-execution recovery quarantines
|
||||
unknown outcomes. This slice does not promise exactly-once external effects or
|
||||
replay an uncertain provider call.
|
||||
|
||||
## Lifecycle and remaining duties
|
||||
|
||||
The shared deadline worker coalesces notifications, preserves changes during a
|
||||
pass, and retains skipped work across idle drain/resume. Startup reconstructs
|
||||
saved approvals. Company deletion invalidates cached recovery and retries through
|
||||
the same transactional notification boundary. Shutdown stops its coordinator registration. With heartbeat
|
||||
scheduling disabled, startup keeps the existing one-off recovery behavior.
|
||||
|
||||
Pending expiry is owned by the preceding slice. Interrupted execution and
|
||||
interaction projection repair remain in the old sweep until their own migrations.
|
||||
The public combined sweep retains all duties for manual/recovery callers.
|
||||
No Cloud eligibility or external-wake proof changes here; worker quiet does not
|
||||
establish deployed database or host suspension.
|
||||
|
||||
## Verification
|
||||
|
||||
Real PostgreSQL service/startup tests cover producer commit and rollback,
|
||||
startup reconstruction, empty/configured/history quiet states, concrete retries,
|
||||
first-write drain/resume, legacy inert approvals, and multi-page recovery.
|
||||
Controlled Test-tab provider tests cover both foreground/recovery race winners,
|
||||
completed legacy outcomes, interrupted legacy calls, target changes, and a failed
|
||||
atomic outcome write followed by quarantine without another provider call.
|
||||
|
||||
The service/startup selection passes 123 tests. The existing gateway acceptance
|
||||
suite passes 92 tests. The strengthened paging and outcome-failure cases pass
|
||||
again after their final edits. Workspace typecheck and build pass. Full CI and
|
||||
review remain separate PR gates; no real external provider was invoked by these
|
||||
fixtures.
|
||||
|
||||
Review found that a one-shot success-write failure could fall into the provider
|
||||
error handler and save a false failure on its second attempt. The added sequence-
|
||||
based PostgreSQL fault reproduces that bug before the fix. Provider execution,
|
||||
pre-dispatch validation, and outcome persistence now have separate error
|
||||
boundaries. Both persistent and one-shot result-write failures retain the claim
|
||||
for quarantine without repeating the call. The complete 123-test selection and
|
||||
92 gateway acceptance cases pass after the repair, as do typecheck and build.
|
||||
|
||||
The existing tool-access service regression now expects a completed Test-tab
|
||||
request to be executed and verifies exactly one provider call before asking for
|
||||
a new approval. Its full 469-test suite passes. The initial local command skipped
|
||||
its database cases; the suite now honors its supplied disposable database through
|
||||
the standard test-database guard independently of embedded Postgres support.
|
||||
|
||||
Company-scoping regression: a real signed/reviewed action in company A previously
|
||||
caused an unfiltered approved-request SELECT over company B's inert legacy action.
|
||||
The captured-SQL test fails before this repair and passes with scoped reads. Added
|
||||
lifecycle cases check retry preservation across committed writes, independent
|
||||
company progress, and disarming after the real company-deletion producer. These
|
||||
checks supplement the earlier execution, transaction, and paging evidence above.
|
||||
|
||||
After company scoping, the complete gateway-service and startup selection passes
|
||||
149 tests against a disposable PostgreSQL database, and server typecheck passes.
|
||||
Stack integration, workspace checks, and fresh remote CI remain separate gates.
|
||||
|
||||
## Preparation failures retain an approved Test-tab action
|
||||
|
||||
A real PostgreSQL policy-read timeout reproduced a permanent-failure bug after
|
||||
claiming a Test-tab action: the provider had never been called, but both request
|
||||
and invocation were saved as failed, so approved recovery could not retry them.
|
||||
|
||||
The preparation boundary now distinguishes explicit validation/access rejections
|
||||
from unexpected dependency errors and transient HTTP errors. The existing
|
||||
`tool-approval-execution` owner releases only its exact claim: invocation and request
|
||||
are locked in the established order, their link/company/claim timestamp must
|
||||
match, and the invocation must still await approval with no recorded start.
|
||||
The request returns to `approved` with a transaction-aware notification. Signed
|
||||
arguments and the human approval remain unchanged; the error propagates so the
|
||||
existing recovery worker retains its retry obligation.
|
||||
|
||||
This boundary ends before `runTestToolInvocation`. Provider dispatch and outcome
|
||||
persistence keep their existing conservative no-replay behavior. A changed claim,
|
||||
started invocation or prior-dispatch evidence cannot be released by this path.
|
||||
If the requeue transaction itself fails, the error propagates and no terminal
|
||||
failure is fabricated; existing interrupted-execution reconciliation remains the
|
||||
fallback. This change does not infer replay safety from a failed requeue.
|
||||
|
||||
Before the fix, both foreground and recovery timeout regressions failed. They
|
||||
now observe the committed claim under a real PostgreSQL table lock, wait for
|
||||
`55P03`, verify unchanged approval and no provider call, then prove automatic
|
||||
recovery invokes the provider once and becomes quiet. Additional real-row cases
|
||||
cover changed ownership, mismatched invocation, a started call, prior-dispatch
|
||||
history, and outer-transaction rollback. A released claim cannot start execution.
|
||||
All 117 gateway-service cases pass on the owning slice, including existing target
|
||||
invalidation, provider-outcome persistence failures and unknown-outcome no-replay
|
||||
cases. Integrated consumer checks, workspace verification and remote PR gates
|
||||
must still be recorded after propagation.
|
||||
|
||||
|
||||
## Agent-origin preparation and dispatch ownership
|
||||
|
||||
Agent-origin approvals use the same pre-dispatch retry contract as Test-tab
|
||||
approvals. Previously, a policy or identity dependency failure after claiming an
|
||||
approved action permanently failed it; earlier issue/signing failures could
|
||||
leave the claim executing. Preparation now has one owning error boundary around
|
||||
issue, signature, identity, target, policy, managed-argument, and content checks.
|
||||
Validation and nontransient access rejections retain their terminal handling.
|
||||
Unexpected dependency failures and transient HTTP errors release only the exact,
|
||||
still-undispatched claim to the existing approved queue and publish its existing
|
||||
transactional work signal. No table, maintenance scan, or retry wrapper per
|
||||
producer is added.
|
||||
|
||||
`tool-approval-execution.ts` owns the shared claim token, release, and dispatch
|
||||
transition. Its Test-tab claim/outcome functions remain specific to that consumer.
|
||||
Both origins use the same invocation-then-request locks and verify invocation,
|
||||
company, request linkage, claim timestamp, awaiting-approval state, and absence
|
||||
of an earlier start before dispatch. An agent that loses its claim during
|
||||
preparation cannot call the provider or overwrite its successor. The Test-tab
|
||||
unapproved direct-call path retains its original behavior.
|
||||
|
||||
Only preparation is retryable here. Recording the invocation start, provider
|
||||
execution, and outcome persistence remain outside that catch. A provider error,
|
||||
unknown outcome, or failed durable release does not establish replay safety.
|
||||
Existing interruption handling remains the conservative fallback. Closed-issue
|
||||
expiry, signature/target invalidation, identity authority, and policy rejection
|
||||
remain enforced, including the final issue check before recording dispatch.
|
||||
|
||||
Real PostgreSQL lock-timeout cases cover foreground approval and recovery:
|
||||
observe the committed executing claim, block the policy read, then require the
|
||||
same signed request to remain approved with no invocation start and zero provider
|
||||
calls. Recovery after releasing the table lock executes once and becomes quiet.
|
||||
A separate contention case replaces the claim while preparation is blocked and
|
||||
proves the old owner cannot dispatch; the surviving approval later executes once.
|
||||
A provider-error case proves that an already-started call is not requeued as a
|
||||
preparation failure. The existing Test-tab timeout and ownership cases exercise
|
||||
the other consumer of the shared boundary.
|
||||
|
||||
|
||||
## New or edited rate rules during approval
|
||||
|
||||
A reviewed invocation already has durable `consumedRateAdmissions` in its policy
|
||||
explanation. The original read-only recheck retained unchanged rules correctly,
|
||||
but new or edited rules only inspected their counters. Two approved actions could
|
||||
therefore both execute against one available slot. The real recovery regression
|
||||
failed for both a newly added limit and an increased limit before this repair.
|
||||
|
||||
The policy service now owns `admitInvocation`: lock the company-scoped invocation,
|
||||
validate its exact actor/task/tool/argument binding, recheck current policy, spend
|
||||
only admissions not already recorded for that policy generation, and save the
|
||||
receipt atomically. The gateway supplies its execution claim; that claim is locked
|
||||
and checked by ID, invocation, company, executing status and timestamp. This also
|
||||
supports historical approved requests whose invocation still says pending, without
|
||||
accepting a stale claimant. Existing claim/start/settlement owners still control
|
||||
provider execution. No queue table, periodic query, or per-route wrapper is added.
|
||||
|
||||
Counter updates preserve the number of already spent slots when the limit changes.
|
||||
A rejection by a later policy or failure to save the receipt rolls back new charges.
|
||||
A retry of a committed admission retains the saved receipt, including concurrent
|
||||
rechecks. Read-only approval preflight and browser authority probes stay read-only;
|
||||
trust-rule usage is not counted again during execution revalidation. The policy
|
||||
service and its metric helper accept only the database capabilities they use, so
|
||||
transaction clients do not require a cast to a root connection.
|
||||
|
||||
The owning gateway and policy suites pass all 181 cases against disposable real
|
||||
PostgreSQL databases. Coverage includes two competing invocations, concurrent
|
||||
retries of one invocation, changed limits, failed receipt writes, rejection after
|
||||
an earlier rate charge, unchanged admissions, historical approvals, scope mismatch,
|
||||
claim loss, quiet recovery and unknown provider outcomes. Server typechecking
|
||||
passes after preparing current runner dependencies.
|
||||
|
||||
Earlier verification failures remain part of this evidence: the first full run
|
||||
was stopped because an old claim-loss fixture held the policy table while trying
|
||||
to revoke the newly locked invocation. It now pauses discovery before admission
|
||||
and retains the no-provider-dispatch assertion. A new receipt assertion was fixed
|
||||
to permit the intended additional receipt while preserving original fields. The
|
||||
next full run exposed an overly strict historical-approval check; exact execution-
|
||||
claim validation fixed it. Typechecking then exposed root-client-only signatures,
|
||||
which were narrowed to their actual database capabilities. The final complete
|
||||
suites and TypeScript check pass; stack integration and current-head CI remain
|
||||
separate gates. This proves worker and admission behavior, not deployed host sleep.
|
||||
|
||||
|
||||
## Managed-argument expiry ordering
|
||||
|
||||
Latest-head CI exposed a preparation-order regression in
|
||||
`tool-gateway.test.ts`: a valid historical Shopify approval whose managed UCP
|
||||
arguments were obsolete became `failed` instead of `expired`. The new consuming
|
||||
admission compared the current invocation identity before the existing managed
|
||||
argument expiry owner ran. It prevented dispatch, but bypassed the correct saved
|
||||
expiry outcome.
|
||||
|
||||
Approved execution now verifies target/signed/managed arguments and content
|
||||
before consuming admission. Exact claim fencing, fresh policy checks, atomic
|
||||
rate receipts, and the final issue-open check remain before dispatch. No check
|
||||
was removed and stale payloads are not permitted to execute.
|
||||
|
||||
The existing regression reproduced the CI failure on this owner in
|
||||
`/tmp/sleep-approved-managed-expiry-red.log`. All 92 gateway acceptance cases
|
||||
pass after the ordering change (`/tmp/sleep-approved-managed-expiry-full.log`),
|
||||
including the historical approval's expired request, failed invocation with
|
||||
`approvalState: expired`, and zero provider requests. Server typecheck passes
|
||||
in `/tmp/sleep-approved-managed-expiry-types.log`. Latest-head remote CI and
|
||||
combined-stack acceptance still need their own results.
|
||||
|
||||
## Delegated Test-tab admission after policy changes
|
||||
|
||||
A delegated Test-tab call has an originating agent but no gateway token. The
|
||||
named-gateway admission path therefore did not evaluate its caller policy. If a
|
||||
rate rule was added while approval was pending, approval charged only the target.
|
||||
A shared company/tool bucket then rejected the caller against that same charge;
|
||||
a caller-specific rule instead allowed execution without recording its charge.
|
||||
|
||||
The Test-tab owner now supplies its caller/target policy evaluation to the existing
|
||||
claim-transaction admission API. Both decisions use the claimed invocation's
|
||||
receipts. The target can share the caller's charge only when the rule and bucket
|
||||
match. Both receipts commit together. A denial rolls back new charges and receipts.
|
||||
Management reauthorization and credential identity checks remain with their owners,
|
||||
and provider execution stays outside the transaction. This adds no queue or timer.
|
||||
|
||||
Three actual HTTP/PostgreSQL cases cover a newly added shared rate rule, separate
|
||||
caller and target rules, and a target denial after caller admission. The unchanged
|
||||
implementation failed the first two cases: the shared rule rejected execution,
|
||||
and separate rules left only one charge. The denial case already passed. After
|
||||
the repair, all three pass, including duplicate approval, both saved receipts,
|
||||
one provider execution, and rejection of a subsequent authorized call.
|
||||
|
||||
The full connection-management suite passes all 78 cases against a fresh database.
|
||||
All 92 existing gateway tests and the server production typecheck also pass.
|
||||
Intermediate fixture failures remain accounted for: the new assertion initially
|
||||
used a nonexistent counter count field instead of remaining; the subsequent-call
|
||||
fixture first reused an ended run and then lacked responsible-user authority.
|
||||
The final fixture uses a new live run with the original responsible user. A reused
|
||||
full-suite database also retained a conflicting Slack bot identity; final acceptance
|
||||
uses a fresh isolated database. These are local checks, not current-head CI or a
|
||||
host-suspension qualification.
|
||||
|
||||
|
||||
## Historical argument identity compatibility repair
|
||||
|
||||
Earlier invocations stored the hash of their redacted argument preview. Modern
|
||||
admission correctly requires the complete request identity, including a keyed hash
|
||||
for sensitive arguments. The gateway already verified signed historical arguments,
|
||||
but admission rejected their saved preview hash even with no rate policy. That
|
||||
permanently failed an otherwise valid approval before the provider call.
|
||||
|
||||
Both agent-origin and Test-tab preparation now bind the verified complete identity
|
||||
for a historical invocation before current policy admission. Signature, reviewed
|
||||
target and managed-argument checks still run first. The approval execution owner
|
||||
performs the identity transition in an invocation-then-request lock transaction.
|
||||
It requires the exact current claim, company and invocation relationship, an
|
||||
undispatched awaiting-approval invocation, and the expected reviewed/preview hash.
|
||||
Unrelated saved hashes are not accepted. Modern matching identities do not add
|
||||
this transaction. Current policies and rate charges still apply afterward.
|
||||
|
||||
Four real-PostgreSQL regressions reproduced the defect: valid sensitive historical
|
||||
arguments, with and without a newly configured rate rule, through each of the two
|
||||
origins. Four controls keep invalid signatures and unrelated invocation hashes
|
||||
rejected without dispatch or identity changes. The provider receives the original
|
||||
verified arguments exactly once, rather than their redacted preview. A separate
|
||||
real-transaction case proves a replaced claim cannot change the saved identity.
|
||||
|
||||
The first draft fixtures used a nonexistent request-level tool name and then
|
||||
non-sensitive arguments whose preview and complete hashes were equal. Those
|
||||
fixture failures are not credited as product regressions. The corrected four
|
||||
positive cases fail before the repair and all eight parameter cases pass after.
|
||||
The complete gateway, policy and startup selection passes 230 cases with no skips;
|
||||
server types pass. The complete assembled-server connection-management suite also
|
||||
passes all 78 cases, including delegated approval and rate-policy behavior. Fresh
|
||||
remote CI and reviews remain required after publication.
|
||||
|
||||
|
||||
The repair was then integrated with the master-based parent, including #17918's
|
||||
tracing changes, at `81231824d773a79fc50c835eccf7d9b3977e79fc`. Server types and the
|
||||
production server build pass. The gateway, policy, startup and preparation-timing
|
||||
suites pass all 242 cases. The complete HTTP management suite passes all 78 cases
|
||||
on a fresh dedicated database, for 320 passing cases across the accepted runs.
|
||||
|
||||
The first combined integrated run passed 319 cases and failed the Slack setup
|
||||
case because its reused HTTP database still held the previous run's bot identity.
|
||||
The existing ownership guard correctly rejected the duplicate connection. A fresh
|
||||
dedicated fixture resolved that environment collision; no product behavior or
|
||||
assertion changed for the rerun. That failed run remains part of the evidence.
|
||||
|
||||
|
||||
## Gateway admission uses its transaction connection
|
||||
|
||||
Review of the master-targeted branch exposed a delegated gateway deadlock: the
|
||||
admission transaction held the sole pool connection while issuer/token authority
|
||||
reads used the root client. Foreground approval and approved recovery each failed
|
||||
the real one-connection regression; a revoked-issuer control already passed.
|
||||
The admission callback now receives its transaction and passes it through the
|
||||
existing issuer authority checks. Both connection authorization checks and the
|
||||
token read use that connection. Ordinary checks retain their root client, and
|
||||
current authority, claim, company and rate enforcement remain in place.
|
||||
The three constrained-pool cases pass, as do all 81 HTTP integration cases and
|
||||
190 gateway, policy, startup and deadline consumer cases. The initial consumer
|
||||
command selected an incorrect policy filename and omitted the gateway database
|
||||
override; it was interrupted and is not counted as passing evidence.
|
||||
|
||||
## Pending historical approvals retain their original admission
|
||||
|
||||
A second review finding concerned the normal foreground approval gate, before
|
||||
execution has claimed and rebound historical arguments. A real PostgreSQL
|
||||
regression with signed sensitive arguments and an already-consumed limit of one
|
||||
failed with `rate_limited`; the Test-tab and invalid-signature controls passed.
|
||||
The gateway now supplies the preview digest only after verifying the signed
|
||||
payload and matching either supported form of the review's canonical digest. Policy may match that digest
|
||||
only in a read-only preflight, with every other invocation-context field and
|
||||
stored rate-rule receipt still checked. The consuming admission continues to
|
||||
require the complete keyed identity, rebound under the exact execution claim.
|
||||
New or changed rate rules still apply. Historical recovery, invalid signatures,
|
||||
foreign invocation hashes, stale claims, and provider argument preservation stay
|
||||
covered. This adds no schema, recurring query or additional connection.
|
||||
|
||||
The preceding CI head also failed E2E shard 3/8. Its failure is retained in the
|
||||
validation record; local gateway checks do not erase it. Fresh-head CI and both
|
||||
required automated reviews remain merge gates.
|
||||
|
||||
The five full gateway/policy/startup/deadline suites passed 329 cases before the
|
||||
additional mixed historical identity control. The first combined attempt is
|
||||
retained: a Test-tab failure-result assertion incorrectly expected a thrown
|
||||
error, and reusing the earlier HTTP database left formal-approval foreign keys
|
||||
that prevented gateway acceptance setup. The assertion now checks the existing
|
||||
failed-result contract, with unchanged no-provider, hash and counter checks;
|
||||
the complete acceptance rerun used an independently migrated gateway database.
|
||||
Server production typechecking and build passed. The earlier E2E failure was a
|
||||
500 during the agent-chat project-plan fixture restoration, with a PostgreSQL
|
||||
deadlock in cancellation; it remains a fresh-head CI gate.
|
||||
@@ -18,7 +18,7 @@ Do not mark an item done from a timer removal or a green check alone.
|
||||
| 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Pending |
|
||||
| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Pending |
|
||||
| 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending |
|
||||
| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged; gateway credentials: [#17450](https://github.com/paperclipai/paperclip/pull/17450), merged; pending tool-review expiry implemented on slice 04; other review duties pending |
|
||||
| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged; gateway credentials: [#17450](https://github.com/paperclipai/paperclip/pull/17450), merged; pending tool-review expiry: [#17453](https://github.com/paperclipai/paperclip/pull/17453), merged; approved recovery: [#17454](https://github.com/paperclipai/paperclip/pull/17454), historical argument identity repair verified and awaiting fresh CI/review; other review duties pending |
|
||||
| 08 | GitHub connection events and continuity | Event discovery/replay and credential continuity separately | Pending |
|
||||
| 09 | Merged-PR confirmations | Candidate writes, webhook activation where supported, outstanding-candidate fallback | Pending |
|
||||
| 10 | External-object refresh | Configuration changes, refresh deadlines, source events | Pending |
|
||||
@@ -117,3 +117,116 @@ index, applies/replays its migration, and preserves approval-time rejection and
|
||||
separate approved/executing/projection recovery. Workspace typecheck/build and all 92 existing gateway acceptance tests also
|
||||
passed. The first acceptance command matched no files; the corrected
|
||||
`tool-gateway.test.ts` selection passed. Fresh PR CI/review remains required.
|
||||
|
||||
|
||||
## Approved-review follow-up repairs (2026-10-26)
|
||||
|
||||
Slice 05 preserves the rate slot admitted before a human review. Agent and
|
||||
Test-tab foreground/recovery paths reuse only the exact invocation/context and
|
||||
unchanged rate-policy admission while rechecking current policy. A one-call
|
||||
limit must not reject the same approved operation against its own occupied slot.
|
||||
|
||||
Recovery now locks and rereads invocation/request history before dispatch.
|
||||
Finished records repair the request without provider replay; executing records
|
||||
retain unknown-outcome recovery. Invalid signatures, unusable formal approvals,
|
||||
and inconsistent invocation states settle once. Missing signing-key configuration
|
||||
and temporary dependencies retain retries. Company mismatches never mutate the
|
||||
foreign invocation. No new table or generic HTTP-error discard rule is introduced.
|
||||
|
||||
Formal approvals awaiting human input have no recovery cadence. The existing
|
||||
approval service publishes commit-aware review notifications for approve/reject
|
||||
and cancellation. Its shared record owner retains the decision validation/CAS
|
||||
contract; hire decisions remain with their lifecycle owner. Revision/resubmission
|
||||
keeps the review parked, and startup rediscovers saved work. Budget approval writes
|
||||
belong to budget incidents, not tool-gateway formal reviews. Company deletion
|
||||
removes the dependent work. These exclusions come from the producer audit.
|
||||
|
||||
Local evidence: the four rate-admission regressions, seven permanent-record
|
||||
regressions, and four formal-approval commit/rollback regressions failed before
|
||||
their respective repairs. The complete gateway, policy, approval-service and
|
||||
signature suites pass (208 tests), and server typechecking passes. Configuration
|
||||
failure, company boundaries, concurrent claims, provider-outcome persistence
|
||||
failure, restart and paging remain covered. The paging fixture now has independent
|
||||
invocations instead of 101 approvals sharing an already-expired invocation.
|
||||
Full remote checks and downstream integration verification remain required;
|
||||
this is not an all-stack acceptance or deployed-sleep claim.
|
||||
|
||||
## Next owner audit: approved tool-action recovery
|
||||
|
||||
The next branch is `codex/sleep-05-approved-tool-recovery`, based on pending
|
||||
expiry PR #17453 at `ce6a260a83`. Keep this separate from interrupted execution
|
||||
and terminal projection repair. The public combined sweep remains compatible.
|
||||
|
||||
- `commitToolActionReview` and accepted-interaction inference in the gateway are
|
||||
approved-state producers. Existing notification ownership from slice 04 can
|
||||
carry the new duty without adding another wrapper around callers.
|
||||
- Agent execute-on-approve uses an atomic approved-to-executing claim before
|
||||
provider execution. Keep that claim and all current authorization, signed
|
||||
argument, current-target, identity, and issue-state checks in the gateway.
|
||||
- Approved history is not uniformly executable. Legacy signed arguments without
|
||||
`executionOnApprove` are deliberately inert. The payload can be encrypted;
|
||||
shared scheduling must not decode it or inspect private provider fields.
|
||||
- Board Test-tab invocations are a separate execution origin. The normal fresh
|
||||
approval calls `runApprovedTestInvocation`, but the already-approved branch
|
||||
currently skips test-origin execution. Test execution can leave the request
|
||||
approved while the invocation holds the completed result. Do not treat those
|
||||
completed rows as perpetual unfinished work.
|
||||
- Before driving test-origin recovery, establish the same single-execution
|
||||
claim on both foreground and recovery paths. The existing test helper does
|
||||
not claim the request before provider dispatch. Simply invoking it from a
|
||||
new worker would risk duplicate effects during foreground/recovery races.
|
||||
- Distinguish saved completed outcomes, never-dispatched approved work, and
|
||||
interrupted execution with unknown outcomes. Known completed provider results
|
||||
must not be converted to unknown-outcome failures because another projection
|
||||
write was interrupted. Retain quarantine for actually uncertain calls.
|
||||
- Concrete unresolved approval work may retain a retry; configured idle,
|
||||
terminal invocation history, and deliberately inert legacy approvals must
|
||||
not cause empty recurring scans. Preserve restart reconstruction and all
|
||||
supported producer paths, including transitions that finish an approved row.
|
||||
|
||||
This is an audit, not an implementation or readiness claim. Tests must exercise
|
||||
real approval producers, foreground/recovery contention, provider failure,
|
||||
commit/rollback, legacy inert approvals, completed Test-tab history, and actual
|
||||
unfinished Test-tab recovery before the next PR is ready.
|
||||
|
||||
Current parent CI note: #17450 at `c5c53fc43f` has Greptile 5/5. Its server shard
|
||||
10/12 failed the unchanged login-cleanup consume test waiting for an idle hold.
|
||||
All 16 login-cleanup tests passed locally on recheck. The run remains active;
|
||||
an attempted single-job rerun was rejected as not rerunnable. Recheck after the
|
||||
run finishes before requesting its one justified retry. Do not report it green.
|
||||
|
||||
|
||||
Slice 05 implementation now separates approved recovery from the aggregate tick.
|
||||
The existing review notification boundary wakes it after outer commit. Legacy
|
||||
inert approvals and completed Test-tab history become quiet; unresolved work
|
||||
retains a retry. Test-tab foreground and recovery share a claim and atomic outcome
|
||||
settlement. No provider call occurs inside that transaction. See
|
||||
[approved recovery](../idle-approved-tool-recovery.md). Local service/startup
|
||||
verification passes 122 tests, gateway acceptance passes 92, and workspace
|
||||
typecheck/build pass. Paging and outcome-failure tests passed again after their
|
||||
final strengthening. Full CI/review is not yet complete.
|
||||
|
||||
Parent CI follow-up: #17450's completed run was retried once for its unchanged
|
||||
login-cleanup timeout after all 16 local tests passed. #17453 has a Runner
|
||||
prepared-launch close failure. Seven focused local Runner cases also fail, so
|
||||
this is under investigation rather than assumed flaky. No retry of that run has
|
||||
been requested. Neither parent is reported green without fresh evidence.
|
||||
|
||||
|
||||
Slice 05 is [#17454](https://github.com/paperclipai/paperclip/pull/17454).
|
||||
Review identified a real one-shot outcome-write error path. A new regression
|
||||
failed before the repair; result persistence now runs outside provider-error
|
||||
handling. The repaired head passes 123 service/startup tests, 92 acceptance
|
||||
tests, workspace typecheck and build. Fresh CI/review remains required.
|
||||
#17450 now has passing checks after its single retry. The local Runner
|
||||
reproduction for #17453 was missing the fake-provider binary: its durable state
|
||||
showed ENOENT hidden by close failure. After building the fixture all seven
|
||||
cases pass; one CI retry is running.
|
||||
|
||||
#17453 now has passing checks after its single Runner retry. #17454's initial
|
||||
server shard found an old expected approved state for a completed Test-tab call;
|
||||
the updated regression requires executed and one provider call. All 469
|
||||
tool-access service tests pass after enabling the existing external database
|
||||
path independently of embedded support. Its initial browser shard failed during
|
||||
unrelated plan-task fixture company deletion with a concurrent activity-log FK
|
||||
violation. Inspect fresh-head CI before deciding whether that needs a repair.
|
||||
@@ -503,6 +503,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
|
||||
"0369_late_logan.sql",
|
||||
"0381_mean_scream.sql",
|
||||
"0382_narrow_the_executioner.sql",
|
||||
"0384_funny_adam_warlock.sql",
|
||||
"0383_tool_review_company_expiry.sql",
|
||||
]);
|
||||
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
CREATE INDEX CONCURRENTLY IF NOT EXISTS "tool_action_requests_approved_idx" ON "tool_action_requests" USING btree ("id") WHERE "tool_action_requests"."status" = 'approved';
|
||||
+83
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "2a9439e3-4a2b-4a20-ab2b-2aee3db8cb03",
|
||||
"prevId": "a0fcd9b8-f125-4985-8795-f567466b9aef",
|
||||
"id": "4f96f70a-fd19-4d35-ad8a-8d5559b83cbd",
|
||||
"prevId": "410457fc-266b-4bd0-8bc0-2b14b0634d07",
|
||||
"version": "7",
|
||||
"dialect": "postgresql",
|
||||
"tables": {
|
||||
@@ -52812,6 +52812,66 @@
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"tool_action_requests_pending_expiry_idx": {
|
||||
"name": "tool_action_requests_pending_expiry_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "expires_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"tool_action_requests\".\"status\" = 'pending'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"tool_action_requests_company_pending_expiry_idx": {
|
||||
"name": "tool_action_requests_company_pending_expiry_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "expires_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"tool_action_requests\".\"status\" = 'pending'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"tool_action_requests_approved_idx": {
|
||||
"name": "tool_action_requests_approved_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"tool_action_requests\".\"status\" = 'approved'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"tool_action_requests_invocation_idx": {
|
||||
"name": "tool_action_requests_invocation_idx",
|
||||
"columns": [
|
||||
@@ -55222,6 +55282,21 @@
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"tool_gateway_sessions_expiry_idx": {
|
||||
"name": "tool_gateway_sessions_expiry_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "expires_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"tool_gateway_sessions_run_idx": {
|
||||
"name": "tool_gateway_sessions_run_idx",
|
||||
"columns": [
|
||||
@@ -55967,6 +56042,12 @@
|
||||
"notNull": true,
|
||||
"default": "''"
|
||||
},
|
||||
"responsible_user_id": {
|
||||
"name": "responsible_user_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"owner_note": {
|
||||
"name": "owner_note",
|
||||
"type": "text",
|
||||
+7
@@ -2675,6 +2675,13 @@
|
||||
"when": 1793060791727,
|
||||
"tag": "0383_tool_review_company_expiry",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 384,
|
||||
"version": "7",
|
||||
"when": 1793060791728,
|
||||
"tag": "0384_funny_adam_warlock",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -836,6 +836,7 @@ export const toolActionRequests = pgTable(
|
||||
.where(sql`${table.status} = 'pending'`),
|
||||
index("tool_action_requests_company_pending_expiry_idx").on(table.companyId, table.expiresAt)
|
||||
.where(sql`${table.status} = 'pending'`),
|
||||
index("tool_action_requests_approved_idx").on(table.id).where(sql`${table.status} = 'approved'`),
|
||||
index("tool_action_requests_invocation_idx").on(table.invocationId),
|
||||
index("tool_action_requests_issue_idx").on(table.companyId, table.issueId),
|
||||
],
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
import { createToolGatewayService } from "../services/tool-gateway.js";
|
||||
import { commitToolActionReview } from "../services/tool-action-review.js";
|
||||
import { googleWorkspaceSelectedScopes, defaultGitHubReviewPolicy, SLACK_CHAT_BOT_SCOPES, SLACK_BOT_TOOL_SCOPES } from "@paperclipai/shared";
|
||||
import { randomUUID, generateKeyPairSync } from "node:crypto";
|
||||
import { join } from "node:path";
|
||||
import { createServer } from "node:http";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { beforeAll, afterAll, describe, expect, it, vi } from "vitest";
|
||||
import { activityLog, agentApiKeys, companies, companySecrets, companySecretVersions, chatConversations, chatDeliveries, chatEndpoints, chatExternalPrincipals, chatIdentityLinks, toolPolicies, connectionGrants, connectionGrantDelegations, toolAccessAuditEvents, toolOauthStates, issueThreadInteractions, connectionIntentDeliveries, agents, companyMemberships, heartbeatRuns, issues, principalPermissionGrants, projects, toolConnections, toolMcpGateways, toolMcpGatewayTokens, toolActionRequests, toolCatalogEntries, toolInvocations, toolCallEvents, toolProfileBindings, plugins, adapterAuthSessions, environments, routines, chatGitHubReviews } from "@paperclipai/db";
|
||||
import { createDb, activityLog, agentApiKeys, companies, companySecrets, companySecretVersions, chatConversations, chatDeliveries, chatEndpoints, chatExternalPrincipals, chatIdentityLinks, toolPolicies, toolRateLimitCounters, connectionGrants, connectionGrantDelegations, toolAccessAuditEvents, toolOauthStates, issueThreadInteractions, connectionIntentDeliveries, agents, companyMemberships, heartbeatRuns, issues, principalPermissionGrants, projects, toolConnections, toolMcpGateways, toolMcpGatewayTokens, toolActionRequests, toolCatalogEntries, toolInvocations, toolCallEvents, toolProfileBindings, plugins, adapterAuthSessions, environments, routines, chatGitHubReviews } from "@paperclipai/db";
|
||||
import { findSatisfiedToolConnection, satisfiedConnectionIntentService } from "../services/satisfied-connection-intents.js";
|
||||
import { connectionManagementService, continuingConnectionManagement } from "../services/connection-management.js";
|
||||
import { redactRunnerApiPayloads } from "../services/native-runtime/runner-api-log-redaction.js";
|
||||
@@ -234,7 +236,7 @@ describe("connection management with production default agents", () => {
|
||||
expect(bindings.map(binding => binding.targetId)).toEqual([f.agentId]);
|
||||
});
|
||||
|
||||
it.each(["none", "other-agent", "other-agent-no-access", "other-agent-allow-policy", "other-agent-explicit-grant", "other-agent-caller-ask", "other-agent-caller-rate", "other-agent-shared-rate", "other-agent-credential", "other-agent-credential-none", "other-agent-caller-revoked", "private-status", "agent", "user", "cancelled", "project-block", "rate-limit", "agent-rate-approval", "authorized-policy-edit"] as const)("keeps delegated test policy and approval continuation bounded (%s)", async revocation => {
|
||||
it.each(["none", "other-agent", "other-agent-no-access", "other-agent-allow-policy", "other-agent-explicit-grant", "other-agent-caller-ask", "other-agent-new-rate", "other-agent-new-shared-rate", "other-agent-new-rate-target-block", "other-agent-caller-rate", "other-agent-shared-rate", "other-agent-credential", "other-agent-credential-none", "other-agent-caller-revoked", "private-status", "agent", "user", "cancelled", "project-block", "rate-limit", "agent-rate-approval", "authorized-policy-edit"] as const)("keeps delegated test policy and approval continuation bounded (%s)", async revocation => {
|
||||
const f = await fixture();
|
||||
let targetAgentId = f.agentId;
|
||||
if (revocation.startsWith("other-agent")) {
|
||||
@@ -247,7 +249,7 @@ describe("connection management with production default agents", () => {
|
||||
await lifecycle.process(targetAgentId);
|
||||
await vi.waitFor(async () => expect((await agentService(server.db).getById(targetAgentId))?.status).toBe("idle"));
|
||||
}
|
||||
const allowed = revocation === "none" || revocation === "other-agent" || revocation === "other-agent-caller-ask";
|
||||
const allowed = revocation === "none" || revocation === "other-agent" || revocation === "other-agent-caller-ask" || ["other-agent-new-rate", "other-agent-new-shared-rate"].includes(revocation);
|
||||
let calls = 0;
|
||||
const remote = createServer(async (req, res) => {
|
||||
const chunks: Buffer[] = [];
|
||||
@@ -264,7 +266,7 @@ describe("connection management with production default agents", () => {
|
||||
const address = remote.address();
|
||||
if (!address || typeof address === "string") throw new Error("Fixture did not listen");
|
||||
const mcpUrl = `http://127.0.0.1:${address.port}/mcp`;
|
||||
const freezeRateClock = ["other-agent-caller-rate", "other-agent-shared-rate", "rate-limit", "agent-rate-approval"].includes(revocation);
|
||||
const freezeRateClock = revocation.startsWith("other-agent-new-") || ["other-agent-caller-rate", "other-agent-shared-rate", "rate-limit", "agent-rate-approval"].includes(revocation);
|
||||
if (freezeRateClock) vi.useFakeTimers({ toFake: ["Date"] });
|
||||
try {
|
||||
const created = await f.request("POST", `/companies/${f.companyId}/tools/connections`, { name: "Governed test", transport: "mcp_remote", authKind: "none", transportConfig: { url: mcpUrl } });
|
||||
@@ -381,6 +383,16 @@ describe("connection management with production default agents", () => {
|
||||
if (revocation === "cancelled") await server.db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId));
|
||||
if (revocation === "other-agent-caller-revoked") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "Caller access revoked", policyType: "block", selectors: { connectionId: created.body.id, agentId: f.agentId }, priority: 0 });
|
||||
if (revocation === "project-block") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New project block", policyType: "block", selectors: { connectionId: created.body.id, projectId: f.projectId }, priority: 0 });
|
||||
if (revocation.startsWith("other-agent-new-")) {
|
||||
await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New delegated test admission", policyType: "rate_limit",
|
||||
selectors: { connectionId: created.body.id, ...(revocation === "other-agent-new-shared-rate" ? {} : { agentId: f.agentId }) },
|
||||
priority: -2, config: { limit: 1, windowSeconds: 60, keyBy: revocation === "other-agent-new-shared-rate" ? ["company", "tool"] : ["agent", "tool"] } });
|
||||
if (revocation === "other-agent-new-rate") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New test target admission", policyType: "rate_limit",
|
||||
selectors: { connectionId: created.body.id, agentId: targetAgentId }, priority: -2,
|
||||
config: { limit: 1, windowSeconds: 60, keyBy: ["agent", "tool"] } });
|
||||
if (revocation === "other-agent-new-rate-target-block") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New test target block", policyType: "block",
|
||||
selectors: { connectionId: created.body.id, agentId: targetAgentId }, priority: -1 });
|
||||
}
|
||||
const cookie = await server.humanSession(f.userId);
|
||||
const approve = () => fetch(server.apiUrl + "/api" + approvalPath, { method: "POST", headers: { cookie, origin: server.apiUrl, "Content-Type": "application/json" }, body: JSON.stringify({ companyId: f.companyId }) });
|
||||
const response = await approve();
|
||||
@@ -390,6 +402,20 @@ describe("connection management with production default agents", () => {
|
||||
else expect(settled, JSON.stringify(settled)).toMatchObject({ status: allowed ? "succeeded" : "failed" });
|
||||
expect(calls).toBe(allowed ? 1 : 0);
|
||||
if (allowed) { expect((await approve()).status).toBe(200); expect(calls).toBe(1); }
|
||||
if (revocation.startsWith("other-agent-new-")) {
|
||||
const counters = await server.db.select().from(toolRateLimitCounters).where(eq(toolRateLimitCounters.companyId, f.companyId));
|
||||
expect(counters).toHaveLength(allowed ? revocation === "other-agent-new-shared-rate" ? 1 : 2 : 0);
|
||||
expect(counters.every(counter => counter.remaining === 0)).toBe(true);
|
||||
if (allowed) {
|
||||
expect(settled.policyExplanation?.consumedRateAdmissions).toHaveLength(1);
|
||||
expect(settled.policyExplanation?.delegatedCallerConsumedRateAdmissions).toHaveLength(1);
|
||||
const [nextRun] = await server.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
|
||||
status: "running", responsibleUserId: f.userId, contextSnapshot: { issueId: f.issueId } }).returning();
|
||||
const nextJwt = createLocalAgentJwt(f.agentId, f.companyId, "paperclip_runner", nextRun.id, f.userId);
|
||||
expect(await f.request("POST", path, body, nextJwt)).toMatchObject({ status: 200, body: { decision: "off", error: { reasonCode: "rate_limited" } } });
|
||||
expect(calls).toBe(1);
|
||||
}
|
||||
}
|
||||
} finally { if (freezeRateClock) vi.useRealTimers(); remote.closeAllConnections(); await new Promise<void>((resolve, reject) => remote.close(error => error ? reject(error) : resolve())); }
|
||||
});
|
||||
|
||||
@@ -1229,7 +1255,7 @@ describe("connection management with production default agents", () => {
|
||||
expect((await use()).status).toBeGreaterThanOrEqual(400);
|
||||
});
|
||||
|
||||
it.each(["block", "ask", "ask-board", "ask-revoked", "ask-rate", "ask-token-revoked", "ask-token-deleted", "ask-token-scope", "ask-grant-revoked", "rate", "shared-rate", "target-block", "credential"] as const)("intersects gateway issuer and target rules (%s)", async mode => {
|
||||
it.each(["block", "ask", "ask-board", "ask-revoked", "ask-rate", "ask-new-rate", "ask-new-shared-rate", "ask-new-rate-target-block", "ask-token-revoked", "ask-token-deleted", "ask-token-scope", "ask-grant-revoked", "ask-one-connection", "ask-one-connection-recovery", "ask-one-connection-revoked", "rate", "shared-rate", "target-block", "credential"] as const)("intersects gateway issuer and target rules (%s)", async mode => {
|
||||
const f = await fixture();
|
||||
const cookie = await server.humanSession(f.userId);
|
||||
const createdTarget = await fetch(`${server.apiUrl}/api/companies/${f.companyId}/agents`, { method: "POST", headers: { cookie, origin: server.apiUrl, "Content-Type": "application/json" },
|
||||
@@ -1302,16 +1328,59 @@ describe("connection management with production default agents", () => {
|
||||
if (!mode.startsWith("ask")) return;
|
||||
const [action] = await server.db.select().from(toolActionRequests).where(eq(toolActionRequests.invocationId, invocation.id));
|
||||
expect(action.requestedByAgentId).toBe(mode === "ask-board" ? null : f.agentId);
|
||||
if (mode.startsWith("ask-new-")) {
|
||||
await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New admission after review", policyType: "rate_limit",
|
||||
selectors: { connectionId: connection.body.id, ...(mode === "ask-new-shared-rate" ? {} : { agentId: f.agentId }) },
|
||||
priority: -2, config: { limit: 1, windowSeconds: 60, keyBy: mode === "ask-new-shared-rate" ? ["company", "tool"] : ["agent", "tool"] } });
|
||||
if (mode === "ask-new-rate") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New target admission", policyType: "rate_limit",
|
||||
selectors: { connectionId: connection.body.id, agentId: target.id }, priority: -1,
|
||||
config: { limit: 1, windowSeconds: 60, keyBy: ["agent", "tool"] } });
|
||||
if (mode === "ask-new-rate-target-block") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New target block", policyType: "block",
|
||||
selectors: { connectionId: connection.body.id, agentId: target.id }, priority: -1 });
|
||||
}
|
||||
if (mode === "ask-revoked") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "Issuer blocked after review", policyType: "block", selectors: { connectionId: connection.body.id, agentId: f.agentId }, priority: -1 });
|
||||
if (mode === "ask-token-revoked") await server.db.update(toolMcpGatewayTokens).set({ revokedAt: new Date() }).where(eq(toolMcpGatewayTokens.id, token.body.id));
|
||||
if (mode === "ask-token-deleted") await server.db.delete(toolMcpGatewayTokens).where(eq(toolMcpGatewayTokens.id, token.body.id));
|
||||
if (mode === "ask-token-scope") await server.db.update(toolMcpGatewayTokens).set({ allowedActions: ["tools/list"] }).where(eq(toolMcpGatewayTokens.id, token.body.id));
|
||||
if (mode === "ask-grant-revoked") await server.db.delete(principalPermissionGrants).where(and(eq(principalPermissionGrants.principalId, f.agentId), eq(principalPermissionGrants.permissionKey, "tools:manage_connections")));
|
||||
const approvalDenied = ["ask-revoked", "ask-token-revoked", "ask-token-deleted", "ask-token-scope", "ask-grant-revoked"].includes(mode);
|
||||
const response = await fetch(`${server.apiUrl}/api/tool-gateway/action-requests/${action.id}/approve`, { method: "POST", headers: { cookie, origin: server.apiUrl, "Content-Type": "application/json" }, body: JSON.stringify({ companyId: f.companyId }) });
|
||||
expect(response.status, JSON.stringify(await response.json())).toBe(approvalDenied ? 403 : 200);
|
||||
if (mode === "ask-grant-revoked" || mode === "ask-one-connection-revoked") await server.db.delete(principalPermissionGrants).where(and(eq(principalPermissionGrants.principalId, f.agentId), eq(principalPermissionGrants.permissionKey, "tools:manage_connections")));
|
||||
const approvalDenied = ["ask-revoked", "ask-new-rate-target-block", "ask-token-revoked", "ask-token-deleted", "ask-token-scope", "ask-grant-revoked", "ask-one-connection-revoked"].includes(mode);
|
||||
if (mode.startsWith("ask-one-connection")) {
|
||||
const limited = createDb(server.databaseUrl, { maxConnections: 1, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
||||
const service = createToolGatewayService(limited);
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
let attempt: Promise<unknown> | undefined;
|
||||
try {
|
||||
if (mode === "ask-one-connection-recovery") await commitToolActionReview(server.db, {
|
||||
companyId: f.companyId, actionRequestId: action.id, decision: "approved", actor: { userId: f.userId },
|
||||
});
|
||||
attempt = mode === "ask-one-connection-recovery" ? service.recoverApprovedActionReviews(f.companyId)
|
||||
: service.approveActionRequest({ companyId: f.companyId, actionRequestId: action.id, actor: { userId: f.userId } });
|
||||
void attempt.catch(() => undefined);
|
||||
const completed = Promise.race([attempt, new Promise<never>((_, reject) => {
|
||||
timer = setTimeout(() => reject(new Error("Single-connection gateway approval did not settle")), 5000);
|
||||
})]);
|
||||
if (approvalDenied) await expect(completed).rejects.toMatchObject({ status: 403 });
|
||||
else await completed;
|
||||
expect((await limited.execute(sql`select 1 as available`))[0].available).toBe(1);
|
||||
} finally {
|
||||
clearTimeout(timer); await limited.$client.end({ timeout: 1 }); await attempt?.catch(() => undefined);
|
||||
}
|
||||
} else {
|
||||
const response = await fetch(`${server.apiUrl}/api/tool-gateway/action-requests/${action.id}/approve`, { method: "POST", headers: { cookie, origin: server.apiUrl, "Content-Type": "application/json" }, body: JSON.stringify({ companyId: f.companyId }) });
|
||||
expect(response.status, JSON.stringify(await response.json())).toBe(approvalDenied ? 403 : 200);
|
||||
}
|
||||
expect(calls).toBe(approvalDenied ? 0 : 1);
|
||||
if (mode === "ask-rate") {
|
||||
if (mode.startsWith("ask-new-")) {
|
||||
const counters = await server.db.select().from(toolRateLimitCounters).where(eq(toolRateLimitCounters.companyId, f.companyId));
|
||||
expect(counters).toHaveLength(approvalDenied ? 0 : mode === "ask-new-shared-rate" ? 1 : 2);
|
||||
expect(counters.every(counter => counter.remaining === 0)).toBe(true);
|
||||
if (!approvalDenied) {
|
||||
const [settled] = await server.db.select().from(toolInvocations).where(eq(toolInvocations.id, invocation.id));
|
||||
expect(settled.policyExplanation?.consumedRateAdmissions).toHaveLength(1);
|
||||
expect(settled.policyExplanation?.delegatedCallerConsumedRateAdmissions).toHaveLength(1);
|
||||
}
|
||||
}
|
||||
if (["ask-rate", "ask-new-rate", "ask-new-shared-rate"].includes(mode)) {
|
||||
await rpc("tools/call", { name: tool.name, arguments: { another: true } });
|
||||
expect(calls).toBe(1);
|
||||
}
|
||||
|
||||
@@ -60,6 +60,7 @@ export async function startRunnerApiTestServer(options: {
|
||||
const apiUrl = `http://127.0.0.1:${address.port}`;
|
||||
setupRunnerPrpWebSocketServer(http, { apiUrl });
|
||||
return {
|
||||
databaseUrl: temporary.connectionString,
|
||||
db, root, apiUrl, storage,
|
||||
async humanSession(userId: string) {
|
||||
if (!auth) throw new Error("Enable humanSessions for this fixture");
|
||||
|
||||
@@ -69,6 +69,8 @@ const {
|
||||
stop: vi.fn(async () => {}),
|
||||
},
|
||||
toolGateway: {
|
||||
recoverApprovedActionReviews: vi.fn(async () => ({ scanned: 0, retry: false })),
|
||||
approvedActionReviewCompanies: vi.fn(async () => []),
|
||||
sweepActionReviews: vi.fn(async () => ({ scanned: 0 })),
|
||||
sweepExpiredActionReviews: vi.fn(async () => ({ scanned: 0 })),
|
||||
nextActionReviewExpiryAt: vi.fn(async () => null),
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
activityLog,
|
||||
@@ -34,8 +34,9 @@ import { toolAccessService } from "../services/tool-access.js";
|
||||
import { createToolGatewayService, ToolGatewayHttpError } from "../services/tool-gateway.js";
|
||||
import { canonicalToolArguments, signToolArguments } from "../services/tool-content-guards.js";
|
||||
|
||||
const externalTestDatabaseUrl = process.env.PAPERCLIP_TOOL_POLICY_TEST_DATABASE_URL;
|
||||
const embeddedPostgresSupport = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TOOL_POLICY_TEST_DATABASE_URL");
|
||||
const embeddedPostgresSupport = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
function legacySideEffectKey(invocation: typeof toolInvocations.$inferSelect, argumentsHash: string) {
|
||||
@@ -216,8 +217,11 @@ describeEmbeddedPostgres("tool access policy service", () => {
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
if (!externalTestDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tool-access-policy-");
|
||||
db = createDb(externalTestDatabaseUrl ?? tempDb!.connectionString);
|
||||
if (externalDatabaseUrl) db = createDb(externalDatabaseUrl);
|
||||
else {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tool-access-policy-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
}
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
@@ -1115,11 +1119,103 @@ describeEmbeddedPostgres("tool access policy service", () => {
|
||||
expect(await svc.decide(input, { admittedInvocationId: invocation.id })).toMatchObject({ allowed: true });
|
||||
expect(await db.select().from(toolRateLimitCounters)).toEqual(counters);
|
||||
expect((await db.select().from(toolRateLimitCounters))[0].remaining).toBe(0);
|
||||
// Neither a new invocation nor an accidental consuming recheck can borrow
|
||||
// another invocation's already-spent slot.
|
||||
// A fresh invocation cannot borrow another invocation's already-spent slot.
|
||||
// Execution revalidation retains its own durable admission.
|
||||
expect(await svc.decide(input, { admittedInvocationId: randomUUID() })).toMatchObject({ decision: "rate_limited" });
|
||||
expect(await svc.decide({ ...input, consumeRateLimit: true }, { admittedInvocationId: invocation.id }))
|
||||
.toMatchObject({ decision: "rate_limited" });
|
||||
expect(await svc.admitInvocation(input, invocation.id)).toMatchObject({ allowed: true });
|
||||
expect(await db.select().from(toolRateLimitCounters)).toEqual(counters);
|
||||
});
|
||||
|
||||
it("atomically charges a new rule once across concurrent retries of the same invocation", async () => {
|
||||
const { company, input, invocation, svc } = await admittedRateLimitedInvocation();
|
||||
const added = await svc.createPolicy(company.id, { name: "New admission", policyType: "rate_limit", priority: 0,
|
||||
selectors: { toolName: "send_email" }, config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } });
|
||||
const oldCounters = await db.select().from(toolRateLimitCounters);
|
||||
expect(await svc.decide(input, { admittedInvocationId: invocation.id })).toMatchObject({ allowed: true });
|
||||
expect(await db.select().from(toolRateLimitCounters)).toEqual(oldCounters);
|
||||
const results = await Promise.all(Array.from({ length: 4 }, () => svc.admitInvocation(input, invocation.id)));
|
||||
expect(results.every(result => result.allowed)).toBe(true);
|
||||
expect(await db.select().from(toolRateLimitCounters).where(eq(toolRateLimitCounters.policyId, added.id)))
|
||||
.toMatchObject([{ limit: 1, remaining: 0 }]);
|
||||
const [saved] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, invocation.id));
|
||||
expect(saved.policyExplanation?.consumedRateAdmissions).toHaveLength(2);
|
||||
const { consumedRateAdmissions: oldAdmissions, ...oldExplanation } = invocation.policyExplanation!;
|
||||
expect(saved.policyExplanation).toMatchObject(oldExplanation);
|
||||
expect(saved.policyExplanation?.consumedRateAdmissions).toEqual(expect.arrayContaining(oldAdmissions as unknown[]));
|
||||
});
|
||||
|
||||
it("allows only one of two admitted invocations to spend a newly added final slot", async () => {
|
||||
const { company, input, invocation, svc } = await admittedRateLimitedInvocation(2);
|
||||
const initial = await svc.decide({ ...input, consumeRateLimit: true });
|
||||
expect(initial.allowed).toBe(true);
|
||||
const { invocation: second } = await svc.recordInvocation(input, initial);
|
||||
await db.update(toolInvocations).set({ status: "executing" }).where(eq(toolInvocations.id, second.id));
|
||||
const added = await svc.createPolicy(company.id, { name: "One competing slot", policyType: "rate_limit", priority: 0,
|
||||
selectors: { toolName: "send_email" }, config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } });
|
||||
const results = await Promise.all([invocation.id, second.id].map(id => svc.admitInvocation(input, id)));
|
||||
expect(results.filter(result => result.allowed)).toHaveLength(1);
|
||||
expect(results.filter(result => result.decision === "rate_limited")).toHaveLength(1);
|
||||
expect(await db.select().from(toolRateLimitCounters).where(eq(toolRateLimitCounters.policyId, added.id)))
|
||||
.toMatchObject([{ limit: 1, remaining: 0 }]);
|
||||
});
|
||||
|
||||
it("rolls back new rate charges when a later policy rejects the invocation", async () => {
|
||||
const { company, input, invocation, svc } = await admittedRateLimitedInvocation();
|
||||
await svc.createPolicy(company.id, { name: "Before rejection", policyType: "rate_limit", priority: 0,
|
||||
selectors: { toolName: "send_email" }, config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } });
|
||||
await svc.createPolicy(company.id, { name: "Later rejection", policyType: "block", priority: 10,
|
||||
selectors: { toolName: "send_email" } });
|
||||
const counters = await db.select().from(toolRateLimitCounters);
|
||||
expect(await svc.admitInvocation(input, invocation.id)).toMatchObject({ decision: "deny" });
|
||||
expect(await db.select().from(toolRateLimitCounters)).toEqual(counters);
|
||||
expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, invocation.id)))[0].policyExplanation)
|
||||
.toEqual(invocation.policyExplanation);
|
||||
});
|
||||
|
||||
it("rolls back rate charges when saving the admission receipt fails and admits a retry once", async () => {
|
||||
const { company, input, invocation, svc } = await admittedRateLimitedInvocation();
|
||||
const added = await svc.createPolicy(company.id, { name: "Receipt failure", policyType: "rate_limit", priority: 0,
|
||||
selectors: { toolName: "send_email" }, config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } });
|
||||
await db.execute(sql`create function sleep_admission_fail() returns trigger language plpgsql as $$ begin
|
||||
raise exception 'receipt write unavailable'; end $$`);
|
||||
await db.execute(sql`create trigger sleep_admission_fail before update of policy_explanation on tool_invocations
|
||||
for each row execute function sleep_admission_fail()`);
|
||||
try {
|
||||
await expect(svc.admitInvocation(input, invocation.id)).rejects.toThrow();
|
||||
expect(await db.select().from(toolRateLimitCounters).where(eq(toolRateLimitCounters.policyId, added.id))).toEqual([]);
|
||||
expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, invocation.id)))[0].policyExplanation)
|
||||
.toEqual(invocation.policyExplanation);
|
||||
} finally {
|
||||
await db.execute(sql`drop trigger sleep_admission_fail on tool_invocations`);
|
||||
await db.execute(sql`drop function sleep_admission_fail()`);
|
||||
}
|
||||
expect(await svc.admitInvocation(input, invocation.id)).toMatchObject({ allowed: true });
|
||||
expect(await svc.admitInvocation(input, invocation.id)).toMatchObject({ allowed: true });
|
||||
expect(await db.select().from(toolRateLimitCounters).where(eq(toolRateLimitCounters.policyId, added.id)))
|
||||
.toMatchObject([{ limit: 1, remaining: 0 }]);
|
||||
});
|
||||
|
||||
it.each([2, 4])("preserves spent slots when a rate limit changes from three to %s", async limit => {
|
||||
const { company, input, ratePolicy, svc } = await admittedRateLimitedInvocation(3);
|
||||
await svc.updatePolicy({ companyId: company.id, policyId: ratePolicy.id,
|
||||
body: { config: { limit, windowSeconds: 3600, keyBy: ["agent", "tool"] } } });
|
||||
const results = await Promise.all(Array.from({ length: limit }, () => svc.decide({ ...input, consumeRateLimit: true })));
|
||||
expect(results.filter(result => result.allowed)).toHaveLength(limit - 1);
|
||||
expect(results.filter(result => result.decision === "rate_limited")).toHaveLength(1);
|
||||
expect(await db.select().from(toolRateLimitCounters)).toMatchObject([{ limit, remaining: 0 }]);
|
||||
});
|
||||
|
||||
it.each(["pending", "approved"] as const)("retains a waiting invocation's consumed slot without bypassing %s approval policy", async approvalState => {
|
||||
const { company, input, invocation, svc } = await admittedRateLimitedInvocation();
|
||||
await db.update(toolInvocations).set({ status: "awaiting_approval", policyDecision: "require_approval", approvalState })
|
||||
.where(eq(toolInvocations.id, invocation.id));
|
||||
await db.insert(toolPolicies).values({ companyId: company.id, name: "Review admitted sends",
|
||||
policyType: "require_approval", priority: 2, selectors: { toolName: "send_email" } });
|
||||
const counters = await db.select().from(toolRateLimitCounters);
|
||||
expect(await svc.decide(input, { admittedInvocationId: invocation.id }))
|
||||
.toMatchObject({ allowed: false, decision: "require_approval" });
|
||||
expect(await db.select().from(toolRateLimitCounters)).toEqual(counters);
|
||||
expect(await svc.decide(input)).toMatchObject({ decision: "rate_limited" });
|
||||
});
|
||||
|
||||
it("rechecks a newly added rate policy rather than borrowing an older policy's consumed slot", async () => {
|
||||
@@ -1172,15 +1268,19 @@ describeEmbeddedPostgres("tool access policy service", () => {
|
||||
{ actorType: "system" }, { actorId: randomUUID() }, { agentId: null }, { runId: null }, { issueId: null }, { gatewayId: null },
|
||||
{ applicationId: null }, { connectionId: null }, { catalogEntryId: null }, { toolName: "different_tool" },
|
||||
{ providerType: null }, { applicationKey: null }, { upstreamToolName: null }, { argumentsHash: "different" },
|
||||
{ status: "authorized" }, { policyDecision: "deny" }, { policyDecision: "require_approval", approvalState: "pending" },
|
||||
{ status: "authorized" }, { status: "succeeded" }, { status: "awaiting_approval" },
|
||||
{ status: "awaiting_approval", policyDecision: "require_approval", approvalState: "rejected" }, { policyDecision: "deny" }, { policyDecision: "require_approval", approvalState: "pending" },
|
||||
];
|
||||
for (const mutation of mutations) {
|
||||
await db.update(toolInvocations).set(mutation).where(eq(toolInvocations.id, invocation.id));
|
||||
expect(await svc.decide(input, options), JSON.stringify(mutation)).toMatchObject({ decision: "rate_limited" });
|
||||
expect(await svc.admitInvocation(input, invocation.id), JSON.stringify(mutation)).toMatchObject({ decision: "deny" });
|
||||
await db.update(toolInvocations).set({ ...invocation, status: "executing" }).where(eq(toolInvocations.id, invocation.id));
|
||||
}
|
||||
expect(await svc.decide({ ...input, request: { ...input.request, arguments: { to: "another@example.test" } } }, options))
|
||||
.toMatchObject({ decision: "rate_limited" });
|
||||
expect(await svc.admitInvocation({ ...input, request: { ...input.request, arguments: { to: "another@example.test" } } }, invocation.id))
|
||||
.toMatchObject({ decision: "deny" });
|
||||
});
|
||||
|
||||
it("rejects unsupported policy semantics at create and update time", async () => {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import * as fs from "node:fs/promises";
|
||||
import { createHash, generateKeyPairSync, randomUUID } from "node:crypto";
|
||||
import { createServer } from "node:http";
|
||||
@@ -97,8 +98,8 @@ import { HttpError } from "../errors.js";
|
||||
import type { VercelConnectClient } from "../services/vercel-connect.js";
|
||||
import { invalidatePaperclipCloudConnectorCapabilities, PaperclipCloudConnectorError, type PaperclipCloudConnector } from "../services/paperclip-cloud-connector.js";
|
||||
|
||||
const externalDatabaseUrl = process.env.PAPERCLIP_TOOL_ACCESS_TEST_DATABASE_URL?.trim();
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TOOL_ACCESS_TEST_DATABASE_URL");
|
||||
const embeddedPostgresSupport = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported
|
||||
? describe
|
||||
: describe.skip;
|
||||
@@ -4119,8 +4120,9 @@ describeEmbeddedPostgres("tool access service", () => {
|
||||
.from(toolActionRequests)
|
||||
.where(eq(toolActionRequests.companyId, company.id));
|
||||
expect(requests).toHaveLength(2);
|
||||
expect(globalThis.fetch).toHaveBeenCalledTimes(1);
|
||||
expect(requests.map((row) => row.status).sort()).toEqual([
|
||||
"approved",
|
||||
"executed",
|
||||
"pending",
|
||||
]);
|
||||
});
|
||||
|
||||
@@ -35,12 +35,15 @@ import {
|
||||
toolGatewaySessions,
|
||||
toolInvocations,
|
||||
toolPolicies,
|
||||
toolRateLimitCounters,
|
||||
} from "@paperclipai/db";
|
||||
import type { PluginToolDispatcher } from "../services/plugin-tool-dispatcher.js";
|
||||
import type { VercelConnectClient } from "../services/vercel-connect.js";
|
||||
import { initializeRunIdentity, reserveSteeredIdentity, reconcileSteeredIdentity } from "../services/run-identity.js";
|
||||
import { issueService } from "../services/issues.js";
|
||||
import { approvalService } from "../services/approvals.js";
|
||||
import { commitToolActionReview } from "../services/tool-action-review.js";
|
||||
import { bindVerifiedToolApprovalArguments, claimToolTestApproval, releaseToolApproval, startToolInvocation } from "../services/tool-approval-execution.js";
|
||||
import { materializeNativeInteractionResponses } from "../services/native-runtime/native-interaction-bridge.js";
|
||||
import { toolActionDeliveryService } from "../services/tool-action-delivery.js";
|
||||
import { secretService } from "../services/secrets.js";
|
||||
@@ -49,7 +52,7 @@ import {
|
||||
createToolGatewayService,
|
||||
ToolGatewayHttpError,
|
||||
} from "../services/tool-gateway.js";
|
||||
import { canonicalToolArguments, signToolArguments } from "../services/tool-content-guards.js";
|
||||
import { canonicalToolArguments, readSignedToolArgumentsPayload, signToolArguments, summarizeToolValue, toolRequestIdentityHash } from "../services/tool-content-guards.js";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
@@ -57,6 +60,7 @@ import {
|
||||
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { registerToolReviewRecoveryWork } from "../services/tool-review-recovery-work.js";
|
||||
import { registerToolReviewExpiryWork } from "../services/tool-review-expiry-work.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
|
||||
@@ -210,7 +214,7 @@ function runSessionInput(company: IdRef, agent: IdRef, run: IdRef): Parameters<R
|
||||
describeEmbeddedPostgres("tool gateway service", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
let stopExpiryWorker: (() => Promise<void>) | undefined;
|
||||
let stopReviewWorker: (() => Promise<void>) | undefined;
|
||||
|
||||
beforeAll(async () => {
|
||||
if (!externalDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tool-gateway-");
|
||||
@@ -221,7 +225,7 @@ describeEmbeddedPostgres("tool gateway service", () => {
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
await stopExpiryWorker?.(); stopExpiryWorker = undefined;
|
||||
await stopReviewWorker?.(); stopReviewWorker = undefined;
|
||||
await clearFixtures();
|
||||
});
|
||||
|
||||
@@ -268,7 +272,7 @@ describeEmbeddedPostgres("tool gateway service", () => {
|
||||
await db.insert(toolPolicies).values(reviewNotePolicyRow(fixture.company));
|
||||
gateway = createTestToolGatewayService(db);
|
||||
draining = false; errors.length = 0;
|
||||
stopExpiryWorker = stopWorker;
|
||||
stopReviewWorker = stopWorker;
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true });
|
||||
});
|
||||
async function stopWorker() {
|
||||
@@ -276,6 +280,7 @@ describeEmbeddedPostgres("tool gateway service", () => {
|
||||
vi.useRealTimers(); vi.restoreAllMocks();
|
||||
await db.execute(sql`drop trigger if exists sleep_review_fail on tool_invocations`);
|
||||
await db.execute(sql`drop function if exists sleep_review_fail()`);
|
||||
await db.execute(sql`drop sequence if exists sleep_review_failure_attempt`);
|
||||
expect(errors).toEqual([]);
|
||||
}
|
||||
async function start() {
|
||||
@@ -313,6 +318,710 @@ describeEmbeddedPostgres("tool gateway service", () => {
|
||||
expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toEqual([]);
|
||||
}
|
||||
|
||||
describe("approved review recovery worker", () => {
|
||||
async function startRecovery() {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
await registerToolReviewRecoveryWork(coordinator, gateway, () => !draining).ready;
|
||||
}
|
||||
const approve = (pending: Awaited<ReturnType<typeof request>>, source = db) => commitToolActionReview(source, {
|
||||
companyId: fixture.company.id, actionRequestId: pending.id, decision: "approved", actor: { userId: "board" },
|
||||
});
|
||||
async function executed(pending: Awaited<ReturnType<typeof request>>) {
|
||||
expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("executed");
|
||||
expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0].status).toBe("succeeded");
|
||||
}
|
||||
|
||||
it("is quiet while configured idle and executes a committed approval once", async () => {
|
||||
await startRecovery(); await quiet();
|
||||
const pending = await request(); await approve(pending); await settle();
|
||||
await executed(pending); await quiet();
|
||||
expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("does not scan another company's approved backlog after review writes", async () => {
|
||||
const other = await createRunFixture(db);
|
||||
await db.insert(toolPolicies).values(reviewNotePolicyRow(other.company));
|
||||
const foreign = await request(gateway, db, other);
|
||||
await db.update(toolActionRequests).set({ signedArguments: signToolArguments({
|
||||
invocationId: foreign.invocationId, toolName: "mcp-remote-fixture:update_note",
|
||||
canonicalArguments: canonicalToolArguments({ noteId: "legacy", body: "inert" }),
|
||||
signingSecret: testToolActionSigningSecret,
|
||||
}) }).where(eq(toolActionRequests.id, foreign.id));
|
||||
await commitToolActionReview(db, { companyId: other.company.id, actionRequestId: foreign.id,
|
||||
decision: "approved", actor: { userId: "board" } });
|
||||
await startRecovery(); await quiet();
|
||||
const reads: Array<{ query: string; params: unknown[] }> = [];
|
||||
const debug = db.$client.options.debug;
|
||||
db.$client.options.debug = (_id, query, params) => {
|
||||
if (/^select/i.test(query) && query.includes('"tool_action_requests"') &&
|
||||
query.includes('"tool_action_requests"."status" =') && params.includes("approved"))
|
||||
reads.push({ query, params: [...params] });
|
||||
};
|
||||
try {
|
||||
for (let index = 0; index < 3; index++) {
|
||||
const pending = await request(); await settle(1500);
|
||||
await approve(pending); await settle(1500); await executed(pending);
|
||||
}
|
||||
} finally { db.$client.options.debug = debug; }
|
||||
expect(reads.length).toBeGreaterThan(0);
|
||||
for (const read of reads) {
|
||||
expect(read.query).toContain('"tool_action_requests"."company_id" =');
|
||||
expect(read.params).toContain(fixture.company.id);
|
||||
expect(read.params).not.toContain(other.company.id);
|
||||
}
|
||||
expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, foreign.id)))[0].status).toBe("approved");
|
||||
expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, foreign.invocationId)))[0].status).toBe("awaiting_approval");
|
||||
expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(3);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it.each(["foreground", "recovery"] as const)("preserves the admitted rate slot for an agent action through %s approval", async mode => {
|
||||
await db.insert(toolPolicies).values({ companyId: fixture.company.id, name: "One reviewed note",
|
||||
policyType: "rate_limit", priority: 1, selectors: { toolName: "mcp-remote-fixture:update_note" },
|
||||
config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } });
|
||||
const pending = await request();
|
||||
const counters = await db.select().from(toolRateLimitCounters);
|
||||
expect(counters).toHaveLength(1); expect(counters[0].remaining).toBe(0);
|
||||
if (mode === "foreground") {
|
||||
await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } });
|
||||
await startRecovery();
|
||||
} else { await approve(pending); await startRecovery(); }
|
||||
await settle(); await executed(pending); await quiet();
|
||||
expect(await db.select().from(toolRateLimitCounters)).toEqual(counters);
|
||||
expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it.each(["new", "increased"] as const)("spends the newly available slot of a %s rate rule during approved recovery", async change => {
|
||||
const rate = { companyId: fixture.company.id, name: "Changed reviewed-note limit",
|
||||
policyType: "rate_limit" as const, priority: 1, selectors: { toolName: "mcp-remote-fixture:update_note" },
|
||||
config: { limit: 2, windowSeconds: 3600, keyBy: ["agent", "tool"] } };
|
||||
const original = change === "increased" ? (await db.insert(toolPolicies).values(rate).returning())[0] : null;
|
||||
const first = await request(), second = await request();
|
||||
if (original) {
|
||||
await db.update(toolPolicies).set({ config: { ...rate.config, limit: 3 }, updatedAt: new Date() })
|
||||
.where(eq(toolPolicies.id, original.id));
|
||||
} else {
|
||||
await db.insert(toolPolicies).values({ ...rate, config: { ...rate.config, limit: 1 } });
|
||||
}
|
||||
await approve(first); await approve(second);
|
||||
await startRecovery(); await settle(); await quiet();
|
||||
const requests = await db.select().from(toolActionRequests);
|
||||
expect(requests.filter(row => row.status === "executed")).toHaveLength(1);
|
||||
expect(requests.filter(row => row.status === "failed")).toHaveLength(1);
|
||||
const invocations = await db.select().from(toolInvocations);
|
||||
expect(invocations.filter(row => row.errorCode === "rate_limited")).toHaveLength(1);
|
||||
expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(1);
|
||||
const counters = await db.select().from(toolRateLimitCounters);
|
||||
expect(counters).toHaveLength(1);
|
||||
expect(counters[0]).toMatchObject({ limit: change === "increased" ? 3 : 1, remaining: 0 });
|
||||
});
|
||||
|
||||
it.each([
|
||||
...["agent", "test-tab"].flatMap(origin => ["valid", "new-rate", "invalid-signature", "foreign-hash", "foreground-rate", "foreground-invalid-signature", "foreground-foreign-hash", "foreground-modern-review-hash"]
|
||||
.map(state => ({ origin, state }))),
|
||||
])("recovers historical preview-hash approval ($origin, $state)", async ({ origin, state }) => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const parameters = { value: "reviewed", apiKey: "synthetic-historical-approval-key" };
|
||||
const foreground = state.startsWith("foreground-");
|
||||
if (foreground) await db.insert(toolPolicies).values({ companyId: fixture.company.id,
|
||||
name: "Already admitted historical action", policyType: "rate_limit", priority: 1,
|
||||
selectors: {}, config: { limit: 1, windowSeconds: 3600, keyBy: ["company", "tool"] } });
|
||||
const pending = origin === "agent" ? await agentRequest(remoteCall.provider, parameters)
|
||||
: (await testRequest(remoteCall.provider, false, parameters)).pending;
|
||||
const [originalInvocation] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId));
|
||||
const payload = readSignedToolArgumentsPayload({ signedArguments: pending.signedArguments,
|
||||
invocationId: pending.invocationId, toolName: originalInvocation.toolName, signingSecret: testToolActionSigningSecret })!;
|
||||
expect(payload).not.toBeNull();
|
||||
const legacyHash = summarizeToolValue(payload.arguments).sha256!;
|
||||
const completeHash = toolRequestIdentityHash(payload.arguments, testToolActionSigningSecret);
|
||||
expect(legacyHash).not.toBe(completeHash);
|
||||
const storedHash = state.endsWith("foreign-hash") ? "e".repeat(64) : legacyHash;
|
||||
await db.update(toolInvocations).set({ argumentsHash: storedHash }).where(eq(toolInvocations.id, pending.invocationId));
|
||||
await db.update(toolActionRequests).set({ canonicalArgumentsHash: state === "foreground-modern-review-hash" ? completeHash : legacyHash,
|
||||
...(["invalid-signature", "foreground-invalid-signature"].includes(state) ? { signedArguments: "unreadable-old-signature" } : {}),
|
||||
}).where(eq(toolActionRequests.id, pending.id));
|
||||
if (state === "new-rate") await db.insert(toolPolicies).values({ companyId: fixture.company.id,
|
||||
name: "New admission for historical approval", policyType: "rate_limit", priority: 1,
|
||||
selectors: {}, config: { limit: 1, windowSeconds: 3600, keyBy: ["company", "tool"] } });
|
||||
const counters = foreground ? await db.select().from(toolRateLimitCounters) : [];
|
||||
if (foreground) {
|
||||
expect(counters).toEqual([expect.objectContaining({ remaining: 0 })]);
|
||||
const approval = gateway.approveActionRequest({ companyId: fixture.company.id,
|
||||
actionRequestId: pending.id, actor: { userId: "board" } });
|
||||
if (state === "foreground-invalid-signature") await expect(approval).rejects.toMatchObject({ reasonCode: "action_request_invalidated" });
|
||||
else if (state === "foreground-foreign-hash" && origin === "agent") await expect(approval).rejects.toMatchObject({ reasonCode: "rate_limited" });
|
||||
else await approval;
|
||||
} else await approve(pending);
|
||||
await startRecovery(); await settle();
|
||||
if (state === "valid" || state === "new-rate" || state === "foreground-rate" || state === "foreground-modern-review-hash") {
|
||||
await executed(pending);
|
||||
expect((await db.select().from(toolInvocations))[0].argumentsHash).toBe(completeHash);
|
||||
expect(remoteCall.provider).toHaveBeenCalledOnce();
|
||||
expect(JSON.parse(String(remoteCall.provider.mock.calls[0][1].body)).params.arguments).toEqual(parameters);
|
||||
if (state === "new-rate") expect(await db.select().from(toolRateLimitCounters))
|
||||
.toEqual([expect.objectContaining({ limit: 1, remaining: 0 })]);
|
||||
} else {
|
||||
expect((await db.select().from(toolActionRequests))[0].status)
|
||||
.toBe(["invalid-signature", "foreground-invalid-signature"].includes(state) ? "cancelled"
|
||||
: state === "foreground-foreign-hash" && origin === "agent" ? "pending" : "failed");
|
||||
expect((await db.select().from(toolInvocations))[0].argumentsHash).toBe(storedHash);
|
||||
expect(remoteCall.provider).not.toHaveBeenCalled();
|
||||
}
|
||||
if (foreground) expect(await db.select().from(toolRateLimitCounters)).toEqual(counters);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("does not rebind historical arguments after the execution claim is replaced", async () => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const pending = await agentRequest(remoteCall.provider, { value: "reviewed", apiKey: "synthetic-stale-claim-key" });
|
||||
const [invocation] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId));
|
||||
const payload = readSignedToolArgumentsPayload({ signedArguments: pending.signedArguments,
|
||||
invocationId: invocation.id, toolName: invocation.toolName, signingSecret: testToolActionSigningSecret })!;
|
||||
const preview = summarizeToolValue(payload.arguments).sha256;
|
||||
const verified = toolRequestIdentityHash(payload.arguments, testToolActionSigningSecret);
|
||||
await db.update(toolInvocations).set({ argumentsHash: preview }).where(eq(toolInvocations.id, invocation.id));
|
||||
await db.update(toolActionRequests).set({ canonicalArgumentsHash: preview }).where(eq(toolActionRequests.id, pending.id));
|
||||
await approve(pending);
|
||||
const claim = (await claimToolTestApproval(db, pending.id, invocation.id))!;
|
||||
expect(claim).not.toBeNull();
|
||||
await db.update(toolActionRequests).set({ updatedAt: new Date(claim.claimedAt.getTime() + 1) })
|
||||
.where(eq(toolActionRequests.id, pending.id));
|
||||
expect(await bindVerifiedToolApprovalArguments(db, claim, { preview, verified })).toBe(false);
|
||||
expect((await db.select().from(toolInvocations))[0].argumentsHash).toBe(preview);
|
||||
expect(remoteCall.provider).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("reconstructs a saved approved action after restart", async () => {
|
||||
const pending = await request(); await approve(pending);
|
||||
await startRecovery(); await settle(); await executed(pending); await quiet();
|
||||
});
|
||||
|
||||
it("retries a failed approval dispatch only while the approved action remains unresolved", async () => {
|
||||
const pending = await request(); await approve(pending);
|
||||
const recover = vi.spyOn(gateway, "approveActionRequest").mockRejectedValue(new Error("Recovery dependency unavailable"));
|
||||
await startRecovery();
|
||||
expect(recover).toHaveBeenCalledTimes(1); expect(coordinator!.nextWakeAt()).not.toBeNull();
|
||||
await settle(10_000); expect(recover).toHaveBeenCalledTimes(1);
|
||||
recover.mockRestore(); await settle(); await executed(pending); await quiet();
|
||||
});
|
||||
|
||||
it("preserves a failed company's retry across writes while another company executes", async () => {
|
||||
const pending = await request(); await approve(pending);
|
||||
const original = gateway.approveActionRequest.bind(gateway);
|
||||
const dispatch = vi.spyOn(gateway, "approveActionRequest").mockImplementation(async input => {
|
||||
if (input.companyId === fixture.company.id) throw new Error("Company dependency unavailable");
|
||||
return original(input);
|
||||
});
|
||||
await startRecovery();
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
const retryAt = coordinator!.nextWakeAt();
|
||||
expect(retryAt).not.toBeNull();
|
||||
const later = await request(); await approve(later);
|
||||
const other = await createRunFixture(db);
|
||||
await db.insert(toolPolicies).values(reviewNotePolicyRow(other.company));
|
||||
const foreign = await request(gateway, db, other);
|
||||
await commitToolActionReview(db, { companyId: other.company.id, actionRequestId: foreign.id,
|
||||
decision: "approved", actor: { userId: "board" } });
|
||||
await settle(5000); await executed(foreign);
|
||||
expect(dispatch.mock.calls.filter(([input]) => input.companyId === fixture.company.id)).toHaveLength(1);
|
||||
expect(coordinator!.nextWakeAt()).toBe(retryAt);
|
||||
dispatch.mockRestore();
|
||||
await settle(); await executed(pending); await executed(later); await quiet();
|
||||
expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(3);
|
||||
});
|
||||
|
||||
it("disarms a failed company's recovery after company deletion commits", async () => {
|
||||
const pending = await request(); await approve(pending);
|
||||
const dispatch = vi.spyOn(gateway, "approveActionRequest").mockRejectedValue(new Error("Company dependency unavailable"));
|
||||
await startRecovery(); expect(coordinator!.nextWakeAt()).not.toBeNull();
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() })
|
||||
.where(eq(heartbeatRuns.id, fixture.run.id));
|
||||
await db.update(agents).set({ status: "terminated", lifecycleState: "terminated" })
|
||||
.where(eq(agents.id, fixture.agent.id));
|
||||
await deleteCompany(db, fixture.company.id);
|
||||
await settle(1500);
|
||||
expect(coordinator!.nextWakeAt()).toBeNull();
|
||||
expect(dispatch).toHaveBeenCalledTimes(1); await quiet();
|
||||
});
|
||||
|
||||
it.each([null, "unreadable-old-signature"])("settles an approved invalid signature (%s) without recurring recovery", async signedArguments => {
|
||||
const pending = await request(); await approve(pending);
|
||||
await db.update(toolActionRequests).set({ signedArguments }).where(eq(toolActionRequests.id, pending.id));
|
||||
const dispatch = vi.spyOn(gateway, "approveActionRequest");
|
||||
await startRecovery(); await settle();
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe("cancelled");
|
||||
expect((await db.select().from(toolInvocations))[0]).toMatchObject({
|
||||
status: "failed", errorCode: "action_request_invalidated",
|
||||
});
|
||||
expect(dispatch).not.toHaveBeenCalled(); await quiet();
|
||||
});
|
||||
|
||||
it.each(["rejected", "cancelled"])("settles a %s formal approval without recurring recovery", async status => {
|
||||
const pending = await request(); await approve(pending);
|
||||
const [formal] = await db.insert(approvals).values({ companyId: fixture.company.id,
|
||||
type: "request_board_approval", status, payload: {} }).returning();
|
||||
await db.update(toolActionRequests).set({ approvalId: formal.id }).where(eq(toolActionRequests.id, pending.id));
|
||||
const dispatch = vi.spyOn(gateway, "approveActionRequest");
|
||||
await startRecovery(); await settle();
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe("cancelled");
|
||||
expect((await db.select().from(toolInvocations))[0]).toMatchObject({
|
||||
status: "denied", errorCode: "formal_approval_unavailable",
|
||||
});
|
||||
expect(dispatch).not.toHaveBeenCalled(); await quiet();
|
||||
});
|
||||
|
||||
it.each([false, true])("parks a formal approval until its decision commits (rollback %s)", async rollback => {
|
||||
const pending = await request(); await approve(pending);
|
||||
const [formal] = await db.insert(approvals).values({ companyId: fixture.company.id,
|
||||
type: "request_board_approval", status: "pending", payload: {} }).returning();
|
||||
await db.update(toolActionRequests).set({ approvalId: formal.id }).where(eq(toolActionRequests.id, pending.id));
|
||||
await startRecovery(); await quiet();
|
||||
let release!: () => void, staged!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const ready = new Promise<void>(resolve => { staged = resolve; });
|
||||
const aborted = new Error("formal approval rollback fixture");
|
||||
const writing = db.transaction(async tx => {
|
||||
await approvalService(tx as unknown as typeof db).approve(formal.id, "board");
|
||||
staged(); await gate; if (rollback) throw aborted;
|
||||
}).catch(error => { if (error !== aborted) throw error; });
|
||||
await Promise.race([ready, writing]);
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe("approved");
|
||||
expect((await db.select().from(toolInvocations))[0].status).toBe("awaiting_approval");
|
||||
} finally { release(); }
|
||||
await writing; await settle();
|
||||
if (rollback) {
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe("approved"); await quiet();
|
||||
await approvalService(db).approve(formal.id, "board"); await settle();
|
||||
}
|
||||
await executed(pending); await quiet();
|
||||
expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it.each(["reject", "cancel"] as const)("wakes a parked revision-requested formal approval on %s", async decision => {
|
||||
const pending = await request(); await approve(pending);
|
||||
const [formal] = await db.insert(approvals).values({ companyId: fixture.company.id,
|
||||
type: "request_board_approval", status: "revision_requested", payload: {} }).returning();
|
||||
await db.update(toolActionRequests).set({ approvalId: formal.id }).where(eq(toolActionRequests.id, pending.id));
|
||||
await startRecovery(); await quiet();
|
||||
const service = approvalService(db);
|
||||
if (decision === "cancel") await service.cancel(formal.id, "Cancelled by user");
|
||||
else await service.reject(formal.id, "board");
|
||||
await settle();
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe("cancelled");
|
||||
expect((await db.select().from(toolInvocations))[0]).toMatchObject({ status: "denied", errorCode: "formal_approval_unavailable" });
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("retains approved work when the signing key is temporarily unavailable", async () => {
|
||||
const pending = await request(); await approve(pending);
|
||||
vi.stubEnv("PAPERCLIP_TOOL_ACTION_SIGNING_SECRET", "");
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "invalid-master-key");
|
||||
gateway = createToolGatewayService(db);
|
||||
await startRecovery();
|
||||
expect(coordinator!.nextWakeAt()).not.toBeNull();
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe("approved");
|
||||
expect((await db.select().from(toolInvocations))[0].status).toBe("awaiting_approval");
|
||||
vi.stubEnv("PAPERCLIP_TOOL_ACTION_SIGNING_SECRET", testToolActionSigningSecret);
|
||||
await settle(); await executed(pending); await quiet();
|
||||
});
|
||||
|
||||
it("does not accept another company's formal approval", async () => {
|
||||
const pending = await request(); await approve(pending);
|
||||
const other = await createRunFixture(db);
|
||||
const [formal] = await db.insert(approvals).values({ companyId: other.company.id,
|
||||
type: "request_board_approval", status: "approved", payload: {} }).returning();
|
||||
await db.update(toolActionRequests).set({ approvalId: formal.id }).where(eq(toolActionRequests.id, pending.id));
|
||||
await startRecovery(); await settle();
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe("cancelled");
|
||||
expect((await db.select().from(toolInvocations))[0]).toMatchObject({ status: "denied", errorCode: "formal_approval_unavailable" });
|
||||
expect((await db.select().from(approvals))[0]).toEqual(formal);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("quarantines a mismatched request without modifying another company's invocation", async () => {
|
||||
const pending = await request(); await approve(pending);
|
||||
const [invocation] = await db.select().from(toolInvocations);
|
||||
const other = await createRunFixture(db);
|
||||
await db.update(toolActionRequests).set({ companyId: other.company.id }).where(eq(toolActionRequests.id, pending.id));
|
||||
await startRecovery(); await settle();
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe("failed");
|
||||
expect((await db.select().from(toolInvocations))[0]).toEqual(invocation);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("repairs completed agent history without executing the action again", async () => {
|
||||
const pending = await request();
|
||||
await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } });
|
||||
await executed(pending);
|
||||
const [invocation] = await db.select().from(toolInvocations);
|
||||
await db.update(toolActionRequests).set({ status: "approved", signedArguments: "unreadable-old-signature" })
|
||||
.where(eq(toolActionRequests.id, pending.id));
|
||||
await startRecovery(); await settle(); await executed(pending); await quiet();
|
||||
expect((await db.select().from(toolInvocations))[0]).toEqual(invocation);
|
||||
expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("keeps deliberately inert legacy approvals quiet", async () => {
|
||||
const pending = await request();
|
||||
await db.update(toolActionRequests).set({ signedArguments: signToolArguments({
|
||||
invocationId: pending.invocationId, toolName: "mcp-remote-fixture:update_note",
|
||||
canonicalArguments: canonicalToolArguments({ noteId: "legacy", body: "inert" }),
|
||||
signingSecret: testToolActionSigningSecret,
|
||||
}) }).where(eq(toolActionRequests.id, pending.id));
|
||||
await approve(pending); await startRecovery(); await quiet();
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe("approved");
|
||||
expect((await db.select().from(toolInvocations))[0].status).toBe("awaiting_approval");
|
||||
});
|
||||
|
||||
it.each([false, true])("waits for the outer approval transaction (rollback %s)", async rollback => {
|
||||
await startRecovery(); const pending = await request();
|
||||
let release!: () => void, staged!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const ready = new Promise<void>(resolve => { staged = resolve; });
|
||||
const aborted = new Error("approval rollback fixture");
|
||||
const writing = db.transaction(async tx => {
|
||||
await approve(pending, tx as unknown as typeof db); staged(); await gate;
|
||||
if (rollback) throw aborted;
|
||||
}).catch(error => { if (error !== aborted) throw error; });
|
||||
await Promise.race([ready, writing]);
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe("pending");
|
||||
} finally { release(); }
|
||||
await writing; await settle();
|
||||
if (rollback) expect((await db.select().from(toolActionRequests))[0].status).toBe("pending");
|
||||
else await executed(pending);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("retains the first approval during drain without passive queries", async () => {
|
||||
await startRecovery(); await quiet(); draining = true;
|
||||
const pending = await request(); await approve(pending); await settle();
|
||||
const reads = vi.spyOn(db, "select");
|
||||
await settle(60 * 60_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore();
|
||||
draining = false; await settle(); await executed(pending); await quiet();
|
||||
});
|
||||
|
||||
async function testRequest(provider: NonNullable<ToolGatewayServiceOptions["remoteHttpRequest"]>, rateLimited = false, parameters: Record<string, unknown> = { value: "reviewed" }) {
|
||||
await db.insert(companyMemberships).values({ companyId: fixture.company.id, principalType: "user",
|
||||
principalId: "board", status: "active", membershipRole: "owner" });
|
||||
const remote = await createRemoteMcpToolFixture(db, fixture.company.id);
|
||||
await db.update(toolCatalogEntries).set({ isReadOnly: false, riskLevel: "write" })
|
||||
.where(eq(toolCatalogEntries.id, remote.catalogEntry.id));
|
||||
await db.insert(toolPolicies).values({ companyId: fixture.company.id, name: "Review connected writes",
|
||||
policyType: "require_approval", priority: 10, selectors: { connectionId: remote.connection.id } });
|
||||
if (rateLimited) await db.insert(toolPolicies).values({ companyId: fixture.company.id, name: "One reviewed test call",
|
||||
policyType: "rate_limit", priority: 1, selectors: { connectionId: remote.connection.id },
|
||||
config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } });
|
||||
gateway = createTestToolGatewayService(db, { remoteHttpRequest: provider });
|
||||
const result = await gateway.executeTestCall({ companyId: fixture.company.id, connectionId: remote.connection.id,
|
||||
agentId: fixture.agent.id, userId: "board", toolName: "needs_input", parameters });
|
||||
expect(result.decision).toBe("ask_first");
|
||||
const [pending] = await db.select().from(toolActionRequests);
|
||||
return { pending, remote };
|
||||
}
|
||||
function providerFixture() {
|
||||
let release!: () => void, entered!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const started = new Promise<void>(resolve => { entered = resolve; });
|
||||
const provider = vi.fn(async (_url, init) => {
|
||||
const payload = JSON.parse(String(init.body));
|
||||
expect(payload.method).toBe("tools/call");
|
||||
entered(); await gate;
|
||||
return Response.json({ jsonrpc: "2.0", id: payload.id,
|
||||
result: { content: [{ type: "text", text: "Saved once" }] } });
|
||||
});
|
||||
return { provider, release, started };
|
||||
}
|
||||
|
||||
it.each(["foreground", "recovery"] as const)("retries a Test-tab %s approval after a real pre-dispatch database timeout", async mode => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const { pending } = await testRequest(remoteCall.provider); await approve(pending);
|
||||
const preparationDb = createDb(externalDatabaseUrl ?? tempDb!.connectionString, {
|
||||
maxConnections: 1, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0,
|
||||
});
|
||||
const preparingGateway = createTestToolGatewayService(preparationDb, { remoteHttpRequest: remoteCall.provider });
|
||||
let unlock!: () => void, locked!: () => void;
|
||||
const gate = new Promise<void>(resolve => { unlock = resolve; });
|
||||
const acquired = new Promise<void>(resolve => { locked = resolve; });
|
||||
const lock = db.transaction(async tx => {
|
||||
await tx.execute(sql`lock table tool_policies in access exclusive mode`);
|
||||
locked(); await gate;
|
||||
});
|
||||
await acquired;
|
||||
try {
|
||||
await preparationDb.execute(sql`set lock_timeout = '2s'`);
|
||||
const attempting = mode === "foreground"
|
||||
? preparingGateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } })
|
||||
: preparingGateway.recoverApprovedActionReviews(fixture.company.id);
|
||||
// Observe the real committed claim while its subsequent policy read
|
||||
// is blocked. Provider execution has not started.
|
||||
void attempting.catch(() => undefined);
|
||||
await vi.waitFor(async () => {
|
||||
expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("executing");
|
||||
});
|
||||
if (mode === "foreground") await expect(attempting).rejects.toMatchObject({ cause: { code: "55P03" } });
|
||||
else expect(await attempting).toEqual({ scanned: 1, retry: true });
|
||||
expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0])
|
||||
.toMatchObject({ status: "approved", signedArguments: pending.signedArguments });
|
||||
expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0])
|
||||
.toMatchObject({ status: "awaiting_approval", startedAt: null });
|
||||
expect(remoteCall.provider).not.toHaveBeenCalled();
|
||||
} finally { unlock(); await lock; await preparationDb.$client.end({ timeout: 1 }); }
|
||||
await startRecovery(); await settle(); await executed(pending); await quiet();
|
||||
expect(remoteCall.provider).toHaveBeenCalledOnce();
|
||||
}, 15_000);
|
||||
|
||||
async function agentRequest(provider: NonNullable<ToolGatewayServiceOptions["remoteHttpRequest"]>, parameters: Record<string, unknown> = { value: "reviewed" }) {
|
||||
const remote = await createRemoteMcpToolFixture(db, fixture.company.id);
|
||||
await db.update(toolCatalogEntries).set({ isReadOnly: false, riskLevel: "write" })
|
||||
.where(eq(toolCatalogEntries.id, remote.catalogEntry.id));
|
||||
await db.insert(toolPolicies).values({ companyId: fixture.company.id, name: "Review agent writes",
|
||||
policyType: "require_approval", selectors: { connectionId: remote.connection.id } });
|
||||
gateway = createTestToolGatewayService(db, { remoteHttpRequest: provider });
|
||||
const session = await gateway.createSession(runSessionInput(fixture.company, fixture.agent, fixture.run));
|
||||
const tool = (await gateway.listToolsForSession(session.token)).find(t => t.connectionId === remote.connection.id)!;
|
||||
await expect(gateway.executeTool({ sessionToken: session.token, tool: tool.name, parameters }))
|
||||
.rejects.toMatchObject({ reasonCode: "approval_required" });
|
||||
const [pending] = await db.select().from(toolActionRequests);
|
||||
return pending;
|
||||
}
|
||||
|
||||
it.each(["foreground", "recovery"] as const)("retries an agent-origin %s approval after a real pre-dispatch database timeout", async mode => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const pending = await agentRequest(remoteCall.provider); await approve(pending);
|
||||
const preparationDb = createDb(externalDatabaseUrl ?? tempDb!.connectionString, {
|
||||
maxConnections: 1, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0,
|
||||
});
|
||||
const preparingGateway = createTestToolGatewayService(preparationDb, { remoteHttpRequest: remoteCall.provider });
|
||||
let unlock!: () => void, locked!: () => void;
|
||||
const gate = new Promise<void>(resolve => { unlock = resolve; });
|
||||
const acquired = new Promise<void>(resolve => { locked = resolve; });
|
||||
const lock = db.transaction(async tx => {
|
||||
await tx.execute(sql`lock table tool_policies in access exclusive mode`);
|
||||
locked(); await gate;
|
||||
});
|
||||
await acquired;
|
||||
try {
|
||||
await preparationDb.execute(sql`set lock_timeout = '2s'`);
|
||||
const attempting = mode === "foreground"
|
||||
? preparingGateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } })
|
||||
: preparingGateway.recoverApprovedActionReviews(fixture.company.id);
|
||||
// Observe the real committed claim while its subsequent policy read
|
||||
// is blocked. Provider execution has not started.
|
||||
void attempting.catch(() => undefined);
|
||||
await vi.waitFor(async () => {
|
||||
expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("executing");
|
||||
});
|
||||
if (mode === "foreground") expect(await attempting).toMatchObject({ status: "approved" });
|
||||
else expect(await attempting).toEqual({ scanned: 1, retry: true });
|
||||
expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0])
|
||||
.toMatchObject({ status: "approved", signedArguments: pending.signedArguments });
|
||||
expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0])
|
||||
.toMatchObject({ status: "awaiting_approval", startedAt: null });
|
||||
expect(remoteCall.provider).not.toHaveBeenCalled();
|
||||
} finally { unlock(); await lock; await preparationDb.$client.end({ timeout: 1 }); }
|
||||
await startRecovery(); await settle(); await executed(pending); await quiet();
|
||||
expect(remoteCall.provider).toHaveBeenCalledOnce();
|
||||
}, 15_000);
|
||||
|
||||
it("refuses agent dispatch after preparation loses its claim", async () => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const pending = await agentRequest(remoteCall.provider); await approve(pending);
|
||||
let unlock!: () => void, locked!: () => void;
|
||||
const gate = new Promise<void>(resolve => { unlock = resolve; });
|
||||
const acquired = new Promise<void>(resolve => { locked = resolve; });
|
||||
const lock = db.transaction(async tx => {
|
||||
// Pause tool discovery before admission locks the invocation. Revocation
|
||||
// cannot legitimately take that lock from an in-flight admission.
|
||||
await tx.execute(sql`lock table tool_catalog_entries in access exclusive mode`);
|
||||
locked(); await gate;
|
||||
});
|
||||
await acquired;
|
||||
const attempting = gateway.approveActionRequest({ companyId: fixture.company.id,
|
||||
actionRequestId: pending.id, actor: { userId: "board" } });
|
||||
void attempting.catch(() => undefined);
|
||||
let successor!: NonNullable<Awaited<ReturnType<typeof claimToolTestApproval>>>;
|
||||
try {
|
||||
await vi.waitFor(async () => {
|
||||
expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("executing");
|
||||
});
|
||||
const [claimed] = await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id));
|
||||
expect(await releaseToolApproval(db, { actionRequestId: pending.id,
|
||||
invocationId: pending.invocationId, claimedAt: claimed.updatedAt })).toBe(true);
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
successor = (await claimToolTestApproval(db, pending.id, pending.invocationId))!;
|
||||
expect(successor).not.toBeNull();
|
||||
expect(successor.claimedAt.getTime()).toBeGreaterThan(claimed.updatedAt.getTime());
|
||||
} finally { unlock(); await lock; }
|
||||
expect(await attempting).toMatchObject({ status: "executing" });
|
||||
expect(remoteCall.provider).not.toHaveBeenCalled();
|
||||
expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0])
|
||||
.toMatchObject({ status: "awaiting_approval", startedAt: null });
|
||||
expect(await releaseToolApproval(db, successor)).toBe(true);
|
||||
await startRecovery(); await settle(); await executed(pending); await quiet();
|
||||
expect(remoteCall.provider).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("does not retry an agent provider error as a preparation failure", async () => {
|
||||
const provider = vi.fn(async () => { throw new Error("Provider outcome is unknown"); });
|
||||
const pending = await agentRequest(provider); await approve(pending);
|
||||
await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } });
|
||||
expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("failed");
|
||||
expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0].startedAt).not.toBeNull();
|
||||
await startRecovery(); await quiet();
|
||||
expect(provider).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it.each(["started", "reclaimed", "mismatched", "prior_dispatch"] as const)("refuses to release a Test-tab preparation claim with %s ownership evidence", async state => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const { pending } = await testRequest(remoteCall.provider); await approve(pending);
|
||||
const claim = await claimToolTestApproval(db, pending.id, pending.invocationId);
|
||||
expect(claim).not.toBeNull();
|
||||
if (state === "started") expect(await startToolInvocation(db, pending.invocationId, claim!)).toBe(true);
|
||||
if (state === "reclaimed") await db.update(toolActionRequests)
|
||||
.set({ updatedAt: new Date(claim!.claimedAt.getTime() + 1000) }).where(eq(toolActionRequests.id, pending.id));
|
||||
if (state === "prior_dispatch") await db.update(toolInvocations)
|
||||
.set({ startedAt: new Date() }).where(eq(toolInvocations.id, pending.invocationId));
|
||||
const beforeRequest = (await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0];
|
||||
const beforeInvocation = (await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0];
|
||||
expect(await releaseToolApproval(db, state === "mismatched" ? { ...claim!, invocationId: randomUUID() } : claim!)).toBe(false);
|
||||
expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0]).toEqual(beforeRequest);
|
||||
expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0]).toEqual(beforeInvocation);
|
||||
expect(remoteCall.provider).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("rolls a Test-tab preparation release back with its transaction and invalidates the released claim", async () => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const { pending } = await testRequest(remoteCall.provider); await approve(pending);
|
||||
const claim = await claimToolTestApproval(db, pending.id, pending.invocationId);
|
||||
expect(claim).not.toBeNull();
|
||||
await expect(db.transaction(async tx => {
|
||||
expect(await releaseToolApproval(tx as unknown as typeof db, claim!)).toBe(true);
|
||||
throw new Error("release rollback");
|
||||
})).rejects.toThrow("release rollback");
|
||||
expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("executing");
|
||||
expect(await releaseToolApproval(db, claim!)).toBe(true);
|
||||
expect(await startToolInvocation(db, pending.invocationId, claim!)).toBe(false);
|
||||
await startRecovery(); await settle(); await executed(pending); await quiet();
|
||||
expect(remoteCall.provider).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it.each(["foreground", "recovery"] as const)("claims a Test-tab action once when %s wins", async winner => {
|
||||
const remoteCall = providerFixture();
|
||||
const { pending } = await testRequest(remoteCall.provider);
|
||||
let running: Promise<unknown>;
|
||||
if (winner === "foreground") {
|
||||
await startRecovery();
|
||||
running = gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } });
|
||||
} else {
|
||||
await approve(pending);
|
||||
running = startRecovery();
|
||||
}
|
||||
await Promise.race([remoteCall.started, running]);
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } });
|
||||
expect(remoteCall.provider).toHaveBeenCalledTimes(1);
|
||||
} finally { remoteCall.release(); }
|
||||
await running; await settle(); await executed(pending); await quiet();
|
||||
expect(remoteCall.provider).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it.each(["foreground", "recovery"] as const)("preserves the admitted rate slot for a Test-tab action through %s approval", async mode => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const { pending } = await testRequest(remoteCall.provider, true);
|
||||
const counters = await db.select().from(toolRateLimitCounters);
|
||||
expect(counters).toHaveLength(1); expect(counters[0].remaining).toBe(0);
|
||||
if (mode === "foreground") {
|
||||
await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } });
|
||||
await startRecovery();
|
||||
} else { await approve(pending); await startRecovery(); }
|
||||
await settle(); await executed(pending); await quiet();
|
||||
expect(await db.select().from(toolRateLimitCounters)).toEqual(counters);
|
||||
expect(remoteCall.provider).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("repairs completed Test-tab history without sending the request again", async () => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const { pending } = await testRequest(remoteCall.provider);
|
||||
await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } });
|
||||
await executed(pending);
|
||||
await db.update(toolActionRequests).set({ status: "approved", signedArguments: "unreadable-old-signature" })
|
||||
.where(eq(toolActionRequests.id, pending.id));
|
||||
remoteCall.provider.mockClear();
|
||||
await startRecovery(); await settle(); await executed(pending); await quiet();
|
||||
expect(remoteCall.provider).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["pending", "authorized", "cancelled"])("settles inconsistent Test-tab %s history without replay", async status => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const { pending } = await testRequest(remoteCall.provider); await approve(pending);
|
||||
await db.update(toolInvocations).set({ status }).where(eq(toolInvocations.id, pending.invocationId));
|
||||
await startRecovery(); await settle();
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe(status === "cancelled" ? "cancelled" : "failed");
|
||||
expect((await db.select().from(toolInvocations))[0]).toMatchObject(status === "cancelled"
|
||||
? { status: "cancelled" }
|
||||
: { status: "failed", errorCode: "tool_recovery_state_invalid" });
|
||||
await quiet(); expect(remoteCall.provider).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not replay an interrupted legacy Test-tab invocation", async () => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const { pending } = await testRequest(remoteCall.provider);
|
||||
await approve(pending);
|
||||
await db.update(toolInvocations).set({ status: "executing", startedAt: new Date(Date.now() - 11 * 60_000) })
|
||||
.where(eq(toolInvocations.id, pending.invocationId));
|
||||
await startRecovery(); await settle();
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe("executing");
|
||||
await gateway.sweepActionReviews({ includeApproved: false, includeExpiry: false });
|
||||
expect((await db.select().from(toolInvocations))[0]).toMatchObject({ status: "failed", errorCode: "tool_execution_outcome_unknown" });
|
||||
await quiet(); expect(remoteCall.provider).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["always", "once"])("does not replay a Test-tab call when persisting its external result fails %s", async mode => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const { pending } = await testRequest(remoteCall.provider);
|
||||
await approve(pending);
|
||||
await db.execute(sql`create sequence sleep_review_failure_attempt`);
|
||||
await db.execute(sql.raw(`create function sleep_review_fail() returns trigger language plpgsql as $$
|
||||
begin
|
||||
if ${mode === "always" ? "true" : "nextval('sleep_review_failure_attempt') = 1"} then
|
||||
raise exception 'approval result fixture failure';
|
||||
end if;
|
||||
return new;
|
||||
end $$`));
|
||||
await db.execute(sql`create trigger sleep_review_fail before update on tool_action_requests
|
||||
for each row when (new.status in ('executed', 'failed')) execute function sleep_review_fail()`);
|
||||
try {
|
||||
await startRecovery();
|
||||
expect((await db.select().from(toolActionRequests))[0].status).toBe("executing");
|
||||
expect((await db.select().from(toolInvocations))[0].status).toBe("executing");
|
||||
expect(remoteCall.provider).toHaveBeenCalledTimes(1);
|
||||
} finally {
|
||||
await db.execute(sql`drop trigger sleep_review_fail on tool_action_requests`);
|
||||
}
|
||||
await settle();
|
||||
await vi.advanceTimersByTimeAsync(11 * 60_000);
|
||||
await gateway.sweepActionReviews({ includeApproved: false, includeExpiry: false });
|
||||
expect((await db.select().from(toolInvocations))[0]).toMatchObject({ status: "failed", errorCode: "tool_execution_outcome_unknown" });
|
||||
await quiet(); expect(remoteCall.provider).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("revalidates the reviewed Test-tab target before recovery dispatch", async () => {
|
||||
const remoteCall = providerFixture(); remoteCall.release();
|
||||
const { pending, remote } = await testRequest(remoteCall.provider);
|
||||
await approve(pending);
|
||||
await db.update(toolCatalogEntries).set({ versionHash: "changed-after-review" }).where(eq(toolCatalogEntries.id, remote.catalogEntry.id));
|
||||
await startRecovery(); await settle();
|
||||
expect((await db.select().from(toolInvocations))[0]).toMatchObject({ status: "failed", errorCode: "approved_tool_target_changed" });
|
||||
await quiet(); expect(remoteCall.provider).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
async function requestWithExpiry(expiresAt: Date, context = fixture) {
|
||||
return db.transaction(async tx => {
|
||||
const source = tx as unknown as typeof db;
|
||||
@@ -876,13 +1585,21 @@ describeEmbeddedPostgres("tool gateway service", () => {
|
||||
await expect(gateway.executeTool({ sessionToken: session.token, tool: "mcp-remote-fixture:update_note", parameters: { noteId: "expire" } })).rejects.toMatchObject({ reasonCode: "approval_required" });
|
||||
const [request] = await db.select().from(toolActionRequests);
|
||||
await db.update(toolActionRequests).set({ expiresAt: new Date(Date.now() - 1000) }).where(eq(toolActionRequests.id, request.id));
|
||||
await db.insert(toolActionRequests).values(Array.from({ length: 100 }, (_, i) => ({
|
||||
const [originalInvocation] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, request.invocationId));
|
||||
const blockedInvocations = Array.from({ length: 101 }, (_, i) => ({ ...originalInvocation,
|
||||
id: `10000000-0000-4000-8000-${String(i).padStart(12, "0")}`, approvalState: "approved", idempotencyKey: null,
|
||||
}));
|
||||
await db.insert(toolInvocations).values(blockedInvocations);
|
||||
await db.insert(toolActionRequests).values(blockedInvocations.map((invocation, i) => ({
|
||||
...request, id: `00000000-0000-4000-8000-${String(i).padStart(12, "0")}`,
|
||||
interactionId: null, status: "approved" as const,
|
||||
invocationId: invocation.id, interactionId: null, status: "approved" as const,
|
||||
signedArguments: signToolArguments({ invocationId: invocation.id, toolName: invocation.toolName,
|
||||
canonicalArguments: canonicalToolArguments({ noteId: "blocked" }), executionOnApprove: true,
|
||||
signingSecret: testToolActionSigningSecret }),
|
||||
})));
|
||||
const recover = vi.spyOn(gateway, "approveActionRequest").mockRejectedValue(new Error("Unavailable approval dependency"));
|
||||
expect(await gateway.sweepActionReviews()).toEqual({ scanned: 101 });
|
||||
expect(recover).toHaveBeenCalledTimes(100);
|
||||
expect(await gateway.sweepActionReviews()).toEqual({ scanned: 102 });
|
||||
expect(recover).toHaveBeenCalledTimes(101);
|
||||
const [settled] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, request.interactionId!));
|
||||
expect(settled.status).toBe("expired");
|
||||
});
|
||||
|
||||
+9
-2
@@ -1,3 +1,4 @@
|
||||
import { registerToolReviewRecoveryWork } from "./services/tool-review-recovery-work.js";
|
||||
import { registerToolReviewExpiryWork } from "./services/tool-review-expiry-work.js";
|
||||
import { registerGatewayCredentialExpiryWork } from "./services/gateway-credential-expiry-work.js";
|
||||
import { registerSecretProposalExpiryWork } from "./services/secret-proposal-expiry-work.js";
|
||||
@@ -1472,7 +1473,13 @@ async function startServerWithDatabaseTeardown(
|
||||
} else {
|
||||
await app.locals.toolGateway.sweepExpiredActionReviews();
|
||||
}
|
||||
return app.locals.toolGateway.sweepActionReviews({ includeExpiry: false });
|
||||
if (config.heartbeatSchedulerEnabled) {
|
||||
await registerToolReviewRecoveryWork(deliveryWork, app.locals.toolGateway,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
} else {
|
||||
await app.locals.toolGateway.recoverApprovedActionReviews();
|
||||
}
|
||||
return app.locals.toolGateway.sweepActionReviews({ includeExpiry: false, includeApproved: false });
|
||||
}).catch((err: unknown) => logger.error({ err }, "startup tool review recovery failed"));
|
||||
await timeStartupRecovery("gateway_sessions", () => config.heartbeatSchedulerEnabled
|
||||
? registerGatewayCredentialExpiryWork(deliveryWork, app.locals.toolGateway,
|
||||
@@ -1903,7 +1910,7 @@ async function startServerWithDatabaseTeardown(
|
||||
logger.error({ err }, "periodic tool connection health sweep failed");
|
||||
}));
|
||||
|
||||
trackHeartbeatSchedulerWork(app.locals.toolGateway.sweepActionReviews({ includeExpiry: false }).catch((err: unknown) => logger.error({ err }, "tool review recovery failed")));
|
||||
trackHeartbeatSchedulerWork(app.locals.toolGateway.sweepActionReviews({ includeExpiry: false, includeApproved: false }).catch((err: unknown) => logger.error({ err }, "tool review recovery failed")));
|
||||
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
if (!(await heartbeat.resolveSchedulingSuppression()).suppressed) {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { writeToolReviewChange } from "./tool-review-work-notifications.js";
|
||||
import { approvalResolutionRecords, resolvableApprovalStatuses as resolvableStatuses } from "../lib/approval-records.js";
|
||||
import { notifyHireApproved } from "./hire-hook.js";
|
||||
|
||||
@@ -12,7 +13,15 @@ import { instanceSettingsService } from "./instance-settings.js";
|
||||
|
||||
export function approvalRecords(db: Db) {
|
||||
const instanceSettings = instanceSettingsService(db);
|
||||
const { getExistingApproval, resolveApproval } = approvalResolutionRecords(db);
|
||||
const { getExistingApproval, resolveApproval: resolveApprovalRecord } = approvalResolutionRecords(db);
|
||||
|
||||
// Tool reviews waiting for a formal decision stay parked until this producer
|
||||
// commits. The shared record owner retains approval validation and CAS rules.
|
||||
async function resolveApproval(id: string, status: "approved" | "rejected", userId: string,
|
||||
note: string | null | undefined, database: Db = db) {
|
||||
const existing = await getExistingApproval(id, database);
|
||||
return writeToolReviewChange(database, existing.companyId, tx => resolveApprovalRecord(id, status, userId, note, tx));
|
||||
}
|
||||
|
||||
function redactApprovalComment<T extends { body: string }>(comment: T, censorUsernameInLogs: boolean): T {
|
||||
return {
|
||||
@@ -64,7 +73,9 @@ export function approvalRecords(db: Db) {
|
||||
// cleanup. Idempotent: a no-op on already-resolved approvals.
|
||||
cancel: async (id: string, reason?: string | null) => {
|
||||
const now = new Date();
|
||||
const updated = await db
|
||||
const [existing] = await db.select({ companyId: approvals.companyId }).from(approvals).where(eq(approvals.id, id));
|
||||
if (!existing) return null;
|
||||
const updated = await writeToolReviewChange(db, existing.companyId, tx => tx
|
||||
.update(approvals)
|
||||
.set({
|
||||
status: "cancelled",
|
||||
@@ -74,7 +85,7 @@ export function approvalRecords(db: Db) {
|
||||
})
|
||||
.where(and(eq(approvals.id, id), inArray(approvals.status, resolvableStatuses)))
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
.then((rows) => rows[0] ?? null));
|
||||
return updated;
|
||||
},
|
||||
|
||||
|
||||
@@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = {
|
||||
executionStatus: "execution-status",
|
||||
executionDisposition: "execution-disposition",
|
||||
environmentCleanup: "environment-cleanup",
|
||||
toolReviewRecovery: "tool-review-recovery",
|
||||
toolReviewExpiry: "tool-review-expiry",
|
||||
gatewayCredentialExpiry: "gateway-credential-expiry",
|
||||
secretProposalExpiry: "secret-proposal-expiry",
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import type { ConnectionManagementContext } from "./connection-management.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { boundRunIssueIdSql } from "./run-task-binding.js";
|
||||
import { and, asc, desc, eq, gt, inArray, isNull, ne, sql } from "drizzle-orm";
|
||||
import type { ToolApprovalClaim } from "./tool-approval-execution.js";
|
||||
import { and, asc, desc, eq, inArray, isNull, ne, sql } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import {
|
||||
agents,
|
||||
@@ -53,6 +54,12 @@ import {
|
||||
import { recordToolRuntimeAuditWriteFailure } from "./tool-runtime-metrics.js";
|
||||
import { canonicalToolArguments, readSignedToolArgumentsPayload, toolRequestIdentityHash } from "./tool-content-guards.js";
|
||||
|
||||
class RejectedInvocationAdmission extends Error {
|
||||
constructor(readonly decision: ToolAccessDecision) {
|
||||
super(decision.explanation);
|
||||
}
|
||||
}
|
||||
|
||||
type ToolAccessContext = {
|
||||
companyId: string;
|
||||
actorType: "agent" | "user" | "system" | "plugin";
|
||||
@@ -717,7 +724,7 @@ function assertSupportedGenericPolicyShape(
|
||||
}
|
||||
}
|
||||
|
||||
async function getGenericPolicyRow(db: Db, companyId: string, policyId: string) {
|
||||
async function getGenericPolicyRow(db: Pick<Db, "select">, companyId: string, policyId: string) {
|
||||
const [policy] = await db
|
||||
.select()
|
||||
.from(toolPolicies)
|
||||
@@ -785,7 +792,7 @@ function scopeAllowsTool(scope: Record<string, unknown> | null, ctx: ToolAccessC
|
||||
return selectorMatches(selectors, ctx);
|
||||
}
|
||||
|
||||
export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecret?: string } = {}) {
|
||||
export function toolAccessPolicyService(db: Pick<Db, "select" | "insert" | "update" | "delete" | "transaction">, options: { toolActionSigningSecret?: string } = {}) {
|
||||
// Policy administration includes authorized self-targeted changes. It cannot
|
||||
// resolve governed action requests or modify principal permission grants.
|
||||
async function listPolicies(companyId: string) {
|
||||
@@ -1301,11 +1308,12 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr
|
||||
],
|
||||
set: {
|
||||
limit: rule.limit,
|
||||
remaining: sql`greatest(0, least(${toolRateLimitCounters.remaining}, ${rule.limit}) - 1)`,
|
||||
// Preserve the number already spent when an operator changes the limit.
|
||||
remaining: sql`greatest(0, ${rule.limit} - (${toolRateLimitCounters.limit} - ${toolRateLimitCounters.remaining}) - 1)`,
|
||||
resetAt,
|
||||
updatedAt: now,
|
||||
},
|
||||
setWhere: gt(toolRateLimitCounters.remaining, 0),
|
||||
setWhere: sql`${toolRateLimitCounters.limit} - ${toolRateLimitCounters.remaining} < ${rule.limit}`,
|
||||
})
|
||||
.returning({ remaining: toolRateLimitCounters.remaining });
|
||||
if (!counter) {
|
||||
@@ -1376,7 +1384,7 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr
|
||||
});
|
||||
}
|
||||
|
||||
async function admittedRateLimitAdmissions(ctx: ToolAccessContext, invocationId: string): Promise<ConsumedRateAdmission[]> {
|
||||
async function admittedRateLimitAdmissions(ctx: ToolAccessContext, invocationId: string, previewHash?: string): Promise<ConsumedRateAdmission[] | null> {
|
||||
const [invocation] = await db.select({
|
||||
actorType: toolInvocations.actorType, actorId: toolInvocations.actorId, agentId: toolInvocations.agentId,
|
||||
runId: toolInvocations.runId, issueId: toolInvocations.issueId, gatewayId: toolInvocations.gatewayId,
|
||||
@@ -1387,6 +1395,9 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr
|
||||
status: toolInvocations.status, policyDecision: toolInvocations.policyDecision, approvalState: toolInvocations.approvalState,
|
||||
policyExplanation: toolInvocations.policyExplanation,
|
||||
}).from(toolInvocations).where(and(eq(toolInvocations.companyId, ctx.companyId), eq(toolInvocations.id, invocationId)));
|
||||
// A parked request already spent its slot before asking for review. Reusing
|
||||
// that admission only skips the same rate rule; current policy and the
|
||||
// gateway's approval/claim owners still decide whether it may execute.
|
||||
const delegated = invocation?.policyExplanation?.delegatedConnectionTest as { agentId?: string } | undefined;
|
||||
const gatewayIssuer = invocation?.gatewayId ? invocation.policyExplanation?.delegatedGatewayIssuer : undefined;
|
||||
const isDelegatedCaller = invocation?.actorType === "agent" && invocation.actorId === ctx.agentId
|
||||
@@ -1401,13 +1412,81 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr
|
||||
&& invocation.catalogEntryId === ctx.catalogEntryId && invocation.toolName === ctx.toolName
|
||||
&& invocation.providerType === ctx.providerType && invocation.applicationKey === ctx.applicationKey
|
||||
&& invocation.upstreamToolName === ctx.upstreamToolName
|
||||
&& invocation.argumentsHash === ctx.argumentsHash;
|
||||
const stored = authorized ? invocation.policyExplanation?.[isDelegatedCaller ? "delegatedCallerConsumedRateAdmissions" : "consumedRateAdmissions"] : null;
|
||||
&& (invocation.argumentsHash === ctx.argumentsHash || invocation.argumentsHash === previewHash);
|
||||
if (!authorized) return null;
|
||||
const stored = invocation.policyExplanation?.[isDelegatedCaller ? "delegatedCallerConsumedRateAdmissions" : "consumedRateAdmissions"];
|
||||
return Array.isArray(stored) ? stored.filter((item): item is ConsumedRateAdmission => isRecord(item)
|
||||
&& typeof item.policyId === "string" && CANONICAL_UUID_RE.test(item.policyId)
|
||||
&& typeof item.versionHash === "string" && /^[a-f0-9]{64}$/.test(item.versionHash)) : [];
|
||||
}
|
||||
|
||||
/** Revalidate claimed work and persist new rate charges before provider dispatch. */
|
||||
async function admitInvocation(
|
||||
input: ToolAccessDecisionInput,
|
||||
invocationId: string,
|
||||
claim?: ToolApprovalClaim,
|
||||
evaluate?: (
|
||||
policy: Pick<ReturnType<typeof toolAccessPolicyService>, "decide" | "writeAudit">,
|
||||
input: ToolAccessDecisionInput,
|
||||
transaction: Db,
|
||||
) => Promise<ToolAccessDecision>,
|
||||
): Promise<ToolAccessDecision> {
|
||||
try {
|
||||
return await db.transaction(async tx => {
|
||||
const [invocation] = await tx.select().from(toolInvocations)
|
||||
.where(and(eq(toolInvocations.companyId, input.companyId), eq(toolInvocations.id, invocationId)))
|
||||
.for("update");
|
||||
// Historical reviewed requests can still have a pending invocation.
|
||||
// Accept that state only with the exact executing claim owned by the
|
||||
// gateway, locked in the same invocation-then-request order as dispatch.
|
||||
let ownsClaim = false;
|
||||
if (claim && claim.invocationId === invocationId) {
|
||||
const [request] = await tx.select().from(toolActionRequests).where(and(
|
||||
eq(toolActionRequests.id, claim.actionRequestId),
|
||||
eq(toolActionRequests.invocationId, invocationId),
|
||||
eq(toolActionRequests.companyId, input.companyId),
|
||||
)).for("update");
|
||||
ownsClaim = request?.status === "executing" && request.updatedAt.getTime() === claim.claimedAt.getTime();
|
||||
}
|
||||
if (!invocation || (claim && !ownsClaim)
|
||||
|| !(invocation.status === "executing" && invocation.policyDecision === "allow"
|
||||
|| ["awaiting_approval", "executing"].includes(invocation.status)
|
||||
&& invocation.policyDecision === "require_approval" && invocation.approvalState === "approved"
|
||||
|| invocation.status === "awaiting_approval" && invocation.policyDecision === "require_approval"
|
||||
&& invocation.approvalState === "pending" && ownsClaim)) {
|
||||
return decision("deny", "deny_policy_block", "Tool invocation admission is unavailable.", [], []);
|
||||
}
|
||||
const admissionPolicy = toolAccessPolicyService(tx, options);
|
||||
const admissionInput = { ...input, consumeRateLimit: true };
|
||||
// Composed caller/target policy must share this claim transaction so
|
||||
// either all new charges and receipts commit or all roll back.
|
||||
const result = evaluate
|
||||
? await evaluate(admissionPolicy, admissionInput, tx as unknown as Db)
|
||||
: await admissionPolicy.decide(admissionInput, { admittedInvocationId: invocationId });
|
||||
if (!result.allowed && result.decision !== "require_approval") {
|
||||
// A later policy can reject after an earlier rate rule spent a slot.
|
||||
// Roll back both the new charges and their receipt on any rejection.
|
||||
throw new RejectedInvocationAdmission(result);
|
||||
}
|
||||
const admissions = result.policyExplanation?.consumedRateAdmissions;
|
||||
const delegatedAdmissions = result.policyExplanation?.delegatedCallerConsumedRateAdmissions;
|
||||
if (Array.isArray(admissions) || Array.isArray(delegatedAdmissions)) {
|
||||
await tx.update(toolInvocations).set({
|
||||
policyExplanation: {
|
||||
...invocation.policyExplanation,
|
||||
...(Array.isArray(admissions) ? { consumedRateAdmissions: admissions } : {}),
|
||||
...(Array.isArray(delegatedAdmissions) ? { delegatedCallerConsumedRateAdmissions: delegatedAdmissions } : {}),
|
||||
},
|
||||
}).where(and(eq(toolInvocations.companyId, input.companyId), eq(toolInvocations.id, invocationId)));
|
||||
}
|
||||
return result;
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof RejectedInvocationAdmission) return error.decision;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Decides one tool access request. Pass the same `cache` to every call of a
|
||||
* batch that shares one actor and run context (for example, one tools/list
|
||||
@@ -1415,7 +1494,7 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr
|
||||
*/
|
||||
async function decide(
|
||||
input: ToolAccessDecisionInput,
|
||||
options: { cache?: ToolAccessDecisionCache; admittedInvocationId?: string; sharedRateDecision?: ToolAccessDecision } = {},
|
||||
options: { cache?: ToolAccessDecisionCache; admittedInvocationId?: string; admittedPreviewHash?: string; sharedRateDecision?: ToolAccessDecision } = {},
|
||||
): Promise<ToolAccessDecision> {
|
||||
const consumedRateAdmissions: ConsumedRateAdmission[] = [];
|
||||
const result = await decideInternal(input, options, consumedRateAdmissions);
|
||||
@@ -1426,7 +1505,7 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr
|
||||
|
||||
async function decideInternal(
|
||||
input: ToolAccessDecisionInput,
|
||||
options: { cache?: ToolAccessDecisionCache; admittedInvocationId?: string; sharedRateDecision?: ToolAccessDecision },
|
||||
options: { cache?: ToolAccessDecisionCache; admittedInvocationId?: string; admittedPreviewHash?: string; sharedRateDecision?: ToolAccessDecision },
|
||||
consumedRateAdmissions: ConsumedRateAdmission[],
|
||||
): Promise<ToolAccessDecision> {
|
||||
// A decision that consumes a rate limit can also record a trust-rule hit
|
||||
@@ -1438,9 +1517,14 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr
|
||||
// A final handoff rechecks current permissions for work already admitted by
|
||||
// the gateway. Its own last consumed slot cannot reject that same work.
|
||||
// Only durable consumption of an unchanged rule retains that slot; new or
|
||||
// edited rules still apply. This option never approves or consumes a call.
|
||||
const admittedRates = input.consumeRateLimit === false && options.admittedInvocationId
|
||||
? await admittedRateLimitAdmissions(ctx, options.admittedInvocationId) : [];
|
||||
// edited rules still apply. admitInvocation owns atomic consuming rechecks;
|
||||
// ordinary preflight and browser handoff callers remain read-only.
|
||||
const admittedRates = options.admittedInvocationId
|
||||
? await admittedRateLimitAdmissions(ctx, options.admittedInvocationId, input.consumeRateLimit === true ? undefined : options.admittedPreviewHash) : [];
|
||||
if (options.admittedInvocationId && input.consumeRateLimit === true && admittedRates === null) {
|
||||
return decision("deny", "deny_policy_block", "Tool invocation no longer matches this admission.", [], []);
|
||||
}
|
||||
if (input.consumeRateLimit === true) consumedRateAdmissions.push(...(admittedRates ?? []));
|
||||
const profileState = await effectiveProfiles(ctx, cache);
|
||||
const effectiveProfileIds = profileState.profiles.map((profile) => profile.id);
|
||||
const permittedByProfile = profileState.profiles.some((profile) => {
|
||||
@@ -1513,7 +1597,7 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr
|
||||
);
|
||||
}
|
||||
const versionHash = ratePolicyVersionHash(policy);
|
||||
if (admittedRates.some(admission => admission.policyId === policy.id && admission.versionHash === versionHash)) continue;
|
||||
if (admittedRates?.some(admission => admission.policyId === policy.id && admission.versionHash === versionHash)) continue;
|
||||
// Paired caller/target checks for one delegated test share a slot only
|
||||
// when the exact policy version and counter bucket are identical.
|
||||
const shared = options.sharedRateDecision?.policyExplanation?.consumedRateAdmissions;
|
||||
@@ -1548,7 +1632,7 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr
|
||||
{ redactionPlan: redaction.redactionPlan, policyExplanation },
|
||||
);
|
||||
}
|
||||
if (input.consumeRateLimit === true) {
|
||||
if (input.consumeRateLimit === true && !options.admittedInvocationId) {
|
||||
await recordTrustRuleHit(policy, ctx, redaction);
|
||||
}
|
||||
return decision("allow", "allow_trust_rule", policy.description ?? "Tool access allowed by trust rule.", effectiveProfileIds, [policy.id], { redactionPlan: redaction.redactionPlan, policyExplanation });
|
||||
@@ -2178,6 +2262,7 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr
|
||||
|
||||
return {
|
||||
decide,
|
||||
admitInvocation,
|
||||
writeAudit,
|
||||
recordInvocation,
|
||||
summarizeAndRedact,
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { toolActionRequests, toolInvocations, type Db } from "@paperclipai/db";
|
||||
import type { ToolRedactedValueSummary } from "@paperclipai/shared";
|
||||
import { notifyToolReviewChange } from "./tool-review-work-notifications.js";
|
||||
|
||||
export type ToolApprovalClaim = { actionRequestId: string; invocationId: string; claimedAt: Date };
|
||||
export type ToolTestOutcome = {
|
||||
status: "succeeded" | "failed" | "timed_out" | "rate_limited";
|
||||
result?: ToolRedactedValueSummary;
|
||||
errorCode?: string;
|
||||
errorMessage?: string;
|
||||
};
|
||||
|
||||
export async function claimToolTestApproval(db: Db, actionRequestId: string, invocationId: string): Promise<ToolApprovalClaim | null> {
|
||||
return db.transaction(async tx => {
|
||||
const [invocation] = await tx.select().from(toolInvocations)
|
||||
.where(eq(toolInvocations.id, invocationId)).for("update");
|
||||
const [request] = await tx.select().from(toolActionRequests)
|
||||
.where(and(eq(toolActionRequests.id, actionRequestId), eq(toolActionRequests.invocationId, invocationId))).for("update");
|
||||
if (!invocation || request?.status !== "approved") return null;
|
||||
await notifyToolReviewChange(tx, request.companyId);
|
||||
const now = new Date();
|
||||
if (invocation.status !== "awaiting_approval") {
|
||||
if (["succeeded", "failed", "denied", "timed_out", "rate_limited"].includes(invocation.status)) {
|
||||
await tx.update(toolActionRequests).set({ status: invocation.status === "succeeded" ? "executed" : "failed",
|
||||
resolvedAt: invocation.completedAt ?? now, updatedAt: now }).where(eq(toolActionRequests.id, actionRequestId));
|
||||
} else if (invocation.status === "executing") {
|
||||
await tx.update(toolActionRequests).set({ status: "executing", updatedAt: invocation.startedAt ?? invocation.updatedAt })
|
||||
.where(eq(toolActionRequests.id, actionRequestId));
|
||||
}
|
||||
return null;
|
||||
}
|
||||
await tx.update(toolActionRequests).set({ status: "executing", updatedAt: now })
|
||||
.where(eq(toolActionRequests.id, actionRequestId));
|
||||
return { actionRequestId, invocationId, claimedAt: now };
|
||||
});
|
||||
}
|
||||
|
||||
/** Return only this still-undispatched claim to its existing approval queue. */
|
||||
export async function releaseToolApproval(db: Db, claim: ToolApprovalClaim) {
|
||||
return db.transaction(async tx => {
|
||||
const [invocation] = await tx.select().from(toolInvocations)
|
||||
.where(eq(toolInvocations.id, claim.invocationId)).for("update");
|
||||
const [request] = await tx.select().from(toolActionRequests)
|
||||
.where(and(eq(toolActionRequests.id, claim.actionRequestId),
|
||||
eq(toolActionRequests.invocationId, claim.invocationId))).for("update");
|
||||
if (invocation?.status !== "awaiting_approval" || invocation.startedAt !== null ||
|
||||
request?.status !== "executing" || request.companyId !== invocation.companyId ||
|
||||
request.updatedAt.getTime() !== claim.claimedAt.getTime()) return false;
|
||||
await notifyToolReviewChange(tx, request.companyId);
|
||||
await tx.update(toolActionRequests).set({ status: "approved", updatedAt: new Date() })
|
||||
.where(eq(toolActionRequests.id, request.id));
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
export async function bindVerifiedToolApprovalArguments(db: Db, claim: ToolApprovalClaim, hashes: { preview: string; verified: string }) {
|
||||
return db.transaction(async tx => {
|
||||
const [invocation] = await tx.select().from(toolInvocations)
|
||||
.where(eq(toolInvocations.id, claim.invocationId)).for("update");
|
||||
const [request] = await tx.select().from(toolActionRequests)
|
||||
.where(and(eq(toolActionRequests.id, claim.actionRequestId),
|
||||
eq(toolActionRequests.invocationId, claim.invocationId))).for("update");
|
||||
if (invocation?.status !== "awaiting_approval" || invocation.startedAt !== null ||
|
||||
request?.status !== "executing" || request.companyId !== invocation.companyId ||
|
||||
request.updatedAt.getTime() !== claim.claimedAt.getTime() ||
|
||||
![hashes.preview, hashes.verified].includes(request.canonicalArgumentsHash) ||
|
||||
![hashes.preview, hashes.verified].includes(invocation.argumentsHash ?? "")) return false;
|
||||
if (invocation.argumentsHash !== hashes.verified) await tx.update(toolInvocations)
|
||||
.set({ argumentsHash: hashes.verified, updatedAt: new Date() }).where(eq(toolInvocations.id, claim.invocationId));
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
export async function startToolInvocation(db: Db, invocationId: string, claim?: ToolApprovalClaim) {
|
||||
const now = new Date();
|
||||
if (!claim) {
|
||||
await db.update(toolInvocations).set({ status: "executing", startedAt: now, updatedAt: now })
|
||||
.where(eq(toolInvocations.id, invocationId));
|
||||
return true;
|
||||
}
|
||||
return db.transaction(async tx => {
|
||||
const [invocation] = await tx.select().from(toolInvocations)
|
||||
.where(eq(toolInvocations.id, invocationId)).for("update");
|
||||
const [request] = await tx.select().from(toolActionRequests)
|
||||
.where(eq(toolActionRequests.id, claim.actionRequestId)).for("update");
|
||||
if (invocationId !== claim.invocationId || invocation?.status !== "awaiting_approval" || invocation.startedAt !== null ||
|
||||
request?.status !== "executing" || request.invocationId !== invocationId || request.companyId !== invocation.companyId ||
|
||||
request.updatedAt.getTime() !== claim.claimedAt.getTime()) return false;
|
||||
await tx.update(toolInvocations).set({ status: "executing", approvalState: "approved", startedAt: now, updatedAt: now })
|
||||
.where(eq(toolInvocations.id, invocationId));
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
export async function persistToolTestOutcome(db: Db, invocationId: string, outcome: ToolTestOutcome, claim?: ToolApprovalClaim) {
|
||||
const now = new Date();
|
||||
const values = { status: outcome.status, ...(outcome.result ? {
|
||||
resultHash: outcome.result.sha256 ?? null, resultSummary: outcome.result,
|
||||
resultSizeBytes: outcome.result.sizeBytes ?? null,
|
||||
} : {}), errorCode: outcome.errorCode, errorMessage: outcome.errorMessage, completedAt: now, updatedAt: now };
|
||||
if (!claim) {
|
||||
await db.update(toolInvocations).set(values).where(eq(toolInvocations.id, invocationId));
|
||||
return true;
|
||||
}
|
||||
return db.transaction(async tx => {
|
||||
const [invocation] = await tx.select({ status: toolInvocations.status }).from(toolInvocations)
|
||||
.where(eq(toolInvocations.id, invocationId)).for("update");
|
||||
const [request] = await tx.select().from(toolActionRequests)
|
||||
.where(eq(toolActionRequests.id, claim.actionRequestId)).for("update");
|
||||
if (!invocation || !["awaiting_approval", "executing"].includes(invocation.status) ||
|
||||
request?.status !== "executing" || request.updatedAt.getTime() !== claim.claimedAt.getTime()) return false;
|
||||
await notifyToolReviewChange(tx, request.companyId);
|
||||
await tx.update(toolInvocations).set(values).where(eq(toolInvocations.id, invocationId));
|
||||
await tx.update(toolActionRequests).set({ status: outcome.status === "succeeded" ? "executed" : "failed",
|
||||
resolvedAt: now, updatedAt: now }).where(eq(toolActionRequests.id, claim.actionRequestId));
|
||||
return true;
|
||||
});
|
||||
}
|
||||
+402
-447
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,33 @@
|
||||
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
||||
import { registerScopedDeadlineWork } from "./scoped-deadline-work.js";
|
||||
import type { createToolGatewayService } from "./tool-gateway.js";
|
||||
|
||||
export function registerToolReviewRecoveryWork(
|
||||
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
|
||||
gateway: Pick<ReturnType<typeof createToolGatewayService>, "recoverApprovedActionReviews" | "approvedActionReviewCompanies">,
|
||||
canRun: () => boolean,
|
||||
) {
|
||||
// Dispatch failures are reported by the domain owner, not thrown. Preserve
|
||||
// their retry deadline when another write reloads the same company's work.
|
||||
const retries = new Map<string, number>();
|
||||
return registerScopedDeadlineWork(coordinator, {
|
||||
queue: DELIVERY_QUEUES.toolReviewRecovery,
|
||||
batchSize: 1,
|
||||
decodeScope: hint => hint === "*" ? undefined : hint,
|
||||
matches: (entry, companyId?: string) => companyId === undefined || entry.companyId === companyId,
|
||||
load: async (companyId?: string) => {
|
||||
const rows = await gateway.approvedActionReviewCompanies(companyId);
|
||||
const retained = new Set(rows.map(row => row.companyId));
|
||||
for (const id of retries.keys()) if ((companyId === undefined || id === companyId) && !retained.has(id)) retries.delete(id);
|
||||
return rows.map(row => ({ ...row, id: row.companyId, dueAt: retries.get(row.companyId) ?? Date.now() }));
|
||||
},
|
||||
check: async entry => {
|
||||
const result = await gateway.recoverApprovedActionReviews(entry.companyId);
|
||||
if (!result.retry) { retries.delete(entry.companyId); return null; }
|
||||
const dueAt = Date.now() + 30_000;
|
||||
retries.set(entry.companyId, dueAt);
|
||||
return { ...entry, dueAt };
|
||||
},
|
||||
}, canRun);
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
import { and, asc, eq, gt } from "drizzle-orm";
|
||||
import { approvals, toolActionRequests, toolInvocations, type Db } from "@paperclipai/db";
|
||||
|
||||
import { readSignedToolArgumentsPayload } from "./tool-content-guards.js";
|
||||
import { notifyToolReviewChange } from "./tool-review-work-notifications.js";
|
||||
|
||||
type ReviewProjection = {
|
||||
actionRequestId: string;
|
||||
status: "executed" | "executing" | "failed" | "cancelled";
|
||||
errorCode?: string | null;
|
||||
errorMessage?: string | null;
|
||||
resultSummary?: string | null;
|
||||
};
|
||||
|
||||
/** Reconcile saved history before dispatch; never infer a safe provider replay. */
|
||||
export async function prepareApprovedToolReview(db: Db, input: {
|
||||
actionRequestId: string;
|
||||
invocationId: string;
|
||||
signingSecret?: string;
|
||||
}) {
|
||||
return db.transaction(async tx => {
|
||||
// Match Test-tab claims/outcome persistence. Recheck after the locks because
|
||||
// a foreground approval may have claimed the row since the recovery scan.
|
||||
const [invocation] = await tx.select().from(toolInvocations)
|
||||
.where(eq(toolInvocations.id, input.invocationId)).for("update");
|
||||
const [request] = await tx.select().from(toolActionRequests)
|
||||
.where(and(eq(toolActionRequests.id, input.actionRequestId),
|
||||
eq(toolActionRequests.invocationId, input.invocationId))).for("update");
|
||||
if (!invocation || request?.status !== "approved") return { kind: "skip" } as const;
|
||||
const now = new Date();
|
||||
|
||||
async function settle(projection: ReviewProjection, failure?: {
|
||||
status: "failed" | "denied";
|
||||
errorCode: string;
|
||||
errorMessage: string;
|
||||
}) {
|
||||
await notifyToolReviewChange(tx, request.companyId);
|
||||
if (failure) await tx.update(toolInvocations).set({ ...failure, completedAt: now, updatedAt: now })
|
||||
.where(eq(toolInvocations.id, invocation.id));
|
||||
await tx.update(toolActionRequests).set({ status: projection.status,
|
||||
...(projection.status === "executing"
|
||||
? { updatedAt: invocation.startedAt ?? invocation.updatedAt }
|
||||
: { resolvedAt: invocation.completedAt ?? now, updatedAt: now }),
|
||||
}).where(eq(toolActionRequests.id, request.id));
|
||||
return { kind: "settled", projection } as const;
|
||||
}
|
||||
function fail(status: "failed" | "cancelled", invocationStatus: "failed" | "denied", errorCode: string, errorMessage: string) {
|
||||
return settle({ actionRequestId: request.id, status, errorCode, errorMessage },
|
||||
{ status: invocationStatus, errorCode, errorMessage });
|
||||
}
|
||||
if (invocation.companyId !== request.companyId) {
|
||||
// Quarantine the request without modifying another company's invocation.
|
||||
return settle({ actionRequestId: request.id, status: "failed",
|
||||
errorCode: "action_context_mismatch", errorMessage: "Tool action and invocation companies do not match." });
|
||||
}
|
||||
if (["succeeded", "failed", "denied", "cancelled", "timed_out", "rate_limited", "executing"].includes(invocation.status)) {
|
||||
return settle({ actionRequestId: request.id,
|
||||
status: invocation.status === "succeeded" ? "executed"
|
||||
: invocation.status === "executing" ? "executing"
|
||||
: invocation.status === "cancelled" ? "cancelled" : "failed",
|
||||
errorCode: invocation.errorCode, errorMessage: invocation.errorMessage,
|
||||
resultSummary: invocation.resultSummary?.summary });
|
||||
}
|
||||
if (invocation.status !== "awaiting_approval") {
|
||||
return fail("failed", "failed", "tool_recovery_state_invalid",
|
||||
"Approved action has inconsistent invocation history. Inspect the provider before retrying; recovery did not replay it.");
|
||||
}
|
||||
|
||||
// Missing-key/configuration errors deliberately propagate as retryable;
|
||||
// only a decoded invalid signature is a permanent request failure.
|
||||
const payload = readSignedToolArgumentsPayload({ signedArguments: request.signedArguments,
|
||||
invocationId: invocation.id, toolName: invocation.toolName, signingSecret: input.signingSecret });
|
||||
if (!payload) return fail("cancelled", "failed", "action_request_invalidated",
|
||||
"Stored tool arguments are no longer verifiable; submit a new action for review.");
|
||||
if (request.approvalId) {
|
||||
const [formal] = await tx.select({ status: approvals.status }).from(approvals)
|
||||
.where(and(eq(approvals.id, request.approvalId), eq(approvals.companyId, request.companyId)));
|
||||
if (!formal || formal.status === "rejected" || formal.status === "cancelled") {
|
||||
return fail("cancelled", "denied", "formal_approval_unavailable",
|
||||
"Required formal board approval is unavailable or was declined; submit a new action for review.");
|
||||
}
|
||||
// Human input has no deadline. The approval owner publishes a committed
|
||||
// resolution/cancellation to this queue; startup also rediscovers the row.
|
||||
if (formal.status === "pending" || formal.status === "revision_requested") {
|
||||
return { kind: "skip" } as const;
|
||||
}
|
||||
}
|
||||
return { kind: "ready", request, invocation, payload } as const;
|
||||
});
|
||||
}
|
||||
|
||||
export function approvedToolReviewCompanies(db: Db, companyId?: string) {
|
||||
return db.select({ companyId: toolActionRequests.companyId }).from(toolActionRequests)
|
||||
.innerJoin(toolInvocations, eq(toolInvocations.id, toolActionRequests.invocationId))
|
||||
.where(and(eq(toolActionRequests.status, "approved"),
|
||||
companyId ? eq(toolActionRequests.companyId, companyId) : undefined))
|
||||
.groupBy(toolActionRequests.companyId);
|
||||
}
|
||||
|
||||
export async function recoverApprovedToolReviews(db: Db, recover: (input: {
|
||||
request: typeof toolActionRequests.$inferSelect;
|
||||
invocation: typeof toolInvocations.$inferSelect;
|
||||
}) => Promise<boolean>, companyId?: string) {
|
||||
let cursor: string | undefined;
|
||||
let scanned = 0, retry = false;
|
||||
for (;;) {
|
||||
const rows = await db.select({ request: toolActionRequests, invocation: toolInvocations })
|
||||
.from(toolActionRequests).innerJoin(toolInvocations, eq(toolInvocations.id, toolActionRequests.invocationId))
|
||||
.where(and(eq(toolActionRequests.status, "approved"),
|
||||
companyId ? eq(toolActionRequests.companyId, companyId) : undefined,
|
||||
cursor ? gt(toolActionRequests.id, cursor) : undefined))
|
||||
.orderBy(asc(toolActionRequests.id)).limit(100);
|
||||
for (const row of rows) retry = await recover(row) || retry;
|
||||
scanned += rows.length;
|
||||
if (rows.length < 100) break;
|
||||
cursor = rows[rows.length - 1].request.id;
|
||||
}
|
||||
return { scanned, retry };
|
||||
}
|
||||
@@ -1,8 +1,9 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export function notifyToolReviewChange(transaction: object, companyId: string) {
|
||||
return notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewExpiry, companyId);
|
||||
export async function notifyToolReviewChange(transaction: object, companyId: string) {
|
||||
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewExpiry, companyId);
|
||||
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewRecovery, companyId);
|
||||
}
|
||||
|
||||
export function writeToolReviewChange<T>(db: Db, companyId: string, write: (transaction: Db) => Promise<T>): Promise<T> {
|
||||
|
||||
@@ -11,7 +11,7 @@ function minuteBucket(date: Date): Date {
|
||||
}
|
||||
|
||||
export async function incrementToolRuntimeMetricCounter(
|
||||
db: Db,
|
||||
db: Pick<Db, "insert">,
|
||||
input: {
|
||||
companyId: string;
|
||||
metric: string;
|
||||
@@ -42,7 +42,7 @@ export async function incrementToolRuntimeMetricCounter(
|
||||
});
|
||||
}
|
||||
|
||||
export async function recordToolRuntimeAuditWriteFailure(db: Db, companyId: string) {
|
||||
export async function recordToolRuntimeAuditWriteFailure(db: Pick<Db, "insert">, companyId: string) {
|
||||
try {
|
||||
await incrementToolRuntimeMetricCounter(db, {
|
||||
companyId,
|
||||
|
||||
Reference in new issue
Block a user