mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 13:20:54 +01:00
Merge latest master and reconcile Coordinator migrations and contracts
Preserve chat milestone notifications alongside review and assignment safeguards. Regenerate contracts and append the preference migration after upstream history, with safe replay coverage. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
scotttongandPaperclip committed
commit
52e75dc4d6
308 files changed
+21708
-1977
No files matched your search
@@ -74,6 +74,8 @@ packages/grok-acp/bin
|
||||
|
||||
# The Copilot build-time identity gate reads these sanitized source declarations.
|
||||
!packages/paperclip-runner/test/fixtures/copilot-profile-v*-identity.json
|
||||
# Grok's source identity is also an input to the image profile drift check.
|
||||
!packages/paperclip-runner/test/fixtures/grok-profile-v*-identity.json
|
||||
# Images build their pinned Copilot assets instead of copying host binaries.
|
||||
/packages/paperclip-runner/provider-assets/
|
||||
/server/provider-assets/
|
||||
|
||||
@@ -2146,6 +2146,13 @@ Lifecycle, timeout, and interrupt grace settings live under **Advanced Run
|
||||
Policy**. Permission selectors with a single valid mode are hidden; a saved
|
||||
unsupported mode still exposes remediation.
|
||||
|
||||
Saving an existing agent compares AI connection bindings by value, independent
|
||||
of PostgreSQL JSONB key order. Identity, thinking-effort and run-policy edits
|
||||
avoid the connection-adoption runtime probe when the connection and execution
|
||||
selection stay the same. Connection, harness, model, provider, engine and
|
||||
environment changes still validate adoption. Explicit runtime tests and
|
||||
required Copilot metadata checks keep their existing behavior.
|
||||
|
||||
Model catalogs and refresh follow the selected provider. ACPX Claude uses the
|
||||
normal Claude catalog and accepts custom model IDs; the exact ID is sent to
|
||||
Claude, which can reject unavailable models. Package/version verification is
|
||||
@@ -2490,6 +2497,36 @@ identity mismatches.
|
||||
|
||||
## App-Shipped Skills Catalog
|
||||
|
||||
Skill-library reads still reconcile bundled releases, folders, and direct local
|
||||
sources on each request. Folder classification reads only its source-kind field,
|
||||
and local-source reconciliation reads only local sources. Bundled reconciliation
|
||||
keeps current skill access checks but does not resolve unused folder paths.
|
||||
Attachment counts read only nonterminated agents' saved desired-skill selections and
|
||||
resolve them once against the complete visible skill list. Detail reads and
|
||||
stored inputs remain complete.
|
||||
|
||||
Run the synthetic HTTP comparison through the production skills router:
|
||||
|
||||
```sh
|
||||
pnpm --filter @paperclipai/server exec tsx scripts/benchmark-skill-list.ts --samples=5 --network-delay-ms=3.5
|
||||
```
|
||||
|
||||
The harness starts disposable PostgreSQL, synthetic skills and agents, and a
|
||||
loopback HTTP server. It checks both normal and conditional responses and records
|
||||
inventory phases, SQL calls, and PostgreSQL result-value bytes. These bytes omit
|
||||
protocol framing and do not measure provider billing. The actor is an authenticated
|
||||
board fixture; the harness does not test login. To use a task-owned local PostgreSQL
|
||||
server instead of embedded PostgreSQL, pass `--postgres-port=PORT` and
|
||||
`--postgres-user=USER`. It creates and removes a separate test database. Keep timing
|
||||
runs separate from builds and tests; repeat both source versions when host noise
|
||||
is material. Repeat with `--capture-values=false` for acceptance timings without
|
||||
the response-value observer. Add `--baseline-ref=COMMIT --capture-values=false`
|
||||
to alternate the original and current skill services against the same database
|
||||
and actor. This mode checks full response equality through equivalent local HTTP
|
||||
handlers. The normal mode uses the production router. Only `company-skills.ts`
|
||||
differs in paired mode; imported helpers stay at the current checkout.
|
||||
|
||||
|
||||
Plugin environment drivers can declare `skillKeys` that reference their own
|
||||
manifest-managed skills. Selecting that environment adds those skills to the
|
||||
normal authorized skill selection, including an explicitly empty optional
|
||||
|
||||
+49
-1
@@ -1,5 +1,43 @@
|
||||
# Paperclip evaluation guide
|
||||
|
||||
The public directory calls full-product testing **E2E tests**. Suites describe
|
||||
what is tested; deployment targets describe which app instance was tested.
|
||||
QA smoke is a named selection of existing journeys against QA, and appears in
|
||||
the same campaign history as other E2E runs. Agent execution (local or Daytona),
|
||||
provider, model and profile are independent filters. Browser-only journeys need
|
||||
no agent or model. Historical records without target metadata remain explicitly
|
||||
"Not recorded" rather than being assigned an inferred environment.
|
||||
|
||||
The history sidebar includes the current suite catalog and retained historical
|
||||
suites. Each suite has its latest matching run and latest full run, with full
|
||||
coverage determined from the recorded suite definition, selection and execution
|
||||
counts. An unexecuted full selection remains unfinished, and a subset
|
||||
can pass without qualifying as a full suite run; failed full selections remain
|
||||
visible. Changing a model filter does not redefine the suite's recorded coverage.
|
||||
Existing `runner-e2e` URLs, test IDs, commands, and measurement timestamps remain
|
||||
stable. **Runner protocol evals** remain a separate category because they test
|
||||
the runner/provider against a mock control plane.
|
||||
|
||||
Sidebar suite links open that suite's campaign history with other filters cleared.
|
||||
The Suite dropdown combines with the current filters instead. Explicit URLs retain
|
||||
their selected filters; when a suite is absent from a preset, the empty state
|
||||
explains the combination and links to the suite's runs.
|
||||
Active filters have blue borders and tinted fields. The filter toolbar counts
|
||||
active values, including a count beside More filters for collapsed controls.
|
||||
Reset all filters clears every campaign filter and the sidebar suite search.
|
||||
|
||||
`node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/history-browser.browser.mjs`
|
||||
exercises the history UI using synthetic metadata and Chromium, with no model calls. It checks suite coverage, correlated
|
||||
suite/target/model/execution filters, saved QA selections, URL history and mobile
|
||||
navigation. Install the existing Playwright Chromium dependency before running.
|
||||
For presentation-only refreshes,
|
||||
`node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/history-refresh.ts --history <history.json> --output <directory>`
|
||||
renders an existing history file. Its optional `--public-campaign`, `--public-url` and
|
||||
`--preset` arguments import a reviewed public measurement without replacing
|
||||
existing campaigns. Private Cloud bundles are rejected. The output contains
|
||||
only index HTML and history/pointer JSON; it does not upload evidence or change
|
||||
retained campaign bundles.
|
||||
|
||||
The explicit-only [chat feature evals](../tests/runner-e2e/CHAT-FEATURES.md) promote
|
||||
two natural-language task scenarios into local native Product E2E cells. They
|
||||
verify saving an unstarted task and delegating a saved announcement back to chat,
|
||||
@@ -331,6 +369,14 @@ compatible complete campaigns. Refreshing an existing report from retained
|
||||
evidence has zero provider calls and is a new presentation of the old
|
||||
measurement, not a new model run.
|
||||
|
||||
Product E2E test details and screenshot galleries show the recorded application
|
||||
URL when it is available. The worker environment, such as Daytona, does not
|
||||
imply an application URL. Public projections can retain the reviewed URL in
|
||||
`targetUrl`. Test details put recorded checks first and keep raw usage, timing,
|
||||
and startup evidence under Diagnostics. Optional `presentation` metadata in a
|
||||
retained campaign supplies its title, description, notes, and evidence links
|
||||
inside the shared report shell when the dashboard is regenerated.
|
||||
|
||||
Existing public histories are available at
|
||||
[Runner protocol history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html)
|
||||
and [Runner Product E2E history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/).
|
||||
@@ -539,5 +585,7 @@ API and download oracles; no live qualification is implied by fixture registrati
|
||||
|
||||
The explicit-only [feedback persistence case](../tests/runner-e2e/README.md#feedback-persistence-explicit-only)
|
||||
checks real native complaint/suggestion submissions and retained bodies with
|
||||
run-attributed tool receipts. It is manual, uses the normal Product E2E report,
|
||||
run-attributed tool receipts on local and Daytona execution. Its four native
|
||||
Claude/Codex cells are explicit-only. Remote registrations remain unqualified
|
||||
until retained live attempts prove them. It uses the normal Product E2E report
|
||||
and keeps synthetic feedback distinct from actionable product findings.
|
||||
@@ -14,6 +14,8 @@ Delegated work and interactions persist their originating context. Retries retai
|
||||
|
||||
Executions with managed GitHub configured receive token-free `git` and `gh` launchers. Each launcher invocation resolves one eligible credential from the current run's accepted identity at operation start. A `gh` command's child Git processes inherit that command's captured identity; later steering does not change already-started operations.
|
||||
|
||||
A `git` command that never uses the network does not ask the broker. The launcher reads past the global options, such as `-C` and `-c`, and checks the subcommand against a short list (for example `status`, `diff`, `log`, `rev-parse`, `add`, `config`, `branch`, `checkout`). An unknown subcommand or option still asks. `commit` and `var` still ask, because the managed author comes from the broker answer. The launcher keeps no broker answer between invocations and never writes a credential to a file.
|
||||
|
||||
GitHub credential capture reads the current task, project, and retained trust
|
||||
policy bindings without transferring saved prompts, runner profiles, results,
|
||||
or log excerpts. It keeps the existing task/run lock order and accepted-identity
|
||||
|
||||
@@ -2073,6 +2073,14 @@ configuration, and controller-owned credential or instruction resources use
|
||||
verified suspension and saved-history resume. This recovery can restart a
|
||||
provider; it does not suspend every healthy warm turn.
|
||||
|
||||
Changing configuration can retire a completed warm runner during instruction
|
||||
preparation, before the next native coordinator is claimed. Its maintenance
|
||||
checkpoints may use the next run's live preparation lease from the same
|
||||
controller process, bound to the same company and task. The original controller
|
||||
generation and committed result must still match. This permits drain and
|
||||
suspension; it grants no new provider work. Expired or foreign preparation
|
||||
owners retain the existing recovery hold.
|
||||
|
||||
Plugin sandbox reuse during acquisition uses the configured startup timeout or
|
||||
the provider's declared acquisition default, with the host RPC margin. Resume
|
||||
attempts and transient retries share that deadline. Providers without either
|
||||
@@ -2089,6 +2097,13 @@ parents. Existing session files, prior activity, missing proof or any saved
|
||||
backup retain the verified-restore requirement. Workspace replacement alone
|
||||
never authorizes a provider-session reset.
|
||||
|
||||
With external controller checkpoints, the local state file is only a marker.
|
||||
Fresh-session bootstrap checks use the registered controller's current canonical
|
||||
state. Marker identity alone cannot prove an untouched session. The same exact
|
||||
identity, zero-activity and pending-bootstrap-command checks apply before the
|
||||
remote directory claim; saved history and existing directories still prevent
|
||||
an unverified fresh start.
|
||||
|
||||
For the native runner, warm mode requests a reusable sandbox lease **before**
|
||||
lease acquisition. The environment's explicit runner lifecycle overrides the
|
||||
agent default; an inherited lifecycle uses the current agent setting on each
|
||||
@@ -2516,3 +2531,15 @@ restored checkpoint must match the normalized session before writing its local
|
||||
projection. This adds a scoped checkpoint read and local reconstruction only
|
||||
when warm admission encounters an external marker or an older projection; it
|
||||
adds no streaming checkpoint writes or recurring idle work.
|
||||
|
||||
Ordinary remote `per_turn` continuations restore a valid external controller
|
||||
marker from the referenced run's company/task-bound checkpoint before reading
|
||||
its durable identity. The restored checkpoint must match the normalized session.
|
||||
An existing prior-run projection also refreshes from that checkpoint, so a
|
||||
rejected attempt cannot retain state that its active owner later advances.
|
||||
Current-run full state stays owned by its controller. Legacy file-backed state
|
||||
retains its existing admission path when no database checkpoint exists.
|
||||
The existing prior-owner, suspended-state and backup checks still decide whether
|
||||
the next turn may start. Missing checkpoints and ambiguous identity remain
|
||||
blocked; restoration does not authorize a new provider session or change the
|
||||
configured lifecycle policy.
|
||||
@@ -67,6 +67,20 @@ its native runtime identity is `native:grok` version 1.0.13. The historical
|
||||
`agentServerPackage`/`agentRuntimePackage` wire fields carry these identities,
|
||||
not npm dependencies. Existing npm-backed ACP bridges retain their package pins.
|
||||
|
||||
Grok launch profile 7 binds the native effort, search, credential, sidecar, and
|
||||
Runner admission source, including Rust backend descriptor writes. An older
|
||||
installed pack with the same ACPX interface version cannot satisfy this profile.
|
||||
The controller stages its current verified
|
||||
pack instead. A generated source-identity comment in the launcher preserves the
|
||||
existing command byte-digest check; no installation guard is relaxed.
|
||||
App source changes alone still permit reuse when the selected
|
||||
Grok profile matches. Historical profile versions remain decodable, but they do
|
||||
not authorize a fresh launch. After changing a bound source, advance the profile
|
||||
and run `node packages/paperclip-runner/scripts/grok-profile-identity.mjs`.
|
||||
The normal `check:acpx-profiles` build gate verifies this source identity.
|
||||
The generator refuses to replace an existing versioned identity with different
|
||||
source. Advance the version and retain the previous version's recovery decoder.
|
||||
|
||||
Provision Grok Build 1.0.13 at
|
||||
`/opt/paperclip/providers/grok/1.0.13/grok` in the selected execution environment.
|
||||
The sandbox provisioning helper is explicit and is never run by npm:
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
# Teams abandoned attempts and idle work
|
||||
|
||||
Teams transport-attempt recovery now runs from committed transfer changes and
|
||||
saved claim deadlines. The shared coordinator and scheduler own its lifecycle.
|
||||
Together with consent expiry, this replaces the periodic Teams maintenance
|
||||
callback. It adds no table, schema or independent timer.
|
||||
|
||||
## Durable obligation
|
||||
|
||||
A transfer in `consent_sending`, `uploading`, `file_info_sending`, or `conflict`
|
||||
with `attemptExpiresAt` has an outstanding claim. The existing protocol writes
|
||||
an attempt ID and a 90-second deadline before provider I/O. A normal receipt
|
||||
clears both. A conflicting callback retains the original in-flight claim until
|
||||
its deadline; the conflict is not proof that the original operation stopped.
|
||||
|
||||
The protocol's central versioned update registers an awaited notification before
|
||||
the write, within the existing transaction. This includes normal claims,
|
||||
settlement, callbacks and caller-owned operator resolution. The DB work-signal
|
||||
boundary owns outer commit, rollback and uncertain transaction settlement.
|
||||
Inserts create unused consent, so they do not create an attempt obligation.
|
||||
|
||||
Startup reconstructs eligible attempts in 100-row UUID pages. A committed hint
|
||||
reloads its transfer ID. PostgreSQL timestamps round upward to a millisecond.
|
||||
An inspection does not move the saved deadline. A changed claim is reloaded
|
||||
before recovery; settled or deleted rows remove the cached obligation.
|
||||
|
||||
## Recovery preserves uncertainty
|
||||
|
||||
At the deadline, the existing Teams owner locks publication then transfer,
|
||||
checks integrity, and rechecks the current phase and deadline. It clears the
|
||||
expired attempt and changes a sending phase to its corresponding unknown outcome.
|
||||
Conflict stays conflict with its original reason and private evidence. Publication
|
||||
projection and transfer mutation remain atomic. Lock waits remain bounded.
|
||||
|
||||
Recovery does not call a provider, resend a card or file, manufacture a receipt,
|
||||
or assert that an unknown effect never occurred. A delayed foreground receipt
|
||||
still has to match its current attempt ID and phase. Existing operator resolution
|
||||
and on-use authorization rules remain unchanged.
|
||||
|
||||
Unknown outcomes with no live attempt are passive review evidence. Unused consent
|
||||
is owned by the separate consent-expiry worker. Neither keeps attempt recovery
|
||||
querying. Failed loads, locks or projections retain bounded retry work; fair
|
||||
batches allow healthy rows and other companies to progress. Worker errors redact
|
||||
SQL parameters and private transfer envelopes.
|
||||
|
||||
## Lifecycle and ownership
|
||||
|
||||
The Teams module selects phases and deadlines. The existing protocol performs
|
||||
recovery. Shared startup only registers the worker and admission gates. The
|
||||
shared scoped worker coalesces writes, preserves dirty scopes through drain,
|
||||
reconstructs after restart, and joins in-flight work during shutdown. An unresolved
|
||||
write may still need transaction-status reconciliation while dispatch is paused;
|
||||
this is not an empty maintenance scan.
|
||||
|
||||
The explicit combined `expireAndRecover` and attempts-only diagnostic methods
|
||||
remain available. Production `processPublicationMaintenance` no longer invokes
|
||||
Teams recovery. Other chat maintenance reasons are separate pending migrations.
|
||||
|
||||
Supported writes use this process's owning DB client. Another active server or
|
||||
independent SQL writer requires a separate notification contract. Cloud external
|
||||
alarm coverage and conservative sleep eligibility are unchanged. A registered
|
||||
local deadline cannot wake a stopped host. This migration proves worker quiet,
|
||||
not measured provider suspension or fleet savings.
|
||||
|
||||
## Verification
|
||||
|
||||
Real PostgreSQL tests exercise all sending stages, conflict during a live send,
|
||||
normal settlement, interrupted receipt persistence, saved deadlines across restart,
|
||||
quiet history, drain/resume, rollback, precision, atomic projection failure,
|
||||
102-entry paging/fairness and shutdown. The real Board producer integration
|
||||
registers the worker before sending, observes the saved claim while the provider
|
||||
receipt is held, and verifies that the committed receipt disarms recovery. Existing
|
||||
Teams authority, consent and publication cases remain covered. See the
|
||||
[audit and failure ledger](plans/2026-10-25-teams-attempt-recovery-sleep-audit.md).
|
||||
|
||||
|
||||
Current parent acceptance is recorded in the audit's
|
||||
[current-parent section](plans/2026-10-25-teams-attempt-recovery-sleep-audit.md#current-consent-parent-preparation-and-acceptance-october-28):
|
||||
239 selected cases pass, with the old inherited rate-window CI failure and its
|
||||
already-landed fixture fix preserved. Actual landed-parent reconciliation and
|
||||
fresh remote checks remain merge requirements; historical counts are not current
|
||||
full-workspace verification.
|
||||
@@ -0,0 +1,72 @@
|
||||
# Teams consent expiry and idle work
|
||||
|
||||
Unused Teams file consent now uses the shared delivery coordinator and scheduler.
|
||||
Existing `chat_teams_file_transfers` rows own its state. There is no new table,
|
||||
independent timer, provider request or external-wake capability.
|
||||
|
||||
## Obligation and trigger
|
||||
|
||||
Only `consent_pending`, `awaiting_consent` and `upload_pending` rows are eligible.
|
||||
Each uses its saved `expiresAt`. Registered startup reconstruction yields after
|
||||
one 100-row UUID page per scope refresh and resumes through the shared scheduler.
|
||||
Registration resets the owner’s cursors so a partial scan restarts safely. The
|
||||
explicit one-shot inventory method still returns the complete inventory. Committed writes reload the affected transfer ID. The service registers
|
||||
intent before insertion and its central versioned update, within the existing
|
||||
transaction. Nested savepoints wait for outer settlement. The shared work-signals
|
||||
protocol retains ambiguous outcomes; callers do not manage recovery wrappers.
|
||||
|
||||
Configured endpoints, completed history, unknown outcomes and active transport
|
||||
attempts do not keep this worker scanning. Pending consent retains its actual
|
||||
future deadline. PostgreSQL submillisecond timestamps round up to the next
|
||||
millisecond; checking a deadline does not extend it.
|
||||
|
||||
## Expiry and recovery
|
||||
|
||||
At the deadline, the worker reloads the current row and delegates to the existing
|
||||
Teams protocol owner. That owner locks publication then transfer, verifies the
|
||||
current authority digest, rechecks phase and expiry, and updates the transfer
|
||||
and publication projection in the same transaction. It does not send a consent
|
||||
card, upload data or send a file-info card. Lock waits remain bounded. A changed
|
||||
phase is re-evaluated; a deleted or settled row removes the cached obligation.
|
||||
|
||||
A failed load, lock or projection retains retry work. Corrupt rows do not starve
|
||||
other transfers or companies. Errors emitted by this worker are generic and do
|
||||
not contain SQL parameters, private transfer envelopes or provider capabilities.
|
||||
The shared worker owns coalescing, batching, dirty hints, drain/resume and shutdown.
|
||||
Shutdown joins an in-flight expiry; restart reconstructs the saved deadline.
|
||||
|
||||
The explicit `expireAndRecover` diagnostic method retains combined behavior.
|
||||
Production periodic maintenance now selects and executes abandoned attempts only.
|
||||
Attempt recovery is a separate remaining migration. Unknown provider outcomes
|
||||
remain quarantined; this change does not authorize a resend.
|
||||
|
||||
## Ownership and limitations
|
||||
|
||||
Provider phase selection and expiry live in the Teams module and existing Teams
|
||||
transfer service. Shared app wiring only registers the owner and lifecycle gate.
|
||||
The queue topic is a notification namespace, not a new durable queue. Other
|
||||
providers and the shared scheduler receive no Teams protocol policy.
|
||||
|
||||
Supported producers use the process's owning DB client. Independent SQL writers
|
||||
or another process require their own wake contract. On-use authorization remains
|
||||
with the protocol. Cloud eligibility and external alarm coverage are unchanged:
|
||||
a future in-process deadline cannot wake stopped compute. This proves the changed
|
||||
worker can become quiet, not fleet savings or complete instance suspension.
|
||||
|
||||
## Verification
|
||||
|
||||
Real PostgreSQL protocol cases cover all three phases, producer activation,
|
||||
rollback and held outer commit, restart, pre-deadline quiet, timestamp precision,
|
||||
projection rollback, fairness, drain/resume and shutdown. Production integration
|
||||
registers the worker before Board publication and proves expiry cancels the
|
||||
publication without another provider call. Existing authority, publication and
|
||||
consent suites remain covered. See the [audit and failure ledger](plans/2026-10-25-teams-consent-expiry-sleep-audit.md).
|
||||
|
||||
|
||||
Current local acceptance covers 223 distinct cases: 209 across five complete
|
||||
protocol/publication/consent/runtime/wiring suites, plus 14 selected real Board
|
||||
producer and Teams-authority integration cases. The other 1,580 integration
|
||||
cases were skipped. A real 105-row regression proves bounded startup, same-owner
|
||||
restart, complete saved-deadline expiry, and quiet SQL. A guarded missing-reset
|
||||
mutation fails that regression. See the audit for exact revisions, earlier
|
||||
fixture failures, current-parent integration status and deployment limits.
|
||||
@@ -0,0 +1,108 @@
|
||||
# Voice call reports and idle work
|
||||
|
||||
Voice report reconciliation uses the shared delivery coordinator and scheduler.
|
||||
It no longer runs from the chat maintenance interval. Existing
|
||||
`chat_voice_sessions` and `chat_voice_reports` rows own its durable state; this
|
||||
change adds no queue table, provider endpoint, or independent timer. A concurrent
|
||||
partial index supports bounded reconstruction of the existing report window.
|
||||
|
||||
## Trigger and deadline
|
||||
|
||||
A terminal session with a provider call ID remains eligible for report refresh
|
||||
for 24 hours after `ended_at`, inclusive. A missing report is due immediately.
|
||||
An existing report uses its saved `next_check_at`. The retry starts at 60 seconds
|
||||
and doubles to one hour, as before. Available reports remain refreshable during
|
||||
this finite window: provider availability is not a promise that transcript or
|
||||
billing data can no longer change.
|
||||
|
||||
Session transitions and signed callbacks notify the report worker before their
|
||||
writes, inside the owning database transaction. The reply sender's `call_ended`
|
||||
result does the same. Healthy active-session observations do not activate report
|
||||
work. Nested transactions wait for the outer commit; rollback does not publish
|
||||
new work. The existing work-signals protocol owns uncertain outcomes.
|
||||
|
||||
Startup reconstructs eligible sessions in pages ordered by `(ended_at, id)`,
|
||||
yielding after at most 100 rows per scope refresh. Re-registering the same owner
|
||||
resets partial scan cursors. The explicit one-shot inventory still loads every
|
||||
page when requested.
|
||||
A partial index selects terminal sessions with provider IDs. The query uses
|
||||
its ended-at range to restrict reads to the current 24-hour window. The cursor retains PostgreSQL timestamp precision. Committed report hints reload the exact changed session. Configuration,
|
||||
authority and connection metadata changes do not alter report eligibility or
|
||||
saved retries, so they do not wake report reconstruction. Each cached entry carries the
|
||||
saved report deadline or the first millisecond after the refresh window ends,
|
||||
whichever comes first. Submillisecond PostgreSQL timestamps round in the safe
|
||||
direction. Expiring the window removes the cached obligation without a provider
|
||||
request. Configured idle endpoints, active calls, old terminal history, and
|
||||
sessions without provider IDs do not cause recurring report SQL or requests.
|
||||
|
||||
## Claims, responses, and projection
|
||||
|
||||
The worker locks and rechecks the session and report, then saves its attempt and
|
||||
next retry before contacting the provider. Network I/O happens outside that
|
||||
transaction. Settlement compares the saved attempt and retry deadline. A newer
|
||||
claim or callback that moves the deadline invalidates the older response.
|
||||
Incomplete or older responses cannot overwrite a retained available report.
|
||||
Eight failures can mark a pending report unavailable; failure cannot downgrade
|
||||
an already available report.
|
||||
|
||||
Guest transcripts remain confined to the call's quarantined intake task. The
|
||||
worker reads the retained report, keeps the task's source-trust metadata, and
|
||||
uses the existing document revision contract. A document lock or conflicting
|
||||
edit leaves the saved report and retry intact. After the lock clears, the next
|
||||
successful refresh retries projection. The existing 24-hour eligibility policy
|
||||
still applies; this PR does not add indefinite transcript repair or extend it.
|
||||
|
||||
Provider credentials, call-report parsing, and delivery diagnostics remain in
|
||||
the voice integration. The shared scheduler receives only normalized work and
|
||||
deadlines. No provider-specific policy is added to app startup.
|
||||
|
||||
## Lifecycle and limitations
|
||||
|
||||
The shared scoped worker coalesces notifications, processes fair company batches,
|
||||
and retains dirty scopes across drain or failed reads. Shutdown joins in-flight
|
||||
work. Restart reconstructs the persisted retry instead of immediately repeating
|
||||
the provider GET. The explicit one-shot voice reconciliation method remains for
|
||||
diagnostics; production maintenance does not call it periodically.
|
||||
|
||||
The supported deployment has one active server process writing voice session
|
||||
and report state through its owning database client. Active-active servers and
|
||||
out-of-band SQL writers are not supported by this worker. Process-local signals
|
||||
do not reach another process; such a deployment needs a durable cross-process
|
||||
wake contract before removing its reconciliation backstop. This PR neither adds
|
||||
that transport nor asserts that startup reconstruction covers concurrent writers. This is worker quiet, not a claim of physical host sleep. Cloud alarm
|
||||
coverage and conservative sleep eligibility are unchanged; a local deadline
|
||||
cannot wake stopped compute.
|
||||
|
||||
## Verification
|
||||
|
||||
`voice-call-reports.test.ts` exercises real PostgreSQL claims, configured-idle
|
||||
quiet, finite expiry, timestamp precision, retries and restart, committed writes,
|
||||
rollback and nesting, drain/resume, callback races, concurrent claims, backlog
|
||||
fairness, result ordering, and shutdown. The full voice integration suite checks
|
||||
real hangup/callback/reply-error producers and transcript projection after a lock.
|
||||
The app wiring suite protects removal of the voice maintenance lane.
|
||||
|
||||
The audit and evidence ledger is
|
||||
[voice call-report scheduling](plans/2026-10-25-voice-call-report-sleep-audit.md).
|
||||
|
||||
|
||||
The [historical parent acceptance](plans/2026-10-25-voice-call-report-sleep-audit.md#current-parent-reconstruction-and-migration-acceptance-october-28)
|
||||
records the earlier 302 distinct passing cases, the bounded startup and same-owner restart
|
||||
regressions, deterministic company fairness, generated migration 0396 and the
|
||||
preserved failures. The report owner keeps its finite provider refresh obligation;
|
||||
this is not an unconditional idle scan or proof of deployed suspension.
|
||||
|
||||
|
||||
The subsequent [configuration-scope repair](plans/2026-10-25-voice-call-report-sleep-audit.md#configuration-scope-review-repair-october-28)
|
||||
records the earlier 303-case acceptance at `9bf20467df`. Five real connection saves over 205 saved
|
||||
reports select zero session rows, after reproducing 1,025 selected rows before
|
||||
the repair. The notification helper accepts only an exact session scope.
|
||||
|
||||
|
||||
The [landed-master acceptance](plans/2026-10-25-voice-call-report-sleep-audit.md#landed-master-migration-and-acceptance-october-28)
|
||||
records the current 185-case acceptance and generated migration
|
||||
`0397_known_madrox.sql`, after landed cleanup migration 0396. It preserves all
|
||||
existing SQL and journal entries, retains snapshots 0393 through 0397, and checks
|
||||
fresh application migration plus concurrent replay and invalid-index recovery.
|
||||
The older 302/303-case sections above are historical evidence, not the current
|
||||
migration filename or a claim that those full groups were rerun on this head.
|
||||
@@ -0,0 +1,111 @@
|
||||
# Inbound voice cleanup without idle scans
|
||||
|
||||
Inbound admission cleanup runs from committed changes and saved deadlines. It
|
||||
uses the shared delivery coordinator and scoped deadline worker. The recurring
|
||||
voice maintenance lane now owns only history retention and reports, which remain
|
||||
separate migrations. Manual reconciliation remains available for diagnostics.
|
||||
|
||||
## Ownership and durable state
|
||||
|
||||
An unbound admission owns the existing provider call until approval or guest
|
||||
intake binds a voice session. The live-session recovery service then owns physical
|
||||
termination. Admission cleanup never also ends that bound call. It can request
|
||||
ending through the session row and copy the owner's confirmed terminal result.
|
||||
|
||||
Approval now commits the admission binding with the authorized session insert or
|
||||
guest promotion. Expiry cannot see an old approving admission after that session
|
||||
commits. Cleanup also locks and rechecks the admission before claiming it; a
|
||||
candidate from an earlier scan is not permission to terminate a call. Both paths
|
||||
lock admission before session. Provider requests run outside SQL transactions.
|
||||
Older interrupted handoffs are repaired only when the existing session matches
|
||||
the approval actor, authority, request key, task, token, generation and credential
|
||||
identity. Conflicting ownership retains an error/retry instead of terminating a
|
||||
call or inventing authority.
|
||||
|
||||
The migration adds three nullable fields to `chat_voice_inbound_calls`, not a new
|
||||
queue table:
|
||||
|
||||
- `cleanupConfirmedAt` separates admission status from physical termination.
|
||||
An ended or denied admission can still have an unconfirmed provider leg.
|
||||
- `cleanupNextAttemptAt` survives crashes and restart. The claim saves the retry
|
||||
before contacting the provider.
|
||||
- `providerAccountId` records the account authenticated by the signed pre-call.
|
||||
Cleanup can use a rotated key after verifying its account. A key for another
|
||||
account cannot inspect or terminate this call. Older admissions lack that
|
||||
evidence and require their original credential fingerprint.
|
||||
|
||||
The migration is replayable. It backfills only historical ended rows whose old
|
||||
writer or signed lifecycle audit proves termination. It leaves old guest-denial
|
||||
rows uncertain when their ended label did not prove physical cleanup. Those rows
|
||||
receive a real check rather than disappearing as completed history. The partial
|
||||
index covers unconfirmed admission identities. Startup reads at most 100 rows
|
||||
per scope refresh and reconstructs deadlines across pages. The historical
|
||||
backfill commits keyset batches of at most 1,000 rows through the migration
|
||||
runner. Migration 0395 preserves the landed checkpoint lineage through 0394.
|
||||
|
||||
## Deadlines and quiet state
|
||||
|
||||
Rejection returns after its decision transaction commits. Its notification wakes
|
||||
the registered cleanup worker; the HTTP request does not wait for provider account
|
||||
verification, inspection, or hangup. The saved denial immediately rejects further
|
||||
unauthorized call input while physical termination remains tracked separately.
|
||||
|
||||
Pending unbound admissions retain their original two-minute approval or ten-minute
|
||||
guest-intake expiry. Rejection is immediately due. A cleanup claim saves a
|
||||
30-second retry before provider I/O. An unconfirmed end receipt keeps that retry;
|
||||
inspection, a confirmed end receipt, or a signed terminal callback clears it.
|
||||
Unexpected failures also receive the shared worker's bounded failure backoff.
|
||||
Neither a timeout nor an end request by itself proves termination.
|
||||
|
||||
Bound live calls leave no admission cleanup deadline. Their existing live-session
|
||||
worker owns expiry, authorization checks and provider cleanup. Its terminal write
|
||||
signals admission settlement. Healthy passive provider inspections do not wake
|
||||
this admission worker. Confirmed history and configured-but-idle endpoints leave
|
||||
no recurring admission SQL or provider requests.
|
||||
|
||||
Feature disable does not discard an outstanding call. Cleanup continues with its
|
||||
saved identity. Missing credentials or account mismatches retain debt; they do not
|
||||
permit use of a replacement account. No recovery path creates or redials a call.
|
||||
|
||||
## Producer audit
|
||||
|
||||
| Producer | Commit boundary and wake |
|
||||
| --- | --- |
|
||||
| Signed pre-call admission | Endpoint-scoped intent before insert in the quota transaction |
|
||||
| Approval or rejection | Admission-scoped intent before the locked state write |
|
||||
| Authorized session insert or guest promotion | Admission binding and intent inside the reservation transaction |
|
||||
| Guest intake task/session binding | Session-scoped voice change before insert; admission binding shares the transaction |
|
||||
| Pending tool detects expiry | Admission intent before the expired-state write |
|
||||
| Signed terminal lifecycle/report callback | Session and admission intents before confirmed termination writes |
|
||||
| Live-session termination, inspection or no-resource expiry | Existing voice-change boundary also signals inbound settlement |
|
||||
| Reply provider confirms the call ended | Inbound session intent before the terminal session write |
|
||||
| Configuration, authority or connection changes | Existing scoped voice-change composition |
|
||||
| Cleanup claim, legacy handoff repair and confirmation | Admission intent in the owning transaction |
|
||||
|
||||
Notifications are process-local hints. Existing database rows remain truth. The
|
||||
shared transaction layer delays hints until outer commit and retains uncertainty.
|
||||
Other replicas and independent SQL writers require an explicit integration.
|
||||
|
||||
History retention cannot delete unconfirmed cleanup. Company deletion locks its
|
||||
endpoints, matching ingress, then asks the voice owner to prove inbound cleanup is
|
||||
settled. It rejects deletion while a provider obligation remains. This prevents
|
||||
cascade deletion from erasing the evidence needed to finish cleanup. Existing
|
||||
voice-history foreign-key restrictions otherwise remain unchanged.
|
||||
|
||||
## Verification and limits
|
||||
|
||||
Real PostgreSQL tests cover confirmed-history quiet, exact expiry, restart,
|
||||
transaction rollback/outer commit, stale discovery, old interrupted handoffs,
|
||||
bound-session ownership, unconfirmed requests, failures, fair paged backlog,
|
||||
drain/resume, shutdown, retention protection, and deletion protection. Production
|
||||
service tests exercise signed ingress, rejection, account mismatch and rotation,
|
||||
feature disable, and atomic ordinary/guest approval. See the dated
|
||||
[inbound cleanup audit](plans/2026-10-25-voice-inbound-cleanup-sleep-audit.md) for
|
||||
commands, counts and preserved failures. The current-parent acceptance records
|
||||
266 distinct passing cases and the guarded batching, restart and approval
|
||||
regressions. Earlier counts in that audit belong to their stated revisions.
|
||||
|
||||
These changes do not migrate history retention or call reports, extend Cloud's
|
||||
external wake coverage, relax sleep eligibility for live calls, or prove actual
|
||||
host/database suspension. A registered in-process deadline alone cannot wake
|
||||
stopped compute.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Inbound voice history retention
|
||||
|
||||
Inbound history retention uses the shared work scheduler. It does not scan on
|
||||
the chat maintenance tick. Call-report reconciliation remains a separate duty.
|
||||
|
||||
An admission is eligible only when it is denied, expired or ended, provider
|
||||
cleanup is confirmed, and neither a session nor an intake task owns it. The
|
||||
existing policy retains it until both expiry and last update are seven days old:
|
||||
`max(expiresAt, updatedAt) + 7 days`. No new table or persisted state is added.
|
||||
|
||||
The voice owner reconstructs eligible deadlines on startup, yielding after at
|
||||
most 100 selected rows per scope refresh. Re-registering the same service resets
|
||||
its page cursor so a partial scan cannot skip old deadlines. The existing
|
||||
`notifyVoiceInbound` boundary now notifies cleanup and retention before writes.
|
||||
Outer transaction settlement wakes the workers. Rolled-back changes do not
|
||||
create retention work. Signed callbacks, rejection cleanup, approval/session
|
||||
handoff and cleanup recovery retain their existing transaction ownership.
|
||||
Configuration, authority and session-only changes do not change admission
|
||||
retention eligibility, so they do not wake retention. Startup reconstructs
|
||||
existing history, and admission writes carry the exact retention hint.
|
||||
|
||||
At the deadline, retention locks the admission and rechecks current eligibility
|
||||
and both timestamps before deleting. PostgreSQL submillisecond timestamps round
|
||||
up for scheduling. Cached candidates cannot authorize deleting a new binding or
|
||||
history whose retention deadline moved. Batches page through history and use the
|
||||
shared worker's company fairness, coalescing, bounded backoff and drain behavior.
|
||||
Deletion failure retains a retry; a paused first write remains pending for resume.
|
||||
Shutdown cancels or joins work, and restart reloads remaining durable deadlines.
|
||||
|
||||
Configured idle endpoints, ineligible history, and future retained history do not
|
||||
cause recurring SQL between changes or actual deadlines. Unconfirmed cleanup
|
||||
remains owned by the cleanup worker. Session/task history is not deleted here.
|
||||
Explicit one-shot `pruneHistory` and diagnostic reconciliation remain available;
|
||||
startup does not register those bulk operations as periodic work.
|
||||
|
||||
Supported writers must participate in the shared transaction notification
|
||||
boundary. Independent SQL writers or additional processes require their own
|
||||
notification/recovery contract. This migration does not extend Cloud alarms or
|
||||
sleep eligibility: an in-process deadline cannot wake stopped compute. The
|
||||
external wake owner must cover retention before that deadline can permit sleep.
|
||||
|
||||
|
||||
The [current-parent acceptance](plans/2026-10-25-voice-inbound-retention-sleep-audit.md#current-parent-paging-and-restart-acceptance-october-28)
|
||||
records bounded paging and restart acceptance. The subsequent
|
||||
[notification review repair](plans/2026-10-25-voice-inbound-retention-sleep-audit.md#configuration-rescan-review-repair-october-28)
|
||||
records 282 current passing cases and repeated configuration saves selecting zero
|
||||
history rows, with the original failures and verification limits preserved.
|
||||
@@ -0,0 +1,156 @@
|
||||
# Voice reply delivery without idle polling
|
||||
|
||||
Phone replies wake from committed publication, reply, session, and configuration
|
||||
changes. The application registers the worker with the shared delivery coordinator.
|
||||
The one-second chat maintenance lane no longer calls `pushReplies`. Inbound voice
|
||||
admission, report reconciliation, and other chat duties remain separate migrations.
|
||||
|
||||
## Durable truth and delivery boundaries
|
||||
|
||||
The existing voice sessions, voice replies, chat publications, and chat actions
|
||||
remain authoritative. No table or migration is added. The worker reconstructs
|
||||
active phone sessions and outstanding dispatch intents at startup. Notifications
|
||||
are scoped hints, never permission to replay a provider action.
|
||||
|
||||
A reply inserted while its publication is streaming is not yet deliverable.
|
||||
Both normal publication completion and operator resolution signal the endpoint
|
||||
before committing its published state. Reply insertion and authorized question
|
||||
retrieval signal their session before writing. The existing transaction boundary
|
||||
holds notifications until outer commit and reconciles uncertain outcomes.
|
||||
|
||||
The dispatcher retains current authorization, session locking, ordered reply and
|
||||
multipart barriers, and its atomic action claim. Provider calls remain outside
|
||||
SQL transactions. An accepted receipt is distinct from spoken playback. Unknown
|
||||
outcomes never authorize resending the answer or appending its remaining parts.
|
||||
|
||||
## Deadlines and disarming
|
||||
|
||||
- A ready ordered reply is immediate work; a bounded pass drains up to eight parts.
|
||||
- A definitive rate limit retains its saved retry deadline.
|
||||
- A dispatch without a durable receipt retains its existing strict 30-second
|
||||
uncertainty deadline. These actions are tracked independently of live sessions,
|
||||
so ending a call cannot strand its abandoned dispatch intent.
|
||||
- Database or provider-path failures retain per-entry backoff. Scoped inventory
|
||||
failures retry their scope; healthy companies retain fair processing access.
|
||||
- No ready reply, a presented question awaiting retrieval, accepted history, or an
|
||||
unknown/failed outcome leaves no recurring reply scan.
|
||||
|
||||
Live-call recovery already checks current authority before provider inspection.
|
||||
Reply delivery remembers sessions denied by its authorization check. A successful
|
||||
subsequent live-call authorization signals only those blocked sessions. This
|
||||
preserves restoration after membership/task authority changes without adding
|
||||
notifications to every authorization table or scanning healthy empty reply queues
|
||||
on every live-call inspection. Startup reconstructs this in-memory state from
|
||||
existing sessions. This dependency must remain explicit if active-call recovery
|
||||
is changed later; a notification alone never bypasses reauthorization.
|
||||
|
||||
## Lifecycle and limits
|
||||
|
||||
The coordinator coalesces changes, serializes work, preserves dirty scopes across
|
||||
failure and drain, and stops work on shutdown. Startup loads identities in pages;
|
||||
processing uses fair bounded batches. Manual `pushReplies` and one-shot `reconcile`
|
||||
remain available for explicit diagnostics/tests and must not become periodic lanes.
|
||||
|
||||
Signals cover the supported process-local database owner. Independent SQL writers
|
||||
or other processes require their own notification/reconciliation contract. These
|
||||
changes do not extend Cloud wake coverage, permit sleep during a live call, or
|
||||
prove deployed instance/database suspension.
|
||||
|
||||
|
||||
Both live-session recovery and reply delivery use `scoped-deadline-work.ts` for
|
||||
coalescing, fair batching, scope reconstruction, retries and drain retention.
|
||||
Their adapters keep domain state and provider operations in the voice owner.
|
||||
The production-wiring regression covers authority returning before live recovery
|
||||
observes the denial; recovery still ends calls when their authority is absent.
|
||||
|
||||
|
||||
## Current-parent paging contract acceptance, October 27
|
||||
|
||||
This slice normally integrates bounded voice recovery #17576 at `712f3862b5`,
|
||||
including receiver `6e88ca0201` and the landed foundations. The original 30-file
|
||||
reply scope adds one shared contract path, `scoped-deadline-work.ts`. No database
|
||||
file changes. The existing helper could represent only a complete inventory;
|
||||
using that contract for the repaired parent would erase its bounded-pass guarantee.
|
||||
|
||||
The real 205-call parent regression fails before the integration repair: startup
|
||||
executes three inventory queries (100, 100, 5 rows) instead of one 100-row query.
|
||||
The shared helper now accepts either its existing complete array or a normalized
|
||||
page result. It retains page generations and seen identities, keeps existing
|
||||
entries until reconstruction completes, limits dirty scopes per pass, and replays
|
||||
writes received during reconstruction. It owns no SQL or voice/provider policy.
|
||||
Recovery delegates to its owner refresh and resets partial owner state on worker
|
||||
registration. Existing complete-array consumers keep their public calls.
|
||||
Connection observations retain the parent's separate receiver/command notification
|
||||
path; actual configuration wakes both voice owners.
|
||||
|
||||
All 445 distinct targeted cases pass: 18 complete parent recovery lifecycle
|
||||
cases, 108 complete voice-store/reply integrations, 224 complete tool-gateway and
|
||||
secret-proposal cases, and 95 shared dispatcher/coordinator/batching/startup/
|
||||
hostname cases. The helper's single existing test also ran in the initial
|
||||
19-case green group; it is counted only in the 95-case group. The 224 cases
|
||||
exercise the four other production consumers of the shared helper, including
|
||||
complete-array empty/expiry/recovery/projection, transaction and drain behavior.
|
||||
Server production typecheck and server build pass. The voice fixtures self-migrate
|
||||
separate initially empty databases. The two other database suites use separate
|
||||
migrated clones. Expected injected provider/wakeup failures and listener warnings
|
||||
remain in logs. Only documentation edits follow these passing production checks.
|
||||
|
||||
This acceptance preserves the parent's startup paging, restart, behind-cursor
|
||||
insert, concurrent feature change, health-only quiet, true disablement, current
|
||||
authority, failure, drain and shutdown contracts. Existing real publication,
|
||||
operator resolution, uncertain send, multipart/question ordering, authority
|
||||
restoration and 105-abandoned-dispatch tests remain in the 108-case suite.
|
||||
Fresh exact-head reviews and all CI remain required. No full local matrix,
|
||||
live provider acceptance, Cloud wake, deployed suspension or savings is claimed.
|
||||
|
||||
|
||||
## Publication inventory and paging acceptance, October 28
|
||||
|
||||
The security review correctly found an endpoint-wide read per publication.
|
||||
Two real publication-path regressions seed 105 unrelated active phone calls and
|
||||
105 future dispatch intents on the same endpoint. Five normal publications select
|
||||
1,065 inventory rows; five operator resolutions select 1,060. Both exceed the
|
||||
125-row total budget. The corrected paths pass that same budget and deliver each
|
||||
answer once. These measurements count selected row results from executed queries,
|
||||
not physical PostgreSQL tuple visits or a deployment cost benchmark.
|
||||
|
||||
Normal publication completion and operator resolution now notify their actual
|
||||
conversation inside the existing transaction. The voice owner handles that
|
||||
normalized conversation scope. Its reader returns at most 100 total session and
|
||||
action headers per refresh, with a saved phase/cursor for the next pass. Startup,
|
||||
configuration changes and abandoned actions retain finite reconstruction. The
|
||||
shared dispatcher retains generations and observed IDs across pages; it also
|
||||
preserves the parent wildcard-failure isolation. SQL and provider decisions stay
|
||||
inside the voice owner. No schema, table, provider-specific shared branch or
|
||||
per-route recovery wrapper is introduced.
|
||||
|
||||
The abandoned-dispatch case now proves a 100-row startup cap, restart of the same
|
||||
owner before its scan completes, restoration of all 105 saved deadlines, no SQL
|
||||
before the deadline, and quarantine without resending. A separate owner-level
|
||||
real-database case reproduces a new denied call behind an older scan cursor:
|
||||
completion of the old scan must not erase the later restoration obligation.
|
||||
Denial revisions fence that cleanup. The case confirms a real 403, then observes
|
||||
the committed wake. The existing actual-service authority-restoration case still
|
||||
verifies the complete production callback path; the new cache test is not a
|
||||
replacement for that integration coverage.
|
||||
|
||||
Guarded mutations broaden publication hints back to endpoints, omit owner reset,
|
||||
and omit the denial revision fence. All fail the corresponding row-budget,
|
||||
100-versus-5 restart, or missing committed-wake assertion, with source restored.
|
||||
An earlier reset probe using already-due actions did not detect the missing reset
|
||||
because its own action writes rediscovered the backlog. That incomplete probe is
|
||||
retained as such; future deadlines make the strengthened regression deterministic.
|
||||
The first amplification fixture also used a shadowed publication function name;
|
||||
that fixture failure preceded the actual 1,065/1,060-row baseline.
|
||||
|
||||
The final branch normally integrates voice parent `ad79c7e805`. Its original
|
||||
30-file scope plus the existing shared dispatcher remains 31 files, with no DB
|
||||
changes. All 449 distinct targeted cases pass: 111 complete voice-store/reply,
|
||||
19 complete parent recovery, and 319 complete shared/legacy-consumer/startup cases.
|
||||
Both voice fixtures self-migrate separate initially empty databases; the two
|
||||
legacy DB suites use separate migrated clones. Production server typecheck and
|
||||
build pass. The current policy checker and its Node tests pass. Only evidence
|
||||
edits follow these checks. Older 445/786-case totals belong to their stated source
|
||||
revisions and are not added here. Current exact-head reviews and all CI remain
|
||||
required. No full local matrix, live provider, external wake, deployed suspension,
|
||||
physical query-cost or savings result is claimed.
|
||||
@@ -0,0 +1,184 @@
|
||||
# Voice session recovery without idle discovery
|
||||
|
||||
The live-session worker replaces the session-discovery portion of the recurring
|
||||
voice maintenance call. It uses the existing delivery coordinator, transaction
|
||||
signals and shared scheduler. It adds no tables or migrations. Reply delivery,
|
||||
inbound admission cleanup, inbound history retention and call reports are separate
|
||||
obligations; their existing maintenance remains until their own migrations land.
|
||||
|
||||
## Work and deadlines
|
||||
|
||||
Existing `chat_voice_sessions` rows are authoritative. Startup reconstructs live
|
||||
sessions in UUID pages of 100, joined to their same-company endpoint and connection.
|
||||
Each background refresh reads at most one inventory page. An unfinished pass has
|
||||
a one-second continuation; it does not create a permanent empty scan. Only timing
|
||||
and eligibility fields are selected for discovery. The explicit foreground
|
||||
reconciliation method keeps its complete inventory behavior.
|
||||
Terminal history does not create a live-session recovery obligation. With no live
|
||||
sessions, the worker also avoids reading experimental settings after reconstruction.
|
||||
|
||||
| State | Next obligation |
|
||||
| --- | --- |
|
||||
| No provider identity yet | Existing `expiresAt` plus two minutes of teardown grace; no provider creation replay |
|
||||
| Known provider, ending or invalidated configuration | Immediate cleanup; a recorded cleanup failure waits 30 seconds |
|
||||
| Connecting or active call | Earlier of session expiry and `updatedAt + 30 seconds` for current authorization and missed-hangup inspection |
|
||||
| Other live state with provider identity | Saved session expiry |
|
||||
| Ended, failed or expired | No live-session work; reports have a separate owner |
|
||||
|
||||
Provider maximum duration remains the final bound for uncertain creation. A
|
||||
late provider response can bind identity while expiry is being considered. The
|
||||
expiration write requires that provider identity is still absent, so a newly
|
||||
known call retains exact-call cleanup rather than being marked terminal prematurely.
|
||||
|
||||
Each due call is reloaded before execution. The voice store checks current
|
||||
company, endpoint, credentials, caller, task, guest sandbox and binding authority.
|
||||
A 403/404 means authority no longer permits the call. A transient lookup error
|
||||
retains a retry; it does not authorize ending an otherwise valid call. The existing
|
||||
voice service owns provider inspection and termination, including single-flight
|
||||
termination and ambiguous-response confirmation. Recovery never creates a call.
|
||||
|
||||
## Producers and transaction ownership
|
||||
|
||||
| Owner | Changes that wake live-session recovery |
|
||||
| --- | --- |
|
||||
| `voice-session-store.ts` | Reservation, guest promotion, connecting-to-active transition and reply cursor writes that change the inspection timestamp |
|
||||
| `voice-session-service.ts` | Creation claim, binding, known/unknown creation failure, user end, cleanup outcome and foreground inspection |
|
||||
| `voice-inbound-service.ts` | Guest session creation, guest denial and signed terminal lifecycle callbacks |
|
||||
| `voice-reply-push.ts` | Confirmed remote end and unknown delivery outcome that changes session state/timestamp |
|
||||
| Chat configuration helper | Endpoint configuration, pause, reconnect, generation/credential changes, connection hints and company removal |
|
||||
| Generic tool/access services | Voice connection API mutation, removal and credential revocation during access removal |
|
||||
| Instance settings | Explicit chat-connector feature enable/disable |
|
||||
|
||||
Existing transactions register the awaited signal before the session/configuration
|
||||
write. Standalone session updates use the small `withVoiceSessionChange` helper.
|
||||
Provider requests remain outside database transactions. Nested writes retain outer
|
||||
commit ownership, rollback does not dispatch work, and uncertain commit settlement
|
||||
stays with the shared database protocol.
|
||||
|
||||
Membership changes, task reassignment, phone preferences and other authorization
|
||||
inputs that do not mutate the session or connection remain checked on the existing
|
||||
30-second active-call inspection deadline, as well as on use. This slice does not
|
||||
promise instant cleanup for every authority mutation or add cross-cutting voice
|
||||
hooks throughout those owners. Session activity that changes `updatedAt` retains
|
||||
the existing inspection-time semantics. The session expiry is always an upper
|
||||
bound for authorized active work.
|
||||
|
||||
The generic chat connection hint does not always carry a transport. It can cause
|
||||
a bounded empty voice lookup after a chat configuration write. It does not create
|
||||
idle polling or a recurring deadline. The shared helper contains only routing;
|
||||
voice eligibility, authority and provider policy stay with the voice owner.
|
||||
|
||||
## Bounded configuration refreshes
|
||||
|
||||
The voice owner remembers the endpoint/connection fields that affect its existing
|
||||
deadline policy: endpoint status, runtime generation, connection enabled state,
|
||||
connection identity, and the chat feature flag. A broad configuration hint first
|
||||
reads a bounded page of these parent records. If those facts are unchanged, the
|
||||
cached call deadlines stay in place and no call inventory is read. Metadata-only
|
||||
writes do not move business deadlines. An unknown or changed parent requires
|
||||
bounded reconstruction; a real disablement still settles every affected call.
|
||||
|
||||
Only the voice owner interprets these facts. The generic health producer now
|
||||
sends a health-observation notification through the existing chat routing owner.
|
||||
It preserves receiver and command signals without generating a voice-configuration
|
||||
wake. An observation changes neither voice eligibility nor the saved call deadline.
|
||||
Supported session producers continue to signal their exact session ID.
|
||||
|
||||
The worker processes at most 25 dirty scopes and 25 provider checks per pass.
|
||||
Scope pagination continues without restarting on every newer hint. Failures retain
|
||||
their backoff and block stale entries in the failed scope. Successful pages can
|
||||
make progress while the remaining inventory is discovered. The existing
|
||||
coordinator can use the actual error deadline, including an already-due healthy
|
||||
backlog, rather than postponing it to an unrelated fallback retry.
|
||||
|
||||
## Failure and lifecycle behavior
|
||||
|
||||
Refreshes coalesce by scope. Paged reconstruction keeps narrower producer hints,
|
||||
including a newly committed session behind the UUID cursor. A scope generation
|
||||
retains configuration changes that arrive between pages. Seen IDs protect
|
||||
concurrent refresh results; absent cached entries are removed only when the
|
||||
scope's inventory is complete. A new worker resets partial inventory state before
|
||||
reconstructing its saved obligations. Changes arriving during refresh remain dirty. Failed refreshes block use of
|
||||
stale cached entries in that scope. Per-scope and per-call failure backoff starts
|
||||
at 30 seconds and caps at five minutes. New hints do not erase failure backoff.
|
||||
Batches rotate across companies, process at most 25 calls, and continue an already
|
||||
due backlog after one second. A failed provider does not stall independent calls.
|
||||
|
||||
Drain and standby retain dirty work without passive SQL. Resume processes that
|
||||
work, and restart reconstructs deadlines from the rows. Shutdown joins admitted
|
||||
provider work and removes the coordinator's subscriptions and timers. Retained
|
||||
terminal history becomes quiet after any already committed hints settle.
|
||||
|
||||
Process-local signals support the existing single active database writer and its
|
||||
shared clients. Independent SQL writers or another replica need an explicit wake
|
||||
contract. This change neither expands Cloud alarm coverage nor relaxes sleep
|
||||
eligibility. Active calls and their real deadlines remain obligations; local worker
|
||||
quiet is not proof of host suspension or fleet savings.
|
||||
|
||||
## Verification
|
||||
|
||||
The voice integration suite exercises actual service producers, guest intake,
|
||||
signed callbacks, current authority, unknown creation and adjacent reply/report
|
||||
behavior. The recovery-owner suite uses real PostgreSQL and controlled provider
|
||||
operations for outer commit/rollback, drain, restart, scope failure, mutation during
|
||||
refresh, paging, company fairness, shutdown and the late-provider-binding race.
|
||||
The latter race failed before the write-time guard and passes afterward.
|
||||
|
||||
The dated audit records exact commands, results, initial fixture failures and
|
||||
remaining acceptance work: [voice migration audit](plans/2026-10-25-voice-maintenance-sleep-audit.md).
|
||||
|
||||
|
||||
## Landed-foundation acceptance, October 27
|
||||
|
||||
The voice repair now integrates receiver parent `6e88ca0201`, including Discord
|
||||
`547fdf2139` and master `f8563d2507`. Exact-patch comparison preserves the repaired
|
||||
22-file scope and master's shared task-drain and native-checkpoint queue names.
|
||||
No database file is part of this slice.
|
||||
|
||||
All 725 distinct targeted cases pass again: 112 complete voice/shared/startup
|
||||
cases, 97 complete voice-store cases, 469 complete tool-access cases, and 47
|
||||
complete health/receiver cases. Server production typecheck and build pass.
|
||||
Voice worker/store and health/receiver fixtures use separate initially empty
|
||||
self-migrating databases. Tool access uses a separate migrated clone. One initial
|
||||
tool-access invocation supplied the generic fixture variable instead of the
|
||||
suite-specific variable: 23 cases passed and 446 skipped. It was not a complete
|
||||
suite result or an assertion failure. The corrected invocation executes and
|
||||
passes all 469. Tests are not counted twice. Expected controlled failures and
|
||||
listener warnings remain in logs. Only documentation changes follow these checks.
|
||||
Fresh current-head reviews, CI and parent landing remain required. No full local
|
||||
matrix, provider acceptance, external wake or deployed sleep result is claimed.
|
||||
|
||||
|
||||
## Wildcard failure and deterministic generation acceptance, October 28
|
||||
|
||||
Current-head CI `40633667141` fails server shard 1 at the feature-change test's
|
||||
34,050 ms result against its 25,000 ms bound; verification fails as an aggregate.
|
||||
This is an actual required assertion failure, distinct from the older interrupted
|
||||
build and missing policy script. The old test mixed automatically advancing fake
|
||||
time with real database latency. It now disables automatic clock advancement,
|
||||
drives 20 explicit scheduler turns, waits for each pass without advancing the
|
||||
clock, and requires every invalidated call to close before the unrelated periodic
|
||||
authority deadline. Removing generation replay still fails with missing closures.
|
||||
The test no longer benchmarks host/database speed or raises its business bound.
|
||||
|
||||
A real committed-session regression also reproduces wildcard starvation: while
|
||||
one broad reconstruction backs off, a newly ending exact call is not closed.
|
||||
The worker now retains the broad retry but considers ready exact hints and cached
|
||||
due calls independently. Each due call still reloads state and authorization in
|
||||
its owning check before effects. Failed exact scopes retain their own fence;
|
||||
the existing stale-cache failure regression still passes. Removing wildcard
|
||||
isolation fails the new exact-call assertion. Broad reconstruction becomes quiet
|
||||
after its successful retry, with no continuing SQL afterward.
|
||||
|
||||
The full-row execution lookup and lean inventory lookup now share the live-state,
|
||||
scope and cursor predicate. Their projections remain intentionally different.
|
||||
An initial edit script rejected the header cursor name before applying the timing
|
||||
and predicate changes; the intermediate 19-case result covers only the earlier
|
||||
worker repair. It is not the final repair evidence. The corrected source passes
|
||||
all 19 lifecycle cases and 191 complete voice-store/shared/startup cases: 210
|
||||
distinct targeted cases. Production server typecheck and build pass. Both guarded
|
||||
regressions fail at their intended effect assertion and restore source afterward.
|
||||
The fresh voice-store database self-migrates. Existing controlled failure logs
|
||||
and signal-listener warnings remain recorded. The older 725-case acceptance
|
||||
belongs to its stated revision and is not added to this count. Fresh current-head
|
||||
reviews and all CI remain required before merging after receiver discovery.
|
||||
@@ -17,7 +17,7 @@ Do not mark an item done from a timer removal or a green check alone.
|
||||
| 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending |
|
||||
| 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Approval retention [#17497](https://github.com/paperclipai/paperclip/pull/17497) and archive delivery [#17475](https://github.com/paperclipai/paperclip/pull/17475) merged; other source retention pending |
|
||||
| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479) merged; effects [#17482](https://github.com/paperclipai/paperclip/pull/17482) merged; targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) merged |
|
||||
| 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | #17554 repaired at `8c1d293d1a`; fresh CI/review in progress |
|
||||
| 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | [#17554](https://github.com/paperclipai/paperclip/pull/17554) merged after all checks and both 5/5 reviews |
|
||||
| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Slices 02–07 merged: [secret proposals](https://github.com/paperclipai/paperclip/pull/17446), [credentials](https://github.com/paperclipai/paperclip/pull/17450), [review expiry](https://github.com/paperclipai/paperclip/pull/17453), [approved recovery](https://github.com/paperclipai/paperclip/pull/17454), [interrupted claims](https://github.com/paperclipai/paperclip/pull/17458), [projection](https://github.com/paperclipai/paperclip/pull/17468) |
|
||||
| 08 | GitHub connection events and continuity | Event discovery/replay and credential continuity separately | Pending |
|
||||
| 09 | Merged-PR confirmations | Candidate writes, webhook activation where supported, outstanding-candidate fallback | Pending |
|
||||
@@ -1483,6 +1483,172 @@ loops or changing domain backoff. All 152 selected command/coordinator/health,
|
||||
producer/Slack, delivery/standby/startup cases and server types/build pass. Publish
|
||||
for fresh CI/review. The full inventory, #17490 CI failure, and stack/master
|
||||
reconciliation remain open; no merge authority is implied.
|
||||
|
||||
|
||||
## Slice 20 receiver implementation
|
||||
|
||||
Receiver discovery now uses scoped committed configuration changes, startup
|
||||
reconstruction, foreign-lease deadlines, and listener-stop completion. Voice and
|
||||
all other maintenance reasons remain separate. See
|
||||
[the receiver audit](2026-10-25-chat-receiver-discovery-sleep-audit.md).
|
||||
A failed-scope deadline regression and a real concurrent listener-stop race were
|
||||
reproduced and repaired. All 218 selected receiver/startup, Discord, Photon,
|
||||
command/delivery and standby cases pass, with workspace typechecking passing.
|
||||
Workspace build also passes; publication and fresh remote gates remain open. This is not
|
||||
completion of the slice or the full stack. Master is refreshed at 0d73505098;
|
||||
final stack/master reconciliation remains required.
|
||||
|
||||
Parent Discord #17557 at d3187f76a4 and company cleanup #17561 at e8febb30eb now
|
||||
have no failed or pending checks (Paperclip Review skipped). No merge was made.
|
||||
The older #17490 E2E cancellation failure remains unresolved. The entire original
|
||||
inventory stays in scope, with voice the next separate reason after receivers.
|
||||
|
||||
|
||||
Receiver discovery is published and attached as [#17572](https://github.com/paperclipai/paperclip/pull/17572)
|
||||
at 643f4766227e15006b684d2cc7bea7ddd5cc2147, based on #17557. Its live CI run is
|
||||
40173962833; security and reviews are pending. All 218 selected local cases,
|
||||
workspace types and build pass. The next audit splits voice into five separate
|
||||
reasons: live-session recovery, reply push, inbound admission expiry/provider
|
||||
cleanup, inbound history retention, and call reports. See
|
||||
[the voice owner audit](2026-10-25-voice-maintenance-sleep-audit.md).
|
||||
Branch codex/sleep-21-voice-session-recovery starts with the first reason; the
|
||||
other four remain explicit required slices. No voice migration is implemented yet.
|
||||
## Receiver review repair
|
||||
|
||||
The real company-removal producer regression reproduced two inventory reads for
|
||||
one observed removal. Discovery-initiated stops now suppress their own completion
|
||||
wake; independent token loss still wakes discovery after listener retirement.
|
||||
A full reconstruction also covers its already-pending scoped hints. Hints that
|
||||
arrive during the load remain dirty, and narrower writes cannot bypass a failed
|
||||
full reconstruction's retry deadline.
|
||||
|
||||
Both redundant-load regressions failed before repair. All 45 selected receiver,
|
||||
startup and real Discord producer tests pass after repair, including the added
|
||||
full-refresh failure/backoff case. Server typecheck and build pass. Prior broad
|
||||
consumer/workspace verification remains recorded above; these 45 overlap that
|
||||
coverage and are not an additional unique total. Publication requires fresh
|
||||
remote CI and review. The full migration inventory remains outstanding.
|
||||
|
||||
|
||||
Voice recovery checkpoint: receiver repair `1604a9fcba` is published to #17572
|
||||
and incorporated into the unpublished voice child. The complete 93-case voice
|
||||
suite, eight new recovery lifecycle tests, and 16 startup wiring cases pass.
|
||||
See the voice audit for precise coverage and remaining work. Receiver CI
|
||||
40175242445 is live with no failures at inspection; no final acceptance or merge
|
||||
is claimed. All remaining reasons, older #17490 failure, and final master
|
||||
reconciliation stay in scope.
|
||||
|
||||
|
||||
Live voice recovery is [#17576](https://github.com/paperclipai/paperclip/pull/17576),
|
||||
head 8f5e861dbd, based on #17572. All 837 selected tests, workspace typecheck and
|
||||
build pass; remote checks and review remain required. The full local test command
|
||||
was stopped after repeated embedded PostgreSQL setup failures, not counted as a
|
||||
pass. Slice 22 owns voice reply delivery and begins with the multipart/outcome
|
||||
policy and publication-settlement producer audit. See
|
||||
[reply delivery audit](2026-10-25-voice-reply-delivery-sleep-audit.md).
|
||||
|
||||
Slice 22 now implements voice reply delivery from committed changes and saved
|
||||
retry/dispatch deadlines. The periodic reply scan and pre-publication direct push
|
||||
are removed. Real normal/operator publication tests prove the final-commit wake;
|
||||
turning that notification off reproduces zero delivered replies. The branch has
|
||||
786 selected passing tests across isolated voice, recovery, startup, and shared
|
||||
consumers, plus workspace typecheck/build. See the reply audit for retained
|
||||
fixture failures and local full-suite infrastructure limitations. Publication and
|
||||
latest-head review/check acceptance remain outstanding. Inbound admission cleanup,
|
||||
inbound history retention, and call reports remain separate next slices.
|
||||
|
||||
Voice reply delivery is [#17578](https://github.com/paperclipai/paperclip/pull/17578),
|
||||
head 439d0a86eb75c129c30742a8cc24bf18e2a6ebf3, stacked on #17576. Local selected
|
||||
checks pass; remote acceptance is pending. Slice 23 starts the inbound admission
|
||||
cleanup audit in `2026-10-25-voice-inbound-cleanup-sleep-audit.md`. It explicitly
|
||||
separates retention and reports and requires proof of approval/cleanup handoff,
|
||||
termination confirmation, and protection against deleting unfinished cleanup.
|
||||
|
||||
|
||||
Voice reply review repairs are published at #17578 head `90caaf3fa9`: shared scoped
|
||||
worker, actual authority-restoration wiring regression, 191 overlapping selected
|
||||
consumer tests and workspace types/build passing. Removing the real notification
|
||||
reproduces the lost reply. Both review threads are resolved; fresh CI/review remain.
|
||||
|
||||
Inbound cleanup slice 23 now makes approval/session handoff atomic, after a real
|
||||
expiry regression reproduced termination of an already authorized call. All 110
|
||||
voice integration cases and server typechecking pass. This slice is not published:
|
||||
locked cleanup claims, legacy handoff recovery, bound-guest ownership, termination
|
||||
confirmation, retention protection and scheduler/producer migration remain required.
|
||||
The full original inventory, older #17490 E2E failure, and stack/master reconciliation
|
||||
remain outstanding. No stack PR has been merged by this work.
|
||||
|
||||
|
||||
Slice 23 inbound cleanup now uses committed admission/session changes and actual
|
||||
expiry/retry deadlines. The admission row separately records physical confirmation
|
||||
and saved provider account. Locked claims, legacy handoff repair, bound-call owner
|
||||
handoff, retention/deletion protection and safe same-account rotation are verified.
|
||||
All 205 selected tests, workspace types and build pass; publication and remote
|
||||
acceptance remain pending. History retention and call reports remain separate next
|
||||
slices. Parent #17578 at 90caaf3fa9 has no pending/failed checks and exact-head
|
||||
Greptile 5/5; the second Paperclip reviewer stopped before completion. The original
|
||||
remaining inventory, #17490 E2E failure and final reconciliation remain in scope.
|
||||
|
||||
|
||||
Inbound cleanup is [#17580](https://github.com/paperclipai/paperclip/pull/17580),
|
||||
head 4b4bf2d8af6e08d3859ada4b720e7fbd3ea27e24, stacked on #17578. It is
|
||||
mergeable; CI 40181994295 and review 40181994134 are live with no initial failure.
|
||||
The final migration applies/replays and its backfill is tested twice. Slice 24
|
||||
starts the separate inbound-history retention audit. Its deadline preserves both
|
||||
seven-day age conditions and requires confirmed cleanup plus no task/session
|
||||
binding. Call reports follow separately. No original remaining scope is removed.
|
||||
|
||||
## Slice 24: inbound voice history retention
|
||||
|
||||
Branch `codex/sleep-24-voice-inbound-retention` follows inbound cleanup #17580
|
||||
including its migration runner repair `4f0364d269`. It removes only periodic
|
||||
history pruning. The existing admission row owns the seven-day deadline and
|
||||
confirmation/binding conditions. The existing notification helper wakes retention
|
||||
on outer transaction settlement; no new table or caller wrapper is added.
|
||||
|
||||
218 selected tests pass: 131 full voice/app wiring, 24 retention/cleanup lifecycle,
|
||||
52 deletion/startup, and 11 parent recovery. Workspace typecheck/build and final
|
||||
server typecheck pass. Tests prove real signed-callback and provider-cleanup
|
||||
reactivation, pre-deadline quiet, exact deadlines, concurrent binding protection,
|
||||
restart, drain, rollback, failure and shutdown. Initial fixture failures remain
|
||||
recorded in the slice audit. Remote checks and review are pending publication.
|
||||
|
||||
Next separate owner: voice call-report reconciliation. Preserve the existing
|
||||
24-hour post-call refresh obligation, transcript ordering, task projection and
|
||||
provider diagnostics while removing scans with no eligible calls. Inspect
|
||||
completion semantics before treating a report marked available as fully settled.
|
||||
|
||||
Slice 24 is published as [#17583](https://github.com/paperclipai/paperclip/pull/17583)
|
||||
at `9c38ddfe53`, based on #17580 `4f0364d269`. It is attached to this chat.
|
||||
CI/review are running; no merge. The next branch is
|
||||
`codex/sleep-25-voice-call-reports`, with its owner audit recorded separately.
|
||||
|
||||
## Slice 25: voice call reports
|
||||
|
||||
Report refresh now owns a scoped deadline worker. Terminal transitions, signed
|
||||
callbacks, and reply call-ended responses activate existing report/session rows.
|
||||
Attempts save the next retry before provider I/O, and callback/new-attempt fences
|
||||
prevent stale settlement. The existing 24-hour refresh obligation is preserved,
|
||||
including available reports. Empty/configured-idle/live/old history is quiet.
|
||||
This removes the last voice duty from recurring chat maintenance.
|
||||
|
||||
Acceptance: 174 selected real-database/wiring cases, workspace typecheck and build
|
||||
pass. A new integration case exposed missing baseRevisionId on an existing guest
|
||||
transcript; the fix uses the document revision contract and preserves lock
|
||||
behavior. Full failure history and external-wake limits are in
|
||||
`2026-10-25-voice-call-report-sleep-audit.md`. Publication and remote checks follow.
|
||||
Parent #17583 has a Discord rollback/drain test failure to repair at its owner;
|
||||
parent #17580 has two new cleanup review findings. Neither is marked complete.
|
||||
The remaining chat duties, original ledger, #17490 E2E failure, and final
|
||||
master/stack reconciliation remain required. No stack merge is authorized.
|
||||
|
||||
Slice 25 published as [#17586](https://github.com/paperclipai/paperclip/pull/17586).
|
||||
Parent Discord test settlement and denial-latency repairs are propagated. At the
|
||||
report tip, 41 Discord tests and 138 full voice/app tests pass; unchanged report,
|
||||
cleanup and retention workers had 37 passing cases. CI/reviews remain required.
|
||||
Next owner audit splits Teams unused-consent expiry from abandoned send/upload
|
||||
attempt recovery, preserving one reason per PR. Company deletion cleanup policy
|
||||
and the earlier E2E failure remain unresolved completion duties.
|
||||
## Acceptance follow-up after the plugin parent merged (October 27)
|
||||
|
||||
Eighteen preceding sleep PRs are merged through plugin scheduling #17518 at
|
||||
@@ -1498,6 +1664,29 @@ status and test counts above are historical. The original remaining inventory
|
||||
is still in scope beyond the 65 published numbered slices; native backups and
|
||||
the remaining aggregate duties have not been migrated by this follow-up.
|
||||
|
||||
## Receiver follow-up with current Discord parent, October 27
|
||||
|
||||
Nineteen sleep PRs are merged through health #17554. Receiver discovery #17572
|
||||
now includes current Discord parent `cedf2030c4`, retaining the original 27-file
|
||||
scope. Its existing slow-failure and stop-wake repairs have guarded mutation
|
||||
proofs, 131 targeted cases pass, and both newly supported external fixtures
|
||||
self-migrate. See [the owning audit](2026-10-25-chat-receiver-discovery-sleep-audit.md#current-parent-and-review-acceptance-october-27)
|
||||
for exact evidence and the older CI runner-loss failure. Fresh CI/reviews and
|
||||
parent landing are still required. The original remaining-duty inventory is unchanged.
|
||||
|
||||
|
||||
## Voice inventory amplification acceptance, October 27
|
||||
|
||||
The voice recovery slice keeps the original single-duty scope and adds the
|
||||
normalized health-observation notification contract. Five real metadata writes
|
||||
and one health check over 205 future calls previously selected 1,230 call rows;
|
||||
they now select zero and preserve the saved deadline. Startup and changed-parent
|
||||
reconstruction use bounded pages and retain concurrent wakes.
|
||||
|
||||
The updated [voice audit](2026-10-25-voice-maintenance-sleep-audit.md#voice-session-inventory-amplification-repair-october-27)
|
||||
records all 725 distinct targeted passing cases, typecheck, build, failed attempts
|
||||
and guarded regressions. This is worker acceptance, not deployed sleep proof.
|
||||
Fresh exact-head review and CI remain required before the voice PR can merge.
|
||||
|
||||
## Discord registration landing compatibility, October 27
|
||||
|
||||
@@ -1505,3 +1694,497 @@ Company deletion has landed. The Discord command slice retains only its original
|
||||
26-file delta against the landed parent and current master at `f8563d2507`. All
|
||||
141 targeted cases, server typecheck and build pass. The [owning audit](2026-10-25-discord-command-registration-sleep-audit.md#acceptance-after-the-company-deletion-parent-landed-october-27)
|
||||
records exact scope, fixture setup and remaining remote gates.
|
||||
|
||||
|
||||
## Receiver integration with landed foundations, October 27
|
||||
|
||||
Receiver discovery keeps its 27-file scope on Discord parent `547fdf2139` and
|
||||
preserves current master queue declarations. All 131 targeted cases, server
|
||||
typecheck and build pass. See the [receiver audit](2026-10-25-chat-receiver-discovery-sleep-audit.md#landed-foundation-integration-acceptance-october-27).
|
||||
Fresh exact-head review, CI and parent landing remain required.
|
||||
|
||||
|
||||
## Voice integration with landed foundations, October 27
|
||||
|
||||
The bounded voice repair now includes receiver `6e88ca0201` and retains its
|
||||
22-file scope. All 725 targeted cases, production typecheck and server build pass
|
||||
again. The [voice audit](2026-10-25-voice-maintenance-sleep-audit.md#landed-foundation-acceptance-october-27)
|
||||
records exact scope, database setup, the initial partial tool-access run and
|
||||
remaining remote gates.
|
||||
|
||||
|
||||
## Reply delivery preserves bounded parent reconstruction, October 27
|
||||
|
||||
The reply child includes voice recovery `712f3862b5`. Its existing shared-worker
|
||||
extraction now supports the parent's paged result contract. The real inherited
|
||||
205-call regression fails before repair and passes afterward; all 445 distinct
|
||||
targeted cases, production typecheck and server build pass. The [reply audit](2026-10-25-voice-reply-delivery-sleep-audit.md#current-parent-paging-contract-acceptance-october-27)
|
||||
records the 31-file scope and effects on the four existing shared consumers.
|
||||
Current-head review, CI and parent landing remain required.
|
||||
|
||||
## Voice trusted-policy checkout repair, October 27
|
||||
|
||||
Receiver `ab39569250` supplies the landed policy checker now required by trusted
|
||||
CI. Voice server/package code and tests remain identical to the 725-case tested
|
||||
head. The policy checker and its Node tests pass. The [voice audit](2026-10-25-voice-maintenance-sleep-audit.md#trusted-policy-checkout-compatibility-october-27)
|
||||
keeps the four old failed gates, including the independent runner shutdown,
|
||||
separate from current local evidence. Fresh review and CI remain required.
|
||||
|
||||
|
||||
## Voice review follow-up, October 28
|
||||
|
||||
Voice recovery isolates a failed broad inventory retry from ready exact hints
|
||||
and cached due calls, while retaining on-use authority checks and exact-scope
|
||||
fences. The generation test uses controlled scheduler turns instead of elapsed
|
||||
SQL time. All 210 current targeted cases, production typecheck and server build
|
||||
pass; guarded mutations catch both regressions. The [voice audit](2026-10-25-voice-maintenance-sleep-audit.md#wildcard-failure-and-deterministic-generation-acceptance-october-28)
|
||||
keeps the actual CI assertion failure and earlier failed attempts separate.
|
||||
|
||||
|
||||
## Reply publication inventory acceptance, October 28
|
||||
|
||||
Reply delivery now scopes final publication wakes to the actual conversation and
|
||||
returns at most 100 identity rows per refresh. Five real publications previously
|
||||
selected 1,065/1,060 rows across unrelated calls; both paths pass the same 125-row
|
||||
budget after repair. The [reply audit](2026-10-25-voice-reply-delivery-sleep-audit.md#publication-inventory-and-paging-acceptance-october-28)
|
||||
records restart, denial-cursor and guarded-mutation coverage. All 449 current
|
||||
targeted cases, server typecheck/build and the policy check pass on parent
|
||||
`ad79c7e805`. Fresh exact-head review, CI and parent landing remain required.
|
||||
|
||||
|
||||
## Inbound cleanup current-parent acceptance, October 28
|
||||
|
||||
Inbound cleanup #17580 now includes reply parent `d2c0a7de8c` and uses its existing
|
||||
paged worker contract. Startup yields after 100 rows per scope refresh, and a
|
||||
same-owner restart reconstructs all 105 saved fixture deadlines. Migration 0395
|
||||
was generated on the parent's 0394 lineage, preserves all historical SQL/journal
|
||||
entries, and commits historical backfill in batches of at most 1,000. The
|
||||
[inbound cleanup audit](2026-10-25-voice-inbound-cleanup-sleep-audit.md#current-parent-acceptance-and-bounded-discovery-october-28)
|
||||
records 266 passing cases, negative probes, migration lineage, preserved failures,
|
||||
and limits. Exact-head review, CI, and the preceding stack still must pass before
|
||||
landing. History retention and call reports remain separate duties.
|
||||
|
||||
|
||||
## Inbound history retention current-parent acceptance, October 28
|
||||
|
||||
Retention #17583 includes cleanup parent `b8b12199b9` and preserves its generated
|
||||
0395 migration. The existing shared page contract now yields after 100 history
|
||||
rows per scope refresh and resets on same-owner restart. All 280 distinct cases,
|
||||
server types/build and module boundaries pass. The [retention audit](2026-10-25-voice-inbound-retention-sleep-audit.md#current-parent-paging-and-restart-acceptance-october-28)
|
||||
records the 13-path scope, real 105-row reconstruction, negative probe, setup and
|
||||
merge failures, and limits. Latest-head review, CI and parent landing remain
|
||||
required. Call-report reconciliation follows as a separate duty.
|
||||
|
||||
|
||||
## Retention notification review follow-up, October 28
|
||||
|
||||
The current retention review found repeated configuration-driven history scans.
|
||||
Five real connection saves selected 1,025 rows across 205 retained admissions.
|
||||
Retention now wakes only for admission writes and startup; cleanup retains all
|
||||
of its broader scopes. The same writes select zero history rows and preserve the
|
||||
saved deadline. Confirmed deletes also avoid a redundant reload. The
|
||||
[retention review ledger](2026-10-25-voice-inbound-retention-sleep-audit.md#configuration-rescan-review-repair-october-28)
|
||||
records 282 passing cases, types/build, failures, writer coverage and limits.
|
||||
Current-head review/CI and the earlier stack remain required.
|
||||
|
||||
|
||||
## Report current-parent acceptance, October 28
|
||||
|
||||
Report scheduling #17586 integrates retention `68b62b6d13` and uses the existing
|
||||
shared paged result contract. It yields after 100 headers per scope refresh,
|
||||
resets same-owner partial scans and retains the finite 24-hour report obligation.
|
||||
Officially generated migration 0396 preserves the preceding 394 SQL files and
|
||||
retains the five newest snapshots. The [report acceptance ledger](2026-10-25-voice-call-report-sleep-audit.md#current-parent-reconstruction-and-migration-acceptance-october-28)
|
||||
records 302 passing cases, types/build, migration/plan checks, guarded restart and
|
||||
fairness probes, failures and limits. Current-head review, all CI and parent
|
||||
landing still gate this slice. The full remaining stack stays in scope.
|
||||
|
||||
|
||||
## Report notification review follow-up, October 28
|
||||
|
||||
Report reconstruction also received configuration hints that could not alter
|
||||
its eligibility or saved retry. Five real connection saves over 205 saved
|
||||
reports selected 1,025 session rows. The report owner now accepts exact session
|
||||
notifications and startup; those saves select zero rows and preserve the deadline.
|
||||
The [report notification ledger](2026-10-25-voice-call-report-sleep-audit.md#configuration-scope-review-repair-october-28)
|
||||
records 303 passing cases, types/build, the narrower contract and limits. Fresh
|
||||
exact-head review and CI are required after the repair.
|
||||
|
||||
|
||||
## Slice 26: Teams unused-consent expiry
|
||||
|
||||
The Teams owner now selects existing unused-consent phases and their saved expiry
|
||||
through scoped deadline work. Production maintenance retains only abandoned
|
||||
attempts; these are the next distinct reason. No new table or Cloud alarm policy
|
||||
is included. The [audit](2026-10-25-teams-consent-expiry-sleep-audit.md) records
|
||||
222 unique selected passing cases, workspace types/build, and initial fixture
|
||||
failures. Publication and fresh remote checks/review remain required.
|
||||
Parent #17586 Actions/security/Greptile pass at 250fca6cff; Paperclip Review is
|
||||
still running. The original full inventory, older E2E failure, deletion-policy
|
||||
finding and master reconciliation remain outstanding. No stack PR is merged.
|
||||
|
||||
|
||||
Slice 26 is published as [#17590](https://github.com/paperclipai/paperclip/pull/17590)
|
||||
at 5ddb4543ed, based on #17586. It is attached; remote gates remain required.
|
||||
Slice 27 begins the separate Teams abandoned transport-attempt audit. The full
|
||||
original inventory, older findings and final master reconciliation remain open.
|
||||
|
||||
|
||||
## Slice 27: Teams abandoned transport attempts
|
||||
|
||||
Attempt recovery now follows committed claims and saved 90-second deadlines.
|
||||
It preserves unknown outcomes and performs no provider I/O. This removes the last
|
||||
Teams periodic maintenance callback; unused consent remains the separate parent
|
||||
owner. The [audit](2026-10-25-teams-attempt-recovery-sleep-audit.md) records 235
|
||||
unique selected passing cases and preserved initial test failures. Workspace
|
||||
typecheck and build pass; publication and remote acceptance remain open.
|
||||
#17590 has Greptile 5/5; CI is running. #17586 has new review findings for writer
|
||||
support and report-query indexing. Fix at that parent and propagate before
|
||||
acceptance. All original remaining work and previous unresolved findings remain.
|
||||
|
||||
|
||||
Teams attempts are published as [#17592](https://github.com/paperclipai/paperclip/pull/17592).
|
||||
Voice report review repair 94c60683b8 adds the bounded report-window index and
|
||||
precise keyset pagination, with the single-active-writer contract explicit. The
|
||||
20,001-old-row cost regression failed before repair, then all 14 report tests,
|
||||
138 full voice/wiring cases and workspace types/build passed. Both review threads
|
||||
are resolved. Consent #17590 incorporates it at e501d6c99d; both combined Teams
|
||||
children pass 30 report/startup tests after propagation. Fresh remote gates still
|
||||
apply. The next new reason is chat run-milestone projection, separate from wake
|
||||
notices, terminal confirmations, interaction continuations and GitHub recovery.
|
||||
The entire original inventory and older unresolved findings remain open.
|
||||
|
||||
|
||||
Slice 28 native finalization checkpoint: existing native finalizer transactions
|
||||
now signal milestone projection; governed review metadata restoration avoids
|
||||
unchanged writes and signals only actual changes. Both missing wake and repeated
|
||||
rewrite were reproduced before repair. All 16 milestone worker cases, nine
|
||||
selected native chat recovery integrations, and server production types pass.
|
||||
See the slice audit for failure logs and coverage limits. The production scan
|
||||
remains until status-decision/effect, admission, lineage, publication, and remaining
|
||||
producer coverage is complete. This is local progress, not a published migration
|
||||
or completion of the full stack.
|
||||
|
||||
|
||||
Slice 28 now also covers new native status-decision commits and pending-effect
|
||||
receipts, after their no-op exits. Two regressions failed first; all 18 milestone
|
||||
worker cases, nine native chat recovery integrations, and server production types
|
||||
pass after repair. Real late continuation receipts now reactivate projection.
|
||||
Next is heartbeat queue/admission/recovery producer coverage. Periodic projection
|
||||
and all remaining inventory/remote acceptance duties remain outstanding.
|
||||
|
||||
|
||||
Slice 28 queue checkpoint: issue admission, ordinary/comment claims, drain
|
||||
release, coalesced lineage and native review claims now notify from their owning
|
||||
transactions. Real missing-wake regressions were reproduced for admission,
|
||||
coalescence and all ordinary/comment claim/release paths. The 24 milestone worker
|
||||
cases and 92 existing queue/policy cases pass, as does server production typecheck.
|
||||
The audit records the initial fixture FK failure, exact logs, remaining raw
|
||||
cancellation/promotion/receipt writers, and native reviewer acceptance gap.
|
||||
Periodic projection remains until producer coverage is complete; no new PR or
|
||||
full-goal completion is claimed.
|
||||
|
||||
|
||||
Slice 28 retry/recovery checkpoint: due retry promotion, suppressed retry and
|
||||
stale-run cancellation, deferred wake promotion and immediate recovery admission
|
||||
now signal from their existing transactions. Five real missing-notice regressions
|
||||
were reproduced before repair; all 166 selected worker/module/policy cases and
|
||||
server production types pass. The slice audit preserves the initial invalid
|
||||
staleness fixture and corrected causal evidence. Periodic projection remains
|
||||
until all supported producers are covered. Remaining original inventory, open
|
||||
PR gates/findings and final master reconciliation are unchanged.
|
||||
|
||||
|
||||
Slice 28 native wake receipt checkpoint: interrupted recovery, same-scope
|
||||
coalescence and fresh admission receipt writes now notify from narrow database
|
||||
transactions. Both missed-recovery and admission-before-receipt races were
|
||||
reproduced. All 56 milestone/queue cases, 13 existing dependency cases and server
|
||||
production types pass; four native cases were rerun after fixing external-test
|
||||
fixture cleanup. The audit preserves setup failures and provider-acceptance
|
||||
limits. No periodic projector has been removed yet. Fresh remote inspection
|
||||
still finds #17590 Paperclip Review queued, #17592 neutral, and #17490 E2E shard
|
||||
3 plus aggregate failing. Those gates and all remaining original scope stay open.
|
||||
|
||||
|
||||
Slice 28 raw queue cancellation checkpoint: general chat-control recovery,
|
||||
superseded case handoffs, stale scheduled retries and reassigned execution owners
|
||||
now signal before terminal writes. Missing notices were reproduced for the first
|
||||
two; replacement-admission tests cover the latter two and their early intent
|
||||
placement is audited. All 60 milestone/queue tests and server production types
|
||||
pass. Queued-comment/lineage and remaining dependency producers still prevent
|
||||
removing periodic projection. No new PR, merge, or full-goal completion is claimed.
|
||||
|
||||
|
||||
Slice 28 lineage checkpoint: queued-comment run-context rewrites, deletion and
|
||||
cancellation signal through their existing adapter transaction. Real queue
|
||||
reconstruction and caller-owned discard regressions failed before repair. The
|
||||
native reviewer claim now has real database integration and causal signal-removal
|
||||
coverage. All 74 selected worker/adapter/policy cases and server production types
|
||||
pass. Remaining publication, interaction, configuration, question and monitor
|
||||
producer coverage still gates periodic-projector removal. Original inventory,
|
||||
remote gates/findings and final master reconciliation remain open.
|
||||
|
||||
## Slice 28 draft and publication/interaction coverage
|
||||
|
||||
[Draft #17606](https://github.com/paperclipai/paperclip/pull/17606) is stacked on
|
||||
#17592. It adds the lifecycle projector worker and covered producer signals;
|
||||
production registration and polling removal remain intentionally unfinished until
|
||||
the remaining producer audit is complete. Native progress stays a separate slice.
|
||||
|
||||
The latest local checkpoint covers real publication success/failure/operator
|
||||
resolution and question answer/cancel/withdraw/skip/terminal-expiry paths,
|
||||
including historical answers that enqueue no continuation. All 160 milestone and
|
||||
interaction-service cases pass on fresh isolated PostgreSQL databases. Four
|
||||
existing chat acceptance cases, publication policy cases, and server production
|
||||
typecheck passed at their recorded checkpoints. See the detailed milestone audit
|
||||
for retained regression/setup failures, controlled ports and remaining coverage.
|
||||
The initial draft CI was running; no final green-check or readiness claim is made.
|
||||
|
||||
|
||||
Slice 28 now also wakes lifecycle projection from confirmation accept/reject and
|
||||
live/historical comment invalidation of questions and confirmations. Six real
|
||||
producer regressions failed before the added intent signals. All 169 cases across
|
||||
the worker and interaction-service suites pass, as does production server
|
||||
typechecking. The draft remains incomplete: other producers and production
|
||||
registration/polling removal are still pending. See the milestone audit for logs.
|
||||
|
||||
|
||||
Slice 28 creation/sibling replacement now registers intent before the first
|
||||
interaction write. Both question and confirmation replacement failed before the
|
||||
fix; all 184 milestone/integration/unit interaction cases and production server
|
||||
typechecking pass. Remaining terminal-interaction writers are document-target
|
||||
expiry, deleted-addressee cancellation and startup supersession; other producer
|
||||
audits and production migration remain pending.
|
||||
|
||||
|
||||
Slice 28 covers the remaining direct terminal-interaction writers in the
|
||||
interaction service: document-target expiry, deleted addressees and startup
|
||||
supersession. Seven real-producer regressions failed before the fix; all 191
|
||||
worker/integration/unit cases and production server typechecking pass. Database
|
||||
fixture setup/cleanup moved to a cohesive helper, keeping the worker test file
|
||||
below 1,000 lines. The broader raw-writer audit still includes external services
|
||||
(secret proposals, tool reviews, conversation reset and native review retirement),
|
||||
plus the previously listed publication/configuration/recovery dependencies.
|
||||
|
||||
|
||||
Slice 28 now covers binding-proposal terminal reflection and tool-review human
|
||||
verdict/expiry reflection. Nine governed-producer cases (including outer rollback
|
||||
and dependent rejection) pass with the lifecycle/interaction suites, 200 total;
|
||||
all 109 existing tool-gateway cases and server production typechecking pass.
|
||||
The audit retains the initial invalid tool-publication fixture assertion and
|
||||
corrected before-fix failures. Native external-review supersession is already
|
||||
covered by its status-decision outer transaction. Other external interaction and
|
||||
lineage producers remain under audit; the draft is not production-wired yet.
|
||||
|
||||
|
||||
Slice 28 covers native automatic/obsolete review retirement and conversation
|
||||
reset with seven additional governed cases, including retained-history and
|
||||
explicit-request controls. Three missing wakes were reproduced before repair;
|
||||
207 lifecycle/governed/interaction cases pass. The draft CI chat-shard failure
|
||||
was reproduced as a test subscribing to unrelated unresolved transaction intent;
|
||||
isolating that test's DB owner preserves the production protocol and passes three
|
||||
native review acceptance cases. Server production types pass. The slice audit
|
||||
retains the invalid fixture and missed test-file selection. Polling removal,
|
||||
remaining producers, fresh CI/review, older findings and the full inventory remain
|
||||
unfinished; no stack merge or whole-goal completion is claimed.
|
||||
|
||||
|
||||
Slice 28 also covers interactive callback recovery of an unknown prompt receipt.
|
||||
Its existing Slack acceptance test failed before the answer-write barrier without
|
||||
the receipt signal, then five Slack/Discord/Telegram cases and server production
|
||||
types passed after the one-line integration. File transport and ordinary tool
|
||||
receipt ordering were audited without adding unnecessary notifications. Remaining
|
||||
configuration/recovery producers and production wiring still gate polling removal.
|
||||
|
||||
|
||||
## Drain review repair and fresh stack gates
|
||||
|
||||
The first broad stack refresh found #17576 Greptile finding 4370544173: dirty
|
||||
in-memory hints kept the coordinator's settled-write activity hold alive during
|
||||
drain. SQL silence alone had missed it. It failed a new real-PostgreSQL counter
|
||||
assertion before repair. The owning parent is fixed at 1b43332c74; all checks now
|
||||
pass and that review thread is resolved. Slice 22 carries the repair into its
|
||||
shared dispatcher at b2af095de2. Open/uncertain writes remain fenced centrally;
|
||||
settled hints and their next deadlines survive resume without holding idle active.
|
||||
|
||||
The fix is propagated through #17580 (0aab051900), #17583 (c46bcbf9ff), #17586
|
||||
(6d99318a8d), #17590 (0bb6de2343), #17592 (6e40ac2570) and #17606 (71a84ee0c0).
|
||||
Report and retention resume tests initially returned early because their flush
|
||||
helper checked only the active counter. Those parent-owned tests now advance the
|
||||
actual retained nextWakeAt and prove the durable result, plus unchanged idle
|
||||
snapshots while paused. See 2026-10-25-voice-drain-repair.md for all retained
|
||||
failure logs, including one invalid intermediate report-field assertion.
|
||||
|
||||
Verified selection at the combined tip:
|
||||
- 99 chat lifecycle/governed/voice recovery cases pass.
|
||||
- 24 Teams consent/abandoned-attempt scheduler cases pass.
|
||||
- 38 inbound cleanup, retention and call-report cases pass after test repairs.
|
||||
- Production server typecheck passes. The 108-case full voice integration suite
|
||||
passed at the shared-dispatch parent; strengthened reply drain passed separately.
|
||||
|
||||
Fresh current-head rollup is saved at
|
||||
/tmp/sleep-stack-checks-after-drain-20261025.json. #17490 still fails E2E shard 3
|
||||
and aggregate at 31bcd19888. #17518 Paperclip Review remains pending. #17578 and
|
||||
newer propagated heads have pending checks, with no failures in that snapshot.
|
||||
The inbound cleanup/company-deletion review and security finding is NOT repaired
|
||||
by this drain change; current pending checks do not discharge that finding.
|
||||
No blind reruns or merges into master occurred.
|
||||
|
||||
Next actionable parent audit: tool health (#17554), Discord command registration
|
||||
(#17557), and receiver discovery (#17572) still contain the same unconditional
|
||||
hasPending dirty-scope predicate. Reproduce their idle-hold behavior, repair at
|
||||
their owning branches, and propagate. Their currently passing CI is not evidence
|
||||
against that missing idle-counter assertion. Then continue milestone configuration
|
||||
and alternate-receipt coverage, production registration/poll removal, the entire
|
||||
original inventory, older E2E/cleanup findings and final master reconciliation.
|
||||
|
||||
## Configuration drain propagation and resource milestone producers
|
||||
|
||||
Discord #17557 at 954a0faa75 and receiver #17572 at 8815dfad38 now carry the
|
||||
confirmed drain-hold repairs, following tool health #17554 at e685a0e7ef.
|
||||
The prerequisite #17561 inherited health at 05ac7835fc. Propagation reached
|
||||
#17576 a4bc3ea50e, #17578 05797e2b9f, #17580 82d023a3a0, #17583 f4f7e14f7a,
|
||||
#17586 920a951505, #17590 d626ff4b13, #17592 33ffff8a12, and #17606 9775b2bc4a.
|
||||
All 109 configuration lifecycle/protocol cases pass together after applying the
|
||||
tip schema to their disposable databases. See the configuration-worker drain
|
||||
repair document for individual causal failures and parent evidence.
|
||||
|
||||
Slice 28 adds real inventory/lifecycle resource reactivation wakes. The two
|
||||
regressions fail before repair; all 90 milestone lifecycle/governed/resource cases
|
||||
and six selected provider acceptance cases pass after it. The owning resource
|
||||
module only wakes reopened issue scopes, never unchanged inventory or completed
|
||||
history. See the milestone audit for full commands/results and remaining producers.
|
||||
|
||||
Fresh gate snapshot /tmp/sleep-stack-gates-configuration-drain-20261025.json has
|
||||
no failures for slices 1–12 and 14–16, plus #17505; #17490 still fails E2E shard 3
|
||||
and its aggregate. #17518/#17554 have pending review and later updated heads have
|
||||
pending checks. New #17576 security finding 4371283330 concerns repeated
|
||||
connection writes loading every live voice session in that connection. Inspect
|
||||
and address it without dropping future deadlines or configuration revocation.
|
||||
The existing #17580 cleanup/company-deletion denial finding remains open. No
|
||||
review blocker is resolved merely by pushing inherited changes. No PRs merged.
|
||||
|
||||
Origin/master refreshed to e335265568 (task-run UI polling removal #17608).
|
||||
Full base reconciliation, remaining recurring-work slices, milestone startup
|
||||
wiring, fresh reviews, and all latest-head CI gates remain required.
|
||||
|
||||
|
||||
Slice 28 Telegram migration now registers affected issue intents in its owning
|
||||
transaction before reactivating/rebinding conversations. The missing wake was
|
||||
reproduced first. All 92 milestone/resource/governed tests, eight existing
|
||||
Telegram migration cases, server typecheck and server build pass. Duplicate
|
||||
migration and unchanged inventory remain quiet. See the milestone audit for
|
||||
failure and verification logs. Inbound activation and remaining producer audits
|
||||
still gate production wiring; no recurring projector has been removed yet.
|
||||
|
||||
Fresh pre-push gates: #17606 at b634353059 has passing checks; #17572 at
|
||||
8815dfad38 and #17490 at 31bcd19888 still fail E2E shard 3 and aggregate.
|
||||
Three local cancellation diagnostic runs passed, including full projects coverage
|
||||
and a database-only delay, without proving the CI deadlock's competing owner.
|
||||
Those failures, parent security/deletion findings, all other original slices,
|
||||
fresh remote review and reconciliation with master 7ebf98eafd remain open.
|
||||
|
||||
|
||||
Slice 28 ordinary ingress now wakes existing milestone debt before reactivating
|
||||
conversation bindings and writing inbound links. A real Slack ingress regression
|
||||
failed before repair and passes without depending on a new run being created.
|
||||
All 12 selected ingress/revocation/Telegram migration cases and server types/build
|
||||
pass. The milestone audit records evidence, isolated DB, and remaining authority/
|
||||
lineage producers. This is another producer checkpoint in draft #17606, not
|
||||
production polling removal or completion of the full requested stack.
|
||||
|
||||
|
||||
Slice 28 now signals successful inbound action settlement and repaired delivery
|
||||
acceptance for already-admitted runs. Both post-close missing notices reproduced
|
||||
before repair. All 41 selected post-close/ingress/revocation/migration cases and
|
||||
server types/build pass; recovery proves it does not admit a second wake. The
|
||||
audit retains the initial contention-sensitive assertion and its correction to
|
||||
honor the existing retry. Identity activation and remaining authority/lineage
|
||||
producers still gate production polling removal. All original slices and older
|
||||
parent CI/review findings remain required.
|
||||
|
||||
|
||||
Slice 28 identity activation now signals the bound issue before changing the
|
||||
responsible user. Two before-fix failures now pass with outer commit/rollback,
|
||||
pending steering, replay and settled quiet checks. All 43 existing identity and
|
||||
GitHub credential tests pass after repairing their external-DB fixture URL;
|
||||
server types/build pass. Recovery terminalization/classification writers are the
|
||||
next audit. Master 34409a92c4 was fetched without disturbing this draft. The
|
||||
worker is still not production-registered and all original stack duties remain.
|
||||
|
||||
|
||||
Slice 28 now covers orphan-run terminalization in stale-lock recovery. Both
|
||||
real-producer missing-notice cases failed before repair and pass afterward;
|
||||
all 40 existing stale-lock ownership/race cases and server types/build pass.
|
||||
The conditional update and provider/process authority retain their existing
|
||||
owners. Adapter-failure classification and remaining producer audits still gate
|
||||
production registration; this does not migrate the stale-lock polling duty itself.
|
||||
|
||||
|
||||
Slice 28 now covers the late adapter-failure classification write, which can
|
||||
return an old failed run to the existing publication window after the issue's
|
||||
earlier wake. The missing notice reproduced before repair; all three recovery
|
||||
milestone cases, 89 recovery-action cases and server types/build pass. Four
|
||||
classification callers reuse their issue identity, with no new queue table or
|
||||
provider policy in shared code. Authority/configuration audit and production
|
||||
wiring remain next; the original full stack and remote gates remain incomplete.
|
||||
|
||||
|
||||
Slice 28 now covers nine real authority-restoration owners: connection/resource
|
||||
reach, unlinked policy, identity confirmation, four membership paths and task
|
||||
visibility. Seven initial causal failures plus connection/resource reproductions
|
||||
now pass; the invalid Discord group-reach fixture is explicitly retained in the
|
||||
failure record. Final local selection: 50 chat integrations, 18 access tests,
|
||||
six action-worker tests, server typecheck/build. No provider authorization was
|
||||
relaxed. Cloud trusted-header membership sync and board claim remain alternate
|
||||
writers to cover before production registration/poll removal. Older parent
|
||||
findings and the complete original stack remain open.
|
||||
|
||||
|
||||
Slice 28 now covers Cloud trusted-header sync and board claim. The missing
|
||||
Cloud notice and absent board-claim intents were reproduced before repair.
|
||||
All 39 post-close cases plus 30 provisioning/auth/claim cases pass; server types
|
||||
and build pass. Unchanged Cloud profile synchronization and active board membership
|
||||
stay silent. See the milestone audit for transaction boundaries and retained logs.
|
||||
The previous pushed head 0f38d1079d has the recurring E2E shard-3 cancellation
|
||||
failure; process-identity event append deadlock evidence is retained for diagnosis.
|
||||
Production registration/poll removal and the entire remaining stack are still open.
|
||||
|
||||
|
||||
Slice 28 production milestone migration: the app now registers the scoped
|
||||
lifecycle worker and the periodic coordinator invokes only native progress.
|
||||
The final GitHub guest/sponsor settings owner signals actual authority changes
|
||||
inside its existing transaction; its missing intent reproduced before repair.
|
||||
159 worker/startup/coordinator cases, four GitHub acceptance cases and all 16
|
||||
assembled-app project-chat E2Es pass; server typecheck passes. The milestone
|
||||
audit has a consolidated alternate-writer disposition table. Native progress
|
||||
remains the next separate reason. Remote gates, parent findings and the rest of
|
||||
the original stack are not complete; local cancellation success does not repair
|
||||
the recorded CI deadlock.
|
||||
|
||||
|
||||
Slice 28 review repair: cover GitHub observed-identity linking, suppress
|
||||
unchanged native-finalization recovery wakes, and repair the voice callback and
|
||||
Cloud-auth test contracts. Restore shared user profile after the Cloud authority
|
||||
test so downstream Slack publication tests see their original identity. Local
|
||||
40 stale-lock, 122 voice, 38 auth/session and 28 combined chat cases pass; server
|
||||
typecheck/build pass. See the milestone audit for failed-run dispositions.
|
||||
Latest-head remote gates and parent failures remain open.
|
||||
|
||||
## Slice 29: native chat progress implementation
|
||||
|
||||
PR [#17639](https://github.com/paperclipai/paperclip/pull/17639), branch
|
||||
`codex/sleep-29-native-chat-progress`, parent #17606 at `ed76256b97`.
|
||||
Native event producers and existing eligibility notifications now wake a scoped
|
||||
worker. Progress leaves the chat timer; retained history becomes quiet. The
|
||||
worker retains only unfinished batches, concrete lock contention and failures.
|
||||
See [the owner audit](2026-10-25-native-chat-progress-sleep-audit.md) for producer
|
||||
coverage, cursor precision repair, authorization boundaries and retained failures.
|
||||
|
||||
Forty-four focused native-progress cases pass, as do 13 app coordinator cases and
|
||||
the server build/typecheck. All 16 browser chat cases pass before the final
|
||||
base-publication hook. The hook fixes a reproduced commit-order wake gap; its
|
||||
19 worker and 13 coordinator cases pass. A second fresh browser run passes all
|
||||
16 cases at `bcf2f38ff5`; final typecheck passes too. Remote checks/review are pending.
|
||||
Parent #17606 now has passing checks and Greptile 5/5 on `ed76256b97`; all three
|
||||
review threads are resolved. This is not completion of the broader stack.
|
||||
@@ -0,0 +1,351 @@
|
||||
# Chat receiver discovery sleep migration audit
|
||||
|
||||
Parent: Discord command registration #17557, head d3187f76a4.
|
||||
Master refreshed during this continuation: e6ec781a1a.
|
||||
Branch: codex/sleep-20-chat-runtime-discovery.
|
||||
Status: implementation and focused lifecycle verification; publication and final acceptance pending.
|
||||
|
||||
## One reason for this PR
|
||||
|
||||
Remove recurring inventory discovery for leased chat receivers from the aggregate
|
||||
chat maintenance timer. Preserve receipt projection, credential fencing, runtime
|
||||
initialization, takeover, and stop semantics. Discord command registration belongs
|
||||
to the parent; voice reconciliation, publication maintenance, milestones, Teams,
|
||||
and GitHub recovery remain separate migration reasons.
|
||||
|
||||
`server/src/app.ts` currently invokes `chatChannels.reconcileProviderRuntimes()`
|
||||
and `chatChannels.voice.reconcile()` in the same recurring maintenance lane.
|
||||
The receiver method in `services/chat-channels.ts` queries every eligible Discord
|
||||
or iMessage Photon endpoint, stops local ownership missing from that inventory,
|
||||
then calls `runtimeFor` and `reconcileDiscordRuntime` for each eligible endpoint.
|
||||
This query runs even when there are no receivers. Separating voice is necessary
|
||||
before removing this callback from the maintenance lane; removing both would lose
|
||||
an unmigrated obligation.
|
||||
|
||||
## Durable truth and owner boundaries
|
||||
|
||||
Existing `chat_endpoints`, `tool_connections`, and `chat_endpoint_leases` contain
|
||||
the necessary configuration and ownership. No additional durable queue table is
|
||||
needed. The receiver owner must retain the current eligibility policy: supported
|
||||
leased provider; verifying/active/attention endpoint; active, enabled connection;
|
||||
and exclusion of attention plus failed connection health.
|
||||
|
||||
The receiver provider policy currently lives in `leasedChatProvider` in
|
||||
`chat-channels.ts` and the lease-key selection inside `runtime/ownership.ts`.
|
||||
Centralize that existing policy with the chat provider/receiver owner before
|
||||
adding another independent Discord/Photon switch in scheduling. Shared delivery
|
||||
coordination should consume normalized receiver identities and deadlines, not
|
||||
provider protocols, credentials, or command-registration rules.
|
||||
|
||||
`runtime/ownership.ts` owns acquisition, renewal, token fencing, and stop order.
|
||||
Acquisition uses insert or expired-token CAS; background acquisition attempts once.
|
||||
Foreground configuration can wait up to twenty seconds, checking every 100 ms.
|
||||
That bounded active-operation wait is separate from idle inventory polling.
|
||||
The current default live lease lasts fifteen seconds and renews every five
|
||||
seconds while owned. These renewals are required active transport work and are
|
||||
not removed in this slice. Existing inbound admission reclaims the exact token
|
||||
after expiry before accepting buffered callbacks; preserve that safety boundary.
|
||||
|
||||
## Proposed wake contract to verify in code
|
||||
|
||||
- Reconstruct eligible receiver obligations at startup with bounded UUID pages.
|
||||
- Notify before supported endpoint/connection eligibility or credential changes
|
||||
in their existing transactions. Decode scoped hints only at the notification
|
||||
boundary; narrow normal refreshes to an endpoint or connection.
|
||||
- Keep locally owned healthy receivers in the inventory cache even when they have
|
||||
no discovery deadline. Otherwise later removal cannot identify the runtime that
|
||||
must stop. A cache of only future deadlines is insufficient for this duty.
|
||||
- A locally healthy unchanged receiver needs no periodic rediscovery query. Its
|
||||
active lease renewal and socket lifecycle remain owned by the receiver runtime.
|
||||
- On a concrete foreign lease, schedule the saved expiry as the next takeover
|
||||
attempt. Reload current authority at that time; a renewed foreign lease yields
|
||||
its new expiry. This is a configured receiver's takeover obligation, not proof
|
||||
that a standby with such an obligation can sleep indefinitely.
|
||||
- Initialization/credential/database failures retain scoped retry/backoff. One
|
||||
failed receiver must not starve independent companies or a healthy due backlog.
|
||||
- After ownership loss, finish listener fencing/stop and lease release, then wake
|
||||
discovery for that endpoint. Do not reinitialize before the old listener stops.
|
||||
Avoid a callback cycle on shutdown or a permanent retry after disable/removal.
|
||||
- Preserve a change arriving during refresh, drain, or stop. Use the existing
|
||||
coordinator's settled hints and memory-only pending predicate; no passive SQL
|
||||
during drain. Join admitted work before returning from shutdown.
|
||||
- Retain the explicit/manual reconciliation entry point used by current callers,
|
||||
or update those callers and their assertions deliberately. Do not leave a
|
||||
second automatic periodic entry point invoking the old full inventory query.
|
||||
|
||||
## Producer audit still required before implementation is accepted
|
||||
|
||||
Existing helpers are starting points, not proof of complete coverage:
|
||||
|
||||
1. `chat-work-notifications.ts:notifyChatEndpointWork` already covers shared
|
||||
connection-origin delivery/command wakes. Inspect its callers individually.
|
||||
2. `chat-channels/configuration.ts`: configure, reconnect, resume, pause, remove,
|
||||
activation and rollback, including foreground acquisition and ownership waits.
|
||||
3. `chat-channels/setup-probes.ts`, `provider-effects.ts`, resource reconciliation,
|
||||
publication failures, and inbound activation: eligibility/health transitions.
|
||||
4. Generic tool access, removal, health observations, credential/binding changes,
|
||||
membership removal, and company deletion: these can change a receiver without
|
||||
calling the endpoint configuration service.
|
||||
5. `runtime/ownership.ts`: renewal failure, token loss, failed initialization,
|
||||
runtime invalidation, lease release, and shutdown. A direct callback is suitable
|
||||
for process-local completion; do not invent a SQL row for that callback.
|
||||
6. Photon runtime callbacks and configuration: independently verify these paths;
|
||||
the Discord parent producer audit is not evidence of Photon coverage.
|
||||
7. Existing live replica/independent-writer assumptions: process-local transaction
|
||||
hints cannot discover another process's arbitrary configuration writes. Retain
|
||||
or explicitly integrate the supported cross-process wake contract rather than
|
||||
silently removing its only recovery path.
|
||||
|
||||
## Required evidence
|
||||
|
||||
Start with a failing empty-instance regression proving that the aggregate chat
|
||||
cadence repeatedly queries receiver inventory. Then verify:
|
||||
|
||||
- empty, disabled, and retained-history states have no discovery SQL after startup;
|
||||
- actual first configure/reconnect/disable/remove producers wake or cancel work;
|
||||
- healthy local ownership adds no inventory scan beyond explicit mutations;
|
||||
- foreign lease expiry, extension, and takeover preserve single ownership;
|
||||
- renewal failure triggers bounded recovery only after fencing the old runtime;
|
||||
- old buffered messages remain fenced after token or credential generation change;
|
||||
- Photon and Discord preserve their existing distinct transport behavior;
|
||||
- scoped changes, company deletion, nested transaction commit/rollback, uncertain
|
||||
commit, page boundaries, failure isolation, concurrent changes, restart,
|
||||
drain/resume, and shutdown retain obligations;
|
||||
- shared coordinator behavior without this worker remains unchanged;
|
||||
- the existing command-registration cases retain saved verification deadlines.
|
||||
|
||||
Use real PostgreSQL for lease CAS and transaction ownership, and controlled local
|
||||
runtime/provider fixtures for lifecycle effects. Existing chat integration tests
|
||||
exercise manual reconciliation, foreign ownership, credential rotations, and
|
||||
Photon restart. Inspect those cases before designing fixtures. SQL silence must
|
||||
be measured after both tracked work and scheduler settlement; a null next-wake
|
||||
value alone can occur during an admitted reload.
|
||||
|
||||
## Sleep limits and remaining stack gates
|
||||
|
||||
Active receiver sockets and lease renewals remain physical sleep blockers. Cloud
|
||||
alarm coverage and admission/sleep-safety proof do not change. This migration
|
||||
removes idle discovery activity, not the receiver's live transport obligation.
|
||||
The full original recurring-server inventory remains in scope in the stack ledger.
|
||||
|
||||
At audit creation, health #17554 has a passing security scan on 9ef6e4f4c8 but
|
||||
other current-head checks remain running. Company cleanup #17561 is e8febb30eb.
|
||||
Discord #17557 is d3187f76a4 with fresh checks required after its failure-path and
|
||||
CI fixture repairs. The older #17490 E2E cancellation failure, final reconciliation
|
||||
with master, and all fresh review/check gates remain outstanding. No stack PR was
|
||||
merged and no new merge authority is implied.
|
||||
|
||||
|
||||
Follow-up acceptance snapshot: health #17554 at 9ef6e4f4c8 now has no pending or
|
||||
failed checks, passing security, and fresh Greptile 5/5. Its three reviewed findings
|
||||
(fair batching, real producer coverage, inventory amplification) are implemented,
|
||||
verified, and their threads resolved. This is acceptance of that current head,
|
||||
not a claim that the full stack is complete or reconciled with moving master.
|
||||
Company cleanup and Discord still have live current-head checks.
|
||||
|
||||
|
||||
## Implementation and verification update
|
||||
|
||||
The aggregate chat lane now calls voice reconciliation only. Receiver discovery
|
||||
registers with the existing delivery coordinator. The receiver owner loads UUID
|
||||
pages of 100, keeps owned identities with no discovery deadline, and processes
|
||||
25 due receivers per fair company batch. Configuration hints refresh only their
|
||||
endpoint, connection, or company. A foreign owner schedules its saved lease
|
||||
expiry; a changed lease is reread before another attempt. Failures retain scoped
|
||||
30-second exponential backoff capped at five minutes. Dirty scopes suppress their
|
||||
cached deadlines until refresh succeeds; otherwise an old takeover deadline could
|
||||
cause repeated useless dispatch while the required refresh is in backoff.
|
||||
|
||||
The generic configuration notifier replaces the misleading command-only helper
|
||||
name. Existing producers now notify both relevant owners. Shared delivery code
|
||||
still receives only a queue topic and scope hint. Provider selection and lease
|
||||
keys remain in the chat receiver owner. No schema or new per-worker table exists.
|
||||
|
||||
Audited producer groups:
|
||||
|
||||
| Owner | Receiver-relevant changes and integration |
|
||||
| --- | --- |
|
||||
| Configuration and setup probes | Existing connection helper covers configure, rotation, resume, pause, removal and completed setup before transaction writes |
|
||||
| Credential persistence and removal | Configuration notifier before reference changes, including Photon |
|
||||
| Inbound activation and resource reconciliation | Endpoint/connection notifier before status and health transitions |
|
||||
| Publication, interaction and provider-effect failures | Endpoint configuration notifier before attention/failed-health writes |
|
||||
| Discord Gateway lifecycle | Receiver hints before recovery/failure transitions; metadata-only writes do not create discovery work |
|
||||
| Photon transport failure and runtime recovery | Failure callback now signals before attention/disable; existing assert-owned recovery uses the shared endpoint helper |
|
||||
| Generic tool update/removal, manual health and access revocation | Existing chat connection helper, broadened configuration dispatch, and SDK transport filter |
|
||||
| Company deletion | Company hint precedes endpoint removal in the owning deletion transaction |
|
||||
| Ownership loss / invalidation | Local completion callback after listener stop and lease release; suppressed during shutdown |
|
||||
|
||||
The additional direct-writer audit found publication-resolution, publication
|
||||
success, provider-effect success, inbound filtering, and Discord root-thread
|
||||
writes that only update timestamps. Endpoint display/reach preferences do not
|
||||
change receiver eligibility or the runtime generation. Photon configuration
|
||||
invalidates the runtime, writes allocation metadata, and proceeds through the
|
||||
credential/configuration owner; its allocation metadata is not an independent
|
||||
receiver identity. Slack/GitHub-only setup writers do not create leased receiver
|
||||
work. Capability projection in Discord reconciliation retains its existing owner.
|
||||
|
||||
A real token-loss regression found a concurrent configuration wake could acquire
|
||||
another lease while the old listener was still stopping. The ownership map now
|
||||
retains the stopping owner until listener shutdown and lease release complete.
|
||||
Acquisition joins that owner's existing stop promise. Current-authority checks
|
||||
still reject stopping owners. The test uses real timing and holds listener stop;
|
||||
an earlier fake-clock observation returned before async SQL completed and did not
|
||||
prove absence of overlap. The strengthened test failed before the ownership fix.
|
||||
|
||||
Current passing evidence: 32 receiver/startup cases, 90 selected Discord
|
||||
integration cases, all 20 Photon channel cases, and server typechecking. The
|
||||
Discord selection includes configuration, credential rotation, pause/resume,
|
||||
company deletion, token-loss stop ordering, and existing takeover/admission fences.
|
||||
The dedicated real-DB worker takeover test also passes, as do 75 command/delivery/
|
||||
standby consumer cases and full workspace typechecking. This is 218 unique selected
|
||||
passing cases. Workspace build passes. Remote review/check gates remain pending.
|
||||
The full local test matrix is not claimed: embedded PostgreSQL is unavailable
|
||||
on this host, so these database suites use validated disposable external URLs.
|
||||
Remote CI must run the full matrix before acceptance.
|
||||
|
||||
Recorded failures remain part of acceptance: initial receiver teardown omitted
|
||||
agent deletion; initial Discord pause used incomplete onboarding; failed-scope
|
||||
backoff initially selected the blocked cached takeover deadline; the real-timing
|
||||
stop race reproduced overlapping listeners. The first two were fixture defects;
|
||||
the latter two required implementation repairs. An initial backoff test expected
|
||||
zero idle holds during failed intent recovery; the correct contract retains the
|
||||
intent hold until reconciliation succeeds, so the test now checks the retry
|
||||
deadline and SQL silence rather than weakening that hold.
|
||||
|
||||
## Writer and sleep boundaries
|
||||
|
||||
Notifications cover the active database owner and its transaction-sharing clients.
|
||||
Warm standby does not run discovery; activation reconstructs its pending startup
|
||||
inventory. An already known foreign receiver remains recoverable from lease expiry,
|
||||
with real token CAS and callback fences. Arbitrary new configuration written by
|
||||
an independent process or direct SQL is not discovered by process-local hints;
|
||||
that deployment needs an explicit wake integration. This follows the existing
|
||||
single active writer contract and must not be described as multi-writer support.
|
||||
The manual reconciliation method remains available for explicit callers.
|
||||
|
||||
Active sockets and lease renewals remain required work, and no Cloud admission
|
||||
exemption is added. Voice, Teams, publication maintenance, milestones and other
|
||||
inventory reasons remain separate PRs. The full stack is not complete.
|
||||
|
||||
Master was fetched again at 0d73505098. Company prerequisite #17561 at e8febb30eb
|
||||
and Discord parent #17557 at d3187f76a4 now have no failed or pending checks;
|
||||
Paperclip Review is skipped. Fresh review-content reconciliation and final stack
|
||||
rebasing/verification remain distinct from these check results.
|
||||
|
||||
|
||||
## Receiver review repair
|
||||
|
||||
The real company-removal producer regression reproduced two inventory reads for
|
||||
one observed removal. Discovery-initiated stops now suppress their own completion
|
||||
wake; independent token loss still wakes discovery after listener retirement.
|
||||
A full reconstruction also covers its already-pending scoped hints. Hints that
|
||||
arrive during the load remain dirty, and narrower writes cannot bypass a failed
|
||||
full reconstruction's retry deadline.
|
||||
|
||||
Both redundant-load regressions failed before repair. All 45 selected receiver,
|
||||
startup and real Discord producer tests pass after repair, including the added
|
||||
full-refresh failure/backoff case. Server typecheck and build pass. Prior broad
|
||||
consumer/workspace verification remains recorded above; these 45 overlap that
|
||||
coverage and are not an additional unique total. Publication requires fresh
|
||||
remote CI and review. The full migration inventory remains outstanding.
|
||||
|
||||
## Healthy receiver progress after a slow failed batch
|
||||
|
||||
The receiver worker already stores a retry deadline for each failed receiver and
|
||||
scope. It now supplies that deadline calculation through the coordinator's
|
||||
`nextRunAtOnError` contract as well. A completed batch can take longer than the
|
||||
one-second batching interval. When that batch contained an error, the coordinator
|
||||
previously ignored an already-due healthy backlog and waited for its generic
|
||||
30-second retry. The changed registration lets healthy due entries proceed while
|
||||
the failed receiver retains its own exponential backoff.
|
||||
|
||||
The existing 105-receiver PostgreSQL regression now runs with both an immediate
|
||||
failure and a controlled 1,100 ms elapsed interval in the failed provider call.
|
||||
The latter failed before the repair: its next wake was 30 seconds away although
|
||||
healthy receivers were due. Afterward, all 105 are attempted before retrying the
|
||||
failed receiver, which is attempted once during the healthy drain. Recovery then
|
||||
settles and two former poll periods add no SQL. The assertion helper advances its
|
||||
own fake clock; the test no longer adds an extra second on every assertion retry.
|
||||
|
||||
All 20 receiver lifecycle cases and the server production typecheck pass on the
|
||||
owning branch. This changes one coordinator registration and adds no provider
|
||||
branch, queue, table, or independent timer. Current-head CI and combined-stack
|
||||
verification remain separate gates.
|
||||
|
||||
## Current-parent and review acceptance, October 27
|
||||
|
||||
The branch normally integrates Discord parent #17557 at
|
||||
`cedf2030c49c3650a41b76a4767176fd78170819` through `3e957d1204`. Only the
|
||||
original 27 receiver-owned paths differ from that parent. Health #17554 is now
|
||||
merged; company cleanup #17561 and Discord #17557 still precede this slice.
|
||||
Older stack status above describes its recorded source revision.
|
||||
|
||||
The latest old review summaries still mention the delay and redundant-stop
|
||||
findings, but the actual source already contains their repairs: stop-wake
|
||||
suppression at `1604a9fcba` and the error-deadline registration at `ab56393acc`.
|
||||
Both were revalidated on the integrated parent rather than duplicated. A guarded
|
||||
mutation removing `nextRunAtOnError` fails the 1,100 ms slow-provider next-deadline
|
||||
assertion. The expected error also remains visible in teardown after that early
|
||||
assertion. A separate mutation reenabling discovery-initiated stop notification
|
||||
fails the real company-removal producer assertion with two inventory reads
|
||||
instead of one. Both source files are restored byte for byte after the checks.
|
||||
Normal worker and real producer suites pass with the original fixes.
|
||||
|
||||
The two newly supported external-database fixtures initially skipped migrations.
|
||||
Actual worker reconstruction and Photon transport-loss cases failed on separate
|
||||
empty PostgreSQL 17 databases with missing-table errors. Both fixtures now apply
|
||||
pending migrations before client use. On those same initially empty databases,
|
||||
the complete receiver file passes all 20 cases and the complete Photon channel
|
||||
file passes all 20 cases. Existing embedded setup and safety validation remain.
|
||||
|
||||
Current acceptance: five complete receiver/coordinator/batching/startup/GitHub
|
||||
error-projection suites pass 99 cases. Twelve actual Discord automatic-command,
|
||||
receiver configure/rotation/pause/resume/removal, ownership-loss/takeover, and Slack
|
||||
held-transaction cases pass separately; 1,581 other chat integration cases are
|
||||
not counted as run. With the 20 Photon cases this is 131 distinct targeted cases.
|
||||
The large chat producer suite uses a separate migrated clone. Server production
|
||||
typecheck and build pass on the integrated source; the later fixture-only changes
|
||||
add migration setup and do not change production code. Startup listener-count
|
||||
warnings and expected controlled provider failures remain in the logs.
|
||||
|
||||
Required CI at the older `ab56393acc` head failed server shard 12. GitHub's
|
||||
check-run annotation says the self-hosted runner lost communication with the
|
||||
server; its grouped-test step is marked cancelled. The retained log ends without
|
||||
a failed assertion or final test summary. That annotation does not establish the
|
||||
underlying host, resource, or network cause, and the interrupted job is not a
|
||||
passing result. The new parent integration and fixture repairs require a fresh
|
||||
full CI run; local passes do not replace it. Fresh exact-head Greptile and
|
||||
Ponytail reviews remain mandatory. No deployed sleep or full local suite is claimed.
|
||||
|
||||
|
||||
## Landed-foundation integration acceptance, October 27
|
||||
|
||||
This slice normally integrates Discord parent `547fdf2139`, which contains
|
||||
master `f8563d2507` and landed company-deletion fixes. The original 27-file
|
||||
scope is unchanged. All owned source matches its precise original patch applied
|
||||
to that parent. The shared queue catalog also preserves master's task-drain and
|
||||
native-checkpoint declarations. An initial whole-file identity assertion rejected
|
||||
those expected parent additions; the corrected proof checks the actual owned
|
||||
patch against the integrated parent. No failed verification is counted as a pass.
|
||||
|
||||
The complete five worker/shared-consumer suites pass 99 cases. Real Discord and
|
||||
Slack producers pass 12 cases; the complete Photon suite passes 20. This is 131
|
||||
distinct targeted cases, with 1,581 chat cases deselected. Receiver and Photon
|
||||
self-migrate separate initially empty databases. The large chat fixture uses a
|
||||
separate migrated clone. Server typecheck and build pass. Only documentation
|
||||
changes follow the passing source. Expected controlled provider failures and
|
||||
startup signal-listener warnings remain recorded. Fresh exact-head review and
|
||||
all CI must pass before this child lands after Discord. No deployed sleep or
|
||||
full local suite result is inferred from these checks.
|
||||
|
||||
|
||||
## Compatibility after Discord squash merge, October 27
|
||||
|
||||
Discord #17557 landed at `5bf0142d45`. The normal merge of that master caused
|
||||
ancestry conflicts in overlapping producer paths. Resolution applies only the
|
||||
recorded 27-file receiver patch from Discord parent `547fdf2139` to actual landed
|
||||
master. Every server and package production/test file is byte-identical to the
|
||||
published `6e88ca0201` source that passes 131 cases, typecheck and build. No source
|
||||
verification is rerun merely for unrelated upstream UI/eval/documentation changes.
|
||||
The PR-owned delta passes its whitespace check. This integration records that
|
||||
proof and requires fresh exact-head review and CI before merging into master.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -43,3 +43,19 @@ followed by registration on resume without another producer. Production server
|
||||
typecheck passes in /tmp/sleep19-drain-typecheck.log. These tests use the migrated
|
||||
paperclip_sleep19_drain_20261025_test database. Remote CI/review must run on the
|
||||
new head; this is not evidence of provider suspension.
|
||||
|
||||
## Receiver discovery — slice 20 / #17572
|
||||
|
||||
The strengthened first-receiver drain case fails with active=1 after commit in
|
||||
/tmp/sleep20-drain-before.log. Apply the same owner-level pending predicate while
|
||||
retaining all dirty scopes and lease/retry deadlines. All 19 receiver lifecycle
|
||||
cases pass in /tmp/sleep20-drain-tests.log, including no SQL and an unchanged idle
|
||||
snapshot for a minute after settlement, then receiver acquisition on resume.
|
||||
Production server typecheck passes in /tmp/sleep20-drain-typecheck.log. The fresh
|
||||
migrated database is paperclip_sleep20_drain_20261025_test.
|
||||
|
||||
The three configuration owners now have measured drain activity as well as traffic
|
||||
checks. Their retained timers remain memory-only while paused; they do not provide
|
||||
external wake coverage. Fresh remote CI/review and downstream propagation remain
|
||||
required. Origin/master was refreshed to e335265568; the latest change removes
|
||||
idle task-run UI polling and does not replace these worker repairs.
|
||||
@@ -0,0 +1,225 @@
|
||||
# Native chat progress: sleep migration audit
|
||||
|
||||
Parent: #17606 at `ed76256b979541b962fa784016a550c32920d670`.
|
||||
Branch: `codex/sleep-29-native-chat-progress`. PR: #17639.
|
||||
Status: implementation, focused PostgreSQL acceptance, and browser acceptance
|
||||
complete; exact-head remote checks and review pending. This is not a deployed sleep claim.
|
||||
|
||||
## Duty and owner
|
||||
|
||||
Native progress used the one-second chat reconciliation cadence. It projects a
|
||||
closed allowlist of durable run event types into coarse provider-safe phrases.
|
||||
It never copies event messages, payloads, tool names, arguments, or results into
|
||||
external publications.
|
||||
|
||||
`chat-native-progress.ts` owns selection, authorization and insertion.
|
||||
`safe-native-chat-progress.ts` owns the normalized allowlist and phrase policy.
|
||||
`chat-run-publication-policy.ts` holds the existing projection window and shared
|
||||
automatic-publication eligibility. It preserves upgraded GitHub bots' tool-owned
|
||||
replies and legacy automatic replies. The original export and explicit combined
|
||||
lifecycle/progress API stay compatible, including their shared insertion budget.
|
||||
Startup registers the worker; it no longer invokes progress on the chat timer.
|
||||
Other chat maintenance duties remain for their own migration slices.
|
||||
|
||||
## Durable contract and triggers
|
||||
|
||||
The candidate requires a native running run, matching company/agent and bound
|
||||
issue, an active/waiting conversation, and an automatic endpoint assigned to the
|
||||
agent. Its existing base working publication must be pending, retry, streaming,
|
||||
or published. The latest allowlisted event must be within the existing 24-hour
|
||||
window and at least 20 seconds after the base publication. Existing lineage,
|
||||
interaction arbitration, final-output suppression and delivery authorization stay
|
||||
with their current owners. The ordinary direct-origin path and stricter
|
||||
post-close lineage path retain their existing different authorization stages.
|
||||
|
||||
There is at most one publication per phase. Spacing compares saved event time
|
||||
with saved publication times. Waiting twenty seconds does not make an old event
|
||||
eligible. No timer retries cadence-suppressed events. Existing events and
|
||||
publications are durable truth; there is no new queue table or schema migration.
|
||||
|
||||
The two event writers register awaited intent before insertion, after duplicate
|
||||
and ownership checks, inside their existing transaction boundaries:
|
||||
|
||||
- `appendHeartbeatRunEvent` covers native ingress through
|
||||
`PaperclipControlPlanePort`, direct allowlisted events and recovered trace diffs.
|
||||
It reads only runtime mode and bound issue alongside the existing binding
|
||||
fields. Ordinary non-progress direct log events retain their atomic-statement
|
||||
path. Direct allowlisted event writes now use a transaction even for legacy
|
||||
runs; only native bound runs signal progress.
|
||||
- `NativeRunCoordinatorStore.appendEvent` has its own insertion path. It signals
|
||||
after binding, sequence and duplicate validation, before persistence.
|
||||
- `persistReprojectedWorkspaceDiffs` delegates to the first writer and is tested
|
||||
through its actual recovered-trace persistence API, including duplicate replay.
|
||||
|
||||
Eligibility mutations share slice 28's issue/company notification boundary.
|
||||
`notifyChatRunMilestones` now invalidates both lifecycle and native progress,
|
||||
without adding recovery wrappers to every caller. Both topics share the outer
|
||||
transaction's identity and commit/rollback outcome. Existing writers cover
|
||||
lineage receipts, run state, interactions, endpoint/resource configuration,
|
||||
identity and membership. Failed-publication replay and operator retry now supply
|
||||
the affected issue to `notifyChatPublicationWork`: they can restore an existing
|
||||
base working publication without any new run event. The lifecycle projector also signals native progress before creating a first
|
||||
working publication. A long transaction can give that publication an older
|
||||
transaction-start timestamp but commit after an event was already inspected.
|
||||
Routine transport claim bookkeeping retains its narrower notification behavior.
|
||||
|
||||
## Scheduling, contention and lifetime
|
||||
|
||||
`chat-native-progress-work.ts` uses `registerScopedDeadlineWork` and the shared
|
||||
delivery coordinator. It reconstructs relevant issue scopes on startup, refreshes
|
||||
only the affected issue/company after a committed change, and projects batches
|
||||
of 100. A full insertion batch retains immediate work. Lock contention retains a
|
||||
one-second retry; failure uses the shared backoff. A settled short pass disarms.
|
||||
The shared worker preserves changes during execution, paused work during drain,
|
||||
startup failures through resume, fair company dispatch and shutdown ownership.
|
||||
|
||||
Projection retains issue-before-run SKIP LOCKED ordering. A skipped lock is a
|
||||
concrete obligation only while the matching row still exists and the run is
|
||||
native/running. Deleted or terminal candidates settle. Authority, latest event,
|
||||
phase deduplication and final output are checked again inside projection.
|
||||
|
||||
The selector also repairs an existing pagination bug: JavaScript Date truncated
|
||||
PostgreSQL microseconds, repeatedly selecting a full rejected page. Cursors now
|
||||
retain PostgreSQL timestamp text and the full timestamp/run/conversation tuple.
|
||||
The regression seeds 30 origin-rejected candidates at one fractional timestamp
|
||||
and an eligible later event. A query-count guard bounds failure of the old code.
|
||||
|
||||
## Evidence and failures retained
|
||||
|
||||
- `/tmp/sleep29-progress-cursor-before.log`: causal pagination failure.
|
||||
- `/tmp/sleep29-progress-cursor-acceptance.log`: 17 selected cases after repair.
|
||||
- `/tmp/sleep29-progress-owner-acceptance.log`: 21 projection cases after extraction.
|
||||
- `/tmp/sleep29-native-producers-before.log`: real event-writer cases fail before
|
||||
notification hooks; startup and lock cases already pass.
|
||||
- `/tmp/sleep29-native-producers-fixed.log`: all five original producer cases pass.
|
||||
- `/tmp/sleep29-native-lifecycle.log`: 14 worker lifecycle cases pass.
|
||||
- `/tmp/sleep29-progress-recovery.log`: trace recovery and 105-run backlog pass.
|
||||
Its attempted publication-mode test was invalid: endpoint update does not
|
||||
accept that field. No production API was expanded to satisfy the test.
|
||||
- `/tmp/sleep29-progress-eligibility.log`: the first identity test incorrectly
|
||||
assumed ordinary direct-origin projection enforced post-close authority.
|
||||
The replacement uses the real closed/reopened conversation and admission
|
||||
receipts, where the existing projection contract requires current authority.
|
||||
- `/tmp/sleep29-progress-eligibility-fixed.log`: five real-service cases pass:
|
||||
post-close identity, resource and visibility restoration, plus failed publication
|
||||
replay and operator retry. No new event or manual projection call is used.
|
||||
- `/tmp/sleep29-progress-worker-acceptance.log`: 13 app coordinator tests pass;
|
||||
the concurrent-event test's quiet assertion initially counted old calls from
|
||||
an already-installed transaction spy. Restoring that barrier spy before taking
|
||||
the quiet baseline corrects the test accounting, without a production change.
|
||||
- `/tmp/sleep29-final-native-acceptance.log`: 44 selected tests pass across the
|
||||
18 worker cases and 26 existing/new integration cases. These cover both real
|
||||
event owners, nested outer commit/rollback, duplicate replay, irrelevant logs,
|
||||
issue/run lock contention and terminalization, startup reconstruction, drain,
|
||||
failed startup through resume, shutdown, trace recovery, issue/company scope,
|
||||
concurrent commits, full batches, cadence, final suppression and provider-safe
|
||||
content. The name filter skips the app file; its 13 cases passed separately.
|
||||
- `/tmp/sleep29-wired-typecheck.log`: production server typecheck passes.
|
||||
- `/tmp/sleep29-server-build.log`: server build passes (production TypeScript;
|
||||
test files are executed by Vitest, not included in the server typecheck).
|
||||
- `/tmp/sleep29-first-base-before.log`: causal missing-wake failure when a
|
||||
first working publication commits after event selection. The test backdates
|
||||
the base inside its transaction to model transaction-start timestamp ordering.
|
||||
- `/tmp/sleep29-base-fixed-workers.log`: all 19 worker and 13 app coordinator
|
||||
cases pass after the lifecycle producer signals native progress.
|
||||
- Browser acceptance: all 16 `agent-chat-projects.spec.ts` cases pass, log
|
||||
`/tmp/sleep29-app-e2e.log`, fresh isolated database, port 32659. This run
|
||||
precedes the final base-publication hook. `/tmp/sleep29-final-e2e.log` then
|
||||
passes all 16 cases at committed implementation `bcf2f38ff5` with a second
|
||||
fresh isolated database. `/tmp/sleep29-final-typecheck.log` also passes with
|
||||
the final hook. No retries were needed in either browser run.
|
||||
|
||||
PostgreSQL is external and isolated because embedded PostgreSQL is unavailable
|
||||
on this host. Worker fixtures clean their own companies. The reused chat fixture
|
||||
DB retains history; tests stop workers and retire endpoints after assertions.
|
||||
Configured-idle and retained-history quiet assertions observe select/transaction
|
||||
calls, including two minutes of controlled time in worker tests. Provider APIs
|
||||
are controlled fixtures, not live Slack/Telegram/Discord accounts.
|
||||
|
||||
## Limits and remaining acceptance
|
||||
|
||||
Process-local notifications retain the single active writer model. Independent
|
||||
SQL writers and other processes need a wake integration. No Cloud sleep gate or
|
||||
external alarm coverage changes here. An internal deadline cannot wake stopped
|
||||
compute. Exact-head CI and review remain required before this slice is ready. The whole stack remains unmerged.
|
||||
|
||||
Fetched master is `19ec787e5b` (not replayed into this stack yet). Newly landed
|
||||
#17616 expands native-harness backup expiration to continuity/quarantine evidence
|
||||
and adds its evidence signal. Its interval remains. Preserve that newer contract
|
||||
when migrating its separate retention owner.
|
||||
|
||||
## CI repair: narrow notification payload (2026-10-25)
|
||||
|
||||
CI at `34df8b0d01` failed native status/finalization consumers across server,
|
||||
chat, Runner, and serialized suites. Local reproduction of the existing
|
||||
same-run Slack response-recovery test failed before the repair. Its persisted
|
||||
`native_run_finalizations.failure_detail.message` identified circular JSON
|
||||
serialization of a Drizzle table through the caller's database handle.
|
||||
|
||||
The new native-progress notifier serialized the entire structurally compatible
|
||||
scope argument. Existing lifecycle callers legitimately pass wider domain inputs
|
||||
(including `db`) through that typed boundary. Serialize only `companyId` and
|
||||
`issueId`, as the lifecycle notifier already does; do not constrain every caller
|
||||
or serialize domain/private fields into work hints.
|
||||
|
||||
After repair, 15 existing chat finalization/response-recovery/authority cases
|
||||
passed against external PostgreSQL, all 32 native-progress worker/startup cases
|
||||
passed, and server typecheck passed. The attempted broader native-finalization
|
||||
and session-resumption selection failed during local embedded PostgreSQL
|
||||
initialization (bootstrap exit 1), before tests could run; it is not acceptance
|
||||
evidence. Fresh CI must verify those native consumers. Before/after logs:
|
||||
`/tmp/sleep29-finalization-repro.log`, `/tmp/sleep29-finalization-fixed.log`,
|
||||
`/tmp/sleep29-notification-fix-workers.log`,
|
||||
`/tmp/sleep29-notification-fix-typecheck.log`, and
|
||||
`/tmp/sleep29-native-finalization-acceptance.log`.
|
||||
|
||||
## CI repair: keep non-progress event admission narrow (2026-10-25)
|
||||
|
||||
CI at `3447973f6f` clears the prior finalization failures. Its sole failing test
|
||||
file is `heartbeat-run-event-atomic-append.test.ts`: native lifecycle and retry
|
||||
exhaustion events unnecessarily select runtime mode and the task-binding SQL,
|
||||
which can read a large context snapshot. Those events do not project progress.
|
||||
The lock query now selects the extra binding only for allowlisted progress
|
||||
kinds. Keep the existing strict SQL/payload regression assertions unchanged.
|
||||
|
||||
The atomic-event suite now supports the repository's guarded external test DB
|
||||
convention, allowing its actual PostgreSQL SQL assertions to run locally despite
|
||||
embedded bootstrap failure. The original implementation reproduces both CI
|
||||
failures in `/tmp/sleep29-atomic-events-before.log`. That first external run also
|
||||
exposed fixture cleanup ordering; cleanup now removes event/run/agent rows before
|
||||
companies and always closes its pools. The repaired atomic-event and progress
|
||||
worker suites pass all 25 cases in `/tmp/sleep29-atomic-progress-fixed.log`;
|
||||
server typecheck passes in `/tmp/sleep29-atomic-progress-typecheck.log`.
|
||||
|
||||
Greptile reviewed `3447973f6f` at 5/5 with no threads. A new full CI/review pass
|
||||
is required for this second repair; do not carry forward readiness from its
|
||||
parent head.
|
||||
|
||||
|
||||
## Current milestone-parent acceptance (October 28)
|
||||
|
||||
The existing native-progress slice is prepared with a normal merge of the verified
|
||||
local milestone parent `eef52b9dd759417bd7a2fea835357fbfeb460111`. Its previous
|
||||
published head is `08d07ead105dc89c2af9b62f4d90bff8cc93540c`, with original own
|
||||
base `7991c2fa2ce7298bee69b25dbc6cb60a5e2fe250`. All unrelated parent changes,
|
||||
including the current migration 0397 and authority, voice, and identity repairs,
|
||||
remain present. The only explicit-base conflict was publication resolution: keep
|
||||
this slice's issue-scoped retry notification and the parent's exact voice
|
||||
conversation scope. No new duty, table, dependency, or scheduler is introduced.
|
||||
|
||||
Fresh separate PostgreSQL fixtures pass **163 distinct selected cases**:
|
||||
**137 across eight complete suites** (native progress, atomic event append,
|
||||
all five milestone worker suites, and app admission wiring), plus **26 real chat
|
||||
integration cases** selected by `native progress`. These include replay/operator
|
||||
retry, microsecond pagination, lock contention, current authority, cadence, phase
|
||||
coalescing, final/question precedence, and redaction. The other **1,595 chat cases
|
||||
were skipped**, not passed. Server production typechecking, build, and module
|
||||
boundaries also pass; test files were executed rather than statically checked.
|
||||
|
||||
Commands, separate fixture names, output, and zero exit-status receipts are in
|
||||
`/tmp/sleep29-current-parent-acceptance-command.json` and
|
||||
`/tmp/sleep29-current-parent-{acceptance,producers,types,build,boundaries}.log`.
|
||||
Earlier verification and failures above keep their stated revision scope. This
|
||||
is local preparation only: final landed-parent reconciliation, all fresh exact-
|
||||
head remote gates, and Greptile 5/5 remain required before merge. This does not
|
||||
qualify a live provider, external alarm, whole-instance suspension, or savings.
|
||||
@@ -0,0 +1,154 @@
|
||||
# Teams abandoned transport-attempt recovery sleep audit
|
||||
|
||||
Baseline: consent expiry PR #17590 at 5ddb4543ed.
|
||||
Branch: codex/sleep-27-teams-attempt-recovery.
|
||||
Status: implemented; targeted local acceptance and workspace types/build pass; publication and remote gates pending.
|
||||
|
||||
## Reason and source of truth
|
||||
|
||||
This slice owns only abandoned Teams transport claims. Existing transfer phases
|
||||
consent_sending, uploading, file_info_sending and conflict with a non-null
|
||||
attemptExpiresAt carry the obligation. claim() saves attemptId and a 90-second
|
||||
attempt expiry before provider I/O. Normal settlement clears the attempt. A
|
||||
conflicting callback can retain an in-flight attempt until it expires.
|
||||
|
||||
Recovery rechecks the row under the publication/transfer lock order, clears the
|
||||
expired claim, and moves a sending phase to its matching unknown outcome. Conflict
|
||||
stays conflict. It must never resend, manufacture a receipt or treat time as proof
|
||||
that provider I/O did not occur. Unknown outcome rows without a deadline are
|
||||
passive review evidence. Unused consent remains owned by #17590.
|
||||
|
||||
## Producers and boundary
|
||||
|
||||
teamsFileTransferService central update() owns claims, settlement, cancellation,
|
||||
conflict and operator resolution. Add an awaited attempt signal there, within the
|
||||
existing transaction. Insert creates consent_pending, not an attempt. Caller-owned
|
||||
resolution must retain outer transaction settlement. Direct SQL/other processes
|
||||
remain outside the local signal contract. Startup reconstructs saved attempts.
|
||||
|
||||
Use the existing scoped deadline helper in the Teams folder. Expose only a narrow
|
||||
per-transfer recovery method and worker registration. Shared app wiring supplies
|
||||
admission gates; provider policy stays in Teams. Remove scheduleTeamsFileMaintenance
|
||||
and its call from processPublicationMaintenance after both duties have owners.
|
||||
Keep the explicit combined diagnostic recovery method compatible.
|
||||
|
||||
## Verification required
|
||||
|
||||
Prove real claim activation and settlement, three sending phases and conflict,
|
||||
quiet unused/unknown/completed history, saved deadline across restart, no early
|
||||
SQL, precision, projection rollback, lock contention and fair backlog, first work
|
||||
during drain, nested rollback/outer commit, changes before deadline, and shutdown.
|
||||
Exercise production wiring and ensure no periodic Teams scan remains. Keep
|
||||
provider calls outside transactions and prove recovery makes no new provider call.
|
||||
No change to external alarms, on-use authority or Cloud sleep eligibility.
|
||||
|
||||
## Implementation and local evidence
|
||||
|
||||
The new Teams-owned worker registers existing attempt deadlines. The central
|
||||
versioned update signals both consent and attempt owners before writes. A narrow
|
||||
recoverAbandonedAttempt method delegates to the original locked recovery logic.
|
||||
The periodic scheduleTeamsFileMaintenance callback and its orchestration call are
|
||||
removed. No other chat duty, schema or external wake policy is changed.
|
||||
|
||||
- Five real-PostgreSQL/wiring suites pass 220 cases: 77 transfer protocol cases
|
||||
and 143 publication, consent, consent-runtime and app cases.
|
||||
- Fifteen selected production integration cases pass, including actual Board
|
||||
claim/receipt activation and the parent expiry case plus existing Teams current
|
||||
authority cases. The other 1,411 cases are skipped, not passed.
|
||||
- These are 235 unique selected passing cases. The production claim test measures
|
||||
120 seconds with no select/transaction calls after the receipt settles. The
|
||||
102-entry recovery case drains healthy work from two companies while corrupt
|
||||
evidence retains recovery. No recovery test causes another provider request.
|
||||
- Server typecheck passes; workspace typecheck passes. Workspace build passes. Typecheck excludes test files.
|
||||
- Full local test:run remains unclaimed due to the earlier embedded-PostgreSQL
|
||||
host limitation. Full remote gates are required. No live Teams/suspension claim.
|
||||
|
||||
Failure ledger: the first five-suite run passed 217 and failed two new assertions.
|
||||
The drain fixture simulated an interrupted receipt transaction, then incorrectly
|
||||
counted its pg_xact_status reconciliation as forbidden idle SQL. The rollback
|
||||
fixture incorrectly rejected any future wake, including the legitimate shared
|
||||
transaction-resolution deadline. It now reads committed eligibility and verifies
|
||||
that no provider call occurred. Its asynchronous rejection is joined in finally.
|
||||
The next run passed 219 and failed the drain fixture's wait for all idle holds to
|
||||
clear while dispatch was deliberately paused. The final drain fixture uses a real
|
||||
committed conflicting callback, retains its live claim through a quiet paused
|
||||
window, and verifies retirement after resume. The interrupted receipt and failed
|
||||
projection cases remain independently covered. Production behavior was not
|
||||
relaxed to satisfy these assertions.
|
||||
|
||||
Logs: /tmp/sleep27-teams-tests.log (initial failures),
|
||||
/tmp/sleep27-teams-fixed-tests.log (drain hold assertion),
|
||||
/tmp/sleep27-teams-final-tests.log (220 pass),
|
||||
/tmp/sleep27-production-tests.log (15 pass),
|
||||
/tmp/sleep27-server-types.log, /tmp/sleep27-workspace-types.log and
|
||||
/tmp/sleep27-workspace-build.log (pass).
|
||||
|
||||
## Stack gates
|
||||
|
||||
Consent parent #17590 at 5ddb4543ed has Greptile 5/5 with no inline findings;
|
||||
CI is still running. Voice-report #17586 now has two new Paperclip findings:
|
||||
explicit single-active-process writer support and indexed report reconstruction.
|
||||
They require owning-parent review/repair and propagation before stack acceptance.
|
||||
Older #17490 E2E and #17580 deletion-policy findings and final master reconciliation
|
||||
remain open. No stack PR is merged or the full objective complete.
|
||||
|
||||
|
||||
Publication: [#17592](https://github.com/paperclipai/paperclip/pull/17592), initial
|
||||
head 6f2f3951ca. Parent voice-report findings are repaired at 94c60683b8 and both
|
||||
review threads resolved. The partial index and precise pagination are propagated
|
||||
through consent parent e501d6c99d. Thirty report-worker/startup cases pass on each
|
||||
combined Teams child; logs /tmp/sleep26-parent-review-tests.log and
|
||||
/tmp/sleep27-parent-review-tests.log. Snapshot retention remains five. Fresh
|
||||
remote gates on the propagated heads remain required; older stack findings remain.
|
||||
|
||||
|
||||
## Current consent-parent preparation and acceptance — October 28
|
||||
|
||||
This existing attempt-recovery slice is prepared on consent candidate
|
||||
`9e84cfab1f795279c8cec413a946631f1d781bd2`. Its complete 13-path patch is unchanged
|
||||
while inheriting the parent's bounded consent reconstruction and retention
|
||||
acceptance repair. The normal merge keeps the complete parent stack history
|
||||
and the attempt appendix. The initial explicit-base composition was recorded
|
||||
at `163a81d1`; the parent follow-up is included at `642095e2`. This preparation may be published on that exact pending parent to run CI in
|
||||
parallel. Merge still requires the consent PR to have landed, an ordinary clean
|
||||
Git composition onto actual master, and proof that only the complete owned
|
||||
attempt patch is added while all master changes survive. A normal ancestry
|
||||
merge remains necessary if that composition is not clean. No additional worker
|
||||
refactor or new background duty is introduced.
|
||||
|
||||
Current local acceptance passes 239 distinct selected cases: 221 across the
|
||||
complete transfer/publication/consent/consent-runtime/app-wiring suites, 16
|
||||
selected production integration cases, and two delegated Slack policy cases.
|
||||
The production cases include the real Board attempt claim and confirmed receipt,
|
||||
retired recovery deadline and quiet SQL, consent expiry and current authority,
|
||||
and the existing timed-out GitHub metadata/retry race. The other 1,579 chat
|
||||
integration cases were skipped. The Slack probe covers linked and invitation
|
||||
recipients; its other 81 cases were skipped. Each database-owning suite uses a
|
||||
new isolated PostgreSQL database. Server production typecheck, build and module
|
||||
boundaries pass. Test files were executed; production typechecking does not
|
||||
statically check them. The unchanged signoff E2E test was not run locally in this
|
||||
acceptance; fresh remote E2E gates remain required.
|
||||
|
||||
The old CI run `40557802358`, server shard 10/12 job `122390238707`, failed the
|
||||
delegated Slack invitation target-rate assertion: the second call was admitted
|
||||
at a real minute boundary. That shard had 2,122 passing tests, one failed and 15
|
||||
skipped. The actual prepared parent already contains the Date-only rate-window
|
||||
fixture fix, so it is preserved rather than duplicated in this PR. Both linked
|
||||
and invitation policy cases now pass through their full caller/target/shared
|
||||
rate scenarios on a fresh server database. No production policy, timeout, test
|
||||
expectation or new owned file was changed for this inherited failure; current
|
||||
remote CI must pass independently. The old failure is not erased by the probe.
|
||||
|
||||
Evidence: `/tmp/sleep27-old-server10-job.log`,
|
||||
`/tmp/sleep27-current-parent-protocols.log` (221 pass),
|
||||
`/tmp/sleep27-current-parent-producers.log` (16 pass),
|
||||
`/tmp/sleep27-inherited-rate-window-probe.log` (two pass),
|
||||
`/tmp/sleep27-current-parent-types.log`,
|
||||
`/tmp/sleep27-current-parent-build.log`, and
|
||||
`/tmp/sleep27-current-parent-boundaries.log`.
|
||||
Earlier 235-case or combined-stack evidence belongs to its stated revisions and
|
||||
is not added to 239. Latest-head Greptile 5/5 and required CI still gate merge;
|
||||
missing or paused Ponytail turns without a verdict follow the user's existing
|
||||
merge authorization, and actual failing findings must be fixed. No full local
|
||||
repository suite, live Teams/Slack tenant, external wake, physical suspension or
|
||||
cost-saving claim is made.
|
||||
@@ -0,0 +1,219 @@
|
||||
# Teams unused-consent expiry sleep audit
|
||||
|
||||
Baseline: voice call-report PR #17586 at 250fca6cff.
|
||||
Branch: codex/sleep-26-teams-consent-expiry.
|
||||
Status: implemented and locally verified; publication and remote gates pending.
|
||||
|
||||
## Separate reasons
|
||||
|
||||
The Teams maintenance callback in microsoft-teams/file-delivery.ts runs from
|
||||
processPublicationMaintenance and repeats a database selection every 30 seconds.
|
||||
It combines two obligations. This slice owns only unused consent expiry:
|
||||
consent_pending, awaiting_consent and upload_pending rows become expired at their
|
||||
existing expiresAt. Abandoned consent_sending/uploading/file_info_sending attempts,
|
||||
and conflict rows with a retained attempt expiry, belong to the next slice.
|
||||
Unknown outcomes without a live deadline are passive operator-review evidence;
|
||||
expiry must not convert them into retries or manufacture delivery receipts.
|
||||
|
||||
## Owner and persistence
|
||||
|
||||
chatTeamsFileTransfers already owns the phase, expiry, company, endpoint,
|
||||
publication, version, authority digest and encrypted private state. No schema or
|
||||
new queue table is needed. teamsFileTransferService is the only production insert
|
||||
and update owner found by the initial writer search. Its issue() insertion and
|
||||
central update() helper are the two notification boundaries. Caller-owned
|
||||
resolveInTransaction uses that helper; its signal must wait for outer commit.
|
||||
Deletion should simply remove a cached entry on the next owned deadline; it must
|
||||
not retain a perpetual scan.
|
||||
|
||||
The existing lock order is publication -> transfer -> projection policy/source
|
||||
locks. Expiry rechecks eligibility after those locks and performs no provider
|
||||
I/O. Integrity checks and projection in the same transaction must remain intact.
|
||||
Maintenance lock waits are bounded to 250ms, with no-wait publication/transfer
|
||||
locking. A bad or locked row must not starve other companies or later rows.
|
||||
Never log private envelopes, SQL parameter errors, provider capabilities or tokens.
|
||||
|
||||
## Intended migration
|
||||
|
||||
- Reuse scoped deadline work, with paged startup reconstruction and transfer-ID
|
||||
hints from committed writes. Register the worker through the Teams owner;
|
||||
app wiring only supplies shared lifecycle and admission gates.
|
||||
- Load only the three unused phases, with ceil(expiresAt milliseconds) so a
|
||||
PostgreSQL submillisecond expiry cannot be executed early or spin.
|
||||
- Recheck and expire one transfer through its owning protocol method, preserving
|
||||
atomic publication projection and no-provider-I/O semantics. Result loading
|
||||
removes settled/changed rows; failed reconstruction or expiry retains retry.
|
||||
- Keep explicit manual expireAndRecover compatibility for tests/diagnostics.
|
||||
Production aggregate maintenance must select and execute only abandoned
|
||||
attempts after this slice. It must not keep expiring unused consent as a hidden
|
||||
second owner. That remaining attempt scan is intentionally still pending.
|
||||
- Preserve configured idle, completed history, unknown outcome and active-attempt
|
||||
quiet for this worker. Real outstanding consent remains a future obligation.
|
||||
- Do not change authority-on-use checks, transport execution, Cloud wake coverage
|
||||
or whole-instance sleep eligibility.
|
||||
|
||||
## Verification plan
|
||||
|
||||
Use the existing real-PostgreSQL Teams protocol fixture, and exercise real issue,
|
||||
process/consent and cancellation/resolution writers. Cover saved expiry and
|
||||
restart, first committed transfer, nested commit/rollback, no early reads,
|
||||
changed phase at claim, provider-call absence, atomic projection rollback,
|
||||
private-error redaction, lock contention/fair backlog, drain/resume, shutdown,
|
||||
and submillisecond deadlines. Existing protocol, publication and consent tests
|
||||
must continue to pass. Exercise production registration and ensure old maintenance
|
||||
cannot expire consent. Full stack checks and master reconciliation remain required.
|
||||
|
||||
## Other outstanding gates
|
||||
|
||||
#17586 is published, attached, and locally verified; remote CI/reviews are pending.
|
||||
#17580's company deletion policy finding and #17490's E2E cancellation failure are
|
||||
still unresolved. No original recurring-server obligation is removed from scope.
|
||||
|
||||
## Implementation and acceptance
|
||||
|
||||
The Teams owner now registers a scoped deadline worker. The existing service
|
||||
signals before issue/update writes, including caller-owned resolution transactions.
|
||||
Startup pages eligible rows; transfer hints reload only that primary key. The
|
||||
owner rechecks expiry with the original publication/transfer lock order and
|
||||
atomic projection. The old periodic selection and execution now cover attempts
|
||||
only. No schema or external wake contract changes are included.
|
||||
|
||||
Local verification at the slice-26 implementation:
|
||||
|
||||
- 65 transfer protocol cases pass on a fresh external PostgreSQL database.
|
||||
These include 102-entry paging/fairness with one corrupt row, shutdown joining
|
||||
a held projection, the three consent phases, precision, rollback, restart,
|
||||
drain/resume, private-error redaction and the remaining maintenance boundary.
|
||||
- The earlier five-suite run passed 206 cases: then-63 transfer cases and 143
|
||||
publication, consent, consent-runtime and app-wiring cases. The final transfer
|
||||
run adds two cases. The latest app-wiring assertions pass all 16 cases again.
|
||||
- 14 selected real production integration cases pass. This includes registration
|
||||
before an actual Board publication, post-commit wake, saved deadline, pre-expiry
|
||||
SQL quiet, atomic expiry/publication cancellation, no extra provider requests,
|
||||
and the existing Teams current-authority/Board-transfer group. The other 1,411
|
||||
integration cases were skipped, not passed.
|
||||
- Together these are 222 unique selected cases. Workspace `pnpm -r typecheck`
|
||||
and `pnpm build` pass. Server typechecking excludes test files; it is not proof
|
||||
that the test files were statically checked.
|
||||
- Local full `pnpm test:run` remains unclaimed because this host's embedded
|
||||
PostgreSQL setup failed earlier in the stack. Full remote gates are required.
|
||||
No live Teams tenant or deployed host-suspension qualification is claimed.
|
||||
|
||||
Failure ledger: the first protocol run had two invalid fixtures. A direct phase
|
||||
mutation set attempt expiry without its required paired attempt ID; another
|
||||
assertion treated the public ISO expiry string as a Date. Both were corrected,
|
||||
then all five suites passed on a fresh database. The first production expiry
|
||||
assertion expected a failed publication; the existing correct expiry projection
|
||||
is cancelled. The corrected test and the authority group pass. These were test
|
||||
expectation/setup errors, not weakened production contracts or hidden retries.
|
||||
|
||||
Evidence logs: `/tmp/sleep26-teams-tests.log` (initial two fixture failures),
|
||||
`/tmp/sleep26-teams-fixed-tests.log` (206 pass),
|
||||
`/tmp/sleep26-teams-final-tests.log` (65 pass),
|
||||
`/tmp/sleep26-producer-tests.log` (initial projection-state assertion),
|
||||
`/tmp/sleep26-production-acceptance.log` (14 pass),
|
||||
`/tmp/sleep26-final-wiring.log` (16 pass), and
|
||||
`/tmp/sleep26-workspace-{types,build}.log` (pass).
|
||||
|
||||
Parent #17586 at 250fca6cff has passing Actions, security and Greptile checks;
|
||||
Paperclip Review is still running at the latest inspection. This does not clear
|
||||
older stack findings or imply merge authority. Refreshed master is b7dc594bfa;
|
||||
final whole-stack reconciliation is still required.
|
||||
|
||||
|
||||
## Bounded reconstruction and current-parent acceptance — October 28
|
||||
|
||||
The published Teams head `aa082e4679` was inspected after the report parent was
|
||||
prepared on landed master. Its worker's registered load still looped through
|
||||
all pages in one scheduler pass. This acceptance repairs that existing slice;
|
||||
it does not introduce another queue, scheduler, database change or duty.
|
||||
|
||||
A local normal merge prepares the slice on report candidate `c28de398`. The
|
||||
explicit-base merge had one stack-plan append conflict. The complete report
|
||||
plan and complete Teams appendix are both preserved; all source blobs match
|
||||
the clean explicit-base composition. An initial literal patch comparison
|
||||
rejected shifted queue line numbers (47 to 49) and blob IDs, then an overly
|
||||
broad merge whitespace check flagged an inherited runner report-lock patch.
|
||||
The actual patch content and owned-scope whitespace check pass; inherited
|
||||
upstream files were not altered. Actual landed-parent reconciliation and fresh
|
||||
remote gates are required before publication and merge.
|
||||
|
||||
The new 105-transfer PostgreSQL regression failed against the original reader:
|
||||
startup selected 105 rows instead of yielding after 100. The worker now splits
|
||||
one-page selection from complete explicit inventory reads. Registered refresh
|
||||
returns the existing shared `{rows, complete}` contract and retains per-scope
|
||||
cursors; registration clears them. Restarting the same service halfway through
|
||||
a scan reconstructs all 105 saved deadlines, expires them at the original time,
|
||||
and leaves no recurring SQL before the deadline or after settlement. No consent
|
||||
card or upload is sent. Removing only cursor reset makes the restart read five
|
||||
rows instead of 100; original source was restored before final acceptance.
|
||||
|
||||
The first regression fixture faked all timers before establishing its first
|
||||
database connection and timed out, including teardown. That was not the intended
|
||||
row-budget failure and is not accepted as regression evidence. Seeding now uses
|
||||
real network timers with only Date frozen, then controls scheduler timers after
|
||||
seeding. The corrected unchanged-reader run fails at 105 versus 100, and the
|
||||
fixed worker passes. Both the failed setup and actual regression remain recorded.
|
||||
|
||||
Current acceptance: all 209 cases in the complete transfer (66), publication,
|
||||
consent, consent-runtime and app-wiring suites pass on a new migrated fixture;
|
||||
14 selected real Board-producer and Teams-authority integration cases pass on a
|
||||
separate new migrated fixture (1,580 others skipped). Total 223 distinct cases;
|
||||
the targeted red/green case is not added again. Server production typecheck,
|
||||
build and module-boundary checks pass. The new source remains within the
|
||||
existing Teams owner and existing test file. No schema, shared-worker contract,
|
||||
workflow or lockfile change is introduced. The old 222-case evidence remains
|
||||
historical. This does not claim a complete local repository suite, a live Teams
|
||||
tenant, external Cloud wake, physical suspension or cost savings.
|
||||
|
||||
Logs: `/tmp/sleep26-bounded-reconstruction-red.log` (invalid timer setup),
|
||||
`/tmp/sleep26-bounded-reconstruction-actual-red.log` (105 vs 100),
|
||||
`/tmp/sleep26-bounded-reconstruction-green.log`,
|
||||
`/tmp/sleep26-bounded-cursor-mutant-red.log` (5 vs 100 after restart),
|
||||
`/tmp/sleep26-bounded-final-protocols.log`,
|
||||
`/tmp/sleep26-bounded-final-producers.log`,
|
||||
`/tmp/sleep26-bounded-final-types.log`, `/tmp/sleep26-bounded-final-build.log`,
|
||||
and `/tmp/sleep26-bounded-final-boundaries.log`.
|
||||
|
||||
Current stack state supersedes the historical outstanding-gates paragraph:
|
||||
cleanup and retention have landed, including the repaired legacy ownership case
|
||||
and the explicitly accepted company-deletion ownership policy. The report parent
|
||||
has a documentation-only follow-up after `c28de398` and still needs its own latest
|
||||
CI/reviews. This local Teams preparation must preserve that actual landed parent
|
||||
before publication; it is not authority to merge into an unmerged feature branch.
|
||||
|
||||
|
||||
## Inherited retention CI fairness repair — October 28
|
||||
|
||||
Current Teams head `36f43922` received exact-head Greptile 5/5 and Ponytail 5/5,
|
||||
but CI run `40708566195`, server shard 6/12 job `122887395220`, failed the existing
|
||||
voice-retention case `pages through more than one batch of history across
|
||||
companies`. The shard had 2,291 passed tests, one failed and one skipped. Its
|
||||
initial assertion expected the other company's row already deleted after one
|
||||
startup batch, even when its random ID put it beyond the first 100-row inventory
|
||||
page. This CI failure is not accepted as green evidence.
|
||||
|
||||
Placing the other company at a fixed final UUID reproduces the exact assertion.
|
||||
The fixture now uses a supplied ID for both insertion and its transaction hint.
|
||||
The test controls scheduler time, requires that row to remain before discovery,
|
||||
then requires its deletion in the next fair batch after page-two discovery. It
|
||||
also checks the first 25 deletions leave exactly 80 eligible rows, drains all
|
||||
remaining history, and verifies SQL quiet. The existing five-second assertion
|
||||
wait budget and global test timeout are unchanged. This corrects the acceptance
|
||||
expectation without changing retention, Teams, or shared scheduler behavior.
|
||||
|
||||
The complete 13-case retention suite passes. Its run spent 96.66 seconds in
|
||||
module transforms but all actual cases completed successfully; no timeout was
|
||||
increased and this was not counted as a failed test or silently retried. A
|
||||
guarded mutation bypassing company rotation fails the new post-discovery fair
|
||||
batch assertion. The original shared helper was restored byte-for-byte. The
|
||||
previous 223-case Teams acceptance remains evidence for unchanged Teams source;
|
||||
these 13 retention cases are a separate CI-repair acceptance group. Earlier
|
||||
current-head Greptile/Ponytail scores become stale after publishing this test-only
|
||||
follow-up, and fresh CI must pass independently. No old-job retry or waiver is
|
||||
claimed.
|
||||
|
||||
Logs: `/tmp/sleep26-36f-server6-job.log`,
|
||||
`/tmp/sleep26-retention-fairness-red.log`,
|
||||
`/tmp/sleep26-retention-fairness-green.log`, and
|
||||
`/tmp/sleep26-retention-fairness-mutant-red.log`.
|
||||
@@ -0,0 +1,446 @@
|
||||
# Voice call-report scheduling sleep audit
|
||||
|
||||
Original baseline: voice history retention PR #17583 at
|
||||
9c38ddfe5349ef1872306a91c7e4c5c1bc083259.
|
||||
Current integrated parent: 68b62b6d13f28ac5f17f35b98adf5488752f9b97.
|
||||
Branch: codex/sleep-25-voice-call-reports.
|
||||
Status: implementation and local acceptance complete; publication and remote gates pending.
|
||||
|
||||
## The obligation
|
||||
|
||||
`voiceSessionService.reconcileReports` is the final voice duty called from the
|
||||
chat maintenance cadence. It scans terminal sessions with a provider call ID and
|
||||
endedAt within the last 24 hours. A missing report is immediately due; an existing
|
||||
report uses nextCheckAt. The existing exponential retry is 30 * 2^attempts seconds,
|
||||
capped at one hour. Its first post-fetch retry is 60 seconds.
|
||||
|
||||
The current policy deliberately revisits even available reports during that
|
||||
24-hour window. Do not assume available means no future obligation: later provider
|
||||
reports can update transcript and billing, and delivery diagnostics may still
|
||||
need their final snapshot. The provider parser's complete flag only means the
|
||||
response has a report object. It is not an explicit final-and-immutable promise.
|
||||
Preserve this finite refresh window while replacing the repeated global scan.
|
||||
No recording URL or provider metadata is stored; the provider parser allowlists
|
||||
spoken turns and safe billing/duration values.
|
||||
|
||||
## Ownership and implementation direction
|
||||
|
||||
Existing chat_voice_sessions and chat_voice_reports remain durable truth. Keep
|
||||
selection, refresh policy, fetch, result ordering and projection within the voice
|
||||
owner. Use the scoped deadline helper for coalescing, fair batches, backoff, drain
|
||||
and shutdown. Register a report topic on the existing voice-change composition;
|
||||
do not add caller recovery wrappers or a queue table.
|
||||
|
||||
- Startup loads paged terminal-session candidates and actual nextCheckAt values.
|
||||
- Missing reports are due at terminal transition. Future nextCheckAt remains
|
||||
unchanged across inspection/restart.
|
||||
- A report entry also has a finite window end. Expire its cached obligation when
|
||||
the 24-hour window ends, even if its next attempt would be later. Define SQL
|
||||
timestamp precision and inclusive boundary consistently; do not spin at cutoff.
|
||||
- Lock/recheck current session eligibility and report deadline before reserving
|
||||
an attempt. Save retry intent before network I/O. Provider GET remains outside
|
||||
database transactions. Inspect whether a concurrent manual reconciliation or
|
||||
callback can invalidate a result; use existing persisted fields for ownership
|
||||
where sufficient. Do not add speculative leases or schema without evidence.
|
||||
- Preserve providerUpdatedAt ordering. An incomplete or older response must not
|
||||
replace a newer available report. Stale responses must still back off.
|
||||
- Preserve the guest-task transcript projection, its advisory lock, low-trust
|
||||
quarantine check, owner access, and activity log. A saved report is not proof
|
||||
that projection completed. Preserve retry after a projection failure.
|
||||
- Preserve bounded diagnostic attempts and never replay speech. Diagnostic
|
||||
failure must not prevent transcript/history reconciliation.
|
||||
- Keep explicit one-shot diagnostic reconciliation compatible. Remove the
|
||||
recurring voice lane only after all report producers are covered.
|
||||
|
||||
## Producer audit
|
||||
|
||||
- `voice-inbound-service.ts` signed terminal and report callbacks call
|
||||
notifyVoiceChange before the session update and report upsert. Existing reports
|
||||
get nextCheckAt = now, which may advance a scheduled refresh.
|
||||
- `voice-session-service.ts` close and inspect paths use withVoiceSessionChange.
|
||||
- `voice-session-recovery.ts` uses full voice-change notification for terminal
|
||||
inspection results. Healthy inspection and error-only updates intentionally
|
||||
notify only recovery; do not make those trigger empty report scans.
|
||||
- `voice-reply-push.ts` handles HTTP 409 by writing state=ended and endedAt. It
|
||||
currently notifies reply, recovery and inbound explicitly; report notification
|
||||
must be included before this terminal write.
|
||||
- Configuration/authority/company changes use notifyVoiceChange composition.
|
||||
- The existing report loop inserts and updates report rows without transaction
|
||||
notifications. Move that ownership into the new report service. Account for
|
||||
callbacks during an in-flight provider response and for any remaining direct
|
||||
writes discovered in the full source audit.
|
||||
|
||||
## Verification required
|
||||
|
||||
- Configured idle, live calls, old terminal history and calls without provider IDs
|
||||
are SQL/network quiet after startup.
|
||||
- Real session end, signed callbacks, and reply call-ended results wake reports.
|
||||
- Saved refresh deadlines survive restart; no early reads, bounded failures, and
|
||||
final window expiry disarm. Available reports retain the existing window.
|
||||
- Nested commit/rollback, moved-earlier callback deadlines, concurrent writes,
|
||||
first work during drain, failed work entering drain, and shutdown keep work.
|
||||
- Multi-page backlog and per-company fairness do not starve later calls.
|
||||
- Provider failure, incomplete responses, older provider timestamps, guest
|
||||
transcript projection failures and diagnostic behavior retain correct data.
|
||||
- Existing full voice integration/history/access tests pass on a fresh database.
|
||||
|
||||
No Cloud alarm coverage or sleep-eligibility expansion is authorized by this
|
||||
migration. A future in-process refresh cannot wake stopped compute. Preserve
|
||||
current external-wake limitations explicitly.
|
||||
|
||||
## Parent state
|
||||
|
||||
#17583 is published and attached. Its 218 selected tests, workspace types/build,
|
||||
and final server typecheck pass locally; fresh CI/reviews are running.
|
||||
#17580 migration-runner repair 4f0364d269 has fresh Greptile 5/5. At the last
|
||||
inspection it had no failing checks and three pending checks. Its prior database
|
||||
setup failure and regression/repair evidence remain in its audit.
|
||||
|
||||
The complete original recurring-server ledger, older #17490 E2E failure,
|
||||
remaining PR reviews and final master/stack reconciliation remain required.
|
||||
|
||||
## Implementation and acceptance, 2026-10-25
|
||||
|
||||
The parent retention repair at `360e1253a0` is incorporated. The report owner now
|
||||
loads scoped, paged obligations, claims each due attempt before network I/O, and
|
||||
uses attempt/deadline ownership for settlement. Report writes notify the same
|
||||
shared coordinator. Existing voice-change producers and the explicit reply
|
||||
`call_ended` path activate it. App startup registers it once and the final voice
|
||||
lane is removed from the chat interval. No schema or Cloud eligibility change.
|
||||
|
||||
The saved report worker suite passed 13 tests. Initial full integration acceptance
|
||||
had 136 passes and one failure: an unlocked existing transcript still could not
|
||||
update because the copied projection omitted the document's base revision ID.
|
||||
The repair supplies the observed `latestRevisionId`, preserving document locks
|
||||
and conflict checks. A focused rerun against the already-used full-suite database
|
||||
verified the document update but failed its global provider-call count because
|
||||
133 retained fixture calls were eligible; this is not accepted as a clean run.
|
||||
Final full acceptance uses a new database migrated through the application
|
||||
migrator. Results are recorded below when terminal.
|
||||
|
||||
Earlier worker fixture failures (assignedAgentId vs agentId) and the production
|
||||
missing `or` import were repaired before the 13-case worker pass. Source wiring
|
||||
assertions still expecting reconcileMaintenance were updated to require its
|
||||
removal. Full-suite tests remain Vitest coverage; server tsconfig excludes tests,
|
||||
so production typecheck is not described as test typechecking.
|
||||
|
||||
Parent #17583 at `360e1253a0` has a failed server shard: the Discord rollback/drain
|
||||
quiet test observes one scoped select after its nextWakeAt becomes null. The job
|
||||
log is retained at `/tmp/sleep24-review-ci-failure.log`. This needs investigation
|
||||
at its owning earlier slice and propagation; it is not silently retried or
|
||||
claimed green. Parent #17580 also has two review findings about cleanup latency
|
||||
and deletion with unconfirmed provider cleanup. These remain open. The older
|
||||
#17490 E2E failure and final master reconciliation remain part of completion.
|
||||
|
||||
Final local acceptance after the revision fix: 137 full voice/app wiring cases
|
||||
passed on `paperclip_sleep25_projection_fixed_20261025_test`; 37 report/cleanup/
|
||||
retention worker cases passed serially on the isolated worker database. Workspace
|
||||
`pnpm -r typecheck` and `pnpm build` passed. Logs: `/tmp/sleep25-voice-fixed-acceptance.log`,
|
||||
`/tmp/sleep25-voice-workers-acceptance.log`, `/tmp/sleep25-workspace-types.log`,
|
||||
and `/tmp/sleep25-workspace-build.log`. These 174 selected cases do not claim the
|
||||
entire `pnpm test:run` suite passed locally. This host's embedded PostgreSQL test
|
||||
initialization limitation remains; CI must run the complete relevant gates.
|
||||
|
||||
## Publication and parent repairs
|
||||
|
||||
Published as #17586. The Discord quiet-test settlement repair is propagated from
|
||||
#17557 and passes all 41 command ownership tests at the report branch. The inbound
|
||||
denial latency fix is also propagated; its held-provider regression is included.
|
||||
After both merges, all 138 full voice/app cases pass on a fresh migrated database
|
||||
(`/tmp/sleep25-denial-acceptance.log`). The earlier 37 worker cases remain valid;
|
||||
those worker implementations are unchanged by the parent repairs. Remote checks
|
||||
and reviews are required for the final pushed head. Parent #17580's company
|
||||
deletion finding remains unresolved; denial latency finding is resolved.
|
||||
|
||||
|
||||
## Paperclip review: report window index and writer scope
|
||||
|
||||
Review findings 4370594150 and 4370594357 identified deployment-contract clarity
|
||||
and expensive report reconstruction. The supported writer model is now explicit:
|
||||
one active server process owns voice writes through its DB client. Active-active
|
||||
servers and independent SQL mutation are unsupported without a separate durable
|
||||
wake contract. This is a documented scope restriction, not enforcement added by
|
||||
this PR, and not a claim that startup recovers concurrent foreign writers.
|
||||
|
||||
The query finding reproduces on fresh PostgreSQL with 20,001 old sessions and
|
||||
205 eligible sessions: an actual reconstruction query filters 20,001 historical
|
||||
rows. The repair adds partial `(ended_at, id)` index 0389 and uses that order and
|
||||
keyset cursor. The cursor retains PostgreSQL text timestamp precision, so sessions
|
||||
100 microseconds apart do not repeat or disappear across a page. All 14 report
|
||||
lifecycle cases pass after the migration; the query-plan regression checks all
|
||||
three pages and each stays below 1,000 rows removed by filtering. Report scope,
|
||||
24-hour policy, and provider behavior are unchanged.
|
||||
|
||||
The generated migration is concurrent and idempotent and registered with the
|
||||
application's top-level migration runner. It applies and a second application is
|
||||
a no-op. Migration numbering/safety checks pass. The generator retained exactly
|
||||
five snapshots, 0385 through 0389, pruning 0384 per repository policy.
|
||||
|
||||
Retained failures: the first probe reused a database with two unrelated eligible
|
||||
fixture sessions and failed its exact fixture count. It is not performance proof.
|
||||
A fresh database reproduced the 20,001-row filtering problem. The first migration
|
||||
invocation used unsupported top-level await in tsx eval and did not apply; its
|
||||
subsequent test still failed without the index. The corrected chained invocation
|
||||
applied/replayed successfully and all 14 cases passed. No assertion was relaxed.
|
||||
Logs: /tmp/sleep25-report-index-before.log (fixture contamination),
|
||||
/tmp/sleep25-report-index-clean-before.log (cost reproduction),
|
||||
/tmp/sleep25-report-index-apply.log (invalid invocation),
|
||||
/tmp/sleep25-report-index-after.log (unapplied index),
|
||||
/tmp/sleep25-report-index-apply-fixed.log (application/replay),
|
||||
/tmp/sleep25-report-index-fixed-tests.log (14 pass).
|
||||
Workspace typecheck and build pass. Fresh full voice integration and app wiring
|
||||
pass all 138 cases after the index repair. Logs:
|
||||
/tmp/sleep25-report-review-{types,build,acceptance}.log.
|
||||
Publish and propagate through Teams #17590/#17592 after acceptance. Fresh remote
|
||||
review/checks and all prior full-stack requirements remain open.
|
||||
|
||||
|
||||
## Current-parent reconstruction and migration acceptance, October 28
|
||||
|
||||
The report branch normally integrates retention `68b62b6d13` and its cleanup,
|
||||
recovery and reply foundations. The retention repair prevents configuration saves
|
||||
from rescanning history. Its real 205-admission, five-save regression is retained
|
||||
and passes in the complete voice suite here. Both report and retention tests were
|
||||
preserved when their additions conflicted at the same point in that suite.
|
||||
The original 20 owned paths remain the complete report slice. Shared scheduler,
|
||||
workflow and lockfile changes are inherited context, not additions by reports.
|
||||
|
||||
### Bounded discovery, restart and fair dispatch
|
||||
|
||||
The old report `load` executed every page before yielding. A new 105-session
|
||||
fixture with saved future report deadlines selected 105 rows in its first pass,
|
||||
failing the 100-row startup budget. The registered worker now returns one page
|
||||
per scope refresh through the existing shared `{ rows, complete }` contract.
|
||||
The domain owner retains full explicit `load` and one-shot reconciliation APIs.
|
||||
The keyset cursor still stores PostgreSQL timestamp text, not a truncated Date.
|
||||
Only successful, non-aborted reads advance a cursor, and registration resets it.
|
||||
|
||||
The real same-owner restart case stops after 100 of 105 saved deadlines,
|
||||
re-registers the service, reconstructs the original deadline, and refreshes each
|
||||
of the 105 reports once at that time. It observes no SQL/provider activity before
|
||||
the deadline or between saved retries afterward. Removing the cursor reset fails
|
||||
at five rows instead of 100 on restart. This measures selected rows, not physical
|
||||
tuple visits or instance cost.
|
||||
|
||||
Fairness applies to discovered work. The prior test demanded that a company
|
||||
beyond the first page run in the initial batch; that assumes an eager inventory.
|
||||
The revised test puts that company deterministically on page two and requires
|
||||
it in the next 25-work batch as soon as it is discovered. It also verifies all
|
||||
105 calls run and the failing provider retains one saved attempt without
|
||||
starving the backlog. Bypassing company rotation makes the next-batch assertion
|
||||
fail. Both guarded mutations restore the exact original source afterward.
|
||||
|
||||
Preserved test failures distinguish this contract change from production defects.
|
||||
The first bounded full run failed the old first-batch assertion. The first
|
||||
controlled-clock revision allowed random UUID order to put the other company
|
||||
on page one; a fixed final key now makes page placement deterministic. An
|
||||
intermediate revision expected zero active intent holds before all dirty scopes
|
||||
settled. The test now uses 20 controlled one-second scheduler turns, retaining
|
||||
those holds until reconciliation completes, and finishes before the provider's
|
||||
60-second retry. It still checks fair dispatch, all 105 calls, one failed attempt,
|
||||
and quiet after settlement. It does not advance time in every assertion retry.
|
||||
|
||||
The external fixture first failed on missing tables. It now applies the actual
|
||||
application migrations with the standard setup timeout before creating data.
|
||||
The failed setup and 105-versus-100 budget remain recorded. No test is skipped
|
||||
and no eligibility or provider assertion was removed to repair setup.
|
||||
|
||||
### Generated migration lineage
|
||||
|
||||
The old unmerged report migration 0393 conflicts with the landed checkpoint and
|
||||
cleanup lineage. The complete retention parent migration tree and client were
|
||||
restored, then the unchanged report-index schema edit was regenerated with
|
||||
`pnpm db:generate`. The result is `0396_careful_blindfold.sql`. It creates the
|
||||
existing proposed partial `(ended_at, id)` index concurrently and replayably,
|
||||
through the application's top-level migration runner. No table or field is added.
|
||||
|
||||
All 394 parent SQL files and the journal prefix are preserved byte-for-byte.
|
||||
Snapshots 0392 through 0396 are the five retained snapshots. The generator's
|
||||
required deletion of 0391 is included; no retained snapshot or ID was hand-edited.
|
||||
Generation used parent `ffe9d47b`; retention review head `68b62b6d13` has the exact
|
||||
same database tree. The snapshot/schema drift regression passes. The report
|
||||
fixture boots through the application migrator, and two later complete migration
|
||||
invocations succeed with the report index valid. The existing query-plan case
|
||||
still tests 20,001 old sessions, 205 eligible sessions and all three precision-safe
|
||||
pages; each query removes fewer than 1,000 rows by filtering.
|
||||
|
||||
### Final local evidence
|
||||
|
||||
302 distinct cases pass on this combined source:
|
||||
|
||||
- 131 complete voice integration cases, including real hangup, callback and reply
|
||||
terminal producers, transcript projection after a document lock, and the parent
|
||||
retention amplification regression. This suite uses an initially empty DB.
|
||||
- 170 cases across 11 complete report, retention, cleanup, recovery, company
|
||||
removal, shared deadline/coordinator/batching, startup and hostname suites.
|
||||
This includes 15 report, 13 retention and 14 cleanup lifecycle cases. Recovery,
|
||||
company removal, retention and cleanup use separate initially empty DBs and
|
||||
migrate themselves. Reports use their dedicated DB after the setup repair.
|
||||
- One complete snapshot/schema drift case. The schema and migration tree are
|
||||
identical to the checked generation after the final retention merge.
|
||||
|
||||
Database and server typechecks/builds, migration numbering/safety and the module
|
||||
boundary check pass. Earlier 174/138-case and workspace-wide claims apply only to
|
||||
their stated heads. No new full local repository suite, live provider, Cloud wake,
|
||||
physical suspension or savings claim is made. The 24-hour mutable-report window,
|
||||
retry, confirmation, callback ownership and quarantined transcript policy remain.
|
||||
|
||||
Evidence logs are `/tmp/sleep25-final-voice-consumers.log`,
|
||||
`/tmp/sleep25-final-lifecycle-consumers.log`,
|
||||
`/tmp/sleep25-final-snapshot-drift.log`,
|
||||
`/tmp/sleep25-final-integrated-{types,build}.log`,
|
||||
`/tmp/sleep25-current-migration-{check,replay}.log`, and
|
||||
`/tmp/sleep25-current-migration-lineage-proof.json`. Failed setup, eager paging,
|
||||
and the three fairness-clock attempts are in `/tmp/sleep25-empty-fixture-red.log`,
|
||||
`/tmp/sleep25-page-budget-red.log`, `/tmp/sleep25-bounded-report-first-full.log`,
|
||||
`/tmp/sleep25-bounded-report-green.log` and
|
||||
`/tmp/sleep25-bounded-report-fairness-green.log`. The passing initial full report
|
||||
run is `/tmp/sleep25-bounded-report-final-green.log`; its 15 cases are included
|
||||
in the final 170 and not counted twice. Negative probes are
|
||||
`/tmp/sleep25-{cursor-reset,company-fairness}-mutant-red.log`.
|
||||
|
||||
Fresh exact-head review, CI and earlier stack landing remain required. Cleanup
|
||||
parent `b8b12199b9` completed green CI after its single recorded E2E retry, but
|
||||
lacks current Ponytail acceptance. Retention's earlier 4/5 review was repaired
|
||||
in `68b62b6d13`; that new parent still requires its own fresh reviews and CI.
|
||||
Neither parent state is a waiver for reports or for remaining stack duties.
|
||||
|
||||
|
||||
## Configuration scope review repair, October 28
|
||||
|
||||
The earlier security thread correctly identified another source of repeated
|
||||
report inventory. A voice connection save reached `notifyVoiceChange`, which
|
||||
notified reports for connection, endpoint, company or global scopes. Bounded
|
||||
paging controlled each burst but still reloaded the whole eligible set after
|
||||
separate saves. A real producer regression seeds 205 terminal sessions with saved
|
||||
future report deadlines, then performs five separate connection saves. It selects
|
||||
1,025 session rows before repair and zero afterward. The original next deadline
|
||||
is unchanged, and no provider GET occurs during the saves.
|
||||
|
||||
The composed notification now signals reports only for actual session changes.
|
||||
The report-specific helper accepts only an exact session scope, preventing a
|
||||
future configuration caller from reintroducing broad hints through that boundary.
|
||||
Startup still uses the shared worker's initial paged reconstruction. Report claims,
|
||||
receipts and signed callbacks keep exact session notifications. Existing terminal
|
||||
hangup and reply-ended paths remain covered through their real producer APIs.
|
||||
Live recovery, replies and inbound cleanup retain their own broader authority
|
||||
and configuration signals; their eligibility can depend on those changes.
|
||||
|
||||
The report predicate depends on session terminal state, provider identity and
|
||||
endedAt, plus saved report state. A configuration save changes none of them.
|
||||
Previously such a save only reloaded nextCheckAt and did not advance a future
|
||||
provider attempt; the repair retains that same saved retry. There is no new
|
||||
credential policy, refresh cadence, schema or scheduler contract. Manual inventory
|
||||
APIs remain complete. Unsupported external SQL still needs its own wake contract.
|
||||
|
||||
303 distinct cases pass after the repair: 132 complete voice integration cases,
|
||||
170 complete lifecycle/shared/startup cases, and the unchanged schema/snapshot
|
||||
case. The voice suite uses a fresh separate DB; the lifecycle suites use their
|
||||
existing dedicated isolated DBs. Server typecheck/build and module boundaries
|
||||
pass. The earlier 302 count applies to `80207ba6bb` and is not added to this count.
|
||||
The current Ponytail 5/5 on that earlier head cannot be reused after this edit;
|
||||
fresh exact-head review and all CI remain required.
|
||||
|
||||
Logs: `/tmp/sleep25-config-amplification-red.log`,
|
||||
`/tmp/sleep25-scope-final-voice-consumers.log`,
|
||||
`/tmp/sleep25-scope-final-lifecycle-consumers.log`,
|
||||
`/tmp/sleep25-scope-final-types.log`, `/tmp/sleep25-scope-final-build.log`, and
|
||||
`/tmp/sleep25-scope-final-module-boundaries.log`. No full local repository suite,
|
||||
live provider, Cloud wake or physical suspension claim is added. The original
|
||||
20-path report scope and the existing finite 24-hour obligation remain intact.
|
||||
|
||||
## Landed-master migration and acceptance — October 28
|
||||
|
||||
The normal merge uses actual master `980775e0e956eecdc28c73c9b4aca0407ad45e1a`
|
||||
after cleanup and retention landed. The reviewed report head was
|
||||
`9bf20467dfd50b81848e1c8dd7ddc905d206418f`, based on retention `68b62b6d13`.
|
||||
An explicit-base composition preserved the complete non-database report patch
|
||||
before the acceptance-test repair below. Only the shared client, journal, and
|
||||
396 snapshot conflicted. Those metadata files were restored from actual master
|
||||
before invoking the official `pnpm db:generate` workflow.
|
||||
|
||||
The generated report migration is now `0397_known_madrox.sql`. It retains the
|
||||
reviewed partial `(ended_at, id)` concurrent, idempotent index SQL byte-for-byte.
|
||||
The existing top-level migration registry adds only the new filename and keeps
|
||||
landed cleanup migration 396 registered. All 395 existing SQL files and the
|
||||
complete journal prefix match actual master byte-for-byte. Snapshots 393 through
|
||||
397 remain, with the generated 397 snapshot pointing to actual parent 396.
|
||||
No landed migration, snapshot contents, or journal entry was renumbered.
|
||||
|
||||
A new empty PostgreSQL database boots through the application migrator with a
|
||||
valid report index. A deliberately failed unique concurrent index build under
|
||||
the report index name leaves an invalid index; two concurrent invocations of
|
||||
the application migrator repair it, restore the intended report index, and
|
||||
record one migration history row. Another replay succeeds. This exercises the
|
||||
new 397 registration and invalid-index recovery rather than assuming that a
|
||||
successful Drizzle generation proves executable migration acceptance.
|
||||
|
||||
The first complete four-suite run failed: 183 passed and one report integration
|
||||
case failed, 184 total. `retains one safe report per call and rejects history
|
||||
after access revocation` expected cost 240000 but retained 120000. The test used
|
||||
a global one-shot transport response while reconciliation correctly visits
|
||||
other due calls. An unrelated call could consume the target's newer response.
|
||||
Adding an explicit earlier-ended competing call reproduced the same assertion
|
||||
in isolation on unchanged production. The test now selects responses by the
|
||||
target provider session ID, asserts the target accepts 240000, asserts the
|
||||
other call independently keeps 120000, and then proves stale cost 1 cannot
|
||||
overwrite the target. Existing access-revocation and one-report-per-call
|
||||
assertions remain. No production report or provider behavior changed.
|
||||
|
||||
The isolated repaired case passes, followed by the complete 132-case voice
|
||||
integration suite on a new empty database. The unchanged complete report (15),
|
||||
cleanup (24), and retention (13) suites passed in the first combined run, each
|
||||
on its separate empty database. These 184 distinct consumer cases plus one
|
||||
complete schema/snapshot drift case yield 185 distinct passing cases for this
|
||||
integration. The isolated case is not added again. Database build, server
|
||||
build, final server typecheck, migration safety/numbering and module-boundary
|
||||
checks pass. The earlier 303-case evidence belongs to the stated previous head;
|
||||
it is not added to this acceptance count or claimed rerun here.
|
||||
|
||||
Logs: `/tmp/sleep25-landed-parent-db-generate.log`,
|
||||
`/tmp/sleep25-landed-parent-db-build.log`,
|
||||
`/tmp/sleep25-landed-parent-real-migration.log`,
|
||||
`/tmp/sleep25-landed-parent-real-consumers.log` (the retained 183-pass/one-fail run),
|
||||
`/tmp/sleep25-report-mock-isolation-red.log`,
|
||||
`/tmp/sleep25-report-mock-isolation-green.log`,
|
||||
`/tmp/sleep25-landed-parent-final-voice.log`,
|
||||
`/tmp/sleep25-landed-parent-snapshot.log`,
|
||||
`/tmp/sleep25-landed-parent-final-types.log`,
|
||||
`/tmp/sleep25-landed-parent-server-build.log`, and
|
||||
`/tmp/sleep25-landed-parent-module-boundaries.log`.
|
||||
|
||||
Fresh current-head CI and Greptile 5/5 remain required. Prior-head review scores
|
||||
are stale after this normal merge. A stopped or paused Ponytail turn without a
|
||||
verdict does not block under the user's authorization; an actual failing
|
||||
verdict must be fixed. No new full local repository suite, live-provider test,
|
||||
Cloud wake, host/database suspension, or cost-saving result is claimed.
|
||||
|
||||
|
||||
## Review documentation and preserved remote timeout — October 28
|
||||
|
||||
Ponytail reviewed `c28de398` at 4/5 advisory, found no must-fix production defect,
|
||||
and said to ship after correcting the design document's stale migration
|
||||
reference. The design document now links the current 185-case acceptance and
|
||||
migration `0397_known_madrox.sql`; the older 302/303-case paragraphs are marked
|
||||
historical. Its two remaining suggestions are explicitly optional: one-shot
|
||||
diagnostic `reconcile(limit)` still inventories the whole eligible window before
|
||||
slicing, and some existing test helpers advance fake time during assertion
|
||||
retries. They are not silently claimed fixed or treated as current idle polling.
|
||||
|
||||
CI run `40703490340`, server shard 8/12 job `122869422044`, failed one unchanged
|
||||
approval-queue test: `resets consecutive failure backoff after a successful batch
|
||||
with remaining debt` exceeded its existing 15-second timeout. Teardown then
|
||||
reported a company foreign-key violation from activity-log work. The shard had
|
||||
2,097 passed tests, one failed and one skipped. No report assertion failed in
|
||||
that job, and the exact CI timing cause is not proven. The complete unchanged
|
||||
28-case approval-queue suite passes on a new isolated PostgreSQL database. Its
|
||||
test, worker, coordinator and transaction-signals source were checked byte-for-
|
||||
byte against `c28de398` before that probe, which ran from the local child checkout.
|
||||
No timeout, workflow, approval logic or production source was changed. The old
|
||||
CI failure remains recorded and the new documentation head requires fresh CI;
|
||||
no blind rerun of the failed old job or waiver is claimed.
|
||||
|
||||
Evidence: `/tmp/sleep25-c28-server8-job.log`,
|
||||
`/tmp/sleep25-approval-timeout-local-probe.log` and its exit status 0. The 28 cases
|
||||
are a separate investigation result, not added to the 185-case report acceptance.
|
||||
Prior-head review scores are stale after this documentation commit.
|
||||
@@ -0,0 +1,72 @@
|
||||
# Voice recovery drain repair
|
||||
|
||||
Owner: slice 21, PR #17576. Reviewed baseline: 1bf70185d2.
|
||||
|
||||
Greptile finding 4370544173 is valid. The worker retained dirty scopes while
|
||||
paused, and reported those in-memory hints as pending work. After the producer
|
||||
committed, the coordinator therefore retained its intent activity hold. No SQL
|
||||
was required to reproduce the blocked drain; the earlier quiet assertion checked
|
||||
SQL only and missed the nonzero idle counter.
|
||||
|
||||
The worker now reports pending dirty scopes only while dispatch is permitted.
|
||||
It still retains every hint and next deadline. The coordinator independently
|
||||
checks databaseWorkPending, so an open/uncertain transaction retains its activity
|
||||
hold until transaction reconciliation establishes settlement. Paused timers then
|
||||
use the existing passive-maintenance path without changing the idle generation.
|
||||
Resume processes the retained first session without another producer signal.
|
||||
No provider behavior or external sleep eligibility rule changes.
|
||||
|
||||
The strengthened real-PostgreSQL test proves: an open write during drain retains
|
||||
an activity hold; commit releases it; a further minute causes neither SQL nor
|
||||
an idle-snapshot change; resume processes the one retained session exactly once.
|
||||
|
||||
- /tmp/sleep21-drain-before.log: original worker fails, active is 1 instead of 0.
|
||||
- /tmp/sleep21-drain-tests.log: all 11 voice recovery lifecycle tests pass.
|
||||
- /tmp/sleep21-drain-typecheck.log: production server typecheck passes.
|
||||
- Dedicated migrated database: paperclip_sleep21_drain_20261025_test.
|
||||
|
||||
Propagate this predicate into slice 22's extracted scoped-deadline worker and
|
||||
all descendants, then rerun its existing consumers. The earlier tool-health,
|
||||
Discord command, and receiver workers have the same predicate pattern; audit
|
||||
and repair those at their own parent PRs as a separate propagation pass. Do not
|
||||
report all stack drain behavior fixed from the voice test alone. The company
|
||||
inbound cleanup/deletion policy, older E2E failure, current chat milestone audit,
|
||||
and full original inventory remain outstanding.
|
||||
|
||||
|
||||
## Shared-worker propagation in slice 22
|
||||
|
||||
Slice 22 extracts voice recovery into scoped-deadline-work. The merge keeps that
|
||||
extraction and applies the predicate in the shared dispatcher, covering both
|
||||
voice session recovery and reply delivery. The phone-reply drain test now also
|
||||
checks zero active work and an unchanged idle snapshot during a further minute,
|
||||
then verifies the saved reply sends on resume without another producer.
|
||||
|
||||
- /tmp/sleep22-shared-drain-tests.log: 11 recovery cases pass.
|
||||
- /tmp/sleep22-shared-drain-acceptance.log: all 108 voice integration cases pass.
|
||||
- /tmp/sleep22-reply-drain-tests.log: the strengthened reply drain case passes
|
||||
on a separate fresh database after the full suite run.
|
||||
- /tmp/sleep22-shared-drain-typecheck.log: production server types pass.
|
||||
|
||||
All later shared-worker consumers inherit this repair through the stack. Their
|
||||
combined-tip checks and PR checks still need fresh verification after propagation.
|
||||
|
||||
## Retention resume assertion
|
||||
|
||||
The combined-tip consumer run exposed a retention test that called flush after
|
||||
resume and assumed a zero activity counter meant the retained retry had run. Its
|
||||
helper checks active work only; zero is now also the correct paused state. The
|
||||
owner test now advances the actual retained nextWakeAt, waits for row removal,
|
||||
and checks disarm. Before resume it also proves a stable idle snapshot through a
|
||||
minute of quiet. All 11 retention cases pass in /tmp/sleep24-drain-tests.log.
|
||||
No production scheduling deadline or dispatch behavior changed for this repair.
|
||||
|
||||
## Report resume assertion
|
||||
|
||||
The report test had the same active-counter-only flush assumption. It now checks
|
||||
idle snapshot stability while paused, advances the retained nextWakeAt on resume,
|
||||
and verifies one fetch and the available durable report before testing rollback.
|
||||
All 14 report cases pass in /tmp/sleep25-drain-tests.log. An intermediate new
|
||||
assertion incorrectly expected state=completed; the schema uses status=available.
|
||||
That test-authoring failure is retained in /tmp/sleep25-drain-invalid-assertion.log.
|
||||
The scheduler repair and business deadlines were not changed to satisfy tests.
|
||||
@@ -0,0 +1,419 @@
|
||||
# Inbound voice admission cleanup sleep audit
|
||||
|
||||
Original baseline: voice reply PR #17578, head 439d0a86eb75c129c30742a8cc24bf18e2a6ebf3.
|
||||
Current integrated parent: d2c0a7de8c3e9cac8382b1268f3f5d883a2067b3.
|
||||
Branch: codex/sleep-23-voice-inbound-cleanup.
|
||||
Status: cleanup implementation and local verification complete; publication and remote acceptance pending.
|
||||
|
||||
## One reason in this slice
|
||||
|
||||
Remove recurring discovery of expired inbound admissions and unfinished provider
|
||||
termination. Keep inbound history retention and call reports as separate PRs.
|
||||
The current `voiceInboundService.reconcile` combines a seven-day delete with a
|
||||
25-row expiry/provider scan. `voiceSessionService.reconcileMaintenance` invokes
|
||||
it from the remaining one-second chat lane.
|
||||
|
||||
Existing `chat_voice_inbound_calls` rows own admission state, expiresAt, updatedAt,
|
||||
providerSessionId, generation, credential fingerprint, requested issue, caller,
|
||||
and optional bound voice session/intake task. Existing voice sessions own the
|
||||
bound call's execution and current authorization. Do not add a table per timer.
|
||||
Decide and document the handoff between those existing owners before coding.
|
||||
|
||||
## Current behavior and supported producers
|
||||
|
||||
- Signed `call.pre_call` inserts an awaiting-approval admission with a two-minute
|
||||
expiry, or a guest-intake admission with a ten-minute expiry. The endpoint lock
|
||||
serializes the quota check and insertion. Repeated provider identities are
|
||||
deduplicated; changed replay content is rejected.
|
||||
- `pendingTool` can mark an expired admission as expired. Guest submit_request
|
||||
creates a low-trust task and active voice session, then saves the session/task
|
||||
binding in the same transaction. These writes must wake the appropriate owner.
|
||||
- `decide` locks the admission and validates current caller, endpoint, credential,
|
||||
approval code, expiry, and task authority. It writes approving or denied/ended.
|
||||
Provider inspection and session start occur afterward. Final approved binding
|
||||
is currently a standalone update guarded only by state=approving.
|
||||
- Signed lifecycle/report callbacks lock admission before session and mark the
|
||||
admission ended. Preserve that order and the callback receipt's replay fence.
|
||||
- The expiry sweep scans guest_intake, awaiting_approval, approving, denied, and
|
||||
expired rows only once their expiresAt is past. It inspects the exact saved
|
||||
provider call, marks ended if inspection confirms termination, otherwise marks
|
||||
expired and requests termination. Errors only update updatedAt.
|
||||
- No production admission writers were found outside voice-inbound-service.ts.
|
||||
Session handoff/approval recovery must also inspect voice-session-store.reserve.
|
||||
|
||||
## Correctness questions that the migration must resolve
|
||||
|
||||
1. **Discovery is not an execution claim.** A row read by cleanup can change before
|
||||
provider I/O. Recheck/claim current due state under the owning row lock. Never
|
||||
overwrite a newer approved or terminal state from a stale candidate snapshot.
|
||||
2. **Approval handoff has a crash window.** Inbound start commits an authorized
|
||||
session with the existing provider ID and returns without creating a new call.
|
||||
The admission's final approved update follows in a different transaction.
|
||||
Expiry or a crash can occur between these writes. Determine whether existing
|
||||
binding evidence should complete handoff or retain cleanup; do not end a valid
|
||||
handed-off call based on an older approving snapshot. Test the exact race.
|
||||
3. **Termination requested is not termination confirmed.** Speko endSession returns
|
||||
confirmed=false for status=ending and true only for already_ended. Keep cleanup
|
||||
debt until an inspect/callback/end receipt establishes termination. Never create
|
||||
or redial a provider call in recovery.
|
||||
4. **Guest denial currently writes ended before provider confirmation.** Without a
|
||||
bound session, a failed end request can lose its cleanup obligation. With a
|
||||
session, live-session recovery also owns ending. Preserve one explicit owner
|
||||
for unfinished termination; do not treat a terminal UI label as physical proof.
|
||||
5. **Bound guest calls have two current scanners.** Admission expiry and live-session
|
||||
recovery can inspect/end the same exact call. Define the handoff and admission
|
||||
settlement contract so a scheduler migration does not create two uncoordinated
|
||||
provider workers for one resource.
|
||||
6. **Retention currently precedes cleanup.** The seven-day delete can remove denied
|
||||
or expired rows with no session/task binding before provider termination is
|
||||
proved. Keep history retention separate, but do not allow it to erase newly
|
||||
explicit unfinished cleanup debt. Record any required narrow predicate change
|
||||
with this slice and keep the retention scheduling migration in its own PR.
|
||||
7. **Configuration and credentials.** Cleanup must remain possible after feature
|
||||
disable and endpoint/credential changes. Do not use a replacement credential
|
||||
to act on an unrelated provider call. Read the existing provider/account
|
||||
identity and deletion contracts before selecting recovery scope.
|
||||
|
||||
## Intended scheduler contract
|
||||
|
||||
The inbound owner exposes normalized identities and actual deadlines to the
|
||||
existing delivery coordinator. The initial deadline comes from saved admission
|
||||
expiry; explicit rejection requests immediate cleanup through the same owner.
|
||||
Unconfirmed termination or errors retain a bounded retry. Completed history and
|
||||
no admissions leave no cleanup deadline. Startup reconstructs saved obligations.
|
||||
Notifications precede writes in existing transaction ownership and cover all
|
||||
producer paths above. No provider request is made inside a SQL transaction.
|
||||
|
||||
Preserve scope coalescing, per-obligation failure isolation, fair bounded batches,
|
||||
startup paging, drain/resume, concurrent changes during a sweep, and shutdown
|
||||
joining in-flight work. Keep public/manual reconciliation compatible while
|
||||
separating the retention operation from the scheduled cleanup operation.
|
||||
|
||||
## Required proof
|
||||
|
||||
- No configuration, configured idle, and confirmed terminal history become quiet.
|
||||
- Real signed pre-call after quiet registers its original expiry.
|
||||
- First guest intake, approval/denial, pending-tool expiry, and lifecycle callbacks
|
||||
signal their existing transactions; outer rollback and commit behave correctly.
|
||||
- Approval completes during cleanup discovery without stale termination or state
|
||||
overwrite. A crash between session binding and final approval remains recoverable.
|
||||
- End receipts with confirmed=false, provider failure, missing callbacks, and restart
|
||||
retain the exact-call obligation without redial or silent deletion.
|
||||
- Bound guest termination has one explicit owner and settles admission history.
|
||||
- Retention cannot erase unfinished provider cleanup.
|
||||
- Backlog, failure isolation, drain/resume, and shutdown behave as claimed.
|
||||
- Existing full voice, reply, and call-report cases continue to pass. Use separate
|
||||
PostgreSQL databases for suites that retain fixtures and global-idle suites.
|
||||
|
||||
## Parent and full-stack obligations
|
||||
|
||||
#17578 is published and attached. Local evidence covers 786 selected tests,
|
||||
workspace typecheck and build. CI run 40179014816 and PR review 40179014437 were
|
||||
live at publication; no check failure was present at the latest inspection.
|
||||
Latest-head checks, Greptile score and review findings still require acceptance.
|
||||
No stack PR is authorized for merge by the active goal. The remaining original
|
||||
ledger, older #17490 failure, and final master reconciliation remain required.
|
||||
|
||||
|
||||
## Atomic handoff checkpoint
|
||||
|
||||
The parent reply review repairs are published at `90caaf3fa929141a13cc17f54d6be846d6e93217`
|
||||
and merged into this branch. Both voice workers now share the scoped deadline
|
||||
mechanism; the next inbound worker should reuse it rather than copy a third loop.
|
||||
Parent CI run 40180022971 is live; no failure was present at the latest inspection.
|
||||
Its old 439d head passed CI/Greptile, but fresh-head acceptance remains required.
|
||||
|
||||
A real-service regression paused after inbound session reservation committed but
|
||||
before its result reached `decide`, then advanced beyond the admission expiry.
|
||||
The original sweep called provider termination on that authorized session. This
|
||||
failed in `/tmp/sleep23-handoff-before.log`. The reservation now writes admission
|
||||
state=approved and sessionId in the same transaction as insertion or guest
|
||||
promotion. `decide` reads that binding after start, rather than creating it in a
|
||||
second transaction. No provider call is moved inside the transaction.
|
||||
|
||||
Both ordinary admission and guest-to-private promotion are covered. All 110 voice
|
||||
integration cases pass on a fresh PostgreSQL database, and server typechecking
|
||||
passes. An initial full run passed 109 and failed the guest test's assertion of
|
||||
no termination calls globally: advancing ten minutes also expired unrelated
|
||||
retained fixture admissions. The assertion now checks the exact provider call
|
||||
whose handoff is under test, preserving legitimate unrelated cleanup. The full
|
||||
rerun passes. No broad workspace/remote verification is claimed for this child.
|
||||
|
||||
This is a prerequisite, not the complete cleanup migration. Still required:
|
||||
- Recheck and claim a discovered admission under its row lock before provider I/O;
|
||||
atomic new handoff alone does not protect a stale discovery snapshot.
|
||||
- Reconstruct legacy interrupted handoffs written before this atomic change.
|
||||
- Define one termination owner for bound guest sessions, and settlement signals.
|
||||
- Preserve unconfirmed/failed termination debt and credential/account isolation.
|
||||
- Prevent history retention from deleting unfinished cleanup.
|
||||
- Wire all producer notifications, actual deadlines, scoped worker registration,
|
||||
lifecycle tests and remove only the cleanup portion of periodic reconciliation.
|
||||
- Keep history retention and call reports as separate forthcoming PRs.
|
||||
|
||||
|
||||
## Cleanup implementation and local acceptance
|
||||
|
||||
The worker is now registered in app startup. `reconcileMaintenance` no longer
|
||||
runs inbound cleanup; it retains history pruning and reports for separate PRs.
|
||||
See [the implementation contract](../idle-voice-inbound-cleanup.md) for the complete
|
||||
producer table, ownership, credential rotation and remaining host-sleep limits.
|
||||
|
||||
The previous checkpoint's remaining implementation items are now handled:
|
||||
locked current-state claims; matching legacy handoff repair; bound-session cleanup
|
||||
ownership; durable termination confirmation/retry; account isolation; retention
|
||||
protection; notifications and shared scheduler registration. A company-delete
|
||||
participant check also prevents cascades from erasing unfinished cleanup and locks
|
||||
endpoints to serialize against signed admission ingress.
|
||||
|
||||
The existing admission row needs separate authorization and cleanup state. Three
|
||||
nullable fields record confirmation, retry and the signed provider account.
|
||||
Migration 0392_lying_trish_tilby.sql was generated with `pnpm db:generate`, made
|
||||
idempotent, and given a concurrent partial index. It backfills only provable old
|
||||
termination. The initial two-field unpublished generation was replaced by this
|
||||
single migration after account-rotation analysis. Five newest snapshots remain;
|
||||
the generated deletion of 0383 is intentional.
|
||||
|
||||
Local evidence:
|
||||
- `voice-inbound-cleanup.test.ts`: 13 tests pass using a separate PostgreSQL DB.
|
||||
Includes migration backfill/replay, company deletion and shutdown joining.
|
||||
- Full voice integration and app wiring: 129 tests pass on a fresh DB (113 voice,
|
||||
16 app wiring). Signed ingress wakes an idle worker; a feature-disabled instance
|
||||
retains an unconfirmed end; a replacement account cannot issue cleanup; a key
|
||||
for the saved account can. Rejection failure survives history pruning.
|
||||
- Company deletion and server startup consumers: 52 tests pass on their isolated DB.
|
||||
- Parent live-session recovery: 11 tests pass on its separate DB.
|
||||
- Total selected coverage: 205 passing cases across these nonoverlapping suites.
|
||||
- `pnpm -r typecheck` and `pnpm build` pass on the current implementation.
|
||||
- Migration safety/numbering checks passed for the initial generated migration;
|
||||
rerun on the final regenerated name before publication. Initial application and
|
||||
replay succeeded on real PostgreSQL. Final generation also applied cleanly;
|
||||
its backfill is exercised twice by the lifecycle test.
|
||||
|
||||
Earlier handoff regression and fixture assertion failures remain recorded above.
|
||||
The repository-wide local test command still has the previously recorded embedded
|
||||
PostgreSQL host limitation; this is not a claim that that command completed. Fresh
|
||||
remote CI and review are required before accepting the PR.
|
||||
|
||||
Parent #17578 at 90caaf3fa9 has all checks terminal with no failures and Greptile
|
||||
5/5 on that exact head. Both requested findings are fixed/resolved. The fresh
|
||||
Paperclip reviewer stopped before completing its turn; do not report a new 5/5
|
||||
assessment from that reviewer. The full original migration ledger, #17490 failure,
|
||||
and final stack/master reconciliation remain outstanding.
|
||||
|
||||
## Migration runner repair after first CI
|
||||
|
||||
The initial head 4b4bf2d8af failed database setup. Greptile identified that the
|
||||
concurrent 0388 index was missing from the migration runner's top-level set.
|
||||
CI production bundle boot and Runner static integration logs independently show
|
||||
`CREATE INDEX CONCURRENTLY cannot run inside a transaction block`. The earlier
|
||||
SQL application/replay evidence bypassed this application runner and did not
|
||||
establish that fresh setup or upgrades could work.
|
||||
|
||||
Added a real-PostgreSQL migration regression that creates a disposable database,
|
||||
boots all migrations through `applyPendingMigrations`, then simulates an invalid
|
||||
concurrent index and replays through two concurrent application migrators. It
|
||||
failed with the exact transaction error before the fix. Registering 0388 in the
|
||||
existing top-level set makes fresh boot, invalid-index repair, single journal
|
||||
record, and repeat replay pass. No migration SQL/history was rewritten.
|
||||
Database typecheck and migration numbering/safety checks also pass. Local logs:
|
||||
`/tmp/sleep23-migrator-before.log`, `/tmp/sleep23-migrator-after.log`, and
|
||||
`/tmp/sleep23-migrator-types.log`. Fresh remote checks remain required; the
|
||||
original CI failure is not classified as a flake.
|
||||
|
||||
## Denial request latency review repair
|
||||
|
||||
Review finding 4370459974 correctly identified that denial awaited account
|
||||
verification, provider inspection and endSession, despite already committing the
|
||||
rejection and its cleanup notification. The request now returns the saved denial;
|
||||
the registered worker owns physical termination and persists its retry. No detached
|
||||
promise or second dispatch path is added. Per-attempt account verification stays
|
||||
with the provider boundary, protecting rotated credentials from a different
|
||||
account. This patch does not cache provider authority indefinitely.
|
||||
|
||||
A new real-producer regression holds verifyAgent unresolved, proves the committed
|
||||
denial returns, checks that cleanup is still unconfirmed with a saved retry, then
|
||||
releases the provider and verifies confirmed termination. It failed before the
|
||||
fix (`/tmp/sleep23-denial-before.log`) and passed after
|
||||
(`/tmp/sleep23-denial-after.log`). Existing explicit one-shot tests now invoke
|
||||
that one-shot path rather than relying on the HTTP decision to do cleanup.
|
||||
All 130 full voice and app wiring cases pass on a fresh migrated PostgreSQL DB;
|
||||
production server typecheck passes. Logs: `/tmp/sleep23-denial-acceptance.log`
|
||||
and `/tmp/sleep23-denial-types.log`.
|
||||
|
||||
Finding 4370459800 about company deletion remains separately unresolved. Missing
|
||||
credentials and account mismatch retain cleanup debt. There is a tested recovery
|
||||
path through restoring the original account, but a permanently unavailable
|
||||
provider needs a deliberate ownership/product policy. A retry cap, elapsed time,
|
||||
or ambiguous 404 is not evidence that a billable call ended. Do not manufacture
|
||||
cleanup confirmation or silently discard the obligation to unblock deletion.
|
||||
|
||||
## Retrying an interrupted approval (2026-10-27)
|
||||
|
||||
Older versions could commit a private session or guest promotion before writing
|
||||
the admission binding. Retrying approval then found the existing session and
|
||||
returned before the new atomic binding helper ran. The inbound API rejected the
|
||||
still-approving admission with “This call ended during approval.”
|
||||
|
||||
Reservation now validates the inbound admission before the existing-session
|
||||
return. It uses the existing session authorizer to check current liveness,
|
||||
expiry, generation, credentials, caller and task access, then checks the exact
|
||||
admission/session identity before binding in the same transaction. Lock order
|
||||
remains endpoint, connection, admission, session. A matching provider session ID
|
||||
alone cannot establish approval. Already approved idempotent calls remain valid
|
||||
and do not rewrite the binding. Browser and outbound reservation behavior stays
|
||||
with the existing path. This adds no table, timer or recovery wrapper.
|
||||
|
||||
Both direct and promoted-guest retry regressions failed through the real inbound
|
||||
API before the correction. They now return the same session without starting or
|
||||
ending another provider call. Ended, expired and token-changed sessions are
|
||||
rejected without binding. Existing atomic handoff and approval-authority cases
|
||||
also pass. An initial negative expiry fixture violated the database timestamp
|
||||
constraint; it was corrected to represent a valid session that has expired.
|
||||
All 132 cases across the complete voice-session and inbound-cleanup suites pass
|
||||
on separate fresh databases, and server typechecking passes. Downstream
|
||||
integration and latest-head CI remain required. The separate company-deletion
|
||||
cleanup findings are not resolved by this correction.
|
||||
|
||||
## Fairness-test clock follow-up (2026-10-27)
|
||||
|
||||
CI's backlog test observed two provider failures where it expected one. A controlled
|
||||
30 ms provider delay reproduced the same failure: the assertion callback advanced
|
||||
fake time by one second on every retry, in addition to Vitest's ordinary clock
|
||||
advancement, reaching the failed provider's retry while the original backlog was
|
||||
still settling through PostgreSQL. Removing that extra clock jump makes the same
|
||||
controlled case pass. The five-second assertion budget and all product code remain
|
||||
unchanged. With the temporary provider delay removed, all 13 lifecycle tests pass.
|
||||
This repairs test clock ownership; it does not weaken provider retries or delivery
|
||||
fairness. Current-head CI remains required.
|
||||
|
||||
|
||||
## Current-parent acceptance and bounded discovery, October 28
|
||||
|
||||
This revision normally merges reply parent `d2c0a7de8c3e9cac8382b1268f3f5d883a2067b3`.
|
||||
It keeps the parent's paged shared worker, failure isolation, denial restoration,
|
||||
and publication-scoped reply signals. It adds no new shared scheduler contract.
|
||||
Domain SQL and termination decisions remain in the inbound voice owner.
|
||||
|
||||
The previous cleanup `load` read every page before yielding. An actual 105-row
|
||||
pending-admission fixture failed its first-pass budget: 105 selected rows instead
|
||||
of 100. The registered worker now reads one page per scope refresh and returns
|
||||
`{ rows, complete }` through the parent's existing contract. The explicit one-shot
|
||||
`load` still returns complete inventory for its foreground callers. Each query is
|
||||
limited to 100 rows. Refresh cursors advance only after a successful, non-aborted
|
||||
read. Registration resets them when the same service owner restarts.
|
||||
|
||||
The new lifecycle case stops after the first page of 105 future admissions,
|
||||
re-registers the same service, reconstructs the original expiry, and confirms
|
||||
all 105 terminations. It checks no SQL or provider calls before the deadline and
|
||||
no recurring work after settlement. Removing the registration cursor reset fails
|
||||
at the real restarted read (5 rows instead of 100). These measurements cover
|
||||
selected rows, not physical PostgreSQL tuple visits or deployed resource cost.
|
||||
|
||||
The fresh external fixture first failed because it assumed the admission schema
|
||||
already existed. It now runs the application migrator before setup, with the
|
||||
standard embedded-database setup budget. This repairs setup ownership; it does
|
||||
not skip any cleanup or authority assertions.
|
||||
|
||||
### Migration lineage and bounded historical backfill
|
||||
|
||||
The parent includes native checkpoint migration 0394. The old, unmerged inbound
|
||||
0392 migration cannot overwrite that history. The complete parent migration tree
|
||||
and client were restored, then the existing schema edit was regenerated with
|
||||
`pnpm db:generate`. The new migration is `0395_married_the_hand.sql`. Its generated
|
||||
snapshot links the parent snapshot. All 393 historical SQL files and all prior
|
||||
journal entries are byte-for-byte preserved. The five retained snapshots are
|
||||
0391 through 0395; the generator's required deletion of 0390 is included.
|
||||
|
||||
The nullable columns and partial index remain replayable. The concurrent index
|
||||
and batch backfill use the application's existing top-level migration protocol.
|
||||
The original unbounded historical UPDATE is now an indexed keyset scan of at
|
||||
most 1,000 unconfirmed admission IDs per transaction. It commits each batch.
|
||||
The first page includes the nil UUID. Each batch retains the original ended-state
|
||||
and same-company signed termination-evidence predicate. It does not confirm a
|
||||
call from its label alone. No new queue table or temporary production index is
|
||||
introduced. The existing activity entity index supports the evidence lookup.
|
||||
|
||||
The real migrator regression creates and drops its own database. It covers fresh
|
||||
migration, invalid concurrent-index repair, two simultaneous replays, and a single
|
||||
migration receipt. It confirms 1,007 eligible historical rows across multiple
|
||||
actual transaction IDs, with at most 1,000 updates in one transaction. Denied,
|
||||
pending, expired, wrong-company and wrong-action evidence stay unconfirmed. A
|
||||
prior confirmation remains unchanged. Reapplying the receipt adds no updates.
|
||||
The transaction probe exists only in that disposable test database. Restoring
|
||||
the old unbounded UPDATE makes the test fail: one transaction instead of at least
|
||||
two. Snapshot/schema drift acceptance also passes.
|
||||
|
||||
### Final local evidence
|
||||
|
||||
266 distinct cases pass on this combined source:
|
||||
|
||||
- 122 complete voice-session integration cases, using an initially empty test DB.
|
||||
- 142 cases across nine complete inbound cleanup, voice recovery, company removal,
|
||||
shared deadline/coordinator/batching, startup and hostname suites. Cleanup and
|
||||
parent recovery use separate initially empty DBs. Inbound cleanup uses its
|
||||
dedicated DB after the fresh-setup repair; all 14 lifecycle cases pass.
|
||||
- Two complete database migration and schema-snapshot cases.
|
||||
|
||||
Database and server typechecking and builds pass. Migration numbering/safety
|
||||
checks pass. The complete voice suite retains both interrupted-approval retry
|
||||
regressions. Removing only the existing-session binding makes those real API
|
||||
requests reject instead of resolve; the source was restored after the probe.
|
||||
The bounded-backfill and restart probes were also restored before final acceptance.
|
||||
|
||||
Commands use `pnpm --filter @paperclipai/server exec vitest run` for the ten named
|
||||
server suites, `pnpm --filter @paperclipai/db exec vitest run` for the two database
|
||||
cases, and each affected package's `typecheck` and `build` scripts. Evidence logs:
|
||||
`/tmp/sleep23-final-voice-consumers.log`,
|
||||
`/tmp/sleep23-final-lifecycle-consumers.log`,
|
||||
`/tmp/sleep23-current-migration-acceptance.log`, and
|
||||
`/tmp/sleep23-current-migration-lineage-proof.json`. The three guarded negative
|
||||
probes have `/tmp/sleep23-*-mutant-red.log` logs. Earlier failed fixture setup,
|
||||
unbounded inventory, and migration failures remain recorded above and in
|
||||
`/tmp/sleep23-empty-fixture-red.log` and `/tmp/sleep23-page-budget-red.log`.
|
||||
|
||||
The older 130/132/205-case and full-workspace claims apply only to their stated
|
||||
revisions. No new full local repository suite, live provider, Cloud alarm,
|
||||
instance/database suspension, or savings claim is made. Latest-head review,
|
||||
all CI, and parent landing remain required. Company deletion still refuses to
|
||||
erase unconfirmed cleanup; an audited tombstone or override requires a separate
|
||||
product and ownership decision.
|
||||
|
||||
|
||||
The module-boundary check passes. The full local Node policy scan reports two
|
||||
ignored, generated Copilot distribution manifests without an engine field after
|
||||
the successful package builds. Neither file is tracked or changed by this PR.
|
||||
The unchanged checker passes on a temporary copy of all its tracked source inputs
|
||||
(all manifests, Dockerfiles, workflows and required policy fragments). The initial
|
||||
full-tree failure remains `/tmp/sleep23-final-policy.log`; the source-input check
|
||||
and its complete input list are `/tmp/sleep23-final-tracked-policy.log` and
|
||||
`/tmp/sleep23-tracked-policy-proof.json`. No checker, workflow, generated vendor
|
||||
manifest, or ignore rule was changed to obtain this result. Remote CI remains a
|
||||
separate gate.
|
||||
|
||||
### Legacy terminal admissions missing their session binding — October 28
|
||||
|
||||
Ponytail found that an old interrupted approval could leave an admission `ended` or `expired` with a NULL session binding even though a matching provider session exists. The approval-only repair rejected that state forever, leaving cleanup unconfirmed and the company-deletion guard armed. Five new regressions reproduced the ownership conflict on the reviewed source before the repair.
|
||||
|
||||
Cleanup now handles confirmed terminal sessions before approval repair. It copies the existing termination timestamp without granting approval or issuing another provider request. Rejected admissions with a matching active inbound session retain their rejection, persist the missing binding and hand the leg to live recovery. Persisting the binding also lets the subsequent session-scoped confirmation wake find the admission. A conflicting existing binding, mode, idempotency key, generation, credential fingerprint or tool-token digest cannot authorize that handoff. Five negative regressions retain cleanup debt and leave the unrelated session active.
|
||||
|
||||
The complete cleanup lifecycle suite passes 24 cases, including quiet after settlement, current ownership, rejected-state preservation, matching terminal confirmation and conflicting identities. The complete voice-store integration suite passes 122 cases on a separate initially empty PostgreSQL 17 database. These are 146 distinct current cases; the isolated transcript probe is not counted again. Production server typecheck and build pass. No local full-repository, real provider, Cloud wake or host-suspension claim is made.
|
||||
|
||||
Failure history: the first regression invocation set the generic database variable rather than this suite's explicit variable and skipped all cases; it is not a pass. The corrected invocation executed and failed all five intended ownership regressions. The first repaired full cleanup run passed 22 and failed two new test calls to low-level chat deletion because retained voice sessions still have endpoint foreign keys. The tests now exercise the owning cleanup deletion guard (409 before cleanup, settled after confirmation); production deletion ordering is unchanged. Starting another disposable PostgreSQL server failed because the host's semaphore capacity was exhausted. The already-owned PostgreSQL server was verified available and reused with new isolated test databases; no shared IPC resources were removed.
|
||||
|
||||
The first complete voice-store run passed 121 cases and timed out at the unchanged 15-second limit in `retains both spoken sides once on the public call task and preserves quarantine`. Its isolated fresh-database probe passed. One subsequent complete run, with builds finished, passed all 122 cases without source or timeout changes. Recorded transform time was 59.09 seconds for the failed full invocation and 34.65 seconds for the passing full invocation; this does not establish the timeout's underlying cause. The earlier timeout remains part of the verification record.
|
||||
|
||||
### Landed-parent integration and regenerated migration — October 28
|
||||
|
||||
Reply delivery #17578 landed at `e93732c3d49f513845543a6c5d61054950969923`. This normal merge uses actual master `e7ea2e2961863171956e70fb05d3d1f3790f7909`, including the unrelated eval ingestion change. The cleanup owner, voice-store consumer suite and inbound admission schema are byte-identical to tested repair `7e4588fa8c`; the cleanup fixture changes only its migration filename. All unrelated master changes remain present.
|
||||
|
||||
Master acquired `0395_tranquil_firestar.sql` while this stack was open. Its SQL, every other landed SQL file and the parent journal prefix are preserved. Running `pnpm db:generate` produced `0396_tan_paladin.sql` and its new snapshot. The previously reviewed idempotent cleanup SQL, concurrent index and bounded historical backfill are retained byte for byte under that generated filename; application registration and both migration fixtures now reference 0396. Exactly snapshots 0392 through 0396 remain, with 0391 pruned by the existing command.
|
||||
|
||||
On the combined source, database build and numbering/safety checks pass. The complete 24-case cleanup lifecycle suite passes on a new empty database. The separate database migration integration case passes, including fresh application boot, interrupted concurrent-index repair, serialized replay and the real transaction-bounded backfill. Production server typecheck and build pass again after integration. The prior 122-case voice-store pass is carried only for its unchanged consumer and cleanup source; it was not rerun merely for the new migration filename. These checks do not establish deployed sleep or provider behavior. Fresh published-head CI and reviews remain separate merge gates.
|
||||
|
||||
### CI connected-tool rate-limit clock repair — October 28
|
||||
|
||||
Published head `48d8356e73` received exact-head Greptile 5/5. Its replacement CI run 40692550133 failed server shard 9/12 (job 122830940435): 1,610 cases passed, two skipped, and the connected MCP policy/approval/rate-limit acceptance case failed with `Expected connected MCP call to be rate limited`. This is a separate real failed check, not the cancelled duplicate workflow run 40692549975 created during retargeting.
|
||||
|
||||
The policy owner uses fixed 60-second buckets. The existing acceptance test performed its first and second calls against the real clock, so a legitimate window change could make the second call allowed. A controlled minute rollover between those calls reproduced the exact failing assertion on unchanged production code; the diagnostic source was restored. The CI log does not directly record the two bucket timestamps, so this reproduction establishes the test's unsafe timing assumption without claiming an independently captured clock trace for the CI attempt.
|
||||
|
||||
The existing acceptance case now freezes only `Date` around the rate-limit assertions, preserves the second-call 429 and persisted invocation assertions, then advances one window and requires a new call to succeed. A `finally` restores the real clock; network and timeout timers remain real. The complete 92-case gateway acceptance suite passes on a dedicated database. A subsequent whitespace-only correction removed an introduced blank-line indentation and changes no tokens. No production limiter, policy, credential, timeout, workflow or assertion was weakened. This additional existing test fixture is the only new source path outside the cleanup slice and fixes the observed CI gate. Fresh CI and exact-head review are required again after publication.
|
||||
@@ -0,0 +1,339 @@
|
||||
# Inbound voice history retention sleep audit
|
||||
|
||||
Original baseline: inbound cleanup PR #17580, head
|
||||
4b4bf2d8af6e08d3859ada4b720e7fbd3ea27e24.
|
||||
Branch: codex/sleep-24-voice-inbound-retention.
|
||||
Current integrated parent: b8b12199b9185e6318319cceb9511f5285734e41.
|
||||
Status: implementation and local acceptance complete; current-head CI/review and parent landing remain required.
|
||||
|
||||
## One reason
|
||||
|
||||
Remove the remaining recurring inbound history delete. Keep call reports in the
|
||||
next separate slice. `voiceInboundService.pruneHistory` currently runs from
|
||||
`voiceSessionService.reconcileMaintenance` in chat's periodic maintenance lane.
|
||||
|
||||
The existing retention policy deletes only denied, expired or ended admissions
|
||||
with no session or intake task, confirmed provider cleanup, and both expiresAt
|
||||
and updatedAt at least seven days old. Preserve that policy. The actual deletion
|
||||
deadline is max(expiresAt, updatedAt) + seven days. Never treat unconfirmed cleanup
|
||||
or a terminal UI label as deletable history.
|
||||
|
||||
## Owners and producer boundary
|
||||
|
||||
The existing admission rows and cleanup confirmation remain truth. No new table
|
||||
or row fields are needed. Reuse the scoped deadline worker and register a separate
|
||||
retention queue. Keep state selection, deadline calculation and deletion in the
|
||||
voice owner. Startup/app wiring only registers the owner.
|
||||
|
||||
All supported admission mutations already use notifyVoiceInbound directly or the
|
||||
voice-change composition. Extend that boundary to signal the separate retention
|
||||
queue before writes. This includes signed terminal callbacks, cleanup confirmation,
|
||||
legacy binding repair, approval/session handoff, expiry and rejection. Bound rows
|
||||
remain ineligible; their history belongs to their other owners. Company deletion
|
||||
can remove confirmed unbound history within its existing transaction.
|
||||
|
||||
At deletion time, lock and recheck the row and deadline. A cached candidate must
|
||||
not delete a row that acquired a binding, lost confirmation, or had its updatedAt
|
||||
moved by a committed write. Preserve fair paging, coalescing, backoff, drain and
|
||||
shutdown through the existing shared mechanism. Own deletes should retire their
|
||||
cached entries without starting another periodic scan.
|
||||
|
||||
## Required verification
|
||||
|
||||
- Configured idle and no eligible history become SQL-quiet.
|
||||
- Confirmed retained history schedules the exact seven-day deadline and does not
|
||||
query before it; restart reconstructs that same deadline.
|
||||
- Real cleanup confirmation and signed callbacks register retention through their
|
||||
transaction boundary; rollback/outer commit behave correctly.
|
||||
- Unconfirmed cleanup and bound history are never deleted.
|
||||
- A changed deadline/binding between discovery and deletion is rechecked.
|
||||
- Paged backlog, first write during drain, failures, and shutdown retain work.
|
||||
- Existing voice, cleanup, deletion and startup consumers pass.
|
||||
- Remove only history pruning from recurring maintenance; call reports remain.
|
||||
|
||||
No Cloud wake or sleep eligibility expansion is included. The external scheduler
|
||||
must cover a deadline before that deadline can justify stopping the host.
|
||||
|
||||
## Stack state
|
||||
|
||||
#17580 is published and attached; 205 selected tests, workspace types/build and
|
||||
migration checks passed locally. Initial CI run 40181994295 and review run
|
||||
40181994134 are live on its exact published head. It is mergeable and had no
|
||||
failures at inspection. Fresh checks/reviews remain required; no merge authority
|
||||
is implied. #17578 at 90caaf3fa9 has passing checks and exact-head Greptile 5/5;
|
||||
its latest Paperclip reviewer stopped before completion.
|
||||
|
||||
The full original recurring-server ledger, older #17490 E2E failure and final
|
||||
stack/master reconciliation remain required.
|
||||
|
||||
## Implementation and initial evidence
|
||||
|
||||
The dedicated voice retention owner now loads eligible deadlines, rounds SQL
|
||||
submillisecond dates up, and locks/rechecks before deletion. The existing producer
|
||||
helper signals both voice admission cleanup and retention. App wiring registers
|
||||
the owner; voice maintenance no longer calls `pruneHistory`. Reports remain in
|
||||
that lane, and manual diagnostic reconciliation remains compatible.
|
||||
|
||||
The initial ten PostgreSQL lifecycle cases had one invalid assertion: an open
|
||||
transaction was expected to have no coordinator deadline. Intent reconciliation
|
||||
correctly retains a deadline until settlement. The test now checks invisible
|
||||
uncommitted state and the row still present within its owning transaction; it
|
||||
requires deletion only after commit and quiet after rollback. All ten then passed.
|
||||
An additional real concurrent binding-lock regression and production service
|
||||
confirmation tests are being verified. Full voice acceptance uses a fresh database
|
||||
created through the repaired application migrator, not direct SQL replay.
|
||||
|
||||
Parent #17580 required repair 4f0364d269: register migration 0388 as top-level
|
||||
because of its concurrent index. A new migration-runner regression reproduced the
|
||||
failure, then proved fresh setup, invalid-index repair and concurrent/repeat
|
||||
replay after the fix. This parent repair was merged into this branch; its fresh
|
||||
CI is running. See the parent's audit for preserved failure evidence.
|
||||
|
||||
## Acceptance evidence
|
||||
|
||||
- 11 retention lifecycle tests and 13 parent cleanup tests pass (24 total).
|
||||
The added concurrent-binding test holds the real row lock, commits a task
|
||||
binding while deletion waits, and proves retention rechecks and keeps the row.
|
||||
- Full voice integration and app wiring pass 131 cases on a fresh database.
|
||||
Two new service tests start retention while the admission is unconfirmed, then
|
||||
use a signed terminal callback or real rejection/provider cleanup to establish
|
||||
confirmation. Each observes its saved seven-day deadline, SQL quiet before it,
|
||||
deletion at it, and quiet afterwards.
|
||||
- The first acceptance run waited 120 seconds before rejection, exceeding the
|
||||
original call's approval window. That correctly rejected the test operation and
|
||||
left an unconfirmed fixture affecting two later global cleanup assertions. The
|
||||
corrected test checks pre-confirmation quiet for ten seconds and the full suite
|
||||
passes on a newly migrated database. Production expiry policy did not change.
|
||||
- Workspace typecheck and build pass. A final server typecheck covers the last
|
||||
added concurrent-binding test. Shared deletion/startup and parent recovery
|
||||
checks are tracked below when their current processes finish.
|
||||
- No repository-wide `pnpm test:run` success is claimed: the local embedded
|
||||
PostgreSQL initialization limitation from earlier slices remains. The selected
|
||||
database suites use isolated real PostgreSQL; fresh remote CI is required.
|
||||
|
||||
Logs: `/tmp/sleep24-retention-cleanup-final.log`,
|
||||
`/tmp/sleep24-voice-acceptance.log` (preserved initial failure),
|
||||
`/tmp/sleep24-voice-acceptance-fixed.log`, `/tmp/sleep24-workspace-types.log`,
|
||||
`/tmp/sleep24-workspace-build.log`, `/tmp/sleep24-final-server-types.log`.
|
||||
|
||||
Final local consumer checks pass: 52 company-deletion/startup tests, 11 parent
|
||||
voice recovery tests, and final server typecheck. Selected nonoverlapping test
|
||||
coverage totals 218 passing cases. The implementation and local verification are
|
||||
complete; remote PR CI/review and the rest of the original stack remain pending.
|
||||
|
||||
## Review repairs
|
||||
|
||||
The first published head 9c38ddfe53 passed all CI and Greptile 5/5. Paperclip
|
||||
Review found unnecessary endpoint-wide retention refresh on every new call and
|
||||
missing behavioral coverage of removal from the maintenance lane.
|
||||
|
||||
Pre-call admission now allocates its ID before insertion and registers the
|
||||
existing intent for that admission only. Retention lookup uses the primary key;
|
||||
new calls no longer scan the endpoint's retained history. Cleanup receives the
|
||||
same precise hint. Existing endpoint/company configuration changes retain their
|
||||
scope, and startup still reconstructs eligible history. No schema change.
|
||||
|
||||
Two service regressions verify the actual signed pre-call scope and drive the
|
||||
real chat coordinator/voice maintenance path with due, confirmed history. The row
|
||||
survives maintenance and is deleted by the retention worker. Temporarily restoring
|
||||
pruning to voice maintenance makes this latter test fail with the row missing;
|
||||
restoring the implementation passes. All 133 full voice/app wiring tests and
|
||||
server typecheck pass after the repair. Logs are /tmp/sleep24-review-acceptance.log,
|
||||
/tmp/sleep24-review-types.log, /tmp/sleep24-maintenance-mutation.log and
|
||||
/tmp/sleep24-maintenance-restored.log. Typecheck covers production sources; the
|
||||
server tsconfig excludes the test directory, whose behavior is verified by Vitest.
|
||||
Fresh remote checks/review are required on the repaired head.
|
||||
|
||||
Parent #17580 has all checks terminal and Greptile 5/5 at 4f0364d269. Its final
|
||||
Paperclip review added two findings: unresolved cleanup can block deletion, and
|
||||
rejection awaits provider calls. These require explicit disposition/repair at the
|
||||
parent; green checks do not resolve them. An arbitrary retry cap or credential
|
||||
failure is not proof that a provider call ended and must not silently discard the
|
||||
cleanup obligation. Preserve that requirement while assessing the deletion path.
|
||||
|
||||
## Denial worker propagation
|
||||
|
||||
The parent denial-latency fix is incorporated. Retention's real cleanup-receipt
|
||||
producer case now registers the cleanup worker and waits for its confirmed result,
|
||||
instead of assuming the decision request itself performs provider I/O. Both
|
||||
retention and denial regression cases are retained. The initial merge verification
|
||||
failed to parse unresolved conflict markers; no voice cases ran. After preserving
|
||||
both sides and completing the merge, all 134 voice/app cases passed on that still
|
||||
unused voice database. Logs: `/tmp/sleep24-denial-acceptance.log` (parse failure),
|
||||
`/tmp/sleep24-denial-acceptance-fixed.log` (134 pass).
|
||||
|
||||
|
||||
## Current-parent paging and restart acceptance, October 28
|
||||
|
||||
This revision normally merges cleanup parent `b8b12199b9185e6318319cceb9511f5285734e41`.
|
||||
It retains that parent's generated migration 0395, cleanup paging and approval
|
||||
repairs. The 13 original owned paths remain the complete slice. No database,
|
||||
shared worker, workflow, lockfile or provider protocol change belongs to retention.
|
||||
The plan merge preserved both dated histories. The automatic source merge added
|
||||
a duplicate DELIVERY_QUEUES test import; the first combined voice suite failed
|
||||
to parse before running any case. Removing only the duplicate import lets all
|
||||
126 full voice cases run and pass on the still-empty dedicated test database.
|
||||
That parse failure remains `/tmp/sleep24-final-voice-consumers.log`.
|
||||
|
||||
### Bound reconstruction without changing retention policy
|
||||
|
||||
The old retention reader performed all 100-row queries inside one load. A new
|
||||
105-row future-history fixture showed it selected 105 rows in startup's first
|
||||
pass, failing the 100-row budget. The worker now uses the parent's existing
|
||||
`{ rows, complete }` contract. It returns at most 100 rows per scope refresh and
|
||||
preserves the full explicit `load` API. It changes no seven-day calculation,
|
||||
submillisecond rounding, confirmation requirement or binding exclusion.
|
||||
|
||||
Per-scope cursors advance only after successful reads and reset on registration.
|
||||
The new case stops after the first 100 of 105 future records, starts the same
|
||||
service owner, checks another initial 100 rows, reconstructs the original deadline,
|
||||
and deletes all 105 rows at that deadline. It observes SQL quiet before the
|
||||
actual deadline and after settlement. Removing just the reset makes restart read
|
||||
five rows and fails the budget assertion. The guarded mutation was restored
|
||||
before the complete lifecycle acceptance. Row measurements are selected rows,
|
||||
not PostgreSQL tuple visits or deployed costs.
|
||||
|
||||
The external retention fixture also assumed a pre-existing schema. Its first
|
||||
empty-database run failed on missing companies/admission tables. It now applies
|
||||
the application migrations with the standard setup timeout. The first-pass
|
||||
budget then failed at the intended 105-versus-100 assertion before the reader
|
||||
repair. Both failures are retained in `/tmp/sleep24-empty-fixture-red.log` and
|
||||
`/tmp/sleep24-page-budget-red.log`.
|
||||
|
||||
### Final evidence and limits
|
||||
|
||||
280 distinct cases pass on the combined source:
|
||||
|
||||
- All 126 voice-session integration cases, including actual signed ingress,
|
||||
cleanup confirmation, scoped retention wake and maintenance-lane behavior.
|
||||
- All 154 cases in ten full retention, cleanup, parent recovery, company removal,
|
||||
shared deadline/coordinator/batching, startup and hostname suites. This includes
|
||||
12 retention and 14 cleanup lifecycle cases. Voice integration, parent recovery
|
||||
and company removal use separate initially empty databases and migrate themselves.
|
||||
Retention uses its dedicated database after the fresh-setup repair. Cleanup
|
||||
uses its already-migrated dedicated database. These suites do not share fixture
|
||||
state with a live application.
|
||||
|
||||
Server typecheck, build and the module-boundary check pass. Logs:
|
||||
`/tmp/sleep24-final-voice-consumers-fixed.log`,
|
||||
`/tmp/sleep24-final-lifecycle-consumers.log`,
|
||||
`/tmp/sleep24-final-current-types.log`, `/tmp/sleep24-final-current-build.log`, and
|
||||
`/tmp/sleep24-final-module-boundaries.log`. The passing initial 12-case retention
|
||||
run is `/tmp/sleep24-bounded-retention-first-green.log`; it is included in the
|
||||
154, not counted twice. The guarded restart failure is
|
||||
`/tmp/sleep24-cursor-reset-mutant-red.log`.
|
||||
|
||||
Older 218/133/134-case and workspace-wide results apply to their stated heads.
|
||||
No new complete local repository suite, live provider, Cloud wake, host/database
|
||||
suspension or savings claim is made. Independent writers still need an explicit
|
||||
wake integration. Call reports remain their own duty. Current-head reviews,
|
||||
all CI and earlier stack landing remain required; an older passing score or
|
||||
inherited local acceptance cannot replace these gates.
|
||||
|
||||
|
||||
The prior published head `daca270a99` had a failed server (4/12) check in run
|
||||
40557790710, job 122386307340. Its actual log records a runner shutdown signal
|
||||
and operation cancellation before the suite completed. This is incomplete
|
||||
acceptance, not a passing run or an established product flake. The trusted runner
|
||||
selection also had a cancelled check. The source failure log is preserved at
|
||||
`/tmp/sleep24-daca-server4-failed.log`. New-head CI must complete independently.
|
||||
|
||||
|
||||
## Configuration rescan review repair, October 28
|
||||
|
||||
Ponytail's current-head 4/5 review of `ffe9d47b7f` identified that completed
|
||||
connection/company/endpoint scans restarted after every configuration save.
|
||||
Paging bounded each burst but did not remove total repeated work. A real
|
||||
producer regression seeds 205 future confirmed admissions and performs five
|
||||
separate `toolAccessService.updateConnection` saves, allowing each scan to settle.
|
||||
It selected 1,025 admission rows before the repair and selects zero afterward.
|
||||
All original retention deadlines remain armed and unchanged.
|
||||
|
||||
`notifyVoiceInbound` still notifies cleanup for every supported scope. Retention
|
||||
now receives only admission-scoped writes. Startup retains its complete paged
|
||||
reconstruction. Configuration, authority and voice-session changes do not modify
|
||||
admission eligibility or its timestamps. Admission creation, confirmation,
|
||||
expiry, approval/binding and signed lifecycle writes retain their exact hints.
|
||||
The schema uses restrictive session/task foreign keys, not implicit SET NULL
|
||||
transitions that could create eligible history behind this boundary. Cascading
|
||||
endpoint/company removal creates no retained work. A previously cached deadline
|
||||
may expire once after a row has been removed; it cannot recreate history or start
|
||||
recurring reconciliation. Arbitrary SQL and other processes still require their
|
||||
own explicit wake contract.
|
||||
|
||||
Successful deletion now uses its returned row identity to settle the cached
|
||||
entry directly. A changed, bound or not-yet-due row still reloads current state
|
||||
so its deadline is preserved. The new real-delete case observes no post-delete
|
||||
SELECT. Guardedly removing that fast path reproduces the extra SELECT; the
|
||||
original source was restored. There is no new helper, queue, schema or scheduler
|
||||
contract.
|
||||
|
||||
All 282 distinct cases pass after this repair: 127 complete voice integration
|
||||
cases and 155 cases in ten full retention/cleanup/recovery/company-removal/shared/
|
||||
startup suites. These include 13 retention cases. The complete voice suite uses
|
||||
a fresh dedicated database; the lifecycle suites use their existing separate
|
||||
isolated databases. Server typecheck and build pass. The earlier 280 count belongs
|
||||
to `ffe9d47b7f`; these cases include that coverage and are not added to it.
|
||||
|
||||
Logs: `/tmp/sleep24-config-amplification-red.log` (1,025 rows),
|
||||
`/tmp/sleep24-review-final-voice-consumers.log` (127 pass and zero-row budget),
|
||||
`/tmp/sleep24-review-final-lifecycle-consumers.log` (155 pass),
|
||||
`/tmp/sleep24-review-final-types.log`, `/tmp/sleep24-review-final-build.log`, and
|
||||
`/tmp/sleep24-deleted-reload-mutant-red.log`. Fresh exact-head reviews and CI remain
|
||||
required after publication; the prior green CI and 4/5 review are not reused for
|
||||
the new commit. No complete local repository suite, live provider, Cloud wake or
|
||||
physical suspension claim is added.
|
||||
|
||||
## Current-master CI repair — October 28
|
||||
|
||||
After normal integration of landed cleanup, head `768c650365362b12b75ca6481f44468132cf599a`
|
||||
received exact-head Greptile 5/5 and Ponytail 5/5, but CI run `40697141057`
|
||||
failed. This is not accepted as green evidence. Server shard 7/12 job
|
||||
`122846941793` reported 1 failed test, 1,750 passed and 6 skipped: cleanup's
|
||||
backlog fairness case expected the other company in the first 25 provider
|
||||
calls even when its random admission ID placed it after the first 100-row
|
||||
inventory page. The secondary afterEach error retained the deliberately
|
||||
injected provider failure because the assertion interrupted settlement.
|
||||
|
||||
Giving the second company a fixed final UUID reproduced the same first-batch
|
||||
assertion on unchanged production scheduling. The fixture now uses its supplied
|
||||
ID for both the inserted row and transaction notification. The corrected case
|
||||
controls fake time, places that company on the second inventory page, checks it
|
||||
is absent before discovery and served within the next 25-call batch, completes
|
||||
all 105 successful calls before the failed provider's saved retry, and checks
|
||||
that failed provider runs exactly once again when its deadline arrives. It also
|
||||
checks idle SQL after settlement. No production worker, policy, deadline,
|
||||
assertion timeout, or workflow changes are made.
|
||||
|
||||
The first controlled-time repair run passed 23 cases but failed the fairness
|
||||
case by demanding idle active=0 between inventory passes. An unresolved
|
||||
transaction intent can legitimately retain that hold while the next pass is
|
||||
scheduled. The final test waits for the next scheduled pass or actual idle
|
||||
settlement while advancing one second at a time, then requires actual idle
|
||||
settlement after the backlog. This failure is retained rather than counted as
|
||||
a pass. A guarded mutation bypassing company rotation fails the corrected
|
||||
fairness assertion with missing second-company delivery; the original shared
|
||||
helper was restored byte-for-byte before final acceptance.
|
||||
|
||||
The same CI run's Canary Dry Run job `122846941357` independently failed
|
||||
installing the standalone Kubernetes plugin: npm returned 404 for the exact
|
||||
`rollup-4.66.0.tgz` requested by Vite through Vitest. At 13:39:46 UTC both that
|
||||
version metadata and exact tarball returned HTTP 200. No dependency pin,
|
||||
workflow exception, or blind CI retry was introduced. New-head CI, including
|
||||
the canary, must pass independently; the original 404 remains recorded.
|
||||
|
||||
Final local acceptance after restoring the guarded shared helper: both complete
|
||||
cleanup (24) and retention (13) lifecycle suites pass, 37 distinct cases total.
|
||||
Retention starts from a new empty dedicated PostgreSQL database and applies the
|
||||
application migrator; cleanup uses its isolated already-migrated database.
|
||||
The earlier 282-case evidence belongs to unchanged retention source at its
|
||||
stated head and is not added to 37. No new full local repository suite, live
|
||||
provider, Cloud wake, suspension, or cost-saving result is claimed.
|
||||
|
||||
Logs: `/tmp/sleep24-768-server7-job.log`, `/tmp/sleep24-768-canary-job.log`,
|
||||
`/tmp/sleep24-cleanup-fairness-red.log`,
|
||||
`/tmp/sleep24-cleanup-fairness-green.log` (the 23/24 failed intermediate run),
|
||||
`/tmp/sleep24-cleanup-fairness-final.log` (24 pass),
|
||||
`/tmp/sleep24-cleanup-fairness-mutant-red.log`,
|
||||
`/tmp/sleep24-ci-fairness-final-consumers.log` (37 pass), and
|
||||
`/tmp/sleep24-rollup-registry-probe.json` (current HTTP availability only).
|
||||
@@ -0,0 +1,297 @@
|
||||
# Voice maintenance sleep migration audit
|
||||
|
||||
Baseline: receiver parent #17572, head 643f4766227e15006b684d2cc7bea7ddd5cc2147.
|
||||
Branch: codex/sleep-21-voice-session-recovery.
|
||||
Status: implementation and local integration/lifecycle verification in progress; unpublished.
|
||||
|
||||
## Separate the reasons before changing the timer
|
||||
|
||||
The app invokes `voice.reconcile()` from the remaining one-second chat lane.
|
||||
The public service is created through `chat-channels/speko/runtime.ts`; the
|
||||
implementation is `voice/voice-session-service.ts`. Its reconcile method hides
|
||||
five distinct obligations. Each needs its own PR, preserving the others until
|
||||
its producer and deadline contracts are implemented.
|
||||
|
||||
| Slice | Existing behavior | Proposed owner and trigger |
|
||||
| --- | --- | --- |
|
||||
| Live session recovery (this branch) | Reads experimental settings, scans live sessions for expiry, ending, disabled endpoints/connections, generation mismatch and 30-second active-state checks | Voice session owner; committed lifecycle/authorization changes plus real expiry, cleanup and active-call inspection deadlines |
|
||||
| Reply push | `pushVoiceReplies` scans for provider delivery work | Existing reply/session records, committed source changes, pending provider delivery and claim/retry deadlines |
|
||||
| Inbound admission expiry and provider cleanup | Scans guest/approval/denied/expired calls past expiresAt, inspects/ends exact provider sessions | Inbound owner; admission/lifecycle/decision changes, saved expiry and unresolved end retries |
|
||||
| Inbound history retention | Deletes denied/expired/ended rows without sessionId or intakeIssueId once both expiresAt and updatedAt are seven days old | Retention owner; deadline max(expiresAt, updatedAt) + seven days, mutation invalidation and startup reconstruction |
|
||||
| Call reports | Fetches reports for terminal sessions with provider IDs within 24 hours of endedAt; follows nextCheckAt backoff | Report owner; terminal/lifecycle/report events and saved report recheck deadlines, preserving late recording/transcript behavior |
|
||||
|
||||
Removing one invocation of the outer reconcile method would drop several owners.
|
||||
First expose the live-session operation independently, register it with the
|
||||
existing coordinator, and leave the other four calls in their existing lane.
|
||||
The outer wrapper can disappear only after the remaining owners migrate.
|
||||
|
||||
## Live-session source of truth and invariants
|
||||
|
||||
Existing chat_voice_sessions, chat_endpoints, tool_connections, access/bindings,
|
||||
and instance settings define eligibility and current authority. No new table is
|
||||
needed. Current live states are reserved, creating, creation_unknown, connecting,
|
||||
active and ending. Terminal states are ended, failed and expired.
|
||||
|
||||
The current pass uses 25 oldest updatedAt rows. A session qualifies when expired,
|
||||
ending, disabled, generation-mismatched, or when a connecting/active session has
|
||||
not been inspected for 30 seconds. Active-session checks validate current task,
|
||||
caller and guest authority, then inspect the exact saved providerSessionId to
|
||||
recover a hangup without a final browser callback. A live call is concrete work;
|
||||
its bounded provider-state inspection is not empty-instance discovery.
|
||||
|
||||
Unknown creation remains reserved until expiresAt plus two minutes of teardown
|
||||
grace. The provider maximum duration is the final bound. Never replay creation
|
||||
because its response was lost. Ending retains exact provider cleanup debt until
|
||||
termination is confirmed. An error records cleanup_pending and updatedAt.
|
||||
Preserve these business boundaries and distinguish a database error from a real
|
||||
revocation before changing recovery behavior.
|
||||
|
||||
An inactive instance with no live sessions must stop this owner's settings and
|
||||
session queries after reconstruction. Retained terminal sessions alone must not
|
||||
create a live-session deadline. Report work for those rows belongs to its own
|
||||
slice and stays active under its existing contract for now.
|
||||
|
||||
## Required producer audit
|
||||
|
||||
- voice-session-store.reserve, including inbound and guest binding paths;
|
||||
- service start claim, successful provider binding, unknown creation, failed
|
||||
creation, concurrent end, and failure to persist provider identity;
|
||||
- closeProvider and user end; inspect and tool callbacks that change session state;
|
||||
- inbound lifecycle, guest intake, approval/denial and end paths that write sessions;
|
||||
- signed lifecycle callbacks and report callbacks (separate saved report debt);
|
||||
- endpoint/connection pause, removal, generation or credential changes;
|
||||
- feature disable, membership/access revocation, task reassignment, binding and
|
||||
environment changes used by authorizeSession;
|
||||
- restart/resume and the supported single active writer contract.
|
||||
|
||||
Use existing transaction ownership. Add intent before the write; transaction
|
||||
outcome uncertainty remains the shared database layer's responsibility. Prefer
|
||||
session/endpoint/company scopes rather than refreshing all sessions per mutation.
|
||||
Authorization stays with the voice store; provider requests stay with the Speko
|
||||
provider. Shared scheduling consumes normalized work identities and deadlines.
|
||||
|
||||
## Required proof
|
||||
|
||||
Reproduce recurring SQL in no-session and retained-terminal-history states before
|
||||
removal. Test first real session creation after quiet; nested transaction commit
|
||||
and rollback; uncertain creation without duplicate provider calls; expiry/grace;
|
||||
ending cleanup failure and retry; active hangup recovery; revoked membership,
|
||||
changed task binding and feature disable; guest isolation; multiple-company
|
||||
fairness and backlog; failed scoped reads; mutation during sweep; drain/resume;
|
||||
shutdown joining admitted provider work. Keep all four adjacent obligations and
|
||||
their existing tests running. Inspect `voice-session-store.integration.test.ts`
|
||||
and current producer fixtures before inventing new test contracts.
|
||||
|
||||
The receiver parent has 218 selected passing tests, workspace typecheck/build,
|
||||
and is published/attached. CI run 40173962833 is live on its exact head; Greptile,
|
||||
security and Paperclip review are pending. This is not stack completion or merge
|
||||
permission. The full original ledger, #17490 failure and final master
|
||||
reconciliation remain required.
|
||||
|
||||
|
||||
## Initial implementation checkpoint
|
||||
|
||||
Live-session recovery now has a voice-owned paged deadline reader and a scoped
|
||||
coordinator worker. The periodic wrapper retains reply, inbound, and report
|
||||
maintenance. Session writes signal their existing transactions; standalone service
|
||||
updates use one small transaction helper. Configuration, feature-disable and
|
||||
voice tool/access removal paths signal relevant scopes. Active sessions retain
|
||||
30-second inspection, unknown creation retains the saved expiry plus two-minute
|
||||
grace, and no call-creation action exists in recovery. Authority lookup failures
|
||||
retry instead of treating an unavailable database as confirmed revocation.
|
||||
|
||||
Four initial actual-service cases pass: first start after quiet and retirement
|
||||
on missed hangup, membership revocation, and feature disable; unknown creation
|
||||
expires at its saved grace deadline without another provider call. Server types
|
||||
pass after fixing the database projection of the internal runtime generation.
|
||||
The startup wiring regression failed before removal. This is a checkpoint only:
|
||||
full producer, lifecycle, race, backlog, and consumer evidence is still required.
|
||||
|
||||
Receiver #17572 has Greptile 5/5 and no failed checks, but Paperclip Review 4/5
|
||||
requests suppressing rediscovery after discovery-initiated stops. Its optional
|
||||
full-refresh coalescing observation will also be addressed in the owning parent
|
||||
before this branch is published. Preserve all unfinished voice changes in this
|
||||
commit while repairing that parent.
|
||||
|
||||
|
||||
## Expanded lifecycle checkpoint
|
||||
|
||||
Receiver review repairs are published in #17572 at `1604a9fcba` and merged into
|
||||
this child. The receiver CI run 40175242445 remains live, with no failing checks
|
||||
at inspection. The voice worker now also coalesces pending scopes under a full
|
||||
refresh and retains its backoff when narrower signals arrive.
|
||||
|
||||
The complete existing voice session integration suite passes all 93 tests on a
|
||||
fresh disposable PostgreSQL database. This includes the four initial real-service
|
||||
producer cases and the adjacent reply, inbound, callback and report consumers.
|
||||
Eight new recovery-owner integration tests pass: configured terminal history,
|
||||
outer rollback/commit, first write during drain, transient authority failure
|
||||
without ending a valid call, restart with the same saved deadline, mutation during
|
||||
full refresh, a 105-session paged backlog with one failed provider, and shutdown
|
||||
joining admitted inspection. Their provider operations are controlled; real
|
||||
service authorization/producer coverage remains in the 93-case suite.
|
||||
|
||||
The 16 startup wiring tests also pass. Initial new-suite failures were fixture
|
||||
errors (missing channel purpose) and a silence assertion starting before the
|
||||
committed cleanup_pending refresh had settled. Both were corrected; the final
|
||||
retry check proves no SQL during the actual backoff. No production failure was
|
||||
concealed or treated as a successful run.
|
||||
|
||||
Outstanding before publication: complete producer/authorization-change audit,
|
||||
current-state races, cross-company fairness and failed-scope coverage, affected
|
||||
shared producer consumers, full workspace verification and remote gates. The
|
||||
other four voice duties and every remaining original stack item remain required.
|
||||
|
||||
|
||||
## Producer and race audit completion
|
||||
|
||||
See [the live-session contract](../idle-voice-session-recovery.md) for the complete
|
||||
producer mapping, existing transaction ownership and retained active authorization
|
||||
inspection contract. All production session writers are covered. Phone preferences,
|
||||
membership and task authority remain checked on the existing active-call deadline
|
||||
and on use; no new recurring empty inventory scan replaces their absence of signals.
|
||||
|
||||
A late provider binding between grace-period discovery and expiration was reproduced:
|
||||
the previous update marked a connecting call expired despite its new provider ID.
|
||||
The expiration update now requires provider identity to remain absent. The regression
|
||||
proves the known call stays live for exact-provider cleanup. All 11 recovery lifecycle
|
||||
cases pass, including company fairness and failed-scope fencing.
|
||||
|
||||
Expanded real-service coverage passes for endpoint pause, generic voice connection
|
||||
disable, task reassignment and signed guest lifecycle settlement. Shared consumer
|
||||
verification passes 469 tool-access, 18 access-removal, and 207 settings/provider
|
||||
cases. Workspace typechecking passes. A fresh final voice/receiver/startup selection
|
||||
and the full local test command are running; build/publication/remote gates remain.
|
||||
No full local matrix result is claimed from targeted tests. Initial fixture errors
|
||||
and the reproduced provider-binding race remain recorded above and in test logs.
|
||||
|
||||
## Voice session inventory amplification repair, October 27
|
||||
|
||||
The voice branch normally integrates receiver parent #17572 at `841d451164`
|
||||
through `f8d7dd2d14`. The initial single-duty scope is 21 files, with no database
|
||||
changes. The health-observation routing repair adds one related production path,
|
||||
`tool-access/catalog/health.ts`, because an observation must not claim that voice
|
||||
configuration changed. The existing routing helper owns receiver and command
|
||||
fanout; the voice service owns eligibility and deadline comparison.
|
||||
|
||||
A real-API regression with 205 future live calls reproduced 1,230 selected call
|
||||
rows across five metadata updates and one health observation. The row count
|
||||
measures executed database selects, rather than timer calls or helper invocations.
|
||||
After repair these operations read zero call rows, preserve the exact saved next
|
||||
deadline, and issue no voice wake for the health-only operation. A separate real
|
||||
SDK health case confirms receiver and command topics still settle after commit.
|
||||
|
||||
The background path now uses one inventory page of at most 100 lean call headers
|
||||
per refresh. Parent state is probed separately, in bounded pages, and unchanged
|
||||
state keeps existing deadlines. Unknown or genuinely changed parent state has a
|
||||
bounded reconstruction. Foreground reconciliation keeps its complete scan API.
|
||||
The worker retains a generation and seen IDs across pages, processes at most 25
|
||||
scopes per pass, rechecks call state before provider effects, and preserves failed
|
||||
scope/call backoff. It resets partial state when the same service starts a new
|
||||
worker. No per-worker table, migration, extra scheduler, or route recovery wrapper
|
||||
is introduced.
|
||||
|
||||
Real database regressions cover all 205 deadlines after paging and partial-scan
|
||||
restart, a new session behind the cursor, a feature change between pages, and
|
||||
connection disablement settling all 205 calls. Guarded production mutations
|
||||
reproduce the page-size violation, lost global generation (cleanup delayed past
|
||||
its prompt-recovery assertion), and health-only wake. Each original file is
|
||||
restored after the mutation. These checks supplement the existing outer-commit,
|
||||
rollback, drain, current-authority, late-provider-binding and shutdown cases.
|
||||
|
||||
Both newly supported external fixture paths skipped migrations. Empty-database
|
||||
runs failed with missing instance_settings before the worker or provider action.
|
||||
The worker and complete voice-store fixtures now apply pending migrations before
|
||||
creating the client and use the shared PostgreSQL bootstrap budget. The initial
|
||||
amplification fixture also omitted its required Speko provider config, then used
|
||||
the wrong public health method name; those failures preceded the actual 1,230-row
|
||||
baseline. Initial quiet checks counted the observer's previous calls; measurement
|
||||
is now stopped before an independent quiet-window spy is created. The quiet
|
||||
assertion itself is unchanged. The first typecheck exposed timing-only rows being
|
||||
used for full provider operations; the timing-header type stays separate from full
|
||||
rows at that operation boundary. All failures remain recorded as such.
|
||||
|
||||
Current targeted acceptance passes 725 distinct cases: 18 voice lifecycle, 94
|
||||
shared coordinator/batching/startup/hostname, 97 complete voice-store, 469
|
||||
tool-access, and 47 health/receiver cases. Production typecheck and server build
|
||||
pass. The 18 lifecycle cases replace the earlier 16-case result; these counts do
|
||||
not add repeated runs. Full suites ran unless a narrower selection is explicitly
|
||||
named above. No full local repository matrix, actual host suspension, Cloud wake,
|
||||
multi-writer support, or savings is inferred from this worker migration. The original remaining-duty inventory stays
|
||||
in scope. Current exact-head reviews, security, CI, and parent landing are still
|
||||
required before merging #17576.
|
||||
|
||||
|
||||
## Landed-foundation acceptance, October 27
|
||||
|
||||
The voice repair now integrates receiver parent `6e88ca0201`, including Discord
|
||||
`547fdf2139` and master `f8563d2507`. Exact-patch comparison preserves the repaired
|
||||
22-file scope and master's shared task-drain and native-checkpoint queue names.
|
||||
No database file is part of this slice.
|
||||
|
||||
All 725 distinct targeted cases pass again: 112 complete voice/shared/startup
|
||||
cases, 97 complete voice-store cases, 469 complete tool-access cases, and 47
|
||||
complete health/receiver cases. Server production typecheck and build pass.
|
||||
Voice worker/store and health/receiver fixtures use separate initially empty
|
||||
self-migrating databases. Tool access uses a separate migrated clone. One initial
|
||||
tool-access invocation supplied the generic fixture variable instead of the
|
||||
suite-specific variable: 23 cases passed and 446 skipped. It was not a complete
|
||||
suite result or an assertion failure. The corrected invocation executes and
|
||||
passes all 469. Tests are not counted twice. Expected controlled failures and
|
||||
listener warnings remain in logs. Only documentation changes follow these checks.
|
||||
Fresh current-head reviews, CI and parent landing remain required. No full local
|
||||
matrix, provider acceptance, external wake or deployed sleep result is claimed.
|
||||
|
||||
|
||||
## Trusted-policy checkout compatibility, October 27
|
||||
|
||||
CI run `40632057899` on `712f3862b5` failed policy because the trusted workflow
|
||||
had acquired `scripts/check-query-error-rendering.mjs` from newer master, while
|
||||
the checked-out PR head lacked that script. The build job separately received
|
||||
a runner shutdown signal during runtime bundling, then reported cancellation.
|
||||
The underlying shutdown cause is not established. The E2E aggregate reports
|
||||
`E2E_SHARDS_RESULT=success` but fails on `POLICY_RESULT=failure`; verification
|
||||
also fails on policy and build. Those four failed gates are not passing results.
|
||||
|
||||
A normal merge of receiver `ab39569250` brings the landed UI and policy tooling
|
||||
into the checkout. Every server/package production and test file remains
|
||||
byte-identical to the 725-case tested `712f3862b5` source. The policy checker now
|
||||
executes successfully, and its complete Node test file passes locally. No sleep
|
||||
implementation, test assertion or workflow gate is weakened. Fresh CI must prove
|
||||
the full build and aggregate gates; the interrupted build is not relabeled as a
|
||||
pass. Fresh Ponytail and Greptile acceptance still precede any merge.
|
||||
|
||||
|
||||
## Wildcard failure and deterministic generation acceptance, October 28
|
||||
|
||||
Current-head CI `40633667141` fails server shard 1 at the feature-change test's
|
||||
34,050 ms result against its 25,000 ms bound; verification fails as an aggregate.
|
||||
This is an actual required assertion failure, distinct from the older interrupted
|
||||
build and missing policy script. The old test mixed automatically advancing fake
|
||||
time with real database latency. It now disables automatic clock advancement,
|
||||
drives 20 explicit scheduler turns, waits for each pass without advancing the
|
||||
clock, and requires every invalidated call to close before the unrelated periodic
|
||||
authority deadline. Removing generation replay still fails with missing closures.
|
||||
The test no longer benchmarks host/database speed or raises its business bound.
|
||||
|
||||
A real committed-session regression also reproduces wildcard starvation: while
|
||||
one broad reconstruction backs off, a newly ending exact call is not closed.
|
||||
The worker now retains the broad retry but considers ready exact hints and cached
|
||||
due calls independently. Each due call still reloads state and authorization in
|
||||
its owning check before effects. Failed exact scopes retain their own fence;
|
||||
the existing stale-cache failure regression still passes. Removing wildcard
|
||||
isolation fails the new exact-call assertion. Broad reconstruction becomes quiet
|
||||
after its successful retry, with no continuing SQL afterward.
|
||||
|
||||
The full-row execution lookup and lean inventory lookup now share the live-state,
|
||||
scope and cursor predicate. Their projections remain intentionally different.
|
||||
An initial edit script rejected the header cursor name before applying the timing
|
||||
and predicate changes; the intermediate 19-case result covers only the earlier
|
||||
worker repair. It is not the final repair evidence. The corrected source passes
|
||||
all 19 lifecycle cases and 191 complete voice-store/shared/startup cases: 210
|
||||
distinct targeted cases. Production server typecheck and build pass. Both guarded
|
||||
regressions fail at their intended effect assertion and restore source afterward.
|
||||
The fresh voice-store database self-migrates. Existing controlled failure logs
|
||||
and signal-listener warnings remain recorded. The older 725-case acceptance
|
||||
belongs to its stated revision and is not added to this count. Fresh current-head
|
||||
reviews and all CI remain required before merging after receiver discovery.
|
||||
@@ -0,0 +1,344 @@
|
||||
# Voice reply delivery sleep migration audit
|
||||
|
||||
Parent: live-session recovery #17576 at 8f5e861dbd6ea5c2b45684dff970ea27723a84cb.
|
||||
Branch: codex/sleep-22-voice-reply-delivery.
|
||||
Status: owner audit and implementation in progress; recurring reply scan remains.
|
||||
|
||||
## Obligation
|
||||
|
||||
`voice-session-service.reconcileMaintenance` invokes `pushReplies` every chat tick.
|
||||
The reply owner first reads settings, quarantines dispatch intents older than 30
|
||||
seconds, and scans up to 25 active phone sessions. Each session can dispatch up to
|
||||
eight chunks. A live phone call with no published reply still repeats this work.
|
||||
Browser calls retrieve updates through their own request path and do not use push.
|
||||
|
||||
Existing `chat_voice_replies`, `chat_publications`, `chat_voice_sessions` and
|
||||
`chat_actions` are the durable truth. The action kind `speko_voice_reply_push`
|
||||
records per-chunk dispatch and outcome. No new queue table is needed. Use one
|
||||
coordinator worker for this delivery obligation, preserving the existing ledger,
|
||||
provider authority, multipart ordering and uncertainty policy.
|
||||
|
||||
## Policy that must not be duplicated or weakened
|
||||
|
||||
- Only active, unexpired phone sessions can receive proactive speech.
|
||||
- The first eligible published undelivered reply controls ordering. An empty text
|
||||
or blocked earlier reply must not be silently skipped to speak later answers.
|
||||
- Synthetic progress is excluded unless it carries an authorized question.
|
||||
- A question notification is a barrier until authorized get_updates marks the
|
||||
reply presented. Accepted notification is not permission to speak later replies.
|
||||
- Multipart acceptance is recorded per part. A changed publication digest cannot
|
||||
append a different suffix to a partly delivered answer.
|
||||
- A dispatching/unknown outcome is never permission to resend. A dispatch intent
|
||||
becomes unknown only at its existing 30-second stale threshold. A known 429
|
||||
retries at its saved retryAt. Failed or uncertain historical actions alone must
|
||||
not create repeated empty delivery scans.
|
||||
- Accepted by the provider is not spoken playback. Preserve both distinctions.
|
||||
- Session authorization remains before claim and immediately before provider I/O.
|
||||
Do not move provider calls inside the transaction or invent a second claim owner.
|
||||
|
||||
Extract the existing multipart decision into a voice-owned pure policy before
|
||||
implementing deadline selection. The dispatcher and scheduler must share that
|
||||
policy rather than reproduce partially overlapping JSON/status predicates.
|
||||
|
||||
## Producers and the publication timing gap
|
||||
|
||||
The Speko publication transport calls `enqueuePublication` while the publication
|
||||
can still be streaming. It schedules a direct `pushReplies`, then the publication
|
||||
worker commits `state = published`. Therefore signaling only the reply insertion
|
||||
is insufficient: that wake can run before publication settlement and miss the
|
||||
reply becoming deliverable. Signal the published-state write in its transaction,
|
||||
with a normalized voice publication scope routed by the owning transport/service
|
||||
boundary. Audit the resolution/recovery published-state writer as well.
|
||||
|
||||
Other producers are live phone-session creation/promotion, missed-reply collection,
|
||||
reply retrieval/presentation, explicit repeat, session termination, action claim,
|
||||
retry/unknown/accepted settlement, and feature/configuration changes. Browser
|
||||
retrieval may unblock the next phone reply and needs a committed hint. Startup
|
||||
must reconstruct both deliverable replies and abandoned dispatch intents, including
|
||||
intents for sessions that are already terminal. Prior-call delivery barriers and
|
||||
missed-publication catch-up must remain with their existing voice-store owner.
|
||||
|
||||
## Required lifecycle and acceptance
|
||||
|
||||
The worker must become quiet for no sessions, active calls with no replies,
|
||||
terminal history, accepted speech, unknown/failed outcomes and presented question
|
||||
barriers. It must wake from each supported producer, retain exact retry/claim
|
||||
expiry deadlines, and preserve changes during refresh/drain. Verify transaction
|
||||
rollback/outer commit, restart, unfair backlogs, failures and shutdown joins with
|
||||
real PostgreSQL and controlled provider operations.
|
||||
|
||||
Keep live-session recovery, inbound admission cleanup, inbound history retention
|
||||
and call reports separate. Do not remove the outer maintenance call until each
|
||||
remaining reason has migrated. Keep manual push entry points compatible where
|
||||
needed, but route normal committed work through the coordinator.
|
||||
|
||||
## Parent verification record
|
||||
|
||||
#17576 is published and attached. Its 837 selected tests, workspace typecheck and
|
||||
build pass. CI run 40176142139 and review run 40176142016 were live at publication.
|
||||
The full local test command was deliberately stopped (session 77073, exit 143)
|
||||
after repeated embedded PostgreSQL initialization failures prevented suites from
|
||||
executing. It is not a full local pass; the PR description records this. Affected
|
||||
database suites did execute against separate disposable PostgreSQL databases.
|
||||
Receiver #17572 has passing latest-head checks and Greptile 5/5 at 1604a9fcba;
|
||||
its second Paperclip reviewer stopped before completion, so do not invent a fresh
|
||||
completed review result. No stack PR was merged. The complete original inventory,
|
||||
older #17490 failure and final master reconciliation remain outstanding.
|
||||
|
||||
|
||||
## Initial implementation checkpoint
|
||||
|
||||
The existing multipart selection now lives in voice-reply-policy.ts. The dispatcher
|
||||
uses that pure policy for chunk identity, digest consistency, question barriers,
|
||||
retry deadlines and no-resend outcomes. Its claim, authorization and provider I/O
|
||||
remain in the existing owner. Twelve selected real-database reply integration
|
||||
cases and server typechecking pass after extraction. This is a foundation only:
|
||||
no recurring reply scan has been removed, and no reply PR is published yet.
|
||||
|
||||
Next implement the shared voice reply-state reader and deadline selection, audit
|
||||
all session/reply/action and published-state producers, register the worker, and
|
||||
prove quiet/reactivation before retiring the recurring and direct normal dispatch
|
||||
paths. Preserve the public/manual push API and all adjacent voice maintenance.
|
||||
|
||||
The ordered reply-state reader is now also shared inside the voice owner. It
|
||||
preserves the caller's session lock, published-only first reply, question barrier,
|
||||
prior action locks and multipart policy. It exposes a saved dispatch-expiry
|
||||
boundary without replaying a dispatch. The same 12 delivery cases and server
|
||||
typechecking pass after this second extraction.
|
||||
|
||||
Implementation direction for the next step:
|
||||
|
||||
- Scoped reconstruction pages identities of active unexpired phone sessions and
|
||||
outstanding dispatching actions. A changed session may receive one immediate
|
||||
inspection; this is a startup/committed-write wake, not a recurring deadline.
|
||||
- Check each session independently: authorize and collect missed replies under
|
||||
the existing session transaction, then use readVoiceReplyState. No reply or a
|
||||
permanent barrier disarms it. A retry or in-flight intent keeps its real deadline.
|
||||
Provider dispatch remains outside the transaction, using the existing claim.
|
||||
- Keep action-expiry entries independent of session eligibility so terminal calls
|
||||
do not strand old dispatch intents. Use the existing strict 30-second threshold
|
||||
and atomic dispatching-to-unknown update. Never resend an unknown action.
|
||||
- A session check that fails must retain its own backoff and not block successful
|
||||
checks for other companies. Do not move all authorization/catch-up into one
|
||||
full-inventory load that fails as a unit.
|
||||
- Add a session scope to manual push execution so the worker cannot rediscover all
|
||||
calls per selected item. Retain the existing manual API for explicit callers.
|
||||
- Route published-state notification in both publications/worker.ts and
|
||||
publications/resolution.ts through a narrow owner helper. Their endpoint identity
|
||||
is already available. Avoid adding voice policy to the shared coordinator.
|
||||
- Notify before reply creation, presentation and action mutation. Wire session and
|
||||
configuration changes without giving reply delivery its own active-call polling
|
||||
deadline. Reconstruct on startup/resume; explicit feature changes invalidate it.
|
||||
|
||||
## Worker and producer implementation checkpoint
|
||||
|
||||
Implemented scoped session/action reconstruction, per-entry reply execution,
|
||||
commit-aware producer notifications, and coordinator registration. The normal
|
||||
periodic push and the pre-publication direct push are removed; manual one-shot
|
||||
entry points remain. Abandoned dispatch recovery pages independently of session
|
||||
state and atomically quarantines only still-expired dispatching actions. Passive
|
||||
live-call inspection does not wake healthy empty reply queues. Previously denied
|
||||
reply checks can reactivate through the already-required successful live-call
|
||||
authorization boundary; no notification fan-out across every authority table.
|
||||
See ../idle-voice-reply-delivery.md for the owned contract.
|
||||
|
||||
Evidence so far: the first three scheduling integration cases passed. A fixture
|
||||
name collision (pushFixture exposes publication as a row, not the factory method)
|
||||
failed one added producer test; corrected to create the publication inside the
|
||||
actual transaction. All 15 selected producer/delivery cases then passed. A fresh
|
||||
PostgreSQL full voice run passed 102 tests and parent recovery passed 11. Server
|
||||
typechecking passed after producer changes and again after final startup wiring.
|
||||
|
||||
The first final wiring command accidentally matched the whole suite through its
|
||||
"voice" describe title while using a retained-fixture database. Eight existing
|
||||
startup recovery assertions observed unrelated saved deadlines; 98 tests passed.
|
||||
That is not acceptance evidence. A fresh isolated database full run is now required
|
||||
and its result must be inspected before a PR-ready claim.
|
||||
|
||||
Remaining before publication: final full voice/startup result, direct published-state
|
||||
producer coverage through the actual shared publication worker/operator resolution,
|
||||
reply worker scope/backlog/failure lifecycle tests, relevant shared consumers,
|
||||
workspace verification, PR template/body and latest-head CI/reviews. The entire
|
||||
remaining stack inventory and older #17490 failure remain outstanding.
|
||||
|
||||
The corrected final wiring run passed all 120 tests: 104 full voice integration
|
||||
cases and 16 startup/hostname wiring cases, against a newly created PostgreSQL
|
||||
database. It includes seven new reply lifecycle cases: publication gating and
|
||||
quiet acceptance, saved retry reconstruction across coordinator restart, abandoned
|
||||
intent after call termination, nested producer rollback/commit, real question
|
||||
retrieval, authority restoration, and drain/resume without passive SQL. Final
|
||||
server typechecking also passed. The earlier retained-fixture failures remain
|
||||
recorded above; shared publication-path and broader acceptance work remains open.
|
||||
|
||||
## Publication-path and broader verification
|
||||
|
||||
Both real chat-service paths now have a commit-boundary regression: normal
|
||||
publication and operator mark-delivered. The tests hold the real final transaction
|
||||
before its voice notification, wait until the early enqueue wake is fully idle,
|
||||
and then verify one send after publication. Disabling only the final notification
|
||||
made both tests fail with zero sends; restoring it passed both. A 105-action real
|
||||
PostgreSQL case also proves startup inventory paging and bounded action recovery.
|
||||
|
||||
Workspace typechecking passed. Seven shared access/connection/settings suites
|
||||
passed all 651 tests. A combined full voice/startup/parent-recovery run used one
|
||||
external database: voice/startup passed 123 tests, but ten parent recovery tests
|
||||
observed the voice suite's intentionally retained sessions and failed their global
|
||||
quiet assertions. This is a test invocation isolation error, not accepted evidence
|
||||
for parent recovery. The parent suite is being rerun against its separate database.
|
||||
Do not combine these two suites under one external database URL.
|
||||
|
||||
The 107 full voice cases now include real publication and operator producers,
|
||||
105 abandoned sends, restart, rollback/outer commit, drain, authority restoration,
|
||||
question barriers and no-resend outcomes. Full workspace build and current-head
|
||||
PR CI/reviews remain required. The local all-suite command has the previously
|
||||
recorded embedded-PostgreSQL initialization limitation; targeted database suites
|
||||
run on real isolated PostgreSQL instead. Do not count skipped or uninitialized
|
||||
database tests as passes.
|
||||
|
||||
The parent recovery suite passed all 11 tests on its isolated database. The new
|
||||
reply-state read failure case also passed: the worker keeps its retry, performs
|
||||
no SQL during backoff, and sends once after recovery. Together with the full run,
|
||||
this verifies 108 voice cases, 16 startup cases, 11 parent recovery cases, and 651
|
||||
shared-consumer cases (786 selected tests). Workspace typecheck and build passed.
|
||||
The final added failure case changed tests only after the typecheck; the workspace
|
||||
build compiled the current tree. No production changes followed the full run.
|
||||
Remote CI/review remains pending until this branch is published.
|
||||
|
||||
|
||||
## Review follow-up: shared scheduling and real authority restoration
|
||||
|
||||
Paperclip review at `439d0a86eb` requested one scheduling implementation and a
|
||||
production-wiring regression. `scoped-deadline-work.ts` now owns coalescing,
|
||||
scoped reconstruction, bounded fair batches, per-entry/scope backoff and drain
|
||||
retention for both voice workers. The adapters retain their own scope matching,
|
||||
load/check contracts, eligibility and provider operations. The helper has no SQL,
|
||||
provider knowledge, or feature flags; absent consumers register no work.
|
||||
|
||||
The restoration test now registers the real voice service's reply and live-call
|
||||
recovery workers. It first lets the reply encounter suspended membership, restores
|
||||
membership before the next live-call authority inspection, and verifies exactly
|
||||
one send through the production authorization callback. Healthy inspections do
|
||||
not resend; confirmed hangup leaves both workers SQL-quiet. This deliberately
|
||||
preserves the documented window: if recovery observes denied authority first,
|
||||
it closes the call rather than waiting indefinitely for access to return.
|
||||
|
||||
A mutation probe removed only the production `authorityAvailable` invocation.
|
||||
The new test failed with zero sends instead of one. The production line was
|
||||
restored in a finally block; the unmodified case passes. This replaces the older
|
||||
test that called the notification directly.
|
||||
|
||||
Full voice/startup/coordinator/batching selection passed 180 tests on a fresh
|
||||
PostgreSQL database. Initial parent recovery verification passed nine and failed
|
||||
two: the extraction captured load methods at registration, bypassing the tests'
|
||||
later injected refresh failures and concurrent-write barriers. Adapters now call
|
||||
the service methods at dispatch time, preserving the original boundary. All 11
|
||||
parent lifecycle tests pass on their separately isolated database. These failures
|
||||
are retained here rather than counted as successful initial verification.
|
||||
|
||||
The optional review suggestion to cache the feature flag across a batch is not
|
||||
applied: current checks retain authority/configuration checks at dispatch, including
|
||||
changes while earlier provider calls are in flight. Any optimization must preserve
|
||||
that behavior. Workspace checks and fresh remote acceptance are recorded below
|
||||
when complete; the prior 5/5 Greptile score does not cover this follow-up yet.
|
||||
|
||||
Workspace `pnpm -r typecheck` and `pnpm build` pass after the review repairs.
|
||||
The 191 selected tests above are overlapping consumer verification, not an
|
||||
additional 191 unique cases on top of the earlier 786. Remote acceptance remains
|
||||
required on the published follow-up head.
|
||||
|
||||
|
||||
## Current-parent paging contract acceptance, October 27
|
||||
|
||||
This slice normally integrates bounded voice recovery #17576 at `712f3862b5`,
|
||||
including receiver `6e88ca0201` and the landed foundations. The original 30-file
|
||||
reply scope adds one shared contract path, `scoped-deadline-work.ts`. No database
|
||||
file changes. The existing helper could represent only a complete inventory;
|
||||
using that contract for the repaired parent would erase its bounded-pass guarantee.
|
||||
|
||||
The real 205-call parent regression fails before the integration repair: startup
|
||||
executes three inventory queries (100, 100, 5 rows) instead of one 100-row query.
|
||||
The shared helper now accepts either its existing complete array or a normalized
|
||||
page result. It retains page generations and seen identities, keeps existing
|
||||
entries until reconstruction completes, limits dirty scopes per pass, and replays
|
||||
writes received during reconstruction. It owns no SQL or voice/provider policy.
|
||||
Recovery delegates to its owner refresh and resets partial owner state on worker
|
||||
registration. Existing complete-array consumers keep their public calls.
|
||||
Connection observations retain the parent's separate receiver/command notification
|
||||
path; actual configuration wakes both voice owners.
|
||||
|
||||
All 445 distinct targeted cases pass: 18 complete parent recovery lifecycle
|
||||
cases, 108 complete voice-store/reply integrations, 224 complete tool-gateway and
|
||||
secret-proposal cases, and 95 shared dispatcher/coordinator/batching/startup/
|
||||
hostname cases. The helper's single existing test also ran in the initial
|
||||
19-case green group; it is counted only in the 95-case group. The 224 cases
|
||||
exercise the four other production consumers of the shared helper, including
|
||||
complete-array empty/expiry/recovery/projection, transaction and drain behavior.
|
||||
Server production typecheck and server build pass. The voice fixtures self-migrate
|
||||
separate initially empty databases. The two other database suites use separate
|
||||
migrated clones. Expected injected provider/wakeup failures and listener warnings
|
||||
remain in logs. Only documentation edits follow these passing production checks.
|
||||
|
||||
This acceptance preserves the parent's startup paging, restart, behind-cursor
|
||||
insert, concurrent feature change, health-only quiet, true disablement, current
|
||||
authority, failure, drain and shutdown contracts. Existing real publication,
|
||||
operator resolution, uncertain send, multipart/question ordering, authority
|
||||
restoration and 105-abandoned-dispatch tests remain in the 108-case suite.
|
||||
Fresh exact-head reviews and all CI remain required. No full local matrix,
|
||||
live provider acceptance, Cloud wake, deployed suspension or savings is claimed.
|
||||
|
||||
The whole-merge whitespace check flagged literal context whitespace inside an
|
||||
inherited `tests/runner-e2e/report-lock.patch` fixture. That fixture matches the
|
||||
parent and is outside this PR delta. The 31-file delta against the pinned parent
|
||||
passes the whitespace check; the inherited fixture is not edited to hide it.
|
||||
|
||||
|
||||
## Publication inventory and paging acceptance, October 28
|
||||
|
||||
The security review correctly found an endpoint-wide read per publication.
|
||||
Two real publication-path regressions seed 105 unrelated active phone calls and
|
||||
105 future dispatch intents on the same endpoint. Five normal publications select
|
||||
1,065 inventory rows; five operator resolutions select 1,060. Both exceed the
|
||||
125-row total budget. The corrected paths pass that same budget and deliver each
|
||||
answer once. These measurements count selected row results from executed queries,
|
||||
not physical PostgreSQL tuple visits or a deployment cost benchmark.
|
||||
|
||||
Normal publication completion and operator resolution now notify their actual
|
||||
conversation inside the existing transaction. The voice owner handles that
|
||||
normalized conversation scope. Its reader returns at most 100 total session and
|
||||
action headers per refresh, with a saved phase/cursor for the next pass. Startup,
|
||||
configuration changes and abandoned actions retain finite reconstruction. The
|
||||
shared dispatcher retains generations and observed IDs across pages; it also
|
||||
preserves the parent wildcard-failure isolation. SQL and provider decisions stay
|
||||
inside the voice owner. No schema, table, provider-specific shared branch or
|
||||
per-route recovery wrapper is introduced.
|
||||
|
||||
The abandoned-dispatch case now proves a 100-row startup cap, restart of the same
|
||||
owner before its scan completes, restoration of all 105 saved deadlines, no SQL
|
||||
before the deadline, and quarantine without resending. A separate owner-level
|
||||
real-database case reproduces a new denied call behind an older scan cursor:
|
||||
completion of the old scan must not erase the later restoration obligation.
|
||||
Denial revisions fence that cleanup. The case confirms a real 403, then observes
|
||||
the committed wake. The existing actual-service authority-restoration case still
|
||||
verifies the complete production callback path; the new cache test is not a
|
||||
replacement for that integration coverage.
|
||||
|
||||
Guarded mutations broaden publication hints back to endpoints, omit owner reset,
|
||||
and omit the denial revision fence. All fail the corresponding row-budget,
|
||||
100-versus-5 restart, or missing committed-wake assertion, with source restored.
|
||||
An earlier reset probe using already-due actions did not detect the missing reset
|
||||
because its own action writes rediscovered the backlog. That incomplete probe is
|
||||
retained as such; future deadlines make the strengthened regression deterministic.
|
||||
The first amplification fixture also used a shadowed publication function name;
|
||||
that fixture failure preceded the actual 1,065/1,060-row baseline.
|
||||
|
||||
The final branch normally integrates voice parent `ad79c7e805`. Its original
|
||||
30-file scope plus the existing shared dispatcher remains 31 files, with no DB
|
||||
changes. All 449 distinct targeted cases pass: 111 complete voice-store/reply,
|
||||
19 complete parent recovery, and 319 complete shared/legacy-consumer/startup cases.
|
||||
Both voice fixtures self-migrate separate initially empty databases; the two
|
||||
legacy DB suites use separate migrated clones. Production server typecheck and
|
||||
build pass. The current policy checker and its Node tests pass. Only evidence
|
||||
edits follow these checks. Older 445/786-case totals belong to their stated source
|
||||
revisions and are not added here. Current exact-head reviews and all CI remain
|
||||
required. No full local matrix, live provider, external wake, deployed suspension,
|
||||
physical query-cost or savings result is claimed.
|
||||
@@ -0,0 +1,168 @@
|
||||
# Inbound queue query proof
|
||||
|
||||
## Contract and cause
|
||||
|
||||
Paperclip's inbound queue discovers its next deadline, selects each conversation
|
||||
head, and selects ordinary ready deliveries. A retained processed delivery can
|
||||
still own the head while an issued or processing `inbound_wakeup` action remains.
|
||||
This work keeps that obligation, company and endpoint scope, lease expiry,
|
||||
retry timestamps, ordering keys, lifecycle dependencies, and provider exclusions.
|
||||
It adds no cache, migration, producer, worker, or timer.
|
||||
|
||||
The two open-work selectors used a correlated `EXISTS` inside an `OR`. PostgreSQL
|
||||
17.11 already used a hashed subplan in the measured fixture. The original
|
||||
hypothesis of repeated physical action scans was wrong. The estimate still
|
||||
included a large correlated alternative. It can enable JIT at the normal 100000
|
||||
cost threshold. The deadline query also evaluated JSON conditions on retained
|
||||
processed history before rejecting rows with no pending action.
|
||||
|
||||
The treatment uses a set of `(company_id, endpoint_id, delivery_id)` for pending
|
||||
ownership. Null delivery IDs cannot match and are excluded, so the predicate
|
||||
still returns a boolean. Duplicate actions do not multiply delivery rows.
|
||||
Ready-only probes retain their original scoped `EXISTS`. This keeps the
|
||||
readiness and retry parsing boundary outside this optimization.
|
||||
|
||||
## Reproduction
|
||||
|
||||
Run from the source checkout with its dependencies built:
|
||||
|
||||
```sh
|
||||
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-inbound-queue-discovery.ts --baseline-ref=09ad9adce2cf4742944353a3707a2f908e9bb35e --require-work-budget
|
||||
```
|
||||
|
||||
The harness starts disposable PostgreSQL through the repository test helper.
|
||||
It does not read a live database URL. It loads the pinned pre-fix queue,
|
||||
entry-point, and ledger sources. Other runtime helpers and dependencies must
|
||||
match. It captures the complete SQL and parameters through the real service,
|
||||
including projections, ordering, limits, and the head probe's lease path.
|
||||
It stops before provider effects. Every selected column is compared as a
|
||||
positional row on one reserved SQL connection. Fixture deadlines are fixed. The small control has 20 completed deliveries
|
||||
and actions. The retained-history case has 1500 of each, two open deliveries,
|
||||
and 32 KiB of synthetic incompressible message text per historical delivery.
|
||||
|
||||
Two warmups precede five alternating samples per arm. Plans use
|
||||
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` outside the timed samples. The explicit
|
||||
fixture work budget rejects JIT for the two changed selectors and more than
|
||||
1000 shared buffer accesses (hits plus reads) for the deadline selection. It has no CI timing threshold.
|
||||
Use `--jit-off` for a diagnostic control.
|
||||
|
||||
Local evidence uses Node 26.4.0 and Linux ARM64 PostgreSQL 17.11 in a disposable
|
||||
Docker container, matching the hosted major and minor version. The normal
|
||||
settings are shared_buffers 128 MiB, work_mem 4 MiB, JIT on, jit_above_cost 100000,
|
||||
and random_page_cost 4. Native macOS PostgreSQL could not start because the host
|
||||
had exhausted its SysV semaphore allocation. The scratch provider replaces only
|
||||
the test PostgreSQL constructor. Schema migrations, clients and query execution
|
||||
remain real. No persistent database is used.
|
||||
|
||||
## Results
|
||||
|
||||
Matched pre-fix baseline: `09ad9adce2cf4742944353a3707a2f908e9bb35e`.
|
||||
An earlier baseline at `96727b3` had identical ledger and conversation-queue
|
||||
sources. Master changed an unrelated health notification import in the service
|
||||
entry-point, so the final comparison was repeated from the newer pre-fix base.
|
||||
The ordinary-ready selector is an unchanged control. The head query still reads
|
||||
its existing JSON ordering/filter fields; do not claim it reduces physical
|
||||
history scans. Local SQL timings do not prove a hosted request speedup.
|
||||
|
||||
Final quiet PostgreSQL 17.11 comparison, in milliseconds. Each arm has two
|
||||
warmups and five alternating samples. Parentheses show minimum–maximum.
|
||||
|
||||
| Actual selector | Baseline median (range) | Treatment median (range) | Ratio | Equality |
|
||||
| --- | ---: | ---: | ---: | --- |
|
||||
| deadline | 18.78 (18.43–20.41) | 1.58 (1.14–1.75) | 11.90x | 2 complete rows equal |
|
||||
| head | 5.71 (5.38–6.40) | 6.18 (5.56–6.72) | 0.92x | 1 complete rows equal |
|
||||
| ready | 9.23 (7.97–9.41) | 8.33 (7.72–9.21) | 1.11x | 1 complete rows equal |
|
||||
|
||||
- deadline: shared buffer hits 10662 → 455; JIT present → absent.
|
||||
- head: shared buffer hits 10600 → 10600; JIT absent → absent.
|
||||
- ready: shared buffer hits 66 → 66; JIT present → present.
|
||||
|
||||
The matched small control is neutral (overlapping sample ranges). The JIT-off
|
||||
control remains diagnostic under other local build load: deadline 9.72 → 2.17 ms;
|
||||
head 9.32 → 9.08 ms; ordinary ready 1.52 → 1.56 ms. The last two ranges overlap.
|
||||
It separates deadline physical work from an earlier JIT-only head improvement.
|
||||
The earlier small-then-large fixture put the head estimate at 100369.79 and
|
||||
enabled JIT. The final pristine retained-only fixture put it at 98866.79, below
|
||||
the threshold. Its head timing is neutral. Plan-cost threshold crossings are
|
||||
data-sensitive; do not claim a head speedup for the final fixture.
|
||||
|
||||
## Feedback loop
|
||||
|
||||
| Question | Loop | Measured elapsed | Decision |
|
||||
| --- | --- | --- | --- |
|
||||
| Can the real source run locally? | Cached offline install, isolated build | 27.7 s install; 141 s build log span | Reuse the built checkout for each narrow attempt |
|
||||
| Does the initial scan hypothesis hold? | Captured queries and real PG plans | 4.0 s inside the instrumented baseline harness, plus module loading | Reject repeated physical scans as the cause |
|
||||
| Does the set improve the actual selectors? | Red work budget, alternating pair, JIT-off control | 28.36 s for all three processes, including module and DB setup | Exploratory variant identified JIT cost; final scope keeps ready-only unchanged |
|
||||
| Does a selective ready probe remain scoped? | Original and treatment SQL with unrelated malformed retry data | 12 s focused attempt; 28 s after replay/module rebuild | Direct head probes return empty; actual OR selectors both return error 22007; keep readiness unchanged |
|
||||
|
||||
Coordination waits for a quiet benchmark interval are separate from these command
|
||||
measurements. They are not a reason to run noisy parallel timing samples. The
|
||||
small real-database regression loop takes about 11 seconds including transforms
|
||||
and startup. Full repository checks and exact-head CI remain required.
|
||||
|
||||
## Scope of benefit
|
||||
|
||||
The discovery sample attributed 4.731 summed SQL seconds to 90 deadline
|
||||
selections and 4.681 seconds to 97 head selections over about five minutes.
|
||||
The final pristine fixture proves a deadline improvement; the head is neutral
|
||||
at that data point. The ordinary-ready control accounted
|
||||
for a further 1.389 seconds over 90 calls. These are background selectors, not a
|
||||
correlated request critical path. The maximum improvement to a single affected
|
||||
selection cannot exceed its observed SQL duration (about 49–53 ms on average).
|
||||
No end-to-end latency or rollout claim follows from adding those overlapping
|
||||
aggregate times. The change is small and avoids a recurring cost under the
|
||||
hosted planner defaults, so it is worth shipping with the other proven fixes.
|
||||
|
||||
## Correctness and failure accounting
|
||||
|
||||
Five real-PostgreSQL tests cover tuple scope, duplicate/null actions, exact
|
||||
boolean results, full payloads, future head ownership, independent reactions,
|
||||
paused/attention/agentmail exclusions, retry/processing deadlines, and provider
|
||||
sequence ordering. Six existing assembled chat tests also pass. They cover
|
||||
lease renewal failure, reverse-arrival Slack and Telegram events, lifecycle
|
||||
retry, edit/admission races, and deadline restoration.
|
||||
|
||||
An exploratory malformed-retry test compared an actual OR selector against an
|
||||
AND-only oracle. That comparison was inconclusive. It must not be presented as
|
||||
proof of an introduced defect. The complete actual old and new OR selectors
|
||||
both return PostgreSQL error 22007 for unrelated malformed pending actions in
|
||||
the measured fixture, including other-company and null-delivery controls. The
|
||||
selective direct head probe returns empty on both versions. The final fix leaves
|
||||
ready-only evaluation unchanged.
|
||||
|
||||
The initial repeated-action-scan hypothesis was also rejected. Both arms scan
|
||||
action history once in the measured plans. The benefits come from query costing
|
||||
and filter order, rather than fewer physical action scans.
|
||||
|
||||
## Final verification
|
||||
|
||||
- `pnpm -r typecheck`: pass.
|
||||
- `pnpm build`: pass.
|
||||
- New and selected existing assembled PostgreSQL 17 suites: 11 passed.
|
||||
- Complete local suite: not run because the native host PostgreSQL startup
|
||||
blocker is already established; complete final-head CI remains the gate.
|
||||
- Final retained-history benchmark work budget: pass, complete positional
|
||||
equality for all three selectors.
|
||||
- The final benchmark process took 34.19 seconds from
|
||||
its instrumented setup through proof. This includes the coordination wait after
|
||||
the fixture and queries were ready. The actual timing phase was brief.
|
||||
|
||||
The PR contains the source, fixture and harness. Raw local JSON includes every
|
||||
sample, query, parameter and plan. No hosted rollout was performed.
|
||||
|
||||
## Additional failures retained
|
||||
|
||||
Greptile's first review found two proof gaps: the budget counted only cache
|
||||
hits, and the payload test read the row again instead of checking the captured
|
||||
selection. Both are fixed. The budget includes hits plus reads. The test compares
|
||||
the captured row's complete normalized payload with its seeded object.
|
||||
|
||||
A prepared timing attempt lost its idle pooled connection while waiting. Its
|
||||
now()-derived fixture value differed between queries, so equality stopped the
|
||||
run before samples. The harness now reserves its timing connection and uses an
|
||||
explicit follower deadline. The final quiet comparison passes.
|
||||
|
||||
The first CI head failed e2e shard 3 in `agent-chat-projects` cleanup: cancelling
|
||||
a heartbeat run timed out on a row lock and returned HTTP 500. This is outside
|
||||
the changed queue selector. The final head must still pass that check. The
|
||||
original log remains at https://github.com/paperclipai/paperclip/actions/runs/40624302918/job/122605509182.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Mine unread badge query
|
||||
|
||||
The Mine badge ranks the user's visible personal top 100, then checks unread state. Correlated participation and privacy predicates can trigger expensive PostgreSQL JIT compilation. The final change builds the same participant set using native UUID keys, applies fresh visibility before the limit, and ranks a narrow ordered candidate projection with the existing canonical activity expression.
|
||||
|
||||
A partial activity index supports the new user lookup: `(company_id, actor_id, entity_id) WHERE actor_type = 'user' AND entity_type = 'issue'`. Activity already covered by creation/assignment is excluded from that branch before deduplication. Legacy text passes an exact lowercase 36-character UUID guard inside CASE. Malformed, uppercase, padded, newline, braced and dashless values still match nothing; the all-zero UUID remains supported. Canonical ordering, ties, archive/resurfacing and the second fresh unread query remain unchanged.
|
||||
|
||||
## Final PostgreSQL 17 results
|
||||
|
||||
These are ordinary warm local service calls, not hosted request timings. SQL capture and EXPLAIN happen outside the timed calls. Each arm has two warmups and five alternating samples. Complete positional rows from both SQL phases and final returned ID order match in all twelve cells.
|
||||
|
||||
| Fixture / visibility / JIT | Baseline ms (range) | Treatment ms (range) | Before / after |
|
||||
| --- | --- | --- | --- |
|
||||
| small / unrestricted / on | 4.22 (3.88–4.50) | 4.77 (4.24–5.53) | 0.88x |
|
||||
| small / unrestricted / off | 3.76 (3.47–4.08) | 4.36 (4.26–4.97) | 0.86x |
|
||||
| small / scoped / on | 4.87 (4.58–4.98) | 5.06 (4.93–5.37) | 0.96x |
|
||||
| small / scoped / off | 4.59 (4.33–5.10) | 5.12 (4.93–5.22) | 0.90x |
|
||||
| dense / unrestricted / on | 18.33 (17.92–18.90) | 25.06 (24.44–28.48) | 0.73x |
|
||||
| dense / unrestricted / off | 20.99 (20.49–34.83) | 28.73 (27.73–30.88) | 0.73x |
|
||||
| dense / scoped / on | 909.30 (887.26–915.52) | 221.10 (219.58–224.21) | 4.11x |
|
||||
| dense / scoped / off | 218.34 (215.13–222.99) | 223.24 (219.44–231.43) | 0.98x |
|
||||
| sparse / unrestricted / on | 13.64 (13.11–14.10) | 5.84 (5.34–6.45) | 2.34x |
|
||||
| sparse / unrestricted / off | 12.41 (12.27–13.00) | 4.90 (4.51–6.02) | 2.53x |
|
||||
| sparse / scoped / on | 719.07 (711.22–761.60) | 10.79 (10.18–11.40) | 66.64x |
|
||||
| sparse / scoped / off | 23.15 (22.64–27.02) | 12.90 (12.33–13.86) | 1.79x |
|
||||
|
||||
The dense scoped path improves 4.11x; sparse paths improve 1.79–66.64x. Dense unrestricted paths remain slower by about 6.7–7.7 ms; dense scoped JIT-off is approximately neutral (218.34 to 223.24 ms). The result is strongest when the old query triggers full JIT or when a small personal set sits among company history. It is not a uniform execution-time improvement.
|
||||
|
||||
The baseline-red work assertion fails on the actual captured full-JIT plan. The treatment-green assertion requires at least 1.5x improvement where the baseline performs full JIT optimization/inlining, no such passes afterward, and at most 25% or 2 ms overhead in small/sparse controls. Dense no-JIT cases are reported diagnostics; a stricter 25% dense guard failed, so that regression is explicitly retained. The original optional 60 ms cap still fails on dense scoped reads (221.10 ms). The captured ranking execution retains about 200 ms of privacy-predicate work when half the personal tasks participate through activity alone; the rewrite removes compiler passes rather than that authority work. The failed cap remains in raw evidence. It is not counted as a pass.
|
||||
|
||||
## Fixture and isolation
|
||||
|
||||
The pinned baseline is `70c864633744cd96c0dc7394d631fc7119cd17d2`. Both owning pre-fix service blobs match the initial baseline. The harness copies the old issue service and participation helper and holds other query dependencies constant. Separate logical databases contain identical IDs, timestamps and payloads. The baseline lacks only the candidate index and its journal row; catalog assertions verify all other index definitions match. The final report hashes both query sources, the schema, migration and runner registration.
|
||||
|
||||
- Small: 12 personal issues, 48 activity rows and 24 comments.
|
||||
- Dense: 1200 personal issues, 60000 issue activities, 240000 unrelated run activities and 24000 comments.
|
||||
- Sparse: 12 personal issues among 1200 company issues, with the same history.
|
||||
- Issue activity includes matching and other-human authors; the actor index is populated. The target user has 3600 activity rows in dense and 36 in sparse. Half the personal tasks are created by the user; the other half participate through user activity alone. Fixture assertions verify the complete declared personal set before timing.
|
||||
- Base activity/comment payloads are 4096 incompressible bytes. Additional run rows carry 256 bytes. History appends across tasks in timestamp order.
|
||||
- The reported workspace's fresh read-only catalog estimates were about 292k activity rows, 1198 issues and 7632 comments. Estimates are not exact counts. The large fixtures match issue/history scale and exceed its comment history.
|
||||
|
||||
Environment: PostgreSQL 17.11 ARM64 Linux Docker, Node 26.4.0, postgres.js 3.4.9, Drizzle 0.45.2, shared_buffers 128 MiB, work_mem 4 MiB, jit on, jit_above_cost 100000 and full optimization/inlining thresholds 500000. Hosted PostgreSQL version/architecture and planner settings match; compute, storage and network differ.
|
||||
|
||||
```sh
|
||||
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-mine-unread-badge.ts --baseline-ref=70c864633744cd96c0dc7394d631fc7119cd17d2 --history-decoys=240000 --verify-migration --assert-performance --evidence-dir=<directory>
|
||||
```
|
||||
|
||||
The default harness starts disposable embedded PostgreSQL and ignores DATABASE_URL. Recorded runs used a scratch constructor provider for Linux PostgreSQL 17 because the host native runtime can hit semaphore limits. Database creation, schema migration, clients and cleanup remain the normal repository path. A saved report can be checked with `--validate-report=<report>`; source hashes must still match. Add `--validate-baseline` for the original compiler-work red. `--diagnostic` runs one pair per visibility/JIT setting with a plain plan first. `--diagnostic-dense` selects dense instead of sparse. `--fixture=sparse --history-decoys=940000` selects a million-row control. `--hold-after-report=<directory>` retains one selected disposable fixture for bounded query iteration, with a private ready file and explicit release.
|
||||
|
||||
## Why an exceptional migration is necessary
|
||||
|
||||
The source-only participant set had no actor index and scanned company history. A plain EXISTS rewrite was decorrelated into that scan. A forced existing-entity-index lateral lookup avoided it but regressed the sparse unrestricted case from 56.64 to 66.47 ms with JIT and 12.74 to 18.68 ms without JIT. Both alternatives are rejected. A time cutoff would change historical participation; existing indexes cannot selectively find a user's issue activity without walking tasks/history. The partial index supplies that missing access path and excludes agent/system/non-issue activity from its storage and write maintenance.
|
||||
|
||||
Migration `0395_tranquil_firestar.sql` uses the existing top-level concurrent-index runner and its invalid-index repair. No new startup script or live migration is used. The actual normal runner is measured against the pre-index schema with synthetic history. Concurrent reads/writes continue while it builds; a deliberate cancellation leaves an invalid index without a journal entry, then normal retry repairs it.
|
||||
|
||||
For the 300k-row upgrade, total first-run wall time was 668.79 ms, rerun 21.83 ms and interrupted-build retry 187.71 ms. The migration has one DDL statement; totals include inspection, lock acquisition, concurrent build/validation, journal write and final verification. Concurrent reads completed in 0.51–6.69 ms and writes in 0.75–3.45 ms. The final partial index was 188416 bytes. Local duration does not guarantee a hosted duration, especially for larger tenants or long-lived transactions.
|
||||
|
||||
## Feedback loop and rejected attempts
|
||||
|
||||
Initial dependency install took 21.2 s. Frozen pnpm failed on upstream patch/lock metadata; local non-frozen resolution was retained privately and the tracked lock restored. The first service import lacked SDK dist; a narrow SDK dependency build fixed it.
|
||||
|
||||
The initial PostgreSQL 18 clustered-history diagnostic took 35.4 s through useful evidence, including 29.3 s fixture setup. Ranking was 96.7 ms with 75.5 ms JIT. Interleaved history and PostgreSQL 17 made the later fixture more representative. An early text-key/grouped-history rewrite won unrestricted but regressed scoped sparse to 392 ms with JIT disabled; it was rejected. A native-key draft exposed an ambiguous Drizzle id alias; an empty-database parse smoke now runs before seeding.
|
||||
|
||||
The original query-only 60k-history matrix improved target JIT cases but regressed dense JIT-off by about 17 ms and missed the 60 ms cap. Review exposed the missing actor index. Fresh catalog estimates sized the 300k and million-row controls. The first indexed draft still regressed dense execution. A narrow ordered candidate phase and skipping already-owned activity reduced that overhead. Retaining the seeded fixture shortened the next diagnostic to about ten seconds of actual calls. Repeated SQL capture had inflated timed service work; capture now occurs outside the sample interval in both arms.
|
||||
|
||||
Scratch 512 MiB and eager dual-fixture 2 GiB runs hit ENOSPC; those are setup failures, not performance red. Fixtures now seed/measure sequentially. Matched pre-index/indexed databases use an owned 4 GiB tmpfs ceiling. The final full matrix took 113.26 s including both databases and fixture loading. Raw failures, plans and samples remain in private evidence.
|
||||
|
||||
## Correctness and deployment limit
|
||||
|
||||
Fourteen real PostgreSQL tests exercise ranking before unread, visibility before the top-100 limit, canonical ties, native assignee and exact legacy IDs, archive resurfacing/deleted comments/status changes, request audience/not_creator/dismissal/completion, provider/relay comment exclusions, company boundaries, inherited grants, private project membership, revocation and fresh badge routes. The final production source and strengthened fixtures passed all fourteen focused PostgreSQL 17 tests in 16.37 s. Full workspace typecheck and build passed. Latest-head CI remains required before handoff.
|
||||
|
||||
The earlier full local suite remained incomplete after native PostgreSQL startup failures. An independent empty initdb reproduced system semaphore exhaustion at semget. All sixteen failed cases passed through task-owned Docker PostgreSQL 17; that is not a full local-suite pass. Full final-head CI is the broad gate.
|
||||
|
||||
Fresh hosted ranking variants averaged 78.8 ms SQL per call. That is the affected serial SQL-time budget, not a promised HTTP saving. Badge HTTP requests averaged 752 ms (p95 1521 ms) without correlated SQL traces. A deployed revision check and comparable fresh query/request window are still necessary after rollout. No merge or deployment is part of this PR.
|
||||
|
||||
## Million-row history control
|
||||
|
||||
The same mixed-participation sparse fixture also contains 940000 unrelated run activities, for one million initial activity rows. Both arms again have two warmups and five alternating samples, and both complete SQL phases/final ID order match.
|
||||
|
||||
| Visibility / JIT | Baseline ms (range) | Treatment ms (range) | Before / after |
|
||||
| --- | --- | --- | --- |
|
||||
| unrestricted / on | 13.44 (13.10–14.10) | 5.76 (5.51–6.17) | 2.33x |
|
||||
| unrestricted / off | 13.45 (13.07–14.34) | 5.77 (5.11–6.84) | 2.33x |
|
||||
| scoped / on | 807.76 (741.30–825.57) | 11.59 (10.39–12.67) | 69.68x |
|
||||
| scoped / off | 19.69 (18.78–20.76) | 10.90 (10.06–11.73) | 1.81x |
|
||||
|
||||
The activity participant branch uses the partial actor index and visits 36 matching actor rows (18 returned, 18 already-owned rows filtered), rather than scanning the million-row company history. Canonical activity and authority work remain separate. The activity relation is about 439 MB heap / 1.05 GB including indexes and TOAST, exceeding the observed hosted relation's 160 MB / 381 MB. The physical fixture remains synthetic.
|
||||
|
||||
The actual normal migration runner completed first run in 255.06 ms, no-op rerun in 20.32 ms and interrupted-index repair/retry in 241.73 ms. The final index is 188416 bytes. Concurrent reads took 0.65–7.83 ms and writes 0.71–2.45 ms, without failures. The full command took 66.90 s including both database fixtures. This validates the missing actor access path above the observed target scale, not an unbounded fleet-wide migration duration.
|
||||
@@ -0,0 +1,124 @@
|
||||
# Task interruption latest-run query performance
|
||||
|
||||
The issue list displays a current interruption state for each visible task.
|
||||
`taskInterruptionPresentations` owns that read-only projection. Its old latest-run
|
||||
query grouped `coalesce(native_issue_id::text, issue_id::text,
|
||||
context_snapshot->>'issueId')` across company history. That expression cannot use
|
||||
the existing individual binding indexes to find a small page of tasks.
|
||||
|
||||
The query now probes the native, first-class, and legacy text indexes separately.
|
||||
Each branch supplies its newest candidate. A lateral query selects the newest
|
||||
candidate across those three branches by `created_at DESC, id DESC`. The final
|
||||
join reads the same interruption fields for that winner in one SQL snapshot.
|
||||
No history is deleted or hidden. This changes only the latest-run enrichment;
|
||||
the main issue-list query and external-conversation state gates are unchanged.
|
||||
|
||||
## Contract and regression proof
|
||||
|
||||
Native binding takes precedence over first-class binding, then legacy JSON text.
|
||||
A non-null mismatching native or first-class binding prevents legacy fallback.
|
||||
A guarded UUID parameter keeps uppercase and noncanonical caller strings from
|
||||
matching a UUID where the former text comparison would not. Legacy text keeps
|
||||
PostgreSQL `->>` behavior for empty, missing, scalar, array, object, and null data.
|
||||
Company boundaries, duplicate inputs, absent runs, and timestamp ties are covered.
|
||||
The existing project-access interruption tests exercise real repair and retry
|
||||
consumers. Presentation remains read-only; execution rechecks its own authority.
|
||||
|
||||
The baseline work regression selects six runs from a two-task page but inspects
|
||||
1,506 run rows. The treatment meets a deterministic bound of fewer than 30 rows
|
||||
in any run-table scan on that same fixture. The regression has no timing gate.
|
||||
The baseline semantics test passes before the optimization.
|
||||
|
||||
## Local measurement
|
||||
|
||||
Run the actual old and new query builders against disposable PostgreSQL:
|
||||
|
||||
```sh
|
||||
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-task-interruption.ts --baseline-ref=09ad9adce2cf4742944353a3707a2f908e9bb35e
|
||||
```
|
||||
|
||||
The pinned pre-fix service is loaded from Git. The harness rejects changed runtime
|
||||
dependencies. Its service source is identical to the original discovery baseline
|
||||
`96727b3cfefb61f85ffe1fbc29a9b6dd6837efe8` (Git blob
|
||||
`eb489f1ef6d35c9c243307e60e4c31d58afddfa3`). Both arms use the same current dependencies.
|
||||
The treatment service SHA-256 is
|
||||
`4607c8382aea8ec9cc76f628caa9b4a031b19dd43ff4beff12e596079742b763`.
|
||||
|
||||
The fixtures use a second company, unrelated histories, mixed binding types,
|
||||
fixed dates, and incompressible random context/result bodies. Mixed fixtures
|
||||
retain historical null bindings by disabling only the fixture database's modern
|
||||
scope-fill trigger during insert. The modern control leaves that trigger enabled.
|
||||
The harness captures complete generated SQL and parameters, compares every
|
||||
positional SQL field and row order, and compares full service maps. It warms each
|
||||
arm twice, alternates five measured samples per arm, and saves local
|
||||
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` plans.
|
||||
|
||||
PostgreSQL 17.11 Linux aarch64 matches the observed hosted major/minor version.
|
||||
Local settings: 128 MiB shared buffers, 4 MiB work memory, JIT enabled. These are
|
||||
warm local SQL timings, including driver round trips, not deployed HTTP timings.
|
||||
|
||||
| Page × history per task | Unrelated runs | JSON noise bytes per field | Before median (range) ms | After median (range) ms | Speedup |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 × 1 | 0 | 128 | 0.61 (0.55–0.88) | 1.04 (0.90–1.92) | 0.58× |
|
||||
| 100 × 1, mixed | 1,200 | 24,576 | 6.99 (5.89–8.41) | 2.59 (2.49–2.79) | 2.70× |
|
||||
| 100 × 12, mixed | 1,200 | 24,576 | 17.17 (16.79–17.48) | 3.00 (2.64–4.28) | 5.72× |
|
||||
| 1 × 120, mixed | 1,200 | 24,576 | 5.05 (4.39–5.89) | 2.45 (2.31–2.88) | 2.06× |
|
||||
| 100 × 40, mixed | 1,200 | 12,288 | 31.51 (30.68–31.61) | 3.05 (2.50–3.23) | 10.32× |
|
||||
| 100 × 40, modern | 1,200 | 12,288 | 26.78 (25.79–27.40) | 3.82 (3.47–4.78) | 7.02× |
|
||||
| 1,000 × 1, modern | 1,200 | 12,288 | 9.99 (9.51–10.79) | 14.11 (13.72–16.33) | 0.71× |
|
||||
|
||||
All seven pairs have exact positional SQL and full service equality. The 100 × 12
|
||||
plan drops shared buffer hits from 30,033 to 3,400. The modern dense control drops
|
||||
45,563 to 7,537. The tiny control adds about 0.43 ms. The maximum-size shallow
|
||||
modern page adds 4.12 ms (9.99 to 14.11 ms), with buffer hits increasing from
|
||||
15,467 to 22,008. The extra probes cost more for such wide, shallow history.
|
||||
These absolute costs are retained as tradeoffs; no adaptive threshold is added.
|
||||
Each branch can still inspect a task's own history. The native index lacks
|
||||
creation-time ordering and must sort native candidates. First-class and legacy
|
||||
indexes also contain native-bound rows; the precedence guards can discard those
|
||||
rows only after fetching them. A deeply native-bound task can therefore cause
|
||||
three per-task history walks, even though each branch returns at most one row.
|
||||
The modern 40-run control includes this effect; it does not establish constant
|
||||
work at 10,000 runs per task.
|
||||
|
||||
Immediate follow-up for that remaining cost: profile a synthetic 10,000-run,
|
||||
mostly native-bound task with complete result equality and per-branch
|
||||
`EXPLAIN (ANALYZE, BUFFERS)` counters. First test a migration-free selection that
|
||||
avoids repeated walks while retaining precedence and timestamp/ID ties. If an
|
||||
index is still necessary, evaluate a company/native-issue/creation-time index,
|
||||
a first-class binding index restricted to `native_issue_id IS NULL`, and a legacy
|
||||
binding index restricted to both native and first-class bindings being null.
|
||||
Measure the complete index builds and blocking on a production-size disposable
|
||||
fixture. If a build exceeds the one-to-two-minute budget, first optimize it and
|
||||
flag the remaining cost for human review. No partial index or startup DDL is
|
||||
included in this PR.
|
||||
|
||||
## Feedback loop and practical limit
|
||||
|
||||
| Question | Environment and loop | Measured elapsed | Result |
|
||||
| --- | --- | --- | --- |
|
||||
| Does coalesced lookup inspect unrelated history? | Actual service SQL, disposable PostgreSQL | Initial red test 13.55 s | 1,506 inspected rows; semantic oracle passes |
|
||||
| Does indexed selection preserve the contract? | Two real PostgreSQL regressions | 5.09 s repeated local test loop | Both pass |
|
||||
| Is the first batched union enough? | Pinned source pair, PostgreSQL 18.6 | 3.73 s inside benchmark | 1.62× at 100 × 12; scans remain |
|
||||
| Is selective lateral lookup better? | Seven paired PostgreSQL 17.11 cases | 8.39 s full command; 2.68 s database setup | 5.72× representative, 7.02× modern dense |
|
||||
|
||||
Dependency setup took 22.1 s. The baseline's patch configuration did not match its
|
||||
committed lockfile, so the isolated install needed a local non-frozen resolution.
|
||||
The lockfile is unchanged in this PR; its effective dependency record is retained
|
||||
with private raw evidence. A larger existing consumer suite initially failed to
|
||||
load because plugin-sdk had not been built. After normal build setup, all 19
|
||||
focused query and authority-consumer cases passed. Full recursive typecheck and
|
||||
build passed too. The full local suite was stopped after repeated native
|
||||
PostgreSQL bootstrap failures; all 16 affected executor cases passed with the
|
||||
Docker PostgreSQL 17 substitute. The full local suite is not counted as a pass.
|
||||
The first Linux CI attempt also had one failure in an unchanged Runner close
|
||||
fixture. One targeted job retry passed. A separate macOS diagnostic of that close
|
||||
fixture still fails; its cause is not diagnosed by this query optimization.
|
||||
|
||||
The affected hosted query averaged about 110 ms in the discovery window. Even
|
||||
eliminating that query can save at most its serial critical-path contribution;
|
||||
other enrichment reads run concurrently. The local measurements do not establish
|
||||
that contribution or explain the whole issue-list request. This bounded change
|
||||
is useful alongside the other measured query fixes. Deployed latency remains to
|
||||
be measured after rollout. No migration, startup job, deployment, or hosted
|
||||
mutation is part of this PR.
|
||||
@@ -88,6 +88,24 @@ turn and event outbox. Server-dependent tools wait for reconnection; work alread
|
||||
accepted into the queue resumes through normal admission. A single offline web
|
||||
server cannot accept new HTTP requests during that interval.
|
||||
|
||||
Cloud operators can read `GET /api/instance/task-drain?upgradeReadiness=1`
|
||||
with existing instance-control authority. The version 1 `upgradeReadiness` report
|
||||
counts current remote executions with a bound current-format encrypted header and a live owner lease.
|
||||
The checkpoint owner authenticates/decrypts that header and delegates version and identity
|
||||
inspection to Runner; the identity must match the run, session and runner. Ordinary
|
||||
sandbox runs bind their workspace identity; plugin tasks bind their active dispatched
|
||||
environment lease and need no workspace-sync descriptor.
|
||||
Legacy/local executions, missing identities/checkpoints, foreign/expired owners,
|
||||
in-flight admission, and an incomplete census prevent a positive report. It
|
||||
returns aggregate counts only. The request does not stop admission, detach a
|
||||
runner, or authorize new work. Ordinary drain reads do not perform this SQL.
|
||||
The explicit probe uses a read-only transaction, a two-second statement timeout,
|
||||
and at most 1,001 run rows. Header reads are capped at 256 KiB each and 8 MiB total;
|
||||
oversized, malformed, legacy-format or local evidence falls back to the ordinary drain.
|
||||
This bounded header probe does not validate every journal record or prove a live sandbox. This is an observation; the existing shutdown and
|
||||
startup owners still perform the actual handoff and enforce Stop and fencing.
|
||||
A positive report alone does not prove hosted upgrade continuity.
|
||||
|
||||
Retain Postgres and the instance secret encryption key across deployments. The
|
||||
private `native_runner_checkpoints` journal holds encrypted authentication state,
|
||||
commands, and acknowledged event cursors. These records are company/run scoped
|
||||
|
||||
@@ -121,6 +121,13 @@ Review: `baseline`.
|
||||
API: `present` — representative operation IDs:
|
||||
|
||||
- `GET /api/health`.
|
||||
- `GET /api/instance/task-drain?upgradeReadiness=1` — instance-admin or signed
|
||||
Cloud control only. Returns aggregate restart eligibility, never checkpoint
|
||||
contents, task identities or credentials. Company members and agent keys are
|
||||
denied. The explicit request performs bounded read-only SQL; ordinary drain
|
||||
status remains an in-memory read. [Route coverage](../../server/src/__tests__/instance-settings-routes.test.ts)
|
||||
and [PostgreSQL coverage](../../server/src/services/heartbeat/upgrade-readiness.test.ts)
|
||||
establish this local boundary; hosted active-run continuity requires separate proof.
|
||||
- `GET /api/liveness` and `HEAD /api/liveness` — public process startup status
|
||||
and build commit only; no database readiness or workspace identity.
|
||||
|
||||
|
||||
@@ -271,6 +271,18 @@ session-control access. Deployed diagnostic usefulness remains unverified.
|
||||
|
||||
## `session`
|
||||
|
||||
Scoped source evidence: completed warm-runner retirement can persist maintenance
|
||||
under the same process's live, task-bound preparation lease before native
|
||||
coordinator admission. Original controller generation, committed-result and
|
||||
no-new-work checks remain required. This changes no actor permission or session
|
||||
reset authority. Follow the [session recipe](../feature-map/runs-adapters.md#session)
|
||||
for configuration invalidation and expired or foreign owner rejection.
|
||||
|
||||
Fresh-session admission with external checkpoints reads current canonical state
|
||||
from the registered controller. A local marker does not grant bootstrap or
|
||||
session-reset authority. Exact run/session/runner/lease binding, no prior activity,
|
||||
backup checks and atomic remote directory admission retain their existing gates.
|
||||
|
||||
Capability ID: `runs-adapters/session`.
|
||||
|
||||
Review: `baseline`.
|
||||
@@ -287,6 +299,16 @@ Policy: Task-session listing and execution-context reset require board agent
|
||||
management at their HTTP routes. These controls are separate from delegated
|
||||
managed AI-account authentication attempts under `model-accounts`.
|
||||
|
||||
Ordinary remote `per_turn` continuation restores an external controller marker
|
||||
through the existing company/task-bound checkpoint store, with the expected
|
||||
normalized session. Prior-owner, lifecycle and backup admission checks remain
|
||||
authoritative. Prior-run projections refresh when that checkpoint exists; legacy
|
||||
file-backed admission remains available when it does not. This changes no agent
|
||||
grant or session-control endpoint. The
|
||||
[executor controls](../../server/src/services/native-runtime/native-session-executor.test.ts)
|
||||
provide source-level restoration and denial evidence; live provider continuity
|
||||
is a separate verification step.
|
||||
|
||||
Evidence: `source-only` — [implementation](../../server/src/routes/agents.ts#L6608) at
|
||||
the reviewed commit above.
|
||||
|
||||
|
||||
@@ -129,5 +129,5 @@ Capability ID: `instance-operations/diagnostics`.
|
||||
|
||||
Scope: cli
|
||||
Storybook: No authored story required for doctor/health command results.
|
||||
Production: Run doctor against an owned data directory; check readiness and nonzero failures. A listening port alone does not prove readiness; repair requires observing the repaired result.
|
||||
Production: Run doctor against an owned data directory; check readiness and nonzero failures. Instance operators can inspect `GET /api/instance/task-drain?upgradeReadiness=1` before a compatible app replacement. Verify that unknown or unsupported executions prevent readiness and that the read does not hold admission; separately prove same-run continuity through an actual upgrade. A listening port alone does not prove readiness; repair requires observing the repaired result.
|
||||
Sources: Source: [doctor.ts](../../cli/src/commands/doctor.ts). Tests: [doctor.test.ts](../../cli/src/__tests__/doctor.test.ts)
|
||||
@@ -74,6 +74,24 @@ mutations. Qualify a useful model task separately from saving the account.
|
||||
|
||||
### `session`
|
||||
|
||||
For a remote OpenCode agent configured with `per_turn`, complete a task, then
|
||||
send two follow-ups without resetting its session. Verify that each continuation
|
||||
restores its external controller checkpoint, retains provider context and saved
|
||||
files, and finishes with no stranded task lock. A missing checkpoint, an active
|
||||
prior owner or a mismatched session must still stop admission. The
|
||||
rejected active-owner attempt must not prevent a later continuation from reading
|
||||
the owner's newer completed checkpoint. The
|
||||
[executor controls](../../server/src/services/native-runtime/native-session-executor.test.ts)
|
||||
cover checkpoint restoration and its guards; real-provider acceptance remains
|
||||
separate evidence.
|
||||
|
||||
After a completed warm turn, change an agent setting that invalidates its
|
||||
instruction copy, then send a follow-up. Verify the old runner suspends during
|
||||
preparation, saved files remain available, and the follow-up completes once.
|
||||
Repeat with an expired or foreign preparation owner and verify that execution
|
||||
stays held. The checkpoint integration regression covers the ownership boundary;
|
||||
a real provider run remains separate acceptance evidence.
|
||||
|
||||
For a repository project that inherits storage-plugin defaults, run a task and
|
||||
send two follow-ups. Start with unset task workspace settings, then repeat with
|
||||
an explicit isolated mode or strategy. Binding must save `reuse_existing` and
|
||||
@@ -95,6 +113,11 @@ reads the retained file and appends one new line. Verify a replacement sandbox,
|
||||
a new provider session, the exact file contents, successful finalization and no
|
||||
stranded task lock. Send another follow-up and verify ordinary session reuse.
|
||||
|
||||
With external controller checkpoints enabled, also admit a fresh session on a
|
||||
reused sandbox lease. Verify startup from canonical controller evidence and
|
||||
reject a marker without a registered controller, changed identity or prior turn
|
||||
activity. An existing remote session directory must retain its recovery gate.
|
||||
|
||||
Repeat without resetting the session: its replacement must restore verified
|
||||
history or remain blocked. Missing backup state must not silently start an
|
||||
existing session again. The [executor controls](../../server/src/services/native-runtime/native-session-executor.test.ts)
|
||||
|
||||
@@ -88,6 +88,8 @@ Use a human/unassigned task to verify persistence without invoking a model.
|
||||
|
||||
Automated: [task-list helpers](../../ui/src/pages/Issues.test.tsx) cover search URLs, pagination, deduplication, and presentation constants. The suite does not render the page or exercise filtering, grouping, opening a task, and returning; those interactions still need a running-browser check.
|
||||
|
||||
Task and inbox rows omit the system interruption labels **Thinking**, **Trying again soon**, and **Retry available**. [Shared row tests](../../ui/src/components/IssueRow.test.tsx) cover working, scheduled-retry, and failed interruptions in both row presentations, including suppression of replacement recovery chips. Task status and other relevant badges remain visible; retry controls remain inside the task conversation.
|
||||
|
||||
Manual: Seed human-created, agent-created, scheduled routine, human-initiated chat-channel, and creator-less tasks. Confirm automation is hidden by default, **show agent-created tasks** reveals it, reload preserves the choice, and Clear filters hides it again. Verify list and board search with automation newer than the human results, and expand contextual ancestors/subtasks without promoting unrelated automated roots. Repeat in project task lists. Agent profiles and task-detail subtasks retain complete history.
|
||||
|
||||
Manual: Filter by status and assignee, change grouping/sort, open a task, and return. Verify the correct records and selected view survive navigation and reload. Include an empty result and a failed request.
|
||||
|
||||
@@ -51,7 +51,7 @@ describe("managed GitHub launchers", () => {
|
||||
else cleanups.push(() => new Promise<void>(resolve => server.close(() => resolve())));
|
||||
const configRoot = path.join(root, "config");
|
||||
if (failure === "config-unwritable") await writeFile(configRoot, "not a directory");
|
||||
const result = await exec(path.join(bin, "git"), ["status", "--porcelain"], { cwd: root, env: {
|
||||
const result = await exec(path.join(bin, "git"), ["remote"], { cwd: root, env: {
|
||||
...process.env, ...githubBrokerEnvironment({ GH_TOKEN: "host-must-not-leak" }, { url: `http://127.0.0.1:${port}`, token: "private-capability" }),
|
||||
GH_CONFIG_DIR: configRoot, PATH: `${bin}:${process.env.PATH}`,
|
||||
} });
|
||||
@@ -182,4 +182,50 @@ process.stdout.write(JSON.stringify({identity, token:process.env.GH_TOKEN ?? nul
|
||||
expect(env.GH_TOKEN).toBe("");
|
||||
expect(env.GIT_AUTHOR_NAME).toBe("");
|
||||
});
|
||||
it("keeps local Git commands off the broker and still asks for the rest", async () => {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-github-local-skip-"));
|
||||
cleanups.push(() => rm(root, { recursive: true, force: true }));
|
||||
const bin = path.join(root, "managed");
|
||||
await mkdir(bin);
|
||||
await writeFile(path.join(bin, "git"), githubLauncherSource(), { mode: 0o700 });
|
||||
let posts = 0;
|
||||
const server = createServer((_req, res) => {
|
||||
posts++;
|
||||
res.setHeader("content-type", "application/json");
|
||||
res.end(JSON.stringify({ status: "available", env: {
|
||||
GH_TOKEN: "managed-token", GIT_AUTHOR_NAME: "Managed", GIT_AUTHOR_EMAIL: "managed@example.test",
|
||||
GIT_COMMITTER_NAME: "Managed", GIT_COMMITTER_EMAIL: "managed@example.test",
|
||||
} }));
|
||||
});
|
||||
await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
|
||||
cleanups.push(() => new Promise<void>(resolve => server.close(() => resolve())));
|
||||
const { port } = server.address() as { port: number };
|
||||
const env = { ...process.env, ...githubBrokerEnvironment({}, { url: `http://127.0.0.1:${port}`, token: "run-capability" }), PATH: `${bin}:${process.env.PATH}` };
|
||||
const git = async (...args: string[]) => {
|
||||
const before = posts;
|
||||
const { stdout } = await exec(path.join(bin, "git"), args, { cwd: root, env }).catch(() => ({ stdout: "" }));
|
||||
return { stdout: stdout.trim(), posts: posts - before };
|
||||
};
|
||||
for (const args of [
|
||||
["init", "-q", "repo"],
|
||||
["-C", "repo", "status", "--short"],
|
||||
["-C", "repo", "-c", "core.quotepath=off", "--no-pager", "diff"],
|
||||
["--git-dir=repo/.git", "rev-parse", "--git-dir"],
|
||||
["--git-dir", "repo/.git", "--work-tree", "repo", "status"],
|
||||
["-C", "repo", "config", "user.useConfigOnly"],
|
||||
["--version"],
|
||||
]) expect((await git(...args)).posts, args.join(" ")).toBe(0);
|
||||
expect((await git("-C", "repo", "config", "user.useConfigOnly")).stdout).toBe("true");
|
||||
for (const args of [
|
||||
["-C", "repo", "remote"],
|
||||
["-C", "repo", "var", "GIT_AUTHOR_IDENT"],
|
||||
["-C", "repo", "fetch", "--dry-run"],
|
||||
["-C", "repo", "-c", "alias.noop=!true", "noop"],
|
||||
["-C", "repo", "--unknown-option", "status"],
|
||||
["-C"],
|
||||
]) expect((await git(...args)).posts, args.join(" ")).toBe(1);
|
||||
expect((await git("-C", "repo", "var", "GIT_AUTHOR_IDENT")).stdout).toMatch(/^Managed <managed@example.test>/);
|
||||
expect((await git("-C", "repo", "commit", "--allow-empty", "-m", "Managed work")).posts).toBe(1);
|
||||
expect((await git("-C", "repo", "log", "-1", "--format=%an <%ae>")).stdout).toBe("Managed <managed@example.test>");
|
||||
}, 20_000);
|
||||
});
|
||||
@@ -17,6 +17,20 @@ if (!['git', 'gh'].includes(program) || !executable) {
|
||||
process.stderr.write('Paperclip: requested GitHub command is not installed.\n');
|
||||
process.exit(127);
|
||||
}
|
||||
const localGitCommands = new Set(['init', 'status', 'add', 'config', 'rev-parse', 'log', 'diff', 'show', 'branch', 'checkout', 'switch', 'restore', 'reset', 'rm', 'mv', 'worktree', 'grep', 'blame', 'describe', 'ls-files', 'ls-tree', 'cat-file', 'rev-list', 'merge-base', 'show-ref', 'for-each-ref', 'symbolic-ref', 'check-ignore', 'version', 'help']);
|
||||
const gitValueOptions = ['-C', '-c', '--git-dir', '--work-tree', '--namespace', '--config-env', '--super-prefix', '--attr-source'];
|
||||
const gitFlagOptions = ['-p', '-P', '--paginate', '--no-pager', '--bare', '--no-replace-objects', '--no-lazy-fetch', '--no-optional-locks', '--no-advice', '--literal-pathspecs', '--glob-pathspecs', '--noglob-pathspecs', '--icase-pathspecs'];
|
||||
function isLocalGit(args) {
|
||||
if (program !== 'git') return false;
|
||||
for (let index = 0; index < args.length; index++) {
|
||||
const arg = args[index];
|
||||
if (gitValueOptions.includes(arg)) { index += 1; continue; }
|
||||
if (gitValueOptions.some(option => arg.startsWith(option + '='))) continue;
|
||||
if (gitFlagOptions.includes(arg)) continue;
|
||||
return arg === '--version' || arg === '--help' || localGitCommands.has(arg);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
async function main() {
|
||||
let env = { ...process.env };
|
||||
const diagnostic = (code) => process.stderr.write('Paperclip: GitHub ' + code + '; continuing without managed credentials.\n');
|
||||
@@ -50,9 +64,10 @@ async function main() {
|
||||
GIT_CONFIG_KEY_4: 'user.useConfigOnly', GIT_CONFIG_VALUE_4: 'true',
|
||||
});
|
||||
const base = env.PAPERCLIP_GITHUB_BROKER_URL || env.PAPERCLIP_API_URL;
|
||||
const askBroker = !isLocalGit(process.argv.slice(2));
|
||||
try {
|
||||
let response;
|
||||
if (base && env.PAPERCLIP_GITHUB_BROKER_TOKEN) {
|
||||
if (askBroker && base && env.PAPERCLIP_GITHUB_BROKER_TOKEN) {
|
||||
const url = base.replace(/\/+$/, '').replace(/\/api$/, '') + '/runtime-tools/github/credentials';
|
||||
// A slow or restarting control plane must not cost the operation its
|
||||
// managed identity, so a failed request is retried before giving up.
|
||||
@@ -96,7 +111,7 @@ async function main() {
|
||||
}
|
||||
}
|
||||
}
|
||||
} else { diagnostic('capability_missing'); }
|
||||
} else if (askBroker) { diagnostic('capability_missing'); }
|
||||
} catch { diagnostic('broker_transport_unavailable'); }
|
||||
}
|
||||
// Only this invocation and its children inherit the captured credential.
|
||||
|
||||
@@ -546,6 +546,9 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
|
||||
"0391_third_roulette.sql",
|
||||
"0392_small_frightful_four.sql",
|
||||
"0393_open_grandmaster.sql",
|
||||
"0395_tranquil_firestar.sql",
|
||||
"0396_tan_paladin.sql",
|
||||
"0397_known_madrox.sql",
|
||||
|
||||
|
||||
"0384_funny_adam_warlock.sql",
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
CREATE INDEX CONCURRENTLY IF NOT EXISTS "activity_log_company_user_issue_actor_idx" ON "activity_log" USING btree ("company_id","actor_id","entity_id") WHERE "activity_log"."actor_type" = 'user' AND "activity_log"."entity_type" = 'issue';
|
||||
@@ -0,0 +1,31 @@
|
||||
ALTER TABLE "chat_voice_inbound_calls" ADD COLUMN IF NOT EXISTS "provider_account_id" text;--> statement-breakpoint
|
||||
ALTER TABLE "chat_voice_inbound_calls" ADD COLUMN IF NOT EXISTS "cleanup_confirmed_at" timestamp with time zone;--> statement-breakpoint
|
||||
ALTER TABLE "chat_voice_inbound_calls" ADD COLUMN IF NOT EXISTS "cleanup_next_attempt_at" timestamp with time zone;--> statement-breakpoint
|
||||
CREATE INDEX CONCURRENTLY IF NOT EXISTS "chat_voice_inbound_cleanup_idx" ON "chat_voice_inbound_calls" USING btree ("id") WHERE "chat_voice_inbound_calls"."cleanup_confirmed_at" is null;
|
||||
--> statement-breakpoint
|
||||
DO $$
|
||||
DECLARE cursor_id uuid; batch_ids uuid[];
|
||||
BEGIN
|
||||
LOOP
|
||||
IF cursor_id IS NULL THEN
|
||||
SELECT array_agg(id ORDER BY id) INTO batch_ids FROM (
|
||||
SELECT id FROM chat_voice_inbound_calls WHERE cleanup_confirmed_at IS NULL ORDER BY id LIMIT 1000
|
||||
) batch;
|
||||
ELSE
|
||||
SELECT array_agg(id ORDER BY id) INTO batch_ids FROM (
|
||||
SELECT id FROM chat_voice_inbound_calls WHERE cleanup_confirmed_at IS NULL AND id > cursor_id ORDER BY id LIMIT 1000
|
||||
) batch;
|
||||
END IF;
|
||||
EXIT WHEN batch_ids IS NULL;
|
||||
UPDATE chat_voice_inbound_calls c
|
||||
SET cleanup_confirmed_at = c.updated_at
|
||||
WHERE c.id = ANY(batch_ids) AND c.cleanup_confirmed_at IS NULL AND c.state = 'ended'
|
||||
AND (c.approved_by_user_id IS NULL OR EXISTS (
|
||||
SELECT 1 FROM activity_log a
|
||||
WHERE a.company_id = c.company_id AND a.entity_type = 'voice_inbound_call'
|
||||
AND a.entity_id = c.id::text AND a.action = 'voice.call.ended'
|
||||
));
|
||||
cursor_id := batch_ids[array_length(batch_ids, 1)];
|
||||
COMMIT;
|
||||
END LOOP;
|
||||
END $$;
|
||||
@@ -0,0 +1 @@
|
||||
CREATE INDEX CONCURRENTLY IF NOT EXISTS "chat_voice_sessions_report_window_idx" ON "chat_voice_sessions" USING btree ("ended_at","id") WHERE "chat_voice_sessions"."state" in ('ended', 'failed', 'expired') and "chat_voice_sessions"."provider_session_id" is not null;
|
||||
+2
-2
@@ -1,4 +1,4 @@
|
||||
CREATE TABLE "user_workspace_preferences" (
|
||||
CREATE TABLE IF NOT EXISTS "user_workspace_preferences" (
|
||||
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
|
||||
"user_id" text NOT NULL,
|
||||
"coordinator_workspace_enabled" boolean DEFAULT false NOT NULL,
|
||||
@@ -6,4 +6,4 @@ CREATE TABLE "user_workspace_preferences" (
|
||||
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE UNIQUE INDEX "user_workspace_preferences_user_uq" ON "user_workspace_preferences" USING btree ("user_id");
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS "user_workspace_preferences_user_uq" ON "user_workspace_preferences" USING btree ("user_id");
|
||||
+29
-64
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"id": "f697db44-a03f-4912-a55c-e26e61cd4435",
|
||||
"id": "66fe02b3-6672-4c19-b447-57acad43bf16",
|
||||
"prevId": "633afb6b-4b89-44ea-81d5-a477a3696eb5",
|
||||
"version": "7",
|
||||
"dialect": "postgresql",
|
||||
@@ -1350,6 +1350,34 @@
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"activity_log_company_user_issue_actor_idx": {
|
||||
"name": "activity_log_company_user_issue_actor_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "actor_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "entity_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"activity_log\".\"actor_type\" = 'user' AND \"activity_log\".\"entity_type\" = 'issue'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
@@ -59681,69 +59709,6 @@
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.user_workspace_preferences": {
|
||||
"name": "user_workspace_preferences",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"user_id": {
|
||||
"name": "user_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"coordinator_workspace_enabled": {
|
||||
"name": "coordinator_workspace_enabled",
|
||||
"type": "boolean",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": false
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"user_workspace_preferences_user_uq": {
|
||||
"name": "user_workspace_preferences_user_uq",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "user_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": true,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.work_assessments": {
|
||||
"name": "work_assessments",
|
||||
"schema": "",
|
||||
|
||||
+157
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "e1ddc6f7-6335-4b3d-aeec-a5df27793874",
|
||||
"prevId": "6974dd25-b8e9-4143-b4b7-ee5a5c3ff54a",
|
||||
"id": "be87a5d8-4164-4d95-89ae-b895701fa21b",
|
||||
"prevId": "66fe02b3-6672-4c19-b447-57acad43bf16",
|
||||
"version": "7",
|
||||
"dialect": "postgresql",
|
||||
"tables": {
|
||||
@@ -1350,6 +1350,34 @@
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"activity_log_company_user_issue_actor_idx": {
|
||||
"name": "activity_log_company_user_issue_actor_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "actor_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "entity_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"activity_log\".\"actor_type\" = 'user' AND \"activity_log\".\"entity_type\" = 'issue'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
@@ -14160,6 +14188,24 @@
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"provider_account_id": {
|
||||
"name": "provider_account_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"cleanup_confirmed_at": {
|
||||
"name": "cleanup_confirmed_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"cleanup_next_attempt_at": {
|
||||
"name": "cleanup_next_attempt_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"expires_at": {
|
||||
"name": "expires_at",
|
||||
"type": "timestamp with time zone",
|
||||
@@ -14182,6 +14228,22 @@
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"chat_voice_inbound_cleanup_idx": {
|
||||
"name": "chat_voice_inbound_cleanup_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"chat_voice_inbound_calls\".\"cleanup_confirmed_at\" is null",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_voice_inbound_expiry_idx": {
|
||||
"name": "chat_voice_inbound_expiry_idx",
|
||||
"columns": [
|
||||
@@ -42987,6 +43049,99 @@
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.native_runner_checkpoints": {
|
||||
"name": "native_runner_checkpoints",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"run_id": {
|
||||
"name": "run_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"issue_id": {
|
||||
"name": "issue_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"part": {
|
||||
"name": "part",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"material": {
|
||||
"name": "material",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"native_runner_checkpoints_headers_idx": {
|
||||
"name": "native_runner_checkpoints_headers_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "run_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"native_runner_checkpoints\".\"part\" = 'header'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"native_runner_checkpoints_company_id_issue_id_run_id_heartbeat_runs_company_id_native_issue_id_id_fk": {
|
||||
"name": "native_runner_checkpoints_company_id_issue_id_run_id_heartbeat_runs_company_id_native_issue_id_id_fk",
|
||||
"tableFrom": "native_runner_checkpoints",
|
||||
"tableTo": "heartbeat_runs",
|
||||
"columnsFrom": [
|
||||
"company_id",
|
||||
"issue_id",
|
||||
"run_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"company_id",
|
||||
"native_issue_id",
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {
|
||||
"native_runner_checkpoints_run_id_part_pk": {
|
||||
"name": "native_runner_checkpoints_run_id_part_pk",
|
||||
"columns": [
|
||||
"run_id",
|
||||
"part"
|
||||
]
|
||||
}
|
||||
},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.pipeline_automation_executions": {
|
||||
"name": "pipeline_automation_executions",
|
||||
"schema": "",
|
||||
+201
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "6974dd25-b8e9-4143-b4b7-ee5a5c3ff54a",
|
||||
"prevId": "e080fc1b-af8e-4012-8ce5-3b5065b65143",
|
||||
"id": "c7947622-f079-44c9-a57a-5a11d1bec5e8",
|
||||
"prevId": "be87a5d8-4164-4d95-89ae-b895701fa21b",
|
||||
"version": "7",
|
||||
"dialect": "postgresql",
|
||||
"tables": {
|
||||
@@ -1350,6 +1350,34 @@
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"activity_log_company_user_issue_actor_idx": {
|
||||
"name": "activity_log_company_user_issue_actor_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "actor_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "entity_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"activity_log\".\"actor_type\" = 'user' AND \"activity_log\".\"entity_type\" = 'issue'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
@@ -14160,6 +14188,24 @@
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"provider_account_id": {
|
||||
"name": "provider_account_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"cleanup_confirmed_at": {
|
||||
"name": "cleanup_confirmed_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"cleanup_next_attempt_at": {
|
||||
"name": "cleanup_next_attempt_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"expires_at": {
|
||||
"name": "expires_at",
|
||||
"type": "timestamp with time zone",
|
||||
@@ -14182,6 +14228,22 @@
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"chat_voice_inbound_cleanup_idx": {
|
||||
"name": "chat_voice_inbound_cleanup_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"chat_voice_inbound_calls\".\"cleanup_confirmed_at\" is null",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_voice_inbound_expiry_idx": {
|
||||
"name": "chat_voice_inbound_expiry_idx",
|
||||
"columns": [
|
||||
@@ -14934,6 +14996,28 @@
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_voice_sessions_report_window_idx": {
|
||||
"name": "chat_voice_sessions_report_window_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "ended_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"chat_voice_sessions\".\"state\" in ('ended', 'failed', 'expired') and \"chat_voice_sessions\".\"provider_session_id\" is not null",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
@@ -41945,6 +42029,28 @@
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"join_requests_pending_retention_idx": {
|
||||
"name": "join_requests_pending_retention_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "updated_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"join_requests\".\"status\" = 'pending_approval'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"join_requests_invite_unique_idx": {
|
||||
"name": "join_requests_invite_unique_idx",
|
||||
"columns": [
|
||||
@@ -42965,6 +43071,99 @@
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.native_runner_checkpoints": {
|
||||
"name": "native_runner_checkpoints",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"run_id": {
|
||||
"name": "run_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"issue_id": {
|
||||
"name": "issue_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"part": {
|
||||
"name": "part",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"material": {
|
||||
"name": "material",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"native_runner_checkpoints_headers_idx": {
|
||||
"name": "native_runner_checkpoints_headers_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "run_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"native_runner_checkpoints\".\"part\" = 'header'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"native_runner_checkpoints_company_id_issue_id_run_id_heartbeat_runs_company_id_native_issue_id_id_fk": {
|
||||
"name": "native_runner_checkpoints_company_id_issue_id_run_id_heartbeat_runs_company_id_native_issue_id_id_fk",
|
||||
"tableFrom": "native_runner_checkpoints",
|
||||
"tableTo": "heartbeat_runs",
|
||||
"columnsFrom": [
|
||||
"company_id",
|
||||
"issue_id",
|
||||
"run_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"company_id",
|
||||
"native_issue_id",
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {
|
||||
"native_runner_checkpoints_run_id_part_pk": {
|
||||
"name": "native_runner_checkpoints_run_id_part_pk",
|
||||
"columns": [
|
||||
"run_id",
|
||||
"part"
|
||||
]
|
||||
}
|
||||
},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.pipeline_automation_executions": {
|
||||
"name": "pipeline_automation_executions",
|
||||
"schema": "",
|
||||
+286
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "e080fc1b-af8e-4012-8ce5-3b5065b65143",
|
||||
"prevId": "f38ffbf3-c0ce-435e-aad8-9e5afa6b9417",
|
||||
"id": "d2fa91e5-af68-4894-81f9-ea228a76337f",
|
||||
"prevId": "c7947622-f079-44c9-a57a-5a11d1bec5e8",
|
||||
"version": "7",
|
||||
"dialect": "postgresql",
|
||||
"tables": {
|
||||
@@ -1350,6 +1350,34 @@
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"activity_log_company_user_issue_actor_idx": {
|
||||
"name": "activity_log_company_user_issue_actor_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "actor_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "entity_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"activity_log\".\"actor_type\" = 'user' AND \"activity_log\".\"entity_type\" = 'issue'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
@@ -5721,6 +5749,28 @@
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"approvals_pending_retention_idx": {
|
||||
"name": "approvals_pending_retention_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "updated_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"approvals\".\"status\" = 'pending'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"approvals_company_status_type_idx": {
|
||||
"name": "approvals_company_status_type_idx",
|
||||
"columns": [
|
||||
@@ -14138,6 +14188,24 @@
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"provider_account_id": {
|
||||
"name": "provider_account_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"cleanup_confirmed_at": {
|
||||
"name": "cleanup_confirmed_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"cleanup_next_attempt_at": {
|
||||
"name": "cleanup_next_attempt_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"expires_at": {
|
||||
"name": "expires_at",
|
||||
"type": "timestamp with time zone",
|
||||
@@ -14160,6 +14228,22 @@
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"chat_voice_inbound_cleanup_idx": {
|
||||
"name": "chat_voice_inbound_cleanup_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"chat_voice_inbound_calls\".\"cleanup_confirmed_at\" is null",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_voice_inbound_expiry_idx": {
|
||||
"name": "chat_voice_inbound_expiry_idx",
|
||||
"columns": [
|
||||
@@ -14912,6 +14996,28 @@
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_voice_sessions_report_window_idx": {
|
||||
"name": "chat_voice_sessions_report_window_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "ended_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"chat_voice_sessions\".\"state\" in ('ended', 'failed', 'expired') and \"chat_voice_sessions\".\"provider_session_id\" is not null",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
@@ -41923,6 +42029,28 @@
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"join_requests_pending_retention_idx": {
|
||||
"name": "join_requests_pending_retention_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "updated_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"join_requests\".\"status\" = 'pending_approval'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"join_requests_invite_unique_idx": {
|
||||
"name": "join_requests_invite_unique_idx",
|
||||
"columns": [
|
||||
@@ -42943,6 +43071,99 @@
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.native_runner_checkpoints": {
|
||||
"name": "native_runner_checkpoints",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"run_id": {
|
||||
"name": "run_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"issue_id": {
|
||||
"name": "issue_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"part": {
|
||||
"name": "part",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"material": {
|
||||
"name": "material",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"native_runner_checkpoints_headers_idx": {
|
||||
"name": "native_runner_checkpoints_headers_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "run_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"native_runner_checkpoints\".\"part\" = 'header'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"native_runner_checkpoints_company_id_issue_id_run_id_heartbeat_runs_company_id_native_issue_id_id_fk": {
|
||||
"name": "native_runner_checkpoints_company_id_issue_id_run_id_heartbeat_runs_company_id_native_issue_id_id_fk",
|
||||
"tableFrom": "native_runner_checkpoints",
|
||||
"tableTo": "heartbeat_runs",
|
||||
"columnsFrom": [
|
||||
"company_id",
|
||||
"issue_id",
|
||||
"run_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"company_id",
|
||||
"native_issue_id",
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {
|
||||
"native_runner_checkpoints_run_id_part_pk": {
|
||||
"name": "native_runner_checkpoints_run_id_part_pk",
|
||||
"columns": [
|
||||
"run_id",
|
||||
"part"
|
||||
]
|
||||
}
|
||||
},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.pipeline_automation_executions": {
|
||||
"name": "pipeline_automation_executions",
|
||||
"schema": "",
|
||||
@@ -59544,6 +59765,69 @@
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.user_workspace_preferences": {
|
||||
"name": "user_workspace_preferences",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"user_id": {
|
||||
"name": "user_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"coordinator_workspace_enabled": {
|
||||
"name": "coordinator_workspace_enabled",
|
||||
"type": "boolean",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": false
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"user_workspace_preferences_user_uq": {
|
||||
"name": "user_workspace_preferences_user_uq",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "user_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": true,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.work_assessments": {
|
||||
"name": "work_assessments",
|
||||
"schema": "",
|
||||
+23
-2
@@ -2756,8 +2756,29 @@
|
||||
{
|
||||
"idx": 395,
|
||||
"version": "7",
|
||||
"when": 1793163148662,
|
||||
"tag": "0395_bizarre_firedrake",
|
||||
"when": 1793159398533,
|
||||
"tag": "0395_tranquil_firestar",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 396,
|
||||
"version": "7",
|
||||
"when": 1793191988458,
|
||||
"tag": "0396_tan_paladin",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 397,
|
||||
"version": "7",
|
||||
"when": 1793195976732,
|
||||
"tag": "0397_known_madrox",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 398,
|
||||
"version": "7",
|
||||
"when": 1793205441783,
|
||||
"tag": "0398_sticky_sue_storm",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { sql } from "drizzle-orm";
|
||||
import { pgTable, uuid, text, timestamp, jsonb, index } from "drizzle-orm/pg-core";
|
||||
import { companies } from "./companies.js";
|
||||
import { agents } from "./agents.js";
|
||||
@@ -33,5 +34,8 @@ export const activityLog = pgTable(
|
||||
),
|
||||
runIdIdx: index("activity_log_run_id_idx").on(table.runId),
|
||||
entityIdx: index("activity_log_entity_type_id_idx").on(table.entityType, table.entityId),
|
||||
companyUserIssueActorIdx: index("activity_log_company_user_issue_actor_idx")
|
||||
.on(table.companyId, table.actorId, table.entityId)
|
||||
.where(sql`${table.actorType} = 'user' AND ${table.entityType} = 'issue'`),
|
||||
}),
|
||||
);
|
||||
@@ -27,6 +27,9 @@ export const chatVoiceInboundCalls = pgTable("chat_voice_inbound_calls", {
|
||||
approvedByUserId: text("approved_by_user_id"), callerAuthority: text("caller_authority"),
|
||||
intakeIssueId: uuid("intake_issue_id"),
|
||||
requestedIssueId: uuid("requested_issue_id"), sessionId: uuid("session_id"),
|
||||
providerAccountId: text("provider_account_id"),
|
||||
cleanupConfirmedAt: timestamp("cleanup_confirmed_at", {withTimezone: true}),
|
||||
cleanupNextAttemptAt: timestamp("cleanup_next_attempt_at", {withTimezone: true}),
|
||||
expiresAt: timestamp("expires_at", {withTimezone: true}).notNull(),
|
||||
createdAt: timestamp("created_at", {withTimezone: true}).notNull().defaultNow(),
|
||||
updatedAt: timestamp("updated_at", {withTimezone: true}).notNull().defaultNow(),
|
||||
@@ -36,6 +39,7 @@ export const chatVoiceInboundCalls = pgTable("chat_voice_inbound_calls", {
|
||||
foreignKey({columns: [t.companyId, t.sessionId], foreignColumns: [chatVoiceSessions.companyId, chatVoiceSessions.id], name: "chat_voice_inbound_session_fk"}),
|
||||
foreignKey({columns: [t.companyId, t.intakeIssueId], foreignColumns: [issues.companyId, issues.id], name: "chat_voice_inbound_intake_fk"}),
|
||||
foreignKey({columns: [t.companyId, t.requestedIssueId], foreignColumns: [issues.companyId, issues.id], name: "chat_voice_inbound_requested_issue_fk"}),
|
||||
index("chat_voice_inbound_cleanup_idx").on(t.id).where(sql`${t.cleanupConfirmedAt} is null`),
|
||||
index("chat_voice_inbound_expiry_idx").on(t.state, t.expiresAt),
|
||||
check("chat_voice_inbound_state_check", sql`${t.state} in ('guest_intake', 'awaiting_approval', 'approving', 'approved', 'denied', 'expired', 'ended')`),
|
||||
check("chat_voice_inbound_token_check", sql`${t.toolTokenHash} ~ '^[a-f0-9]{64}$'`),
|
||||
|
||||
@@ -36,6 +36,8 @@ export const chatVoiceSessions = pgTable("chat_voice_sessions", {
|
||||
uniqueIndex("chat_voice_sessions_request_uq").on(t.companyId, t.callerId, t.idempotencyKey),
|
||||
uniqueIndex("chat_voice_sessions_live_caller_task_uq").on(t.companyId, t.callerId, t.issueId).where(sql`${t.state} not in ('ended', 'failed', 'expired')`),
|
||||
index("chat_voice_sessions_expiry_idx").on(t.state, t.expiresAt),
|
||||
index("chat_voice_sessions_report_window_idx").on(t.endedAt, t.id)
|
||||
.where(sql`${t.state} in ('ended', 'failed', 'expired') and ${t.providerSessionId} is not null`),
|
||||
foreignKey({ columns: [t.companyId, t.endpointId], foreignColumns: [chatEndpoints.companyId, chatEndpoints.id], name: "chat_voice_sessions_endpoint_fk" }),
|
||||
foreignKey({ columns: [t.companyId, t.conversationId], foreignColumns: [chatConversations.companyId, chatConversations.id], name: "chat_voice_sessions_conversation_fk" }),
|
||||
foreignKey({ columns: [t.companyId, t.issueId], foreignColumns: [issues.companyId, issues.id], name: "chat_voice_sessions_issue_fk" }),
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import postgres from "postgres";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./test-embedded-postgres.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const describePostgres = support.supported ? describe : describe.skip;
|
||||
|
||||
describePostgres("workspace preferences migration", () => {
|
||||
it("defaults off and preserves existing preferences when the renumbered migration replays", async () => {
|
||||
const database = await startEmbeddedPostgresTestDatabase("paperclip-workspace-preferences-replay-");
|
||||
const sql = postgres(database.connectionString, { max: 1, onnotice: () => {} });
|
||||
try {
|
||||
const userId = randomUUID();
|
||||
const [original] = await sql`
|
||||
INSERT INTO user_workspace_preferences (user_id) VALUES (${userId}) RETURNING *
|
||||
`;
|
||||
expect(original.coordinator_workspace_enabled).toBe(false);
|
||||
await sql`UPDATE user_workspace_preferences SET coordinator_workspace_enabled = true WHERE user_id = ${userId}`;
|
||||
const [before] = await sql`SELECT * FROM user_workspace_preferences WHERE user_id = ${userId}`;
|
||||
|
||||
// An isolated build may already have applied this table under an older
|
||||
// migration number before master claimed that number for another table.
|
||||
const migration = await readFile(new URL("./migrations/0398_sticky_sue_storm.sql", import.meta.url), "utf8");
|
||||
for (const statement of migration.split("--> statement-breakpoint").map(value => value.trim()).filter(Boolean)) {
|
||||
await sql.unsafe(statement);
|
||||
}
|
||||
|
||||
const [after] = await sql`SELECT * FROM user_workspace_preferences WHERE user_id = ${userId}`;
|
||||
expect(after).toEqual(before);
|
||||
await expect(sql`INSERT INTO user_workspace_preferences (user_id) VALUES (${userId})`)
|
||||
.rejects.toMatchObject({ code: "23505", constraint_name: "user_workspace_preferences_user_uq" });
|
||||
} finally {
|
||||
await sql.end();
|
||||
await database.cleanup();
|
||||
}
|
||||
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
||||
});
|
||||
@@ -0,0 +1,120 @@
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import postgres from "postgres";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { applyPendingMigrations } from "./client.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js";
|
||||
|
||||
const external = process.env.PAPERCLIP_MIGRATION_TEST_DATABASE_URL;
|
||||
if (external && !new URL(external).pathname.endsWith("_test")) throw new Error("Migration test URL must name a test database");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
const migrationFile = "0396_tan_paladin.sql";
|
||||
const indexName = "chat_voice_inbound_cleanup_idx";
|
||||
|
||||
(support.supported ? describe : describe.skip)("inbound voice cleanup migration", () => {
|
||||
it("boots through the application migrator and repairs an interrupted index on concurrent replay", async () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let admin: ReturnType<typeof postgres> | undefined;
|
||||
let databaseName: string | undefined;
|
||||
let connectionString: string;
|
||||
let client: ReturnType<typeof postgres> | undefined;
|
||||
try {
|
||||
if (external) {
|
||||
admin = postgres(external, { max: 1, onnotice: () => {} });
|
||||
databaseName = `voice_migration_${randomUUID().replaceAll("-", "")}_test`;
|
||||
await admin.unsafe(`CREATE DATABASE "${databaseName}"`);
|
||||
const url = new URL(external); url.pathname = `/${databaseName}`;
|
||||
connectionString = url.toString();
|
||||
await applyPendingMigrations(connectionString);
|
||||
} else {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("voice-inbound-migration-");
|
||||
connectionString = temporary.connectionString;
|
||||
}
|
||||
client = postgres(connectionString, { max: 1, onnotice: () => {} });
|
||||
const columns = await client`SELECT column_name FROM information_schema.columns
|
||||
WHERE table_name = 'chat_voice_inbound_calls'`;
|
||||
expect(columns.map(row => row.column_name)).toEqual(expect.arrayContaining([
|
||||
"cleanup_confirmed_at", "cleanup_next_attempt_at", "provider_account_id",
|
||||
]));
|
||||
const content = await readFile(new URL(`./migrations/${migrationFile}`, import.meta.url), "utf8");
|
||||
const hash = createHash("sha256").update(content).digest("hex");
|
||||
await client`DELETE FROM drizzle.__drizzle_migrations WHERE hash = ${hash}`;
|
||||
await client.unsafe(`DROP INDEX ${indexName}`);
|
||||
const [company, otherCompany] = await client`INSERT INTO companies (name, issue_prefix)
|
||||
VALUES ('First', 'VM1'), ('Second', 'VM2') RETURNING id`;
|
||||
const agentId = randomUUID(), applicationId = randomUUID(), connectionId = randomUUID(), endpointId = randomUUID();
|
||||
await client`INSERT INTO agents (id, company_id, name, adapter_type) VALUES (${agentId}, ${company.id}, 'Migration voice', 'paperclip_runner')`;
|
||||
await client`INSERT INTO tool_applications (id, company_id, name, type) VALUES (${applicationId}, ${company.id}, 'Voice', 'chat')`;
|
||||
await client`INSERT INTO tool_connections (id, company_id, application_id, name, uid, transport, connection_purpose)
|
||||
VALUES (${connectionId}, ${company.id}, ${applicationId}, 'Voice', ${connectionId}, 'voice', 'channel')`;
|
||||
await client`INSERT INTO chat_endpoints (id, company_id, connection_id, assigned_agent_id, provider, public_id, status)
|
||||
VALUES (${endpointId}, ${company.id}, ${connectionId}, ${agentId}, 'speko', ${endpointId}, 'active')`;
|
||||
await client`WITH candidates AS (SELECT gen_random_uuid() AS id FROM generate_series(1, 1005))
|
||||
INSERT INTO chat_voice_inbound_calls (id, company_id, endpoint_id, provider_session_id, state,
|
||||
approval_code, generation, credential_fingerprint, tool_token_hash, request_fingerprint, expires_at)
|
||||
SELECT id, ${company.id}, ${endpointId}, 'migration:' || id::text, 'ended', '123456', 1,
|
||||
'migration', repeat('a', 64), id::text, now() FROM candidates`;
|
||||
const nilId = '00000000-0000-0000-0000-000000000000', signed = randomUUID();
|
||||
const denied = randomUUID(), wrongCompany = randomUUID(), wrongAction = randomUUID();
|
||||
const pending = randomUUID(), expired = randomUUID(), confirmed = randomUUID();
|
||||
const confirmedAt = new Date('2020-01-01T00:00:00Z');
|
||||
const rows = [
|
||||
{ id: nilId, state: 'ended', actor: null }, { id: signed, state: 'ended', actor: 'approver' },
|
||||
{ id: denied, state: 'ended', actor: 'denier' }, { id: wrongCompany, state: 'ended', actor: 'approver' },
|
||||
{ id: wrongAction, state: 'ended', actor: 'approver' }, { id: pending, state: 'awaiting_approval', actor: null },
|
||||
{ id: expired, state: 'expired', actor: null }, { id: confirmed, state: 'ended', actor: null },
|
||||
].map(row => ({ id: row.id, company_id: company.id, endpoint_id: endpointId,
|
||||
provider_session_id: `migration:${row.id}`, state: row.state, approved_by_user_id: row.actor,
|
||||
approval_code: '123456', generation: 1, credential_fingerprint: 'migration', tool_token_hash: 'a'.repeat(64),
|
||||
request_fingerprint: row.id, expires_at: new Date(), cleanup_confirmed_at: row.id === confirmed ? confirmedAt : null }));
|
||||
await client`INSERT INTO chat_voice_inbound_calls ${client(rows, 'id', 'company_id', 'endpoint_id',
|
||||
'provider_session_id', 'state', 'approved_by_user_id', 'approval_code', 'generation',
|
||||
'credential_fingerprint', 'tool_token_hash', 'request_fingerprint', 'expires_at', 'cleanup_confirmed_at')}`;
|
||||
await client`INSERT INTO activity_log (company_id, actor_id, action, entity_type, entity_id)
|
||||
VALUES (${company.id}, 'migration', 'voice.call.ended', 'voice_inbound_call', ${signed}),
|
||||
(${otherCompany.id}, 'migration', 'voice.call.ended', 'voice_inbound_call', ${wrongCompany}),
|
||||
(${company.id}, 'migration', 'voice.call.requested', 'voice_inbound_call', ${wrongAction})`;
|
||||
await client.unsafe(`CREATE TABLE voice_cleanup_backfill_probe (call_id uuid NOT NULL, transaction_id bigint NOT NULL);
|
||||
CREATE FUNCTION record_voice_cleanup_backfill() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN INSERT INTO voice_cleanup_backfill_probe VALUES (NEW.id, txid_current()); RETURN NEW; END $$;
|
||||
CREATE TRIGGER record_voice_cleanup_backfill AFTER UPDATE ON chat_voice_inbound_calls
|
||||
FOR EACH ROW WHEN (OLD.cleanup_confirmed_at IS NULL AND NEW.cleanup_confirmed_at IS NOT NULL)
|
||||
EXECUTE FUNCTION record_voice_cleanup_backfill()`);
|
||||
|
||||
// A failed unique build deterministically leaves an invalid index under
|
||||
// the migration's name, as cancellation does, without timing a cancel.
|
||||
await expect(client.unsafe(`CREATE UNIQUE INDEX CONCURRENTLY ${indexName} ON companies ((true))`)).rejects.toThrow();
|
||||
const [invalid] = await client`SELECT indisvalid FROM pg_index WHERE indexrelid = ${indexName}::regclass`;
|
||||
expect(invalid.indisvalid).toBe(false);
|
||||
await Promise.all([applyPendingMigrations(connectionString), applyPendingMigrations(connectionString)]);
|
||||
const [rebuilt] = await client`SELECT indisvalid, pg_get_indexdef(indexrelid) AS definition
|
||||
FROM pg_index WHERE indexrelid = ${indexName}::regclass`;
|
||||
expect(rebuilt.indisvalid).toBe(true);
|
||||
expect(rebuilt.definition).toContain("chat_voice_inbound_calls");
|
||||
expect(rebuilt.definition).toContain("cleanup_confirmed_at IS NULL");
|
||||
const [history] = await client`SELECT count(*)::int AS count FROM drizzle.__drizzle_migrations WHERE hash = ${hash}`;
|
||||
expect(history.count).toBe(1);
|
||||
const [backfill] = await client`SELECT count(*)::int AS count, count(DISTINCT transaction_id)::int AS transactions
|
||||
FROM voice_cleanup_backfill_probe`;
|
||||
expect(backfill.count).toBe(1007); expect(backfill.transactions).toBeGreaterThanOrEqual(2);
|
||||
const [batch] = await client`SELECT max(size)::int AS size FROM (
|
||||
SELECT count(*) AS size FROM voice_cleanup_backfill_probe GROUP BY transaction_id
|
||||
) groups`;
|
||||
expect(batch.size).toBeLessThanOrEqual(1000);
|
||||
const proof = await client`SELECT id, cleanup_confirmed_at FROM chat_voice_inbound_calls
|
||||
WHERE id = ANY(${[nilId, signed, denied, wrongCompany, wrongAction, pending, expired, confirmed]})`;
|
||||
for (const id of [nilId, signed]) expect(proof.find(row => row.id === id)?.cleanup_confirmed_at).toBeInstanceOf(Date);
|
||||
for (const id of [denied, wrongCompany, wrongAction, pending, expired])
|
||||
expect(proof.find(row => row.id === id)?.cleanup_confirmed_at).toBeNull();
|
||||
expect(proof.find(row => row.id === confirmed)?.cleanup_confirmed_at.getTime()).toBe(confirmedAt.getTime());
|
||||
await applyPendingMigrations(connectionString);
|
||||
const [unchanged] = await client`SELECT count(*)::int AS count FROM voice_cleanup_backfill_probe`;
|
||||
expect(unchanged.count).toBe(1007);
|
||||
} finally {
|
||||
await client?.end();
|
||||
if (admin && databaseName) await admin.unsafe(`DROP DATABASE "${databaseName}" WITH (FORCE)`);
|
||||
await admin?.end();
|
||||
await temporary?.cleanup();
|
||||
}
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -5,12 +5,12 @@
|
||||
"protocolVersion": 1,
|
||||
"profiles": {
|
||||
"grok": {
|
||||
"agentProfileVersion": 1,
|
||||
"agentProfileVersion": 7,
|
||||
"agentServerPackage": "builtin:grok-acp",
|
||||
"agentServerVersion": "1",
|
||||
"agentRuntimePackage": "native:grok",
|
||||
"agentRuntimeVersion": "1.0.13",
|
||||
"commandDigest": "sha256:f0b698395a3704ed2ffaf84ea19bdb20c36c8a0a70b7c629c7b6ffe144e59e55",
|
||||
"commandDigest": "sha256:6fcf754bf85aaca0b35ba6bad9926f11c60bdda5f7b180158c8f8b1be5e57d36",
|
||||
"requiresProviderPolicy": false
|
||||
},
|
||||
"pi": {
|
||||
@@ -32,12 +32,12 @@
|
||||
"requiresProviderPolicy": true
|
||||
},
|
||||
"copilot": {
|
||||
"agentProfileVersion": 70,
|
||||
"agentProfileVersion": 71,
|
||||
"agentServerPackage": "@github/copilot",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"agentRuntimePackage": null,
|
||||
"agentRuntimeVersion": null,
|
||||
"commandDigest": "sha256:618c3b69bc98b9a7e3c1b3411700eab8b90fd9245ba35d0bc85484319247fdbc",
|
||||
"commandDigest": "sha256:b7a25fe627daf3857ac8097d9a374b1ba15355176389b92457838bf19e9d91bd",
|
||||
"requiresProviderPolicy": true
|
||||
},
|
||||
"claude": {
|
||||
|
||||
@@ -1 +1 @@
|
||||
[{"annotations":{"exposure":"always","operationId":"get_task_context","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the active task and actor, including the exact approved Markdown revision when this issue has an accepted plan.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_task_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"get_task_history","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded comments on the active task.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":[],"type":"object"},"name":"get_task_history","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_documents","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List revisioned documents on the active task.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_documents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"read_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the current revision of one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"}},"required":["key"],"type":"object"},"name":"read_document","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_document_revisions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded revision history for one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":["key"],"type":"object"},"name":"list_document_revisions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"report_progress","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append a durable progress comment to the active task.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Multiline progress update.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"report_progress","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"answer_status_question","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append the answer to a status-only wake without changing task disposition.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Concise status answer.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"answer_status_question","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"write_document","requiredClaimLine truncated
|
||||
[{"annotations":{"exposure":"always","operationId":"get_task_context","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the active task and actor, including the exact approved Markdown revision when this issue has an accepted plan.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_task_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"get_task_history","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded comments on the active task.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":[],"type":"object"},"name":"get_task_history","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_documents","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List revisioned documents on the active task.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_documents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"read_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the current revision of one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"}},"required":["key"],"type":"object"},"name":"read_document","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_document_revisions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded revision history for one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":["key"],"type":"object"},"name":"list_document_revisions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"report_progress","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append a durable progress comment to the active task.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Multiline progress update.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"report_progress","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"answer_status_question","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append the answer to a status-only wake without changing task disposition.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Concise status answer.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"answer_status_question","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"write_document","requiredClaimLine truncated
|
||||
@@ -85,7 +85,7 @@
|
||||
"typecheck:browser": "tsc -p tsconfig.browser.json --noEmit",
|
||||
"test": "pnpm run test:typescript && pnpm run test:rust",
|
||||
"test:typescript": "pnpm run test:typescript:prep && vitest run",
|
||||
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/provider-pack-runtime-tree.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-runtime-dependency-roots.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs scripts/provider-pack-executable-shims.test.mjs scripts/controlled-evals/controller.test.mjs scripts/controlled-evals/portable.test.mjs && node --test scripts/build-copilot-distribution.test.mjs scripts/copilot-inner-distribution.test.mjs scripts/copilot-smoke-resources.test.mjs scripts/discover-copilot-acp.test.mjs scripts/materialize-copilot-binary.test.mjs scripts/install-copilot-assets.test.mjs scripts/probe-copilot-acp.test.mjs scripts/qualify-copilot-acp.test.mjs test/acpx-message-boundaries-package-contract.test.mjs",
|
||||
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/grok-profile-identity.test.mjs scripts/provider-pack-runtime-tree.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-runtime-dependency-roots.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs scripts/provider-pack-executable-shims.test.mjs scripts/controlled-evals/controller.test.mjs scripts/controlled-evals/portable.test.mjs && node --test scripts/build-copilot-distribution.test.mjs scripts/copilot-inner-distribution.test.mjs scripts/copilot-smoke-resources.test.mjs scripts/discover-copilot-acp.test.mjs scripts/materialize-copilot-binary.test.mjs scripts/install-copilot-assets.test.mjs scripts/probe-copilot-acp.test.mjs scripts/qualify-copilot-acp.test.mjs test/acpx-message-boundaries-package-contract.test.mjs",
|
||||
"test:typescript:vitest": "node ./scripts/run-pr-vitest-lane.mjs",
|
||||
"test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace",
|
||||
"test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider",
|
||||
|
||||
@@ -24,7 +24,7 @@
|
||||
"prpVersion": 1,
|
||||
"nativeExecutionVersion": 1,
|
||||
"catalogVersion": 1,
|
||||
"catalogSha256": "sha256:fe1bac862fa91b51997fb77423aad0ffcf24094ec449ac7eb39bf0f10fbd3e55",
|
||||
"catalogSha256": "sha256:72ced94fe0d081422eddb9dbf9e34ce9b5212e6007887e088590c22c37ae5294",
|
||||
"driverContractVersion": 1,
|
||||
"driverKind": "paperclip-deterministic",
|
||||
"driverVersion": "1.0.0"
|
||||
|
||||
@@ -180,7 +180,7 @@
|
||||
},
|
||||
{
|
||||
"path": "fixtures/evals/native-execution-seeded.json",
|
||||
"sha256": "ae5614f8a1f1404394c86e303326e735b1abd23701a07db9d7376db64387d360",
|
||||
"sha256": "64b6f6f618bb6f6182d2a6b066b1f2775204857a2c460156931346c69a8e46fc",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "canonical"
|
||||
},
|
||||
|
||||
@@ -20,7 +20,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option<AcpxReleaseProfile> {
|
||||
agent_runtime_package: Some("native:grok"),
|
||||
agent_runtime_version: Some("1.0.13"),
|
||||
command_digest:
|
||||
"sha256:f0b698395a3704ed2ffaf84ea19bdb20c36c8a0a70b7c629c7b6ffe144e59e55",
|
||||
"sha256:6fcf754bf85aaca0b35ba6bad9926f11c60bdda5f7b180158c8f8b1be5e57d36",
|
||||
requires_provider_policy: false,
|
||||
},
|
||||
"pi" => AcpxReleaseProfile {
|
||||
@@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option<AcpxReleaseProfile> {
|
||||
agent_runtime_package: None,
|
||||
agent_runtime_version: None,
|
||||
command_digest:
|
||||
"sha256:618c3b69bc98b9a7e3c1b3411700eab8b90fd9245ba35d0bc85484319247fdbc",
|
||||
"sha256:b7a25fe627daf3857ac8097d9a374b1ba15355176389b92457838bf19e9d91bd",
|
||||
requires_provider_policy: true,
|
||||
},
|
||||
"claude" => AcpxReleaseProfile {
|
||||
|
||||
@@ -95,6 +95,7 @@ const safeAdmissionReasons = new Set([
|
||||
'Invalid operating overhead component', 'Invalid operating overhead fields', 'Invalid controller observability allocation',
|
||||
'Observability capacity requires its operating bound evidence', 'Hosted manifest bytes changed',
|
||||
'Cell is outside the controlled selection', 'Hosted controller admission does not match the current first-attempt worker',
|
||||
'Hosted controller requires GitHub-hosted operating origin',
|
||||
]);
|
||||
export function hostedAdmissionDiagnostic(error) {
|
||||
if (error instanceof Error && safeAdmissionReasons.has(error.message)) return error.message;
|
||||
|
||||
@@ -6,7 +6,7 @@ import { cp, mkdir, readFile, rm, writeFile } from 'node:fs/promises';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
import { fixture, success } from './controller-fixture.mjs';
|
||||
import { canonical, sha256 } from './manifest.mjs';
|
||||
import { canonical, sha256, validateManifest } from './manifest.mjs';
|
||||
const runtime = process.env.PAPERCLIP_CONTROLLER_TEST_RUNTIME ? pathToFileURL(resolve(process.env.PAPERCLIP_CONTROLLER_TEST_RUNTIME) + '/') : import.meta.url;
|
||||
const { hostedPaths, hostedPolicyConfiguration, prepareHostedManifest, verifyHostedCell, hostedAdmissionDiagnostic } = await import(new URL('./hosted.mjs', runtime));
|
||||
const { packControlledAttempt, summarizeControlledManifest } = await import(new URL('./archive.mjs', runtime));
|
||||
@@ -96,6 +96,25 @@ test('hosted preparation rejects selection drift, unsupported cells and unreview
|
||||
assert.throws(() => prepareHostedManifest(input), /budget/);
|
||||
});
|
||||
|
||||
test('hosted admission rejects local and fleet origins while preserving retained manifests', async t => {
|
||||
const c = await campaign(t);
|
||||
c.input.policy.overhead.accountingReadiness = 'model_requests';
|
||||
for (const component of c.input.policy.overhead.components) delete component.provisionalBound;
|
||||
delete c.input.policy.capacityBound;
|
||||
const original = c.build().manifest;
|
||||
for (const runner of ['local', 'aws-fleet']) {
|
||||
const input = structuredClone(c.input); input.policy.overhead.runner = runner;
|
||||
assert.throws(() => prepareHostedManifest(input), /GitHub-hosted operating origin/);
|
||||
const retained = structuredClone(original); retained.overhead.runner = runner;
|
||||
assert.equal(validateManifest(retained, Date.now(), { admission: false }).manifest.overhead.runner, runner);
|
||||
await writeFile(c.paths.manifest, canonical(retained));
|
||||
await assert.rejects(verifyHostedCell({ manifestPath: c.paths.manifest, manifestSha256: sha256(canonical(retained)),
|
||||
cellId: 'execution-0', environment: { ...c.environment, PAPERCLIP_RUNNER_EVAL_EXECUTION_ID: 'execution-0' } }), /GitHub-hosted operating origin/);
|
||||
}
|
||||
const local = structuredClone(original); delete local.hosted; local.overhead.runner = 'local';
|
||||
assert.equal(validateManifest(local).manifest.overhead.runner, 'local');
|
||||
});
|
||||
|
||||
test('worker preflight verifies identical manifest bytes and current cell/run before credentials', async t => {
|
||||
const c = await campaign(t), { manifest, manifestSha256 } = c.build();
|
||||
await writeFile(c.paths.manifest, canonical(manifest));
|
||||
|
||||
@@ -46,7 +46,7 @@ export function validateManifest(value, now = Date.now(), { admission = true } =
|
||||
fields(artifact, ['path', 'sha256']);
|
||||
if (!isAbsolute(artifact.path ?? '') || !hash(artifact.sha256)) throw new Error('Invalid controller executable provenance');
|
||||
}
|
||||
validateHostedIdentity(value);
|
||||
validateHostedIdentity(value, { admission });
|
||||
const overhead = costRuntime.validateOperatingOverhead(value.overhead, value.maxUsd);
|
||||
if (admission && Date.parse(value.overhead.verifiedAt) > now || Date.parse(value.overhead.expiresAt) <= deadline
|
||||
|| value.overhead.accountingReadiness !== 'model_requests' && value.overhead.components.some(component => !component.provisionalBound)) throw new Error('Controller requires current full-lifecycle operating bounds');
|
||||
@@ -83,7 +83,7 @@ export function validateManifest(value, now = Date.now(), { admission = true } =
|
||||
validateObservability(value);
|
||||
return { manifest: structuredClone(value), manifestSha256: sha256(canonical(value)), allocatedUsd: usd(allocated), overheadUsd: usd(overhead) };
|
||||
}
|
||||
function validateHostedIdentity(manifest) {
|
||||
function validateHostedIdentity(manifest, { admission = true } = {}) {
|
||||
const hosted = manifest.hosted;
|
||||
if (hosted === undefined) return;
|
||||
fields(hosted, ['repository', 'runId', 'runAttempt', 'workflowRevision', 'policySha256', 'controllerLockSha256']);
|
||||
@@ -93,6 +93,7 @@ function validateHostedIdentity(manifest) {
|
||||
|| hosted.controllerLockSha256 !== undefined && !hash(hosted.controllerLockSha256)
|
||||
|| hosted.runAttempt !== 1 || typeof hosted.workflowRevision !== 'string' || !/^[a-f0-9]{40}$/.test(hosted.workflowRevision)
|
||||
|| manifest.campaignId !== `gha-${hosted.runId}-1`) throw new Error('Invalid hosted controller identity');
|
||||
if (admission && manifest.overhead?.runner !== 'github-hosted') throw new Error('Hosted controller requires GitHub-hosted operating origin');
|
||||
}
|
||||
|
||||
export function assertHostedAdmission(manifest, attempt, environment) {
|
||||
|
||||
@@ -92,7 +92,7 @@ test('copied controller runs without source dependencies, retains native evidenc
|
||||
const hosted = await exec(process.execPath, ['--test', '--test-reporter=tap', join(source, 'hosted.test.mjs')], {
|
||||
cwd: root, env: { ...environment, PAPERCLIP_CONTROLLER_TEST_RUNTIME: join(directory, 'controlled-evals') }, timeout: 30000, maxBuffer: 1_048_576,
|
||||
});
|
||||
assert.match(hosted.stdout, /# pass 6\b/);
|
||||
assert.match(hosted.stdout, /# pass 7\b/);
|
||||
|
||||
|
||||
const cli = await exec(process.execPath, [join(directory, 'controlled-evals/archive.mjs'), 'report-attempts', join(root, 'empty-attempts')], {
|
||||
|
||||
@@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url";
|
||||
import { resolve, dirname } from "node:path";
|
||||
|
||||
const root = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const fixture = resolve(root, "test/fixtures/copilot-profile-v70-identity.json");
|
||||
const fixture = resolve(root, "test/fixtures/copilot-profile-v71-identity.json");
|
||||
const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8"));
|
||||
const declaration = { ...historical.declaration, agentProfileVersion: 70 };
|
||||
const declaration = { ...historical.declaration, agentProfileVersion: 71 };
|
||||
delete declaration.permissionIdentitySourceSha256;
|
||||
const sources = {
|
||||
displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts",
|
||||
|
||||
@@ -2,6 +2,7 @@ import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { grokProfileIdentity, GROK_PROFILE_VERSION } from "./grok-profile-identity.mjs";
|
||||
|
||||
const root = new URL("../", import.meta.url);
|
||||
const readJson = async path => JSON.parse(await readFile(new URL(path, root), "utf8"));
|
||||
@@ -11,6 +12,11 @@ const contract = await readJson("cursor-contract.json");
|
||||
const pkg = await readJson("package.json");
|
||||
assert.equal(manifest.schema, "paperclip.acpx-profiles.v1");
|
||||
assert.equal(manifest.acpxVersion, pkg.dependencies.acpx);
|
||||
const grokIdentity = grokProfileIdentity();
|
||||
assert.deepEqual(await readJson(`test/fixtures/grok-profile-v${GROK_PROFILE_VERSION}-identity.json`), grokIdentity, "Grok source identity is stale; advance the profile and regenerate its identity");
|
||||
assert.equal(manifest.profiles.grok.agentProfileVersion, GROK_PROFILE_VERSION);
|
||||
assert.equal(manifest.profiles.grok.commandDigest, grokIdentity.commandDigest);
|
||||
assert.equal(`sha256:${createHash("sha256").update(await readFile(new URL("src/providers/grok/launcher.cjs", root))).digest("hex")}`, grokIdentity.commandDigest, "Grok launcher source stamp is stale");
|
||||
const entries = Object.entries(manifest.profiles);
|
||||
for (const [agent, profile] of entries) {
|
||||
assert.match(agent, /^[a-z]+$/);
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { existsSync, readFileSync, realpathSync, writeFileSync } from "node:fs";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const repositoryRoot = resolve(packageRoot, "../..");
|
||||
// Versions 1–6 remain decodable historical identities. Versioned fixtures are
|
||||
// immutable; advance this version before changing a versioned declaration.
|
||||
export const GROK_PROFILE_VERSION = 7;
|
||||
// Contract surfaces: ACPX wire patch; sidecar parsing and error translation;
|
||||
// native effort/search/credential admission; host/sandbox/tool-bridge execution;
|
||||
// launcher integrity and platform pins; Rust descriptor writes, session state,
|
||||
// and transport classification. This is an explicit boundary list, not the
|
||||
// sidecar's complete import graph.
|
||||
export const GROK_PROFILE_SOURCE_FILES = [
|
||||
"patches/acpx@0.13.1.patch",
|
||||
...[
|
||||
"src/cli/acpx-runtime-sidecar.ts", "src/cli/acpx-sidecar-input.ts",
|
||||
"src/drivers/acpx/admission-diagnostics.ts", "src/drivers/acpx/grok-effort.ts",
|
||||
"src/drivers/acpx/grok-web-search.ts", "src/drivers/acpx/grok-credentials.ts",
|
||||
"src/drivers/acpx/codex-runtime-adapter.ts", "src/drivers/acpx/runtime-host.ts",
|
||||
"src/drivers/acpx/runtime-sandbox.ts", "src/drivers/acpx/sidecar-protocol.ts",
|
||||
"src/drivers/acpx/native-distribution-integrity.ts",
|
||||
"src/providers/grok/launcher.cjs", "src/providers/grok/platforms.json",
|
||||
"runner/crates/runner-core/src/acpx_provider_backend.rs",
|
||||
"runner/crates/runner-core/src/acpx_provider_session.rs",
|
||||
"runner/crates/runner-core/src/acpx_sidecar_transport.rs",
|
||||
].map(path => `packages/paperclip-runner/${path}`),
|
||||
];
|
||||
const canonical = value => value && typeof value === "object"
|
||||
? Array.isArray(value) ? `[${value.map(canonical).join(",")}]`
|
||||
: `{${Object.keys(value).sort().map(key => `${JSON.stringify(key)}:${canonical(value[key])}`).join(",")}}`
|
||||
: JSON.stringify(value);
|
||||
const sha = bytes => `sha256:${createHash("sha256").update(bytes).digest("hex")}`;
|
||||
const launcherPath = "packages/paperclip-runner/src/providers/grok/launcher.cjs";
|
||||
const launcherBody = root => readFileSync(resolve(root, launcherPath), "utf8")
|
||||
.replace(/^\/\/ Grok admission source identity: sha256:[a-f0-9]{64}\r?\n/u, "");
|
||||
const stampedLauncher = (root, sourceDigest) => `// Grok admission source identity: ${sourceDigest}\n${launcherBody(root)}`;
|
||||
|
||||
export function grokProfileIdentity(root = repositoryRoot) {
|
||||
const manifest = JSON.parse(readFileSync(resolve(root, "packages/paperclip-runner/acpx-profiles.json"), "utf8"));
|
||||
const { commandDigest: _digest, ...profile } = manifest.profiles.grok;
|
||||
const declaration = {
|
||||
schema: "paperclip.grok-profile-identity.v1", agent: "grok",
|
||||
...profile, agentProfileVersion: GROK_PROFILE_VERSION, acpxVersion: manifest.acpxVersion,
|
||||
sources: Object.fromEntries(GROK_PROFILE_SOURCE_FILES.map(path => [path,
|
||||
sha(path === launcherPath ? launcherBody(root) : readFileSync(resolve(root, path)))])),
|
||||
};
|
||||
const sourceDigest = sha(canonical(declaration));
|
||||
// Preserve the command's physical byte-digest contract. The source stamp
|
||||
// changes those bytes when the Grok admission implementation changes.
|
||||
return { commandDigest: sha(stampedLauncher(root, sourceDigest)), sourceDigest, declaration };
|
||||
}
|
||||
|
||||
if (process.argv[1] && realpathSync(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
const identity = grokProfileIdentity();
|
||||
const manifestPath = resolve(packageRoot, "acpx-profiles.json");
|
||||
const fixturePath = resolve(packageRoot, `test/fixtures/grok-profile-v${GROK_PROFILE_VERSION}-identity.json`);
|
||||
const manifest = JSON.parse(readFileSync(manifestPath, "utf8"));
|
||||
if (process.argv.includes("--check")) {
|
||||
assert.deepEqual(JSON.parse(readFileSync(fixturePath, "utf8")), identity, "Grok source identity is stale");
|
||||
assert.equal(manifest.profiles.grok.commandDigest, identity.commandDigest);
|
||||
assert.equal(manifest.profiles.grok.agentProfileVersion, GROK_PROFILE_VERSION);
|
||||
assert.equal(readFileSync(resolve(repositoryRoot, launcherPath), "utf8"), stampedLauncher(repositoryRoot, identity.sourceDigest));
|
||||
} else {
|
||||
if (existsSync(fixturePath) && canonical(JSON.parse(readFileSync(fixturePath, "utf8"))) !== canonical(identity)) {
|
||||
throw new Error(`Grok profile ${GROK_PROFILE_VERSION} identity is immutable; advance GROK_PROFILE_VERSION before changing bound source`);
|
||||
}
|
||||
writeFileSync(resolve(repositoryRoot, launcherPath), stampedLauncher(repositoryRoot, identity.sourceDigest));
|
||||
manifest.profiles.grok.commandDigest = identity.commandDigest;
|
||||
manifest.profiles.grok.agentProfileVersion = GROK_PROFILE_VERSION;
|
||||
writeFileSync(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`);
|
||||
writeFileSync(fixturePath, `${JSON.stringify(identity, null, 2)}\n`);
|
||||
}
|
||||
execFileSync(process.execPath, [resolve(packageRoot, "scripts/generate-acpx-profiles.mjs"),
|
||||
...(process.argv.includes("--check") ? ["--check"] : [])], { stdio: "inherit" });
|
||||
console.log(identity.commandDigest);
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { copyFileSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import test from "node:test";
|
||||
import { grokProfileIdentity, GROK_PROFILE_SOURCE_FILES, GROK_PROFILE_VERSION } from "./grok-profile-identity.mjs";
|
||||
|
||||
const root = resolve(dirname(fileURLToPath(import.meta.url)), "../../..");
|
||||
test("the active Grok profile attests its exact admission source", () => {
|
||||
const identity = grokProfileIdentity(root);
|
||||
const manifest = JSON.parse(readFileSync(join(root, "packages/paperclip-runner/acpx-profiles.json"), "utf8"));
|
||||
const fixture = JSON.parse(readFileSync(join(root, `packages/paperclip-runner/test/fixtures/grok-profile-v${GROK_PROFILE_VERSION}-identity.json`), "utf8"));
|
||||
assert.deepEqual(identity, fixture);
|
||||
assert.equal(manifest.profiles.grok.commandDigest, identity.commandDigest);
|
||||
assert.equal(manifest.profiles.grok.agentProfileVersion, GROK_PROFILE_VERSION);
|
||||
});
|
||||
|
||||
for (const contract of [
|
||||
"src/cli/acpx-runtime-sidecar.ts",
|
||||
"runner/crates/runner-core/src/acpx_provider_backend.rs",
|
||||
]) test(`changing the Grok field in ${contract} invalidates the identity`, () => {
|
||||
const copy = mkdtempSync(join(tmpdir(), "grok-profile-source-"));
|
||||
try {
|
||||
for (const path of ["packages/paperclip-runner/acpx-profiles.json", ...GROK_PROFILE_SOURCE_FILES,
|
||||
`packages/paperclip-runner/${contract}`]) {
|
||||
mkdirSync(dirname(join(copy, path)), { recursive: true });
|
||||
copyFileSync(join(root, path), join(copy, path));
|
||||
}
|
||||
const before = grokProfileIdentity(copy);
|
||||
const contractPath = join(copy, "packages/paperclip-runner", contract);
|
||||
const source = readFileSync(contractPath, "utf8");
|
||||
assert.ok(source.includes('"grokEffort"'));
|
||||
writeFileSync(contractPath, source.replace('"grokEffort"', '"obsoleteGrokEffort"'));
|
||||
assert.notEqual(grokProfileIdentity(copy).commandDigest, before.commandDigest);
|
||||
} finally {
|
||||
rmSync(copy, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("the generator cannot overwrite a versioned identity after source changes", () => {
|
||||
const copy = mkdtempSync(join(tmpdir(), "grok-profile-immutable-"));
|
||||
const prefix = "packages/paperclip-runner/";
|
||||
const manifest = JSON.parse(readFileSync(join(root, prefix, "acpx-profiles.json"), "utf8"));
|
||||
const fixture = `${prefix}test/fixtures/grok-profile-v${GROK_PROFILE_VERSION}-identity.json`;
|
||||
const launcher = `${prefix}src/providers/grok/launcher.cjs`;
|
||||
try {
|
||||
for (const path of new Set([...GROK_PROFILE_SOURCE_FILES, ...[
|
||||
"acpx-profiles.json", "cursor-distributions.json", "cursor-contract.json", "package.json",
|
||||
"scripts/grok-profile-identity.mjs", "scripts/generate-acpx-profiles.mjs",
|
||||
`test/fixtures/copilot-profile-v${manifest.profiles.copilot.agentProfileVersion}-identity.json`,
|
||||
`test-fixtures/pi-acp/profile-v${manifest.profiles.pi.agentProfileVersion}-identity.json`,
|
||||
].map(path => prefix + path), fixture])) {
|
||||
mkdirSync(dirname(join(copy, path)), { recursive: true });
|
||||
copyFileSync(join(root, path), join(copy, path));
|
||||
}
|
||||
const protectedFiles = [fixture, launcher, `${prefix}acpx-profiles.json`];
|
||||
const before = protectedFiles.map(path => readFileSync(join(copy, path), "utf8"));
|
||||
const generate = () => spawnSync(process.execPath, [join(copy, prefix, "scripts/grok-profile-identity.mjs")],
|
||||
{ cwd: copy, env: { PATH: process.env.PATH }, encoding: "utf8", timeout: 10000 });
|
||||
const unchanged = generate();
|
||||
assert.equal(unchanged.status, 0, unchanged.stderr);
|
||||
assert.deepEqual(protectedFiles.map(path => readFileSync(join(copy, path), "utf8")), before);
|
||||
const sidecar = join(copy, prefix, "src/cli/acpx-runtime-sidecar.ts");
|
||||
writeFileSync(sidecar, readFileSync(sidecar, "utf8").replace('"grokEffort"', '"obsoleteGrokEffort"'));
|
||||
const result = generate();
|
||||
assert.deepEqual(protectedFiles.map(path => readFileSync(join(copy, path), "utf8")), before, "a changed identity must be refused before any write");
|
||||
assert.notEqual(result.status, 0, "source changes must require a new profile version");
|
||||
assert.match(result.stderr, /identity is immutable/);
|
||||
} finally {
|
||||
rmSync(copy, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
@@ -1221,6 +1221,21 @@ describe("qualified ACPX runtime sidecar", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it.each([true, false])("admits the current Grok effort/search fields before host checks (search disabled: %s)", async grokDisableWebSearch => {
|
||||
const sidecar = startSidecar(), model = "grok-4.7";
|
||||
sidecar.write(initializeRequest(1, "grok", model));
|
||||
expect(await sidecar.next(frame => frame.id === 1)).toMatchObject({ ok: true,
|
||||
result: { profile: { agent: "grok", commandDigest: resolveQualifiedAcpxProfile("grok", model).commandDigest } } });
|
||||
sidecar.write({ protocolVersion: ACPX_SIDECAR_PROTOCOL_VERSION, id: 2, command: "session.open",
|
||||
params: { runtimeDirectory: "/unused", workingDirectory: "/", normalizedSessionId: "grok-effort-open",
|
||||
agent: "grok", model, permissionMode: "approve-all", grokEffort: "high", grokDisableWebSearch,
|
||||
systemInstructions: "Test instructions", tools: [] } });
|
||||
// The working-directory gate stops this real process before installation,
|
||||
// credentials, or inference. The old pack rejected the new fields instead.
|
||||
expect(await sidecar.next(frame => frame.id === 2)).toMatchObject({ ok: false,
|
||||
error: { code: "ACPX_ADMISSION_BINDING", message: "ACPX working directory must be a non-root directory" } });
|
||||
});
|
||||
|
||||
it.each([
|
||||
["pi", { piThinkingLevel: "medium" }, "Pi thinking level"],
|
||||
["codex", { piThinkingLevel: "low" }, "only supported by Pi"],
|
||||
|
||||
@@ -31,6 +31,7 @@ import { digestPaperclipSemanticContent } from "../semantic-tools/receipts.js";
|
||||
import {
|
||||
DurablePrpControlPlane,
|
||||
decodeDurablePrpCheckpoint,
|
||||
inspectDurablePrpCheckpointHeader,
|
||||
type DurablePrpCheckpoint,
|
||||
durableRecoveryInternals,
|
||||
SemanticToolNotDispatchedError,
|
||||
@@ -174,6 +175,10 @@ it("restores the previous checkpoint format and upgrades it on the next capture"
|
||||
await core.flushCheckpoint();
|
||||
const records = new Map(checkpoint.records);
|
||||
const header = JSON.parse(records.get("header")!);
|
||||
expect(inspectDurablePrpCheckpointHeader(records.get("header")!)).toEqual(identity);
|
||||
expect(inspectDurablePrpCheckpointHeader("{}")).toBeNull();
|
||||
expect(inspectDurablePrpCheckpointHeader("{")).toBeNull();
|
||||
expect(inspectDurablePrpCheckpointHeader(JSON.stringify({ ...header, schema: "paperclip.runner.controller-checkpoint.v1" }))).toBeNull();
|
||||
const cursor = JSON.parse(records.get("cursor")!);
|
||||
records.delete("cursor");
|
||||
records.set("header", JSON.stringify({ ...header, schema: "paperclip.runner.controller-checkpoint.v1",
|
||||
|
||||
@@ -377,12 +377,33 @@ function captureCheckpoint(state: StoredCoreState, runnerVersion: string, runner
|
||||
};
|
||||
}
|
||||
|
||||
function checkpointState(records: ReadonlyMap<string, string>) {
|
||||
const header: unknown = JSON.parse(records.get("header") ?? "null");
|
||||
function parseCheckpointHeader(contents: string): Record<string, unknown> & {
|
||||
runnerVersion: string; runnerDigest: string;
|
||||
state: Record<string, unknown> & { identity: DurableRecoveryIdentity };
|
||||
} {
|
||||
const header: unknown = JSON.parse(contents);
|
||||
if (!isRecord(header) || !["paperclip.runner.controller-checkpoint.v1", "paperclip.runner.controller-checkpoint.v2"].includes(String(header.schema))
|
||||
|| typeof header.runnerVersion !== "string" || !header.runnerVersion
|
||||
|| typeof header.runnerDigest !== "string" || !runnerDigestPattern.test(header.runnerDigest)
|
||||
|| !isRecord(header.state) || !exactIdentity(header.state.identity)) throw new Error("native_runner_checkpoint_invalid");
|
||||
return { ...header, runnerVersion: header.runnerVersion, runnerDigest: header.runnerDigest,
|
||||
state: { ...header.state, identity: header.state.identity } };
|
||||
}
|
||||
|
||||
/** Bounded header evidence for rollout policy, not permission to restore or attach.
|
||||
* Full restoration still validates every record through decodeDurablePrpCheckpoint. */
|
||||
export function inspectDurablePrpCheckpointHeader(contents: string): DurableRecoveryIdentity | null {
|
||||
try {
|
||||
const header = parseCheckpointHeader(contents);
|
||||
if (header.schema !== "paperclip.runner.controller-checkpoint.v2"
|
||||
|| header.state.checkpointEventPolicy !== "control-v1"
|
||||
|| ![coreStateSchema, transitionCoreStateSchema].includes(String(header.state.schema))) return null;
|
||||
return { ...header.state.identity };
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
function checkpointState(records: ReadonlyMap<string, string>) {
|
||||
const header = parseCheckpointHeader(records.get("header") ?? "null");
|
||||
const read = (key: string): unknown => {
|
||||
const value = records.get(key);
|
||||
if (value === undefined) throw new Error("native_runner_checkpoint_incomplete");
|
||||
|
||||
@@ -87,9 +87,21 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi
|
||||
|
||||
describe("Copilot build-owned installation", () => {
|
||||
it("admits the current profile with environment launch preparation", () => {
|
||||
expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(70);
|
||||
expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(71);
|
||||
expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined();
|
||||
expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:618c3b69bc98b9a7e3c1b3411700eab8b90fd9245ba35d0bc85484319247fdbc");
|
||||
expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:b7a25fe627daf3857ac8097d9a374b1ba15355176389b92457838bf19e9d91bd");
|
||||
});
|
||||
|
||||
it("preserves v70 while binding historical Grok recovery decoding in v71", async () => {
|
||||
const bytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v70-identity.json", import.meta.url));
|
||||
expect(createHash("sha256").update(bytes).digest("hex")).toBe("7ab63e2832f9d9c3273b5f799b25a02cba18718e956966c3b79052d74b4bd304");
|
||||
const prior = JSON.parse(bytes.toString("utf8"));
|
||||
const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v71-identity.json", import.meta.url), "utf8"));
|
||||
expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort())
|
||||
.toEqual(["agentProfileVersion", "profileCompatibilitySourceSha256"]);
|
||||
expect(isSupportedAcpxProfileVersion("copilot", 70)).toBe(true);
|
||||
await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot,
|
||||
agentProfileVersion: 70, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned");
|
||||
});
|
||||
|
||||
it("preserves v69 while binding closed startup diagnostics in v70", async () => {
|
||||
|
||||
@@ -9,12 +9,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = {
|
||||
"protocolVersion": 1,
|
||||
"acpxVersion": "0.13.1",
|
||||
"agent": "grok",
|
||||
"agentProfileVersion": 1,
|
||||
"agentProfileVersion": 7,
|
||||
"agentServerPackage": "builtin:grok-acp",
|
||||
"agentServerVersion": "1",
|
||||
"agentRuntimePackage": "native:grok",
|
||||
"agentRuntimeVersion": "1.0.13",
|
||||
"commandDigest": "sha256:f0b698395a3704ed2ffaf84ea19bdb20c36c8a0a70b7c629c7b6ffe144e59e55",
|
||||
"commandDigest": "sha256:6fcf754bf85aaca0b35ba6bad9926f11c60bdda5f7b180158c8f8b1be5e57d36",
|
||||
"permissionPolicy": "interactive"
|
||||
},
|
||||
"pi": {
|
||||
@@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = {
|
||||
"protocolVersion": 1,
|
||||
"acpxVersion": "0.13.1",
|
||||
"agent": "copilot",
|
||||
"agentProfileVersion": 70,
|
||||
"agentProfileVersion": 71,
|
||||
"agentServerPackage": "@github/copilot",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"agentRuntimePackage": null,
|
||||
"agentRuntimeVersion": null,
|
||||
"commandDigest": "sha256:618c3b69bc98b9a7e3c1b3411700eab8b90fd9245ba35d0bc85484319247fdbc",
|
||||
"commandDigest": "sha256:b7a25fe627daf3857ac8097d9a374b1ba15355176389b92457838bf19e9d91bd",
|
||||
"permissionPolicy": "interactive"
|
||||
},
|
||||
"claude": {
|
||||
|
||||
@@ -17,10 +17,14 @@ describe("historical ACPX profile decoding", () => {
|
||||
expect(isSupportedAcpxProfileVersion(agent, current)).toBe(true);
|
||||
expect(isSupportedAcpxProfileVersion(agent, Math.max(16, current) + 1)).toBe(false);
|
||||
}
|
||||
for (const agent of ["claude", "codex", "grok"]) {
|
||||
for (const agent of ["claude", "codex"]) {
|
||||
expect(isSupportedAcpxProfileVersion(agent, 5)).toBe(true);
|
||||
expect(isSupportedAcpxProfileVersion(agent, 6)).toBe(false);
|
||||
}
|
||||
for (const version of [1, 2, 3, 4, 5, QUALIFIED_ACPX_PROFILE_DATA.grok.agentProfileVersion]) {
|
||||
expect(isSupportedAcpxProfileVersion("grok", version)).toBe(true);
|
||||
}
|
||||
expect(isSupportedAcpxProfileVersion("grok", QUALIFIED_ACPX_PROFILE_DATA.grok.agentProfileVersion + 1)).toBe(false);
|
||||
});
|
||||
it("decodes retained Copilot revisions and the current profile without widening other providers", () => {
|
||||
for (const version of [1, 5, 6, 12, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40]) {
|
||||
@@ -48,6 +52,15 @@ describe("historical ACPX profile decoding", () => {
|
||||
expect(isSupportedAcpxProfileVersion("copilot", identity.declaration.agentProfileVersion), file).toBe(true);
|
||||
}
|
||||
});
|
||||
it("decodes every retained Grok identity so saved runs can reach recovery", () => {
|
||||
const root = new URL("../../../test/fixtures/", import.meta.url);
|
||||
const files = readdirSync(root).filter(name => /^grok-profile-v\d+-identity\.json$/u.test(name));
|
||||
expect(files.length).toBeGreaterThan(0);
|
||||
for (const file of files) {
|
||||
const identity = JSON.parse(readFileSync(new URL(file, root), "utf8"));
|
||||
expect(isSupportedAcpxProfileVersion("grok", identity.declaration.agentProfileVersion), file).toBe(true);
|
||||
}
|
||||
});
|
||||
it.each([0, 1.5, "11", null, undefined, NaN])("rejects invalid revisions: %s", version => {
|
||||
expect(isSupportedAcpxProfileVersion("cursor", version)).toBe(false);
|
||||
});
|
||||
|
||||
@@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js";
|
||||
|
||||
/** Decodable historical profile revisions, not permission to launch them.
|
||||
* Exact current profile/package/digest admission is checked separately. */
|
||||
type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69;
|
||||
type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70;
|
||||
type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"];
|
||||
export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion;
|
||||
|
||||
const historicalVersions: Readonly<Record<string, readonly HistoricalAcpxProfileVersion[]>> = {
|
||||
pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5],
|
||||
grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
|
||||
grok: [1, 2, 3, 4, 5, 6], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70],
|
||||
cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { createServer } from "node:http";
|
||||
import { mkdtemp, mkdir, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
import { claudeCliPath } from "@agentclientprotocol/claude-agent-acp/dist/acp-agent.js";
|
||||
import { expect, it } from "vitest";
|
||||
import { requestHumanInputAction } from "../protocol-actions/request-human-input.js";
|
||||
import { startRunnerToolBridge } from "./runner-tool-bridge.js";
|
||||
|
||||
it("keeps the human-input tool available in the pinned Claude client", async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), "claude-question-schema-"));
|
||||
const config = join(root, "config"), cwd = join(root, "workspace");
|
||||
await Promise.all([mkdir(config), mkdir(cwd)]);
|
||||
const offered: Array<Array<{ name: string; input_schema: Record<string, unknown> }>> = [];
|
||||
const api = createServer(async (request, response) => {
|
||||
let raw = "";
|
||||
for await (const chunk of request) raw += chunk;
|
||||
const body = JSON.parse(raw || "{}");
|
||||
if (Array.isArray(body.tools)) offered.push(body.tools);
|
||||
response.writeHead(200, { "content-type": "text/event-stream" });
|
||||
// The real client reaches only this local API, which never asks it to run tools.
|
||||
const frames = [
|
||||
{ type: "message_start", message: { id: "msg_fixture", type: "message", role: "assistant", model: "claude-sonnet-5", content: [], stop_reason: null, stop_sequence: null, usage: { input_tokens: 1, output_tokens: 0 } } },
|
||||
{ type: "content_block_start", index: 0, content_block: { type: "text", text: "" } },
|
||||
{ type: "content_block_delta", index: 0, delta: { type: "text_delta", text: "Local fixture complete." } },
|
||||
{ type: "content_block_stop", index: 0 },
|
||||
{ type: "message_delta", delta: { stop_reason: "end_turn", stop_sequence: null }, usage: { output_tokens: 1 } },
|
||||
{ type: "message_stop" },
|
||||
];
|
||||
for (const frame of frames) response.write(`event: ${frame.type}\ndata: ${JSON.stringify(frame)}\n\n`);
|
||||
response.end();
|
||||
});
|
||||
const bridge = await startRunnerToolBridge({ tools: [
|
||||
{ name: "request_human_input", description: "Ask a durable question.", inputSchema: requestHumanInputAction.live.descriptor.inputSchema },
|
||||
{ name: "write_document", description: "Discovery control.", inputSchema: { type: "object", properties: { key: { type: "string" } } } },
|
||||
], handler: async () => { throw new Error("This fixture must not execute tools"); } });
|
||||
try {
|
||||
await new Promise<void>(resolve => api.listen(0, "127.0.0.1", resolve));
|
||||
const address = api.address() as { port: number };
|
||||
const mcpConfig = { mcpServers: { paperclip: { type: "http", url: bridge.url, headers: { Authorization: `Bearer ${bridge.secret}` } } } };
|
||||
await promisify(execFile)(await claudeCliPath(), [
|
||||
"--bare", "--print", "--model", "claude-sonnet-5", "--strict-mcp-config", "--mcp-config", JSON.stringify(mcpConfig),
|
||||
"--setting-sources", "", "--permission-mode", "dontAsk", "--no-session-persistence", "--tools", "", "--max-turns", "1",
|
||||
"Report this local fixture's result. Do not call tools.",
|
||||
], { cwd, timeout: 20_000, env: {
|
||||
PATH: process.env.PATH, TMPDIR: root, CLAUDE_CONFIG_DIR: config,
|
||||
ANTHROPIC_API_KEY: "local-fixture-only", ANTHROPIC_BASE_URL: `http://127.0.0.1:${address.port}`,
|
||||
CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: "1", DISABLE_TELEMETRY: "1", DISABLE_ERROR_REPORTING: "1", ENABLE_TOOL_SEARCH: "false",
|
||||
} });
|
||||
expect(offered.length).toBeGreaterThan(0);
|
||||
expect(offered.at(-1)?.map(tool => tool.name)).toContain("mcp__paperclip__write_document");
|
||||
const question = offered.at(-1)?.find(tool => tool.name === "mcp__paperclip__request_human_input");
|
||||
expect(question).toBeDefined();
|
||||
expect(question!.input_schema).toMatchObject({ type: "object", properties: {
|
||||
idempotencyKey: { type: "string" }, interactionKind: { enum: expect.arrayContaining(["questions", "confirmation"]) },
|
||||
title: { type: "string" }, prompt: { type: "string" }, payload: { type: "object" },
|
||||
}, required: expect.arrayContaining(["idempotencyKey", "interactionKind", "title", "prompt", "continuationPolicy"]) });
|
||||
} finally {
|
||||
api.closeAllConnections();
|
||||
await new Promise<void>(resolve => api.close(() => resolve()));
|
||||
await bridge.close();
|
||||
await rm(root, { recursive: true, force: true, maxRetries: 5, retryDelay: 50 });
|
||||
}
|
||||
}, 30_000);
|
||||
@@ -40,6 +40,7 @@ export {
|
||||
type DurablePrpControlPlaneOptions,
|
||||
type DurablePrpCheckpoint,
|
||||
decodeDurablePrpCheckpoint,
|
||||
inspectDurablePrpCheckpointHeader,
|
||||
type PrpWireConnection,
|
||||
type PrpWireAttachment,
|
||||
type TransportCloseReason,
|
||||
|
||||
@@ -16,6 +16,7 @@ describe("core Paperclip protocol action contracts", () => {
|
||||
const input = { ...action.examples.call.input, interactionKind: "questions", payload };
|
||||
if (schema === action.scenario!.descriptor.inputSchema) delete (input as { idempotencyKey?: string }).idempotencyKey;
|
||||
expect(ajv.validate(schema, input), JSON.stringify(ajv.errors)).toBe(true);
|
||||
expect(ajv.validate(schema, { ...input, payload: undefined })).toBe(false);
|
||||
expect(ajv.validate(schema, { ...input, payload: { version: 1 } })).toBe(false);
|
||||
expect(ajv.validate(schema, { ...input, payload: { version: 1, questionSet: { ...payload.questionSet, questions: [{ ...payload.questionSet.questions[0], options: [{ id: "a", label: "A" }] }] } } })).toBe(false);
|
||||
}
|
||||
|
||||
@@ -86,7 +86,9 @@ export const requestHumanInputAction = {
|
||||
"skill_test"
|
||||
],
|
||||
"inputSchema": {
|
||||
"allOf": [questionsPayloadRequirement],
|
||||
// The pinned Claude client drops a tool with this conditional in root allOf.
|
||||
// Keep the equivalent direct condition on the concrete object root.
|
||||
...questionsPayloadRequirement,
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"idempotencyKey": {
|
||||
@@ -220,7 +222,7 @@ export const requestHumanInputAction = {
|
||||
"title": "Request human input",
|
||||
"description": "Create a typed confirmation, checkbox, question, task suggestion, or item-verdict request.",
|
||||
"inputSchema": {
|
||||
"allOf": [questionsPayloadRequirement],
|
||||
...questionsPayloadRequirement,
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"interactionKind": {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
// Grok admission source identity: sha256:d7cf13acdb5cb424e8023729fa46d07764de06893db23694fc9780334b0aed9e
|
||||
// This launcher is admitted by digest and receives a verified native executable.
|
||||
// Never resolve an executable through PATH or accept command-line overrides.
|
||||
const { isAbsolute, join } = require("node:path");
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
{
|
||||
"commandDigest": "sha256:b7a25fe627daf3857ac8097d9a374b1ba15355176389b92457838bf19e9d91bd",
|
||||
"declaration": {
|
||||
"schema": "paperclip.acpx_profile_declaration.v1",
|
||||
"agent": "copilot",
|
||||
"agentProfileVersion": 71,
|
||||
"acpxVersion": "0.13.1",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"protocolPolicyRevision": "copilot-agent-manual-v1",
|
||||
"acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1",
|
||||
"policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3",
|
||||
"distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107",
|
||||
"agentFilesBinding": "registered-runtime-context-v1",
|
||||
"systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1",
|
||||
"sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1",
|
||||
"innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a",
|
||||
"upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0",
|
||||
"patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679",
|
||||
"messageIdentityContract": "copilot-native-message-id-v1",
|
||||
"ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1",
|
||||
"permissionContextContract": "copilot-edit-permission-context-v1",
|
||||
"permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b",
|
||||
"permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668",
|
||||
"permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9",
|
||||
"permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f",
|
||||
"permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4",
|
||||
"semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2",
|
||||
"semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791",
|
||||
"semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5",
|
||||
"toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706",
|
||||
"semanticNormalizedInputContract": "validated-forwarded-input-commit-v1",
|
||||
"semanticSidecarSourceSha256": "7d6e948b7cff4df52a9afc916db46ace0f42b4b2d3134ba0c21ee0a928857e62",
|
||||
"semanticDirectDriverSourceSha256": "d683429427dd8a1a5efe50952018ea486f808bdbf0a92fc5946ee267fe2c72ee",
|
||||
"semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975",
|
||||
"semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e",
|
||||
"semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59",
|
||||
"semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9",
|
||||
"semanticSchemaBundleSourceSha256": "4cae579aac836762fdbf92c5447425d8b756e87a5766bc0c7a1885a90541c8d6",
|
||||
"semanticSidecarProtocolSourceSha256": "26dd0cfbef8b2b401de6b7862a4af7f243fb703e23678a1a5ed8fa68d4784d42",
|
||||
"semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1",
|
||||
"displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877",
|
||||
"displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e",
|
||||
"configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff",
|
||||
"configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994",
|
||||
"profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282",
|
||||
"profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571",
|
||||
"profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88",
|
||||
"attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2",
|
||||
"singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f",
|
||||
"nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce",
|
||||
"nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842",
|
||||
"nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1",
|
||||
"nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90",
|
||||
"harnessSessionContractSourceSha256": "baba1a210b863518f01e0c875606ed37f9467c433222251438578bd999bc43b5",
|
||||
"harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249",
|
||||
"harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94",
|
||||
"transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523",
|
||||
"nativeBackendSourceSha256": "7495d9898d22dca1ebf6336361d332bafefc4498f6e49f1135025fad846c8586",
|
||||
"nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44",
|
||||
"nativeInstructionsSourceSha256": "82ac6e5f7275b4ef79fb166c7330efc504e80cab3d7e9081e4e40a8b570bc50b",
|
||||
"providerLaunchSourceSha256": "c744235b3067ea75973523f78f1e13399337bc155f1775af694e34782d5c9bef",
|
||||
"launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296",
|
||||
"runtimeRequestBridgeSourceSha256": "93a631b8439df1590b0cf442cf84a9b05f004f12bfee90291651ccfc0861effe",
|
||||
"runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295",
|
||||
"runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211",
|
||||
"guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f",
|
||||
"nativeExecutionSourceSha256": "d60b8ea63bad975c0144f0702b2749d7db9a7da50680a6147f20d4d11f2d5297",
|
||||
"profileCompatibilitySourceSha256": "8a79f4b6f38e4781a056a2e9a7a07cc609c72638993ae16a68853a6189f575fe",
|
||||
"installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d",
|
||||
"assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c",
|
||||
"assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f",
|
||||
"credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161",
|
||||
"admissionDiagnosticsSourceSha256": "df8f1aa4f2d1bc63b2635bdd09cb8165c8b7681ddcc2618435c7c08ca90475af",
|
||||
"grokAdmissionSourceSha256": "f1e46cf2e1e36b7704bd797b678e3db8db2003f97bdb9d827cfa3d7662698777",
|
||||
"sidecarErrorSourceSha256": "17948ad74350dc487ab20d24a0fb89daf0b426b98396454fff0e3856e922d82c",
|
||||
"profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456",
|
||||
"metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727",
|
||||
"permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1",
|
||||
"semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811",
|
||||
"runtimeHostSourceSha256": "c779acab9ae4bbab4a13002d2e12c8c1acb208d0c4b575e7c5697fc3e7bb44c3",
|
||||
"runtimeAdapterSourceSha256": "7c47c05306ebe7d1ca6b5d046a96f5726a48adec4d27e7c16788f505f25b5db2",
|
||||
"runtimeSandboxSourceSha256": "981449a4c6d10f682ecf86d834eed07faca30a7d9b419fb8db4973a442a343ec",
|
||||
"cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca",
|
||||
"agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba",
|
||||
"nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"commandDigest": "sha256:93004d5a7647235e53ab741ba5639e8f20186ebaaccef8b9a7834593a1f5febe",
|
||||
"sourceDigest": "sha256:f1d43562ca126e027147b73d1abf04849130dc4dfddb71b7a233a6c21c32539a",
|
||||
"declaration": {
|
||||
"schema": "paperclip.grok-profile-identity.v1",
|
||||
"agent": "grok",
|
||||
"agentProfileVersion": 6,
|
||||
"agentServerPackage": "builtin:grok-acp",
|
||||
"agentServerVersion": "1",
|
||||
"agentRuntimePackage": "native:grok",
|
||||
"agentRuntimeVersion": "1.0.13",
|
||||
"requiresProviderPolicy": false,
|
||||
"acpxVersion": "0.13.1",
|
||||
"sources": {
|
||||
"patches/acpx@0.13.1.patch": "sha256:00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1",
|
||||
"packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts": "sha256:7d6e948b7cff4df52a9afc916db46ace0f42b4b2d3134ba0c21ee0a928857e62",
|
||||
"packages/paperclip-runner/src/cli/acpx-sidecar-input.ts": "sha256:17948ad74350dc487ab20d24a0fb89daf0b426b98396454fff0e3856e922d82c",
|
||||
"packages/paperclip-runner/src/drivers/acpx/admission-diagnostics.ts": "sha256:df8f1aa4f2d1bc63b2635bdd09cb8165c8b7681ddcc2618435c7c08ca90475af",
|
||||
"packages/paperclip-runner/src/drivers/acpx/grok-effort.ts": "sha256:f1e46cf2e1e36b7704bd797b678e3db8db2003f97bdb9d827cfa3d7662698777",
|
||||
"packages/paperclip-runner/src/drivers/acpx/grok-web-search.ts": "sha256:754b5ca2345c045421f6007f34999508d996e1cea035f0f1952b3d5f00072b9c",
|
||||
"packages/paperclip-runner/src/drivers/acpx/grok-credentials.ts": "sha256:1c4ff0be6da798b6fcc604d20a9198dcc8523a3806d563b5b5345b7260549f73",
|
||||
"packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts": "sha256:7c47c05306ebe7d1ca6b5d046a96f5726a48adec4d27e7c16788f505f25b5db2",
|
||||
"packages/paperclip-runner/src/drivers/acpx/runtime-host.ts": "sha256:c779acab9ae4bbab4a13002d2e12c8c1acb208d0c4b575e7c5697fc3e7bb44c3",
|
||||
"packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.ts": "sha256:981449a4c6d10f682ecf86d834eed07faca30a7d9b419fb8db4973a442a343ec",
|
||||
"packages/paperclip-runner/src/drivers/acpx/sidecar-protocol.ts": "sha256:26dd0cfbef8b2b401de6b7862a4af7f243fb703e23678a1a5ed8fa68d4784d42",
|
||||
"packages/paperclip-runner/src/drivers/acpx/native-distribution-integrity.ts": "sha256:aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b",
|
||||
"packages/paperclip-runner/src/providers/grok/launcher.cjs": "sha256:f0b698395a3704ed2ffaf84ea19bdb20c36c8a0a70b7c629c7b6ffe144e59e55",
|
||||
"packages/paperclip-runner/src/providers/grok/platforms.json": "sha256:931eed3abbf2bf041e6c72e6e9486d976f4352613cf069b216ddf010c9a8e704",
|
||||
"packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs": "sha256:e97a7c4d2cf9be9c6e0ada78076a5551b01401518597ab81cb6fbd5a1faf4a5e",
|
||||
"packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs": "sha256:43d09d1a7cb081e26dd3cad228577cf72f8b0d28c4076cdea722bb9c01044778"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"commandDigest": "sha256:6fcf754bf85aaca0b35ba6bad9926f11c60bdda5f7b180158c8f8b1be5e57d36",
|
||||
"sourceDigest": "sha256:d7cf13acdb5cb424e8023729fa46d07764de06893db23694fc9780334b0aed9e",
|
||||
"declaration": {
|
||||
"schema": "paperclip.grok-profile-identity.v1",
|
||||
"agent": "grok",
|
||||
"agentProfileVersion": 7,
|
||||
"agentServerPackage": "builtin:grok-acp",
|
||||
"agentServerVersion": "1",
|
||||
"agentRuntimePackage": "native:grok",
|
||||
"agentRuntimeVersion": "1.0.13",
|
||||
"requiresProviderPolicy": false,
|
||||
"acpxVersion": "0.13.1",
|
||||
"sources": {
|
||||
"patches/acpx@0.13.1.patch": "sha256:00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1",
|
||||
"packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts": "sha256:7d6e948b7cff4df52a9afc916db46ace0f42b4b2d3134ba0c21ee0a928857e62",
|
||||
"packages/paperclip-runner/src/cli/acpx-sidecar-input.ts": "sha256:17948ad74350dc487ab20d24a0fb89daf0b426b98396454fff0e3856e922d82c",
|
||||
"packages/paperclip-runner/src/drivers/acpx/admission-diagnostics.ts": "sha256:df8f1aa4f2d1bc63b2635bdd09cb8165c8b7681ddcc2618435c7c08ca90475af",
|
||||
"packages/paperclip-runner/src/drivers/acpx/grok-effort.ts": "sha256:f1e46cf2e1e36b7704bd797b678e3db8db2003f97bdb9d827cfa3d7662698777",
|
||||
"packages/paperclip-runner/src/drivers/acpx/grok-web-search.ts": "sha256:754b5ca2345c045421f6007f34999508d996e1cea035f0f1952b3d5f00072b9c",
|
||||
"packages/paperclip-runner/src/drivers/acpx/grok-credentials.ts": "sha256:1c4ff0be6da798b6fcc604d20a9198dcc8523a3806d563b5b5345b7260549f73",
|
||||
"packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts": "sha256:7c47c05306ebe7d1ca6b5d046a96f5726a48adec4d27e7c16788f505f25b5db2",
|
||||
"packages/paperclip-runner/src/drivers/acpx/runtime-host.ts": "sha256:c779acab9ae4bbab4a13002d2e12c8c1acb208d0c4b575e7c5697fc3e7bb44c3",
|
||||
"packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.ts": "sha256:981449a4c6d10f682ecf86d834eed07faca30a7d9b419fb8db4973a442a343ec",
|
||||
"packages/paperclip-runner/src/drivers/acpx/sidecar-protocol.ts": "sha256:26dd0cfbef8b2b401de6b7862a4af7f243fb703e23678a1a5ed8fa68d4784d42",
|
||||
"packages/paperclip-runner/src/drivers/acpx/native-distribution-integrity.ts": "sha256:aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b",
|
||||
"packages/paperclip-runner/src/providers/grok/launcher.cjs": "sha256:f0b698395a3704ed2ffaf84ea19bdb20c36c8a0a70b7c629c7b6ffe144e59e55",
|
||||
"packages/paperclip-runner/src/providers/grok/platforms.json": "sha256:931eed3abbf2bf041e6c72e6e9486d976f4352613cf069b216ddf010c9a8e704",
|
||||
"packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs": "sha256:fbfd9c2d51915dbb5590b2caf291092ae83024479c322422ef2b56e23a02a217",
|
||||
"packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs": "sha256:e97a7c4d2cf9be9c6e0ada78076a5551b01401518597ab81cb6fbd5a1faf4a5e",
|
||||
"packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs": "sha256:43d09d1a7cb081e26dd3cad228577cf72f8b0d28c4076cdea722bb9c01044778"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -27,6 +27,7 @@ Notes:
|
||||
|
||||
- The current published Daytona SDK package is `@daytonaio/sdk`.
|
||||
- The driver supports both `snapshot`-based and `image`-based sandbox creation. If both are set, validation rejects the config as ambiguous.
|
||||
- Optional `ttlMinutes` sets a positive whole-minute wall-clock lifetime in the creation request. It applies to image and snapshot sandboxes, including probes and interactive setup. The driver requires the provider to confirm the expiry before it returns the sandbox. It preserves an earlier creation expiry when a caller supplies a later deadline. Daytona destroys the sandbox and its files at expiry, even while active, stopped, paused, or archived. Omit this setting to keep existing lifetime behavior. This is separate from inactivity cleanup.
|
||||
- Each cold create uses a unique provider name and ownership labels. If creation fails after Daytona has allocated a sandbox, the driver looks up that exact name, checks every ownership label, and waits for deletion. Failed, missing, or timed-out lookups and failed deletions report unconfirmed cleanup with the provider name; they never return a usable lease. A missing name lookup after an uncertain create is not proof that a delayed provider request cannot create a resource.
|
||||
- For plugin-backed sandbox lease acquisition, unresolved creation cleanup crosses the worker RPC as a validated ownership envelope. The host records `pending_cleanup` before retrying deletion. If only a creation name is known, the provider first returns an ownership-verified sandbox ID without deleting it. The host saves that observation before authorizing deletion. Its existing cleanup sweep and durable spool preserve retries across controller restarts and environment deletion. A missing observed ID confirms cleanup after a lost deletion reply or database update; a name that has never been observed remains unresolved. Provider exceptions and resolved credentials are excluded from that envelope. This requires the matching host and plugin SDK update; probe and custom-image interactive-setup calls still report unconfirmed immediate cleanup without this lease-recovery path.
|
||||
- Reusable leases map to Daytona stop/start semantics. Non-reusable leases are deleted on release. A provider-resolved `target` does not change the identity of an existing sandbox. Release closes the same scoped lease that a later sentinel-verified resume reopens.
|
||||
|
||||
@@ -15,7 +15,7 @@ export const DEFAULT_DAYTONA_OPERATION_TIMEOUT_MS = 300_000;
|
||||
// 0.1.6 adds private authenticated WebSocket ingress for paperclip_runner.
|
||||
// 0.1.7 exposes host-owned warm/cold runner lifecycle controls.
|
||||
// 0.1.8 declares the default provider acquisition budget to the host.
|
||||
const PLUGIN_VERSION = "0.1.8";
|
||||
const PLUGIN_VERSION = "0.1.9";
|
||||
|
||||
const manifest: PaperclipPluginManifestV1 = {
|
||||
id: PLUGIN_ID,
|
||||
@@ -151,6 +151,11 @@ const manifest: PaperclipPluginManifestV1 = {
|
||||
"Per-call timeout in milliseconds for the sandbox liveness read (refreshData). A silently unresponsive sandbox connection surfaces as a fast error instead of stalling until the outer RPC ceiling. The start and recovery calls derive their own deadline from timeoutMs, not this bound. `0` or less disables the bound. Defaults to 30000 when unset.",
|
||||
default: 30000,
|
||||
},
|
||||
ttlMinutes: {
|
||||
type: "integer",
|
||||
minimum: 1,
|
||||
description: "Optional wall-clock lifetime in minutes from creation. Daytona destroys the sandbox and its files at expiry in any state, even while active. Omit to leave the lifetime unchanged.",
|
||||
},
|
||||
autoStopInterval: {
|
||||
type: "number",
|
||||
description:
|
||||
|
||||
@@ -537,7 +537,7 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
// The bundled-plugin boot reconcile refreshes the stored manifest for an
|
||||
// existing install only when the version changes. The acquisition budget needs
|
||||
// the bump to reach an existing install.
|
||||
expect(manifest.version).toBe("0.1.8");
|
||||
expect(manifest.version).toBe("0.1.9");
|
||||
});
|
||||
|
||||
it.each([false, true])("closes duplex routes on lease release even when bridge drain hangs: %s", async (hangDrain) => {
|
||||
@@ -732,6 +732,7 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
autoStopInterval: 15,
|
||||
autoArchiveInterval: 60,
|
||||
autoDeleteInterval: -1,
|
||||
ttlMinutes: null,
|
||||
reuseLease: true,
|
||||
archiveOnRelease: false,
|
||||
},
|
||||
@@ -1329,6 +1330,119 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
expect(lease?.expiresAt ?? null).toBeNull();
|
||||
});
|
||||
|
||||
it.each([{ image: "node:20" }, { snapshot: "base-snapshot" }])("sets the wall-clock ttl in the create request for %j", async source => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const autoDestroyAt = new Date(Date.now() + 30 * 60_000).toISOString();
|
||||
const sandbox = createMockSandbox({ autoDestroyAt });
|
||||
mockCreate.mockResolvedValue(sandbox);
|
||||
const lease = await plugin.definition.onEnvironmentAcquireLease?.({
|
||||
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", runId: "run-1",
|
||||
config: { ...source, ttlMinutes: 30, reuseLease: false },
|
||||
});
|
||||
expect(mockCreate).toHaveBeenCalledWith(expect.objectContaining({ ...source, ttlMinutes: 30 }), expect.anything());
|
||||
expect(lease?.expiresAt).toBe(autoDestroyAt);
|
||||
expect(sandbox.setTtl).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("treats a cleared creation ttl as unset", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const config = { image: "node:20", ttlMinutes: "", reuseLease: false };
|
||||
expect(await plugin.definition.onEnvironmentValidateConfig?.({ driverKey: "daytona", config }))
|
||||
.toMatchObject({ ok: true, normalizedConfig: { ttlMinutes: null } });
|
||||
const sandbox = createMockSandbox();
|
||||
mockCreate.mockResolvedValue(sandbox);
|
||||
const lease = await plugin.definition.onEnvironmentAcquireLease?.({
|
||||
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", runId: "run-1", config,
|
||||
});
|
||||
expect(mockCreate.mock.calls[0][0]).not.toHaveProperty("ttlMinutes");
|
||||
expect(lease?.expiresAt ?? null).toBeNull();
|
||||
});
|
||||
|
||||
it.each(["acquire", "probe", "interactive"])("bounds a stalled creation expiry read for %s", async mode => {
|
||||
vi.useFakeTimers();
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox({ autoDestroyAt: new Date(Date.now() + 60_000).toISOString() });
|
||||
let finishRead: (() => void) | undefined;
|
||||
sandbox.refreshData.mockImplementation(() => new Promise<void>(resolve => { finishRead = resolve; }));
|
||||
mockCreate.mockImplementation(async params => Object.assign(sandbox, { name: params.name, labels: params.labels }));
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
const params = { driverKey: "daytona", companyId: "company-1", environmentId: "env-1", runId: "run-1", sessionId: "setup-1",
|
||||
config: { image: "node:20", ttlMinutes: 1, timeoutMs: 60_000, livenessTimeoutMs: 5, reuseLease: false } };
|
||||
const pending = mode === "probe" ? plugin.definition.onEnvironmentProbe!(params)
|
||||
: mode === "interactive" ? plugin.definition.onEnvironmentStartInteractiveSetup!(params)
|
||||
: plugin.definition.onEnvironmentAcquireLease!(params);
|
||||
let settled = false;
|
||||
const observed = pending.then(value => ({ value, error: null }), error => ({ value: null, error }));
|
||||
void observed.then(() => { settled = true; });
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(20);
|
||||
expect(settled).toBe(true);
|
||||
const result = await observed;
|
||||
expect(result.error?.message ?? (result.value as { metadata?: { error?: string } })?.metadata?.error)
|
||||
.toContain("did not respond within 5 ms");
|
||||
expect(sandbox.delete).toHaveBeenCalled();
|
||||
expect(sandbox.process.executeCommand).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
finishRead?.();
|
||||
await observed;
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it.each([0, -1, 1.5, "invalid", Infinity])("rejects invalid creation ttl %s before allocation", async ttlMinutes => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const config = { image: "node:20", ttlMinutes, reuseLease: false };
|
||||
expect(await plugin.definition.onEnvironmentValidateConfig?.({ driverKey: "daytona", config }))
|
||||
.toMatchObject({ ok: false, errors: expect.arrayContaining([expect.stringContaining("ttlMinutes")]) });
|
||||
await expect(plugin.definition.onEnvironmentAcquireLease?.({
|
||||
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", runId: "run-1", config,
|
||||
})).rejects.toThrow("ttlMinutes");
|
||||
expect(mockCreate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([undefined, "not-a-date", new Date(Date.now() - 60_000).toISOString(), new Date(Date.now() + 60 * 60_000).toISOString()])(
|
||||
"deletes a new sandbox whose configured creation ttl is not confirmed: %s", async autoDestroyAt => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox({ autoDestroyAt });
|
||||
mockCreate.mockImplementation(async params => Object.assign(sandbox, { name: params.name, labels: params.labels }));
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
await expect(plugin.definition.onEnvironmentAcquireLease?.({
|
||||
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", runId: "run-1",
|
||||
config: { image: "node:20", ttlMinutes: 30, reuseLease: false },
|
||||
})).rejects.toThrow("TTL");
|
||||
expect(sandbox.delete).toHaveBeenCalled();
|
||||
expect(sandbox.process.executeCommand).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
|
||||
it("does not extend the creation ttl for a later caller deadline", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const autoDestroyAt = new Date(Date.now() + 5 * 60_000).toISOString();
|
||||
const sandbox = createMockSandbox({ autoDestroyAt });
|
||||
mockCreate.mockResolvedValue(sandbox);
|
||||
const lease = await plugin.definition.onEnvironmentAcquireLease?.({
|
||||
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", runId: "run-1",
|
||||
config: { image: "node:20", ttlMinutes: 30, reuseLease: false },
|
||||
requestedExpiresAt: new Date(Date.now() + 10 * 60_000).toISOString(),
|
||||
});
|
||||
expect(lease?.expiresAt).toBe(autoDestroyAt);
|
||||
expect(sandbox.setTtl).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("shortens a configured creation ttl for an earlier caller deadline", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox({ autoDestroyAt: new Date(Date.now() + 30 * 60_000).toISOString() });
|
||||
sandbox.setTtl.mockImplementation(async minutes => { sandbox.autoDestroyAt = new Date(Date.now() + minutes * 60_000).toISOString(); });
|
||||
mockCreate.mockResolvedValue(sandbox);
|
||||
const requestedExpiresAt = new Date(Date.now() + 5 * 60_000 + 30_000).toISOString();
|
||||
const lease = await plugin.definition.onEnvironmentAcquireLease?.({
|
||||
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", runId: "run-1",
|
||||
config: { image: "node:20", ttlMinutes: 30, reuseLease: false }, requestedExpiresAt,
|
||||
});
|
||||
expect(sandbox.setTtl).toHaveBeenCalledWith(5);
|
||||
expect(Date.parse(lease!.expiresAt!)).toBeLessThanOrEqual(Date.parse(requestedExpiresAt));
|
||||
});
|
||||
|
||||
it("configures a provider ttl at or before the requested expiry and returns the provider expiry", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const autoDestroyAt = new Date(Date.now() + 30 * 60_000).toISOString();
|
||||
|
||||
@@ -184,6 +184,7 @@ interface DaytonaDriverConfig {
|
||||
autoStopInterval: number | null;
|
||||
autoArchiveInterval: number | null;
|
||||
autoDeleteInterval: number | null;
|
||||
ttlMinutes: number | null;
|
||||
reuseLease: boolean;
|
||||
archiveOnRelease: boolean;
|
||||
}
|
||||
@@ -307,6 +308,7 @@ function parseDriverConfig(raw: Record<string, unknown>): DaytonaDriverConfig {
|
||||
autoStopInterval: parseOptionalInteger(raw.autoStopInterval) ?? DEFAULT_AUTO_STOP_INTERVAL_MINUTES,
|
||||
autoArchiveInterval: parseOptionalInteger(raw.autoArchiveInterval) ?? DEFAULT_AUTO_ARCHIVE_INTERVAL_MINUTES,
|
||||
autoDeleteInterval: parseOptionalInteger(raw.autoDeleteInterval) ?? DEFAULT_AUTO_DELETE_INTERVAL_MINUTES,
|
||||
ttlMinutes: raw.ttlMinutes == null || raw.ttlMinutes === "" ? null : Number(raw.ttlMinutes),
|
||||
reuseLease: raw.reuseLease === true,
|
||||
archiveOnRelease: raw.archiveOnRelease === true,
|
||||
};
|
||||
@@ -354,6 +356,7 @@ function buildCreateParams(
|
||||
autoStopInterval: config.autoStopInterval ?? undefined,
|
||||
autoArchiveInterval: config.autoArchiveInterval ?? undefined,
|
||||
autoDeleteInterval: config.autoDeleteInterval ?? undefined,
|
||||
...(config.ttlMinutes != null ? { ttlMinutes: config.ttlMinutes } : {}),
|
||||
};
|
||||
if (config.image) {
|
||||
return {
|
||||
@@ -385,6 +388,12 @@ function validateRuntimeResourceRequest(config: DaytonaDriverConfig): string | n
|
||||
return "Daytona resource settings require image-backed sandbox creation; default sandbox creation cannot override CPU, memory, disk, or GPU.";
|
||||
}
|
||||
|
||||
function validateCreationTtl(config: DaytonaDriverConfig): string | null {
|
||||
return config.ttlMinutes != null && (!Number.isSafeInteger(config.ttlMinutes) || config.ttlMinutes <= 0)
|
||||
? "ttlMinutes must be a positive integer when configured."
|
||||
: null;
|
||||
}
|
||||
|
||||
function buildSandboxLabels(input: {
|
||||
companyId: string;
|
||||
environmentId: string;
|
||||
@@ -689,6 +698,7 @@ function leaseMetadata(input: {
|
||||
...(input.snapshotIdentity ? { snapshotIdentity: input.snapshotIdentity } : {}),
|
||||
target: input.sandbox.target,
|
||||
timeoutMs: input.config.timeoutMs,
|
||||
...(input.config.ttlMinutes != null ? { ttlMinutes: input.config.ttlMinutes } : {}),
|
||||
reuseLease: input.config.reuseLease,
|
||||
// Persisted so the release path (which rebuilds config from lease
|
||||
// metadata) still knows to archive instead of delete.
|
||||
@@ -721,22 +731,17 @@ function expiresAtForMinutes(minutes: number): string {
|
||||
return new Date(Date.now() + minutes * 60_000).toISOString();
|
||||
}
|
||||
|
||||
// Configure a provider-side time-to-live so Daytona destroys the sandbox at or
|
||||
// before the caller-requested deadline, even after a Paperclip crash or outage.
|
||||
// `setTtl` counts wall-clock time regardless of the sandbox state, so the destroy
|
||||
// happens even when the sandbox is stopped, paused, or archived. The function
|
||||
// returns the real provider destroy time (`autoDestroyAt`) as evidence of the
|
||||
// provider-side bound. It returns null when the caller sets no deadline, when the
|
||||
// deadline is invalid, or when the deadline is less than one minute away (Daytona
|
||||
// TTL granularity is one minute, so a nearer deadline maps to no valid TTL). The
|
||||
// server then fails closed on a null expiry and releases the lease.
|
||||
async function configureSandboxExpiry(input: {
|
||||
sandbox: Sandbox;
|
||||
requestedExpiresAt: string | null | undefined;
|
||||
nowMs: number;
|
||||
preserveCreationExpiry: boolean;
|
||||
}): Promise<string | null> {
|
||||
const creationExpiry = input.preserveCreationExpiry ? input.sandbox.autoDestroyAt : undefined;
|
||||
const creationExpiryMs = creationExpiry ? Date.parse(creationExpiry) : Number.NaN;
|
||||
const requestedMs = input.requestedExpiresAt ? Date.parse(input.requestedExpiresAt) : Number.NaN;
|
||||
if (!Number.isFinite(requestedMs)) return null;
|
||||
if (!Number.isFinite(requestedMs)) return Number.isFinite(creationExpiryMs) ? creationExpiry! : null;
|
||||
if (Number.isFinite(creationExpiryMs) && creationExpiryMs <= requestedMs) return creationExpiry!;
|
||||
// Round DOWN so the provider destroy time never lands after the deadline.
|
||||
const ttlMinutes = Math.floor((requestedMs - input.nowMs) / 60_000);
|
||||
if (ttlMinutes < 1) return null;
|
||||
@@ -965,7 +970,7 @@ async function createSandbox(
|
||||
onSnapshotIdentity?: (identity: SnapshotIdentity) => void;
|
||||
} = {},
|
||||
): Promise<Sandbox> {
|
||||
const resourceRequestError = validateRuntimeResourceRequest(config);
|
||||
const resourceRequestError = validateCreationTtl(config) ?? validateRuntimeResourceRequest(config);
|
||||
if (resourceRequestError) {
|
||||
throw new Error(resourceRequestError);
|
||||
}
|
||||
@@ -1014,6 +1019,15 @@ async function createSandbox(
|
||||
timeout: toTimeoutSeconds(config.timeoutMs),
|
||||
}));
|
||||
idleActivity.observe(allocation, providerActivityKey(params, config, sandbox.id));
|
||||
if (config.ttlMinutes != null) {
|
||||
cleanup.observedProviderLeaseId = sandbox.id;
|
||||
await withLivenessTimeout("sandbox.refreshData", config.livenessTimeoutMs, () => sandbox.refreshData());
|
||||
const expiry = Date.parse(sandbox.autoDestroyAt ?? "");
|
||||
const nowMs = Date.now();
|
||||
if (!Number.isFinite(expiry) || expiry <= nowMs || expiry > nowMs + config.ttlMinutes * 60_000) {
|
||||
throw new Error("Daytona did not confirm the configured creation TTL.");
|
||||
}
|
||||
}
|
||||
return sandbox;
|
||||
} catch (createError) {
|
||||
// Observe the create failure before compensating deletion can stall or fail.
|
||||
@@ -2534,6 +2548,8 @@ const plugin = definePlugin(trackDaytonaHooks({
|
||||
if (!config.apiKey && !(process.env.DAYTONA_API_KEY?.trim())) {
|
||||
errors.push("Daytona sandbox environments require an API key in config or DAYTONA_API_KEY.");
|
||||
}
|
||||
const ttlError = validateCreationTtl(config);
|
||||
if (ttlError) errors.push(ttlError);
|
||||
const resourceRequestError = validateResourceRequest(config);
|
||||
if (resourceRequestError) {
|
||||
errors.push(resourceRequestError);
|
||||
@@ -2676,6 +2692,7 @@ const plugin = definePlugin(trackDaytonaHooks({
|
||||
sandbox,
|
||||
requestedExpiresAt: params.requestedExpiresAt,
|
||||
nowMs: Date.now(),
|
||||
preserveCreationExpiry: config.ttlMinutes != null,
|
||||
});
|
||||
assertActive();
|
||||
phase = "sentinel";
|
||||
|
||||
@@ -6,6 +6,7 @@ import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import test from 'node:test';
|
||||
import { grokPublicInstallVersion } from '../grok-public-install-sandbox.mjs';
|
||||
import { startPackedRegistry, writePackedRegistryCatalog } from '../public-install-registry.mjs';
|
||||
|
||||
const script = fileURLToPath(new URL('../public-install-registry.mjs', import.meta.url));
|
||||
@@ -15,8 +16,8 @@ const scratch = async body => {
|
||||
};
|
||||
const environment = root => ({ ...process.env, HOME: root, npm_config_cache: join(root, 'cache'),
|
||||
npm_config_userconfig: join(root, 'no-npmrc'), npm_config_audit: 'false', npm_config_fund: 'false', npm_config_update_notifier: 'false', npm_config_fetch_retries: '0', npm_config_fetch_timeout: '10000' });
|
||||
const run = (command, args, cwd) => new Promise((resolve, reject) => {
|
||||
const child = spawn(command, args, { cwd, env: environment(cwd), stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
const run = (command, args, cwd, env = environment(cwd)) => new Promise((resolve, reject) => {
|
||||
const child = spawn(command, args, { cwd, env, stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
let output = '';
|
||||
for (const stream of [child.stdout, child.stderr]) stream.on('data', chunk => { output += chunk; });
|
||||
child.once('error', reject);
|
||||
@@ -59,6 +60,37 @@ const installedFiles = directory => {
|
||||
return entries.sort(([a], [b]) => a.localeCompare(b));
|
||||
};
|
||||
|
||||
test('synthetic Grok versions install from packed bytes even when the hash starts with leading-zero digits', { timeout: 30_000 }, async () => scratch(async root => {
|
||||
const sourceRevision = '096091501644cf20f2e35d1ab02b28c35579b973';
|
||||
const cases = [
|
||||
{ version: `0.0.0-grok-verify.${sourceRevision.slice(0, 12)}`, rejected: true },
|
||||
{ version: grokPublicInstallVersion(sourceRevision) },
|
||||
{ version: grokPublicInstallVersion('8fd48a8d52038fd0d8b286f3742ea13ce40a7ab9') },
|
||||
];
|
||||
for (const [index, { version, rejected }] of cases.entries()) {
|
||||
const cwd = join(root, String(index)), source = join(cwd, 'source'), assets = join(cwd, 'assets'), consumer = join(cwd, 'consumer');
|
||||
for (const directory of [join(source, 'package'), assets, consumer]) mkdirSync(directory, { recursive: true });
|
||||
writeFileSync(join(source, 'package/package.json'), JSON.stringify({ name: '@ci/grok-version', version }));
|
||||
execFileSync('tar', ['-czf', join(assets, 'package.tgz'), '-C', source, 'package']);
|
||||
await writePackedRegistryCatalog(assets);
|
||||
writeFileSync(join(consumer, 'package.json'), '{"private":true}');
|
||||
const userconfig = join(cwd, 'user.npmrc'), globalconfig = join(cwd, 'global.npmrc');
|
||||
writeFileSync(userconfig, ''); writeFileSync(globalconfig, '');
|
||||
const env = { PATH: process.env.PATH, HOME: cwd, npm_config_cache: join(cwd, 'cache'),
|
||||
npm_config_userconfig: userconfig, npm_config_globalconfig: globalconfig,
|
||||
npm_config_audit: 'false', npm_config_fund: 'false', npm_config_update_notifier: 'false', npm_config_fetch_retries: '0' };
|
||||
const registry = await startPackedRegistry(assets);
|
||||
try {
|
||||
const install = run('npm', ['install', '--registry', registry.url, '--replace-registry-host=never', '--ignore-scripts', '--omit=dev', ...registry.specs], consumer, env);
|
||||
if (rejected) await assert.rejects(install, /ETARGET/);
|
||||
else {
|
||||
await install;
|
||||
assert.equal(JSON.parse(readFileSync(join(consumer, 'node_modules/@ci/grok-version/package.json'))).version, version);
|
||||
}
|
||||
} finally { await registry.close(); }
|
||||
}
|
||||
}));
|
||||
|
||||
test('real npm installs identical packed bytes, bundles, optional dependencies and bins without running scripts early', { timeout: 90_000 }, async () => scratch(async root => {
|
||||
const assets = fixture(root, { publicDependency: true }), catalog = await writePackedRegistryCatalog(assets);
|
||||
assert.ok(catalog.every(entry => entry.manifest.version === '1.0.0'), 'metadata must come from packed bytes, not changed source');
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { createDb, startEmbeddedPostgresTestDatabase, type Db } from "../packages/db/src/index.js";
|
||||
import { captureInboundQuery, runQuery, seedInboundDiscovery, type QueueFactory, type ServiceFactory } from "../server/src/__tests__/helpers/inbound-discovery-fixture.js";
|
||||
|
||||
const ref = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15);
|
||||
assert.ok(ref, "Pass an explicit pre-change --baseline-ref=<commit>");
|
||||
const baselineRevision = execFileSync("git", ["rev-parse", "--verify", `${ref}^{commit}`], { encoding: "utf8" }).trim();
|
||||
const root = fileURLToPath(new URL("../", import.meta.url));
|
||||
const sources = ["server/src/services/chat-channels/inbound/delivery-ledger.ts", "server/src/services/chat-channels/inbound/conversation-queue.ts", "server/src/services/chat-channels.ts"];
|
||||
const runtimeChanges = execFileSync("git", ["diff", "--name-only", baselineRevision, "--", "server/src/services", "packages/db/src", "packages/shared/src", "pnpm-lock.yaml"], { encoding: "utf8" }).trim().split("\n").filter(Boolean);
|
||||
assert.deepEqual(runtimeChanges.filter(name => !sources.includes(name)), [], "Baseline must not import changed runtime helpers or dependencies");
|
||||
const dir = await fs.mkdtemp(path.join(root, "server/scripts/.inbound-baseline-"));
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: Db | undefined;
|
||||
const keepAlive = setInterval(() => {}, 1000);
|
||||
function nodes(plan: Record<string, unknown>): Record<string, unknown>[] {
|
||||
return [plan, ...((plan.Plans ?? []) as Record<string, unknown>[]).flatMap(nodes)];
|
||||
}
|
||||
const started = performance.now();
|
||||
try {
|
||||
const targets = new Map(sources.map(source => [path.join(root, source).replace(/\.ts$/, ".js"), path.join(dir, path.basename(source))]));
|
||||
for (const source of sources) {
|
||||
const original = path.join(root, source);
|
||||
const text = execFileSync("git", ["show", `${baselineRevision}:${source}`], { encoding: "utf8" });
|
||||
const rewritten = text.replace(/from "(\.{1,2}\/[^\"]+)"/g, (_match, specifier: string) => {
|
||||
const resolved = path.resolve(path.dirname(original), specifier);
|
||||
return `from ${JSON.stringify(targets.get(resolved) ?? resolved)}`;
|
||||
});
|
||||
await fs.writeFile(path.join(dir, path.basename(source)), rewritten);
|
||||
}
|
||||
const oldQueue = await import(pathToFileURL(path.join(dir, "conversation-queue.ts")).href) as { createChatConversationDeliveryQueue: QueueFactory };
|
||||
const oldService = await import(pathToFileURL(path.join(dir, "chat-channels.ts")).href) as { chatChannelService: ServiceFactory };
|
||||
database = await startEmbeddedPostgresTestDatabase("inbound-discovery-benchmark-");
|
||||
db = createDb(database.connectionString, { maxConnections: 1, idleTimeoutSeconds: 0 });
|
||||
const [{ version }] = await db.$client.unsafe("select version()");
|
||||
const settings = await db.$client.unsafe("select name, setting from pg_settings where name in ('shared_buffers', 'work_mem', 'jit', 'jit_above_cost', 'jit_inline_above_cost', 'random_page_cost', 'max_connections') order by name");
|
||||
const results = [];
|
||||
const malformedReadyControls = [];
|
||||
let baselineExceedsWorkBudget = false;
|
||||
const budgetFailures: string[] = [];
|
||||
for (const fixture of [{ name: "small", history: 20, payloadBytes: 0 }, { name: "retained-history", history: 1500, payloadBytes: 32 * 1024 }].filter(value => !process.argv.includes("--retained-only") || value.name === "retained-history")) {
|
||||
const f = await seedInboundDiscovery(db, fixture.history, fixture.payloadBytes);
|
||||
await db.$client.unsafe("begin");
|
||||
if (process.argv.includes("--jit-off")) await db.$client.unsafe("set local jit = off");
|
||||
try {
|
||||
const pairs = [];
|
||||
for (const kind of ["deadline", "head", "ready"] as const) {
|
||||
const before = await captureInboundQuery(db, f, kind, oldQueue.createChatConversationDeliveryQueue, oldService.chatChannelService);
|
||||
const after = await captureInboundQuery(db, f, kind);
|
||||
pairs.push({ kind, before, after });
|
||||
}
|
||||
const connection = await db.$client.reserve();
|
||||
const timedDb = new Proxy(db, { get(target, key) { return key === "$client" ? connection : Reflect.get(target, key); } });
|
||||
try {
|
||||
if (process.argv.includes("--wait-for-slot")) {
|
||||
process.stderr.write(`READY ${fixture.name}: database seeded, all real queries captured; press Enter to time\n`);
|
||||
await new Promise<void>(resolve => { process.stdin.once("data", () => { process.stdin.pause(); resolve(); }); process.stdin.resume(); });
|
||||
}
|
||||
for (const { kind, before, after } of pairs) {
|
||||
const samples = { before: [] as number[], after: [] as number[] };
|
||||
const expected = await runQuery(timedDb, before);
|
||||
assert.deepEqual(await runQuery(timedDb, after), expected, `${fixture.name}/${kind}: complete positional output differs`);
|
||||
for (let n = 0; n < 7; n++) {
|
||||
for (const arm of n % 2 ? ["after", "before"] as const : ["before", "after"] as const) {
|
||||
const start = performance.now();
|
||||
await runQuery(timedDb, arm === "before" ? before : after);
|
||||
if (n >= 2) samples[arm].push(performance.now() - start);
|
||||
}
|
||||
}
|
||||
const plans = {} as Record<"before" | "after", unknown>;
|
||||
const actionLoops = {} as Record<"before" | "after", number>;
|
||||
for (const arm of ["before", "after"] as const) {
|
||||
const query = arm === "before" ? before : after;
|
||||
const [row] = await connection.unsafe(`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${query.sql}`, query.params as never[]);
|
||||
plans[arm] = row["QUERY PLAN"];
|
||||
actionLoops[arm] = Math.max(0, ...nodes(row["QUERY PLAN"][0].Plan).filter(node => node["Relation Name"] === "chat_actions").map(node => Number(node["Actual Loops"])));
|
||||
}
|
||||
if (fixture.name === "retained-history") {
|
||||
const oldPlan = (plans.before as { JIT?: unknown; Plan: Record<string, number> }[])[0]!;
|
||||
const newPlan = (plans.after as { JIT?: unknown; Plan: Record<string, number> }[])[0]!;
|
||||
const oldBuffers = Number(oldPlan.Plan["Shared Hit Blocks"] ?? 0) + Number(oldPlan.Plan["Shared Read Blocks"] ?? 0);
|
||||
const newBuffers = Number(newPlan.Plan["Shared Hit Blocks"] ?? 0) + Number(newPlan.Plan["Shared Read Blocks"] ?? 0);
|
||||
baselineExceedsWorkBudget ||= Boolean(oldPlan.JIT) || (kind === "deadline" && oldBuffers > 1000);
|
||||
if (kind !== "ready" && newPlan.JIT) budgetFailures.push(`${kind}: unnecessary JIT on the retained-history fixture`);
|
||||
if (kind === "deadline" && newBuffers > 1000) budgetFailures.push(`${kind}: ${newBuffers} shared buffer accesses exceeds fixture budget 1000`);
|
||||
}
|
||||
const median = (values: number[]) => [...values].sort((a, b) => a - b)[Math.floor(values.length / 2)]!;
|
||||
results.push({ fixture, kind, equalPositionalRows: expected.length, before, after, samples, beforeMedianMs: median(samples.before), afterMedianMs: median(samples.after), actionLoops, plans });
|
||||
process.stderr.write(`${fixture.name}/${kind}: ${median(samples.before).toFixed(2)} -> ${median(samples.after).toFixed(2)} ms; action scans ${actionLoops.before} -> ${actionLoops.after}\n`);
|
||||
}
|
||||
} finally { connection.release(); }
|
||||
if (fixture.name === "small") {
|
||||
const unrelated = await f.delivery({ state: "filtered" });
|
||||
const other = await seedInboundDiscovery(db, 0);
|
||||
for (const input of [
|
||||
{ scope: "same-company-unrelated-delivery", companyId: f.companyId, endpointId: f.endpointId, deliveryId: unrelated.id },
|
||||
{ scope: "other-company", companyId: other.companyId, endpointId: other.endpointId, deliveryId: other.pending.id },
|
||||
{ scope: "null-delivery", companyId: f.companyId, endpointId: f.endpointId, deliveryId: null },
|
||||
]) {
|
||||
const action = await f.action(input.deliveryId, { companyId: input.companyId, endpointId: input.endpointId, result: { retryAt: "invalid timestamp" } });
|
||||
const before = await captureInboundQuery(db, f, "ready", oldQueue.createChatConversationDeliveryQueue, oldService.chatChannelService, f.pending.id);
|
||||
const after = await captureInboundQuery(db, f, "ready", undefined, undefined, f.pending.id);
|
||||
const outcome = async (query: typeof before) => {
|
||||
await db!.$client.unsafe("savepoint malformed_probe");
|
||||
try { return { rows: await runQuery(db!, query) }; }
|
||||
catch (error) { await db!.$client.unsafe("rollback to savepoint malformed_probe"); return { error: (error as { code?: string }).code }; }
|
||||
finally { await db!.$client.unsafe("release savepoint malformed_probe"); }
|
||||
};
|
||||
const oldResult = await outcome(before), newResult = await outcome(after);
|
||||
assert.deepEqual(newResult, oldResult, `${input.scope}: malformed ready selection outcome differs`);
|
||||
malformedReadyControls.push({ scope: input.scope, before: oldResult, after: newResult });
|
||||
await db.$client.unsafe("delete from chat_actions where id = $1::uuid", [action.id]);
|
||||
}
|
||||
}
|
||||
} finally { await db.$client.unsafe("rollback"); }
|
||||
await db.$client.unsafe("delete from chat_endpoints where id = $1::uuid", [f.endpointId]);
|
||||
}
|
||||
const runtimeSources = sources.map(source => ({ path: source,
|
||||
baselineSha256: createHash("sha256").update(execFileSync("git", ["show", `${baselineRevision}:${source}`])).digest("hex"),
|
||||
treatmentDiffSha256: createHash("sha256").update(execFileSync("git", ["diff", "--no-ext-diff", baselineRevision, "--", source])).digest("hex"),
|
||||
}));
|
||||
console.log(JSON.stringify({ baselineRevision, treatmentRevision: execFileSync("git", ["rev-parse", "HEAD"], { encoding: "utf8" }).trim(), runtimeSources,
|
||||
version, settings, jitOffControl: process.argv.includes("--jit-off"), baselineExceedsWorkBudget, budgetFailures,
|
||||
malformedReadyControls, warmups: 2, timedSamplesPerArm: 5, setupAndProofMs: performance.now() - started, results }, null, 2));
|
||||
if (process.argv.includes("--require-work-budget")) assert.deepEqual(budgetFailures, [], "Fixture work budget failed");
|
||||
} finally {
|
||||
try { await db?.$client.end({ timeout: 0 }); await database?.cleanup(); await fs.rm(dir, { recursive: true, force: true }); }
|
||||
finally { clearInterval(keepAlive); }
|
||||
}
|
||||
@@ -0,0 +1,384 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { createHash, randomBytes, randomUUID } from "node:crypto";
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { sql } from "../server/node_modules/drizzle-orm/index.js";
|
||||
import { companies, createDb, applyPendingMigrations, inspectMigrations, startEmbeddedPostgresTestDatabase, type Db } from "../packages/db/src/index.js";
|
||||
import { issueService } from "../server/src/services/issues.js";
|
||||
import { issueReadSqlCondition } from "../server/src/services/authorization.js";
|
||||
|
||||
const repositoryRoot = fileURLToPath(new URL("../", import.meta.url));
|
||||
const repositoryPath = (relative: string) => path.join(repositoryRoot, relative);
|
||||
const gitOutput = (args: string[]) => execFileSync("git", args, { cwd: repositoryRoot, encoding: "utf8" });
|
||||
const baselineRef = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15);
|
||||
assert.ok(baselineRef, "Pass a pinned pre-fix --baseline-ref=<commit>");
|
||||
const baselineRevision = gitOutput(["rev-parse", `${baselineRef}^{commit}`]).trim();
|
||||
const diagnostic = process.argv.includes("--diagnostic");
|
||||
const diagnosticDense = process.argv.includes("--diagnostic-dense");
|
||||
const fixtureFilter = process.argv.find(arg => arg.startsWith("--fixture="))?.slice(10);
|
||||
assert.ok(fixtureFilter === undefined || ["small", "dense", "sparse"].includes(fixtureFilter), "--fixture must be small, dense or sparse");
|
||||
const verifyMigration = process.argv.includes("--verify-migration");
|
||||
const migrationName = "0395_tranquil_firestar.sql";
|
||||
const indexName = "activity_log_company_user_issue_actor_idx";
|
||||
const migrationSql = await fs.readFile(repositoryPath(`packages/db/src/migrations/${migrationName}`), "utf8");
|
||||
const migrationHash = createHash("sha256").update(migrationSql).digest("hex");
|
||||
const holdAfterReport = process.argv.find(arg => arg.startsWith("--hold-after-report="))?.slice(20);
|
||||
const historyDecoys = Number(process.argv.find(arg => arg.startsWith("--history-decoys="))?.slice(17) ?? 0);
|
||||
assert.ok(Number.isSafeInteger(historyDecoys) && historyDecoys >= 0, "--history-decoys must be a nonnegative integer");
|
||||
const treatmentIsBaseline = process.argv.includes("--treatment-is-baseline");
|
||||
const enforceBudget = process.argv.includes("--assert-performance");
|
||||
const maxDenseArgument = process.argv.find(arg => arg.startsWith("--max-dense-ms="))?.slice(15);
|
||||
const maxDenseMs = maxDenseArgument === undefined ? Infinity : Number(maxDenseArgument);
|
||||
const validateBaseline = process.argv.includes("--validate-baseline");
|
||||
const validateReport = process.argv.find(arg => arg.startsWith("--validate-report="))?.slice(18);
|
||||
const sampleGateDir = process.argv.find(arg => arg.startsWith("--sample-gate-dir="))?.slice(18);
|
||||
const evidenceDir = process.argv.find(arg => arg.startsWith("--evidence-dir="))?.slice(15);
|
||||
const started = performance.now();
|
||||
const modulePaths = ["server/src/services/issues.ts", "server/src/services/issue-participation.ts"];
|
||||
const proofPaths = [...modulePaths, "packages/db/src/client.ts", "packages/db/src/schema/activity_log.ts", `packages/db/src/migrations/${migrationName}`];
|
||||
const changedDependencies = gitOutput(["diff", "--name-only", baselineRevision, "--", "server/src/services", "packages/db/src", "packages/shared/src"]).trim().split("\n").filter(Boolean);
|
||||
assert.deepEqual(changedDependencies.filter(file => !modulePaths.includes(file) && file !== "packages/db/src/client.ts" && file !== "packages/db/src/schema/activity_log.ts" && !file.startsWith("packages/db/src/migrations/")), [], "Baseline must use unchanged service dependencies");
|
||||
type PerformanceCase = {
|
||||
jit: string; readMode: string; baseline: { medianMs: number }; treatment: { medianMs: number }; speedup: number;
|
||||
plans: Array<{ arm: string; queries: Array<{ plan: Array<{ JIT?: { Options?: { Inlining?: boolean; Optimization?: boolean } } }> }> }>;
|
||||
};
|
||||
type PerformanceResult = { fixture: { name: string }; cases: PerformanceCase[] };
|
||||
function assertPerformance(results: PerformanceResult[], arm: "baseline" | "treatment" = "treatment") {
|
||||
const comparisons = results.flatMap(result => result.cases.map(control => ({ fixture: result.fixture.name, ...control })));
|
||||
const jitOptions = (result: PerformanceCase, selected: "baseline" | "treatment") => result.plans.find(plan => plan.arm === selected)!.queries[0]!.plan[0]!.JIT?.Options;
|
||||
const isCompilationTarget = (result: PerformanceCase) => result.jit === "on" && (jitOptions(result, "baseline")?.Inlining || jitOptions(result, "baseline")?.Optimization);
|
||||
const targets = comparisons.filter(isCompilationTarget);
|
||||
assert.ok(targets.length > 0, "Fixture must reproduce expensive baseline JIT compilation");
|
||||
for (const result of targets) {
|
||||
const options = jitOptions(result, arm);
|
||||
assert.ok(!options?.Inlining && !options?.Optimization, `${arm} must avoid full JIT compilation passes`);
|
||||
if (arm === "treatment") assert.ok(result.speedup >= 1.5, `Default-JIT target requires at least 1.5x improvement, observed ${result.speedup.toFixed(2)}x`);
|
||||
}
|
||||
if (arm === "baseline") return;
|
||||
for (const result of comparisons) {
|
||||
if (result.fixture === "dense" && result.jit === "on" && Number.isFinite(maxDenseMs)) assert.ok(result.treatment.medianMs <= maxDenseMs, `Dense default-JIT fixture exceeds ${maxDenseMs} ms local budget: ${result.treatment.medianMs.toFixed(2)} ms`);
|
||||
if (result.fixture !== "dense" && !isCompilationTarget(result)) assert.ok(result.treatment.medianMs <= Math.max(result.baseline.medianMs * 1.25, result.baseline.medianMs + 2), `Control regressed beyond 25% and 2 ms allowance: ${result.fixture}/${result.readMode}/${result.jit}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (validateReport) {
|
||||
const report = JSON.parse(await fs.readFile(validateReport, "utf8"));
|
||||
assert.equal(report.baselineRevision, baselineRevision);
|
||||
assert.deepEqual(Object.keys(report.sourceHashes).sort(), [...proofPaths].sort(), "Report must include the measured index and runner sources");
|
||||
for (const [file, hash] of Object.entries(report.sourceHashes)) assert.equal(createHash("sha256").update(await fs.readFile(repositoryPath(file))).digest("hex"), hash, `Source changed since measured report: ${file}`);
|
||||
assertPerformance(report.results, validateBaseline ? "baseline" : "treatment");
|
||||
console.log(JSON.stringify({ status: "performance budget passed", report: validateReport, baselineRevision, sourceHashes: report.sourceHashes }));
|
||||
process.exit(0);
|
||||
}
|
||||
const baselineDir = await fs.mkdtemp(path.join(fileURLToPath(new URL("../server/scripts/", import.meta.url)), ".mine-unread-baseline-"));
|
||||
const rewrite = (source: string, originalPath: string) => source.replace(/from "(\.{1,2}\/[^\"]+)"/g, (_match, specifier: string) => {
|
||||
const resolved = fileURLToPath(new URL(specifier, pathToFileURL(repositoryPath(originalPath))));
|
||||
const target = resolved.endsWith("/services/issue-participation.js")
|
||||
? path.join(baselineDir, "issue-participation.ts") : resolved;
|
||||
return `from ${JSON.stringify(target)}`;
|
||||
});
|
||||
for (const modulePath of modulePaths) {
|
||||
const source = gitOutput(["show", `${baselineRevision}:${modulePath}`]);
|
||||
await fs.writeFile(path.join(baselineDir, path.basename(modulePath)), rewrite(source, modulePath));
|
||||
}
|
||||
const baseline = await import(pathToFileURL(path.join(baselineDir, "issues.ts")).href) as typeof import("../server/src/services/issues.js");
|
||||
type Query = { sql: string; params: unknown[]; rows: unknown[]; elapsedMs: number };
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: Db | undefined;
|
||||
let baselineDb: Db | undefined;
|
||||
let baselineConnectionString: string;
|
||||
const keepAlive = setInterval(() => {}, 1000);
|
||||
try {
|
||||
database = await startEmbeddedPostgresTestDatabase("mine-unread-benchmark-");
|
||||
db = createDb(database.connectionString, { maxConnections: 1 });
|
||||
await db.$client.unsafe('CREATE DATABASE "mine_unread_baseline"');
|
||||
const baselineUrl = new URL(database.connectionString);
|
||||
baselineUrl.pathname = "/mine_unread_baseline";
|
||||
baselineConnectionString = baselineUrl.href;
|
||||
await applyPendingMigrations(baselineConnectionString);
|
||||
baselineDb = createDb(baselineConnectionString, { maxConnections: 1 });
|
||||
await baselineDb.$client.unsafe("set client_min_messages = warning");
|
||||
await baselineDb.$client.unsafe(`DROP INDEX "${indexName}"`);
|
||||
await baselineDb.$client.unsafe("DELETE FROM drizzle.__drizzle_migrations WHERE hash=$1", [migrationHash]);
|
||||
const indexDefinitions = async (source: Db) => source.$client.unsafe("SELECT indexname,indexdef FROM pg_indexes WHERE schemaname='public' ORDER BY indexname");
|
||||
const baselineIndexes = await indexDefinitions(baselineDb), treatmentIndexes = await indexDefinitions(db);
|
||||
assert.deepEqual(treatmentIndexes.filter(index => index.indexname !== indexName), [...baselineIndexes]);
|
||||
await db.$client.unsafe("set client_min_messages = warning");
|
||||
const [settings] = await db.$client.unsafe("select version(), current_setting('shared_buffers') shared_buffers, current_setting('work_mem') work_mem, current_setting('jit') jit, current_setting('jit_above_cost') jit_above_cost");
|
||||
await issueService(db).unreadInboxIssueIds(randomUUID(), "parse-smoke", sql<boolean>`true`);
|
||||
const setupMs = performance.now() - started;
|
||||
const captured = new Error("Ranking query captured before execution");
|
||||
function captureDb(queries: Query[], captureOnly = false, sourceDb = db!): Db {
|
||||
function captureBuilder(builder: any) {
|
||||
const from = builder.from.bind(builder);
|
||||
builder.from = (...args: any[]) => {
|
||||
const query = from(...args), execute = query.execute.bind(query);
|
||||
query.execute = async () => {
|
||||
const statement = query.toSQL(), started = performance.now();
|
||||
if (captureOnly) { queries.push({ ...statement, rows: [], elapsedMs: 0 }); throw captured; }
|
||||
const rows = await execute();
|
||||
queries.push({ ...statement, rows, elapsedMs: performance.now() - started });
|
||||
return rows;
|
||||
};
|
||||
return query;
|
||||
};
|
||||
return builder;
|
||||
}
|
||||
return new Proxy(sourceDb, { get(target, key) {
|
||||
if (key === "select") return (...fields: any[]) => captureBuilder((target.select as any)(...fields));
|
||||
if (key === "with") return (...ctes: any[]) => {
|
||||
const builder = (target.with as any)(...ctes), select = builder.select.bind(builder);
|
||||
builder.select = (...fields: any[]) => captureBuilder(select(...fields));
|
||||
return builder;
|
||||
};
|
||||
return Reflect.get(target, key);
|
||||
} });
|
||||
}
|
||||
const fixtureSeeds = new Map<string, { companyId: string; ids: string[]; payload: string }>();
|
||||
async function seed(name: string, count: number, participatedCount: number, logsPerIssue: number, commentsPerIssue: number, target = db!) {
|
||||
const fixtureStart = performance.now();
|
||||
const inputs = fixtureSeeds.get(name) ?? { companyId: randomUUID(), ids: Array.from({ length: count }, () => randomUUID()), payload: randomBytes(3072).toString("base64") };
|
||||
fixtureSeeds.set(name, inputs);
|
||||
const { companyId, ids, payload } = inputs, userId = `synthetic-${name}`;
|
||||
await target.insert(companies).values({ id: companyId, name: `Synthetic Mine ${name}`, issuePrefix: name.toUpperCase() });
|
||||
await target.$client.unsafe(`insert into issues(id,company_id,title,status,created_by_user_id,created_at,updated_at)
|
||||
select id,$2::uuid,'Synthetic '||n,'todo',case when n <= $4 and n%2=0 then $3 else 'unrelated-human' end,
|
||||
'2026-01-01'::timestamptz,'2026-01-01'::timestamptz+n*interval '1 second'
|
||||
from unnest($1::uuid[]) with ordinality as f(id,n)`, [ids, companyId, userId, participatedCount]);
|
||||
await target.$client.unsafe(`insert into activity_log(company_id,entity_type,entity_id,actor_type,actor_id,action,details,created_at)
|
||||
select $1::uuid,'issue',i.id::text,case when n=1 or n%10=0 then 'user' else 'agent' end,
|
||||
case when (n=1 or n%20=0) and i.id=ANY($5::uuid[]) then $4 when n=1 or n%10=0 then 'other-human' else 'synthetic-worker' end,
|
||||
case when n=1 or n%10=0 then 'issue.updated' when n%5=0 then 'issue.read_marked' else 'issue.updated' end,
|
||||
jsonb_build_object('padding',$2::text),'2026-01-01'::timestamptz+n*interval '1 second'
|
||||
from issues i cross join generate_series(1,$3::int)n where company_id=$1::uuid order by n,i.id`, [companyId, payload, logsPerIssue, userId, ids.slice(0, participatedCount)]);
|
||||
await target.$client.unsafe(`insert into issue_comments(company_id,issue_id,author_user_id,body,created_at)
|
||||
select $1::uuid,id,null,$2,'2026-01-02'::timestamptz+n*interval '1 second'
|
||||
from issues cross join generate_series(1,$3::int)n where company_id=$1::uuid order by n,id`, [companyId, payload, commentsPerIssue]);
|
||||
if (historyDecoys > 0 && name !== "small") await target.$client.unsafe(`insert into activity_log(company_id,entity_type,entity_id,actor_type,actor_id,action,details,created_at)
|
||||
select $1::uuid,'heartbeat_run',gen_random_uuid()::text,'agent','synthetic-worker','heartbeat.run.created',
|
||||
jsonb_build_object('padding',left($2,256)),'2026-01-01'::timestamptz+n*interval '1 second'
|
||||
from generate_series(1,$3::int)n`, [companyId, payload, historyDecoys]);
|
||||
await target.$client.unsafe("ANALYZE");
|
||||
const [counts] = await target.$client.unsafe(`SELECT count(*)::int participants FROM issues i WHERE company_id=$1 AND
|
||||
(created_by_user_id=$2 OR EXISTS(SELECT 1 FROM activity_log a WHERE a.company_id=$1 AND a.entity_type='issue' AND a.entity_id=i.id::text AND a.actor_type='user' AND a.actor_id=$2 AND a.action='issue.updated'))`, [companyId, userId]);
|
||||
assert.equal(counts!.participants, participatedCount, "Fixture must include its complete declared personal set");
|
||||
const [actorCounts] = await target.$client.unsafe("SELECT count(*)::int rows FROM activity_log WHERE company_id=$1 AND actor_type='user' AND actor_id=$2 AND entity_type='issue'", [companyId, userId]);
|
||||
return { name, companyId, userId, issues: count, participatedCount, activityRows: count * logsPerIssue,
|
||||
commentRows: count * commentsPerIssue, targetUserActivityRows: actorCounts!.rows, createdParticipation: Math.floor(participatedCount / 2), activityParticipation: Math.ceil(participatedCount / 2), historyDecoys: name === "small" ? 0 : historyDecoys, payloadBytes: Buffer.byteLength(payload), fixtureMs: performance.now() - fixtureStart };
|
||||
}
|
||||
async function run(factory: typeof issueService, fixture: Awaited<ReturnType<typeof seed>>, readCondition: Awaited<ReturnType<typeof issueReadSqlCondition>>, sourceDb: Db) {
|
||||
const queries: Query[] = [], started = performance.now();
|
||||
const result = await factory(captureDb(queries, false, sourceDb)).unreadInboxIssueIds(fixture.companyId, fixture.userId, readCondition);
|
||||
const elapsedMs = performance.now() - started;
|
||||
return { result, queries, elapsedMs, serialSqlMs: queries.reduce((total, query) => total + query.elapsedMs, 0) };
|
||||
}
|
||||
const stats = (values: number[]) => ({ samplesMs: values, medianMs: [...values].sort((a, b) => a - b)[Math.floor(values.length / 2)], minMs: Math.min(...values), maxMs: Math.max(...values) });
|
||||
let migrationProof: unknown;
|
||||
async function measureMigration(fixture: Awaited<ReturnType<typeof seed>>) {
|
||||
const pending = await inspectMigrations(baselineConnectionString);
|
||||
assert.deepEqual(pending.pendingMigrations, [migrationName]);
|
||||
const monitor = createDb(baselineConnectionString, { maxConnections: 2 });
|
||||
await monitor.$client.unsafe("SET lock_timeout='1s'");
|
||||
await monitor.$client.unsafe("SET statement_timeout='5s'");
|
||||
const size = await monitor.$client.unsafe("SELECT reltuples::bigint estimated_rows,pg_relation_size(oid)::text heap_bytes,pg_total_relation_size(oid)::text total_bytes FROM pg_class WHERE oid='activity_log'::regclass");
|
||||
let completed = false;
|
||||
const reads: number[] = [], writes: number[] = [], progress: unknown[] = [];
|
||||
const load = (async () => {
|
||||
while (!completed) {
|
||||
let start = performance.now();
|
||||
await monitor.$client.unsafe("INSERT INTO activity_log(company_id,entity_type,entity_id,actor_type,actor_id,action) VALUES($1,'issue',$2,'user','synthetic-migration-writer','issue.updated')", [fixture.companyId, fixtureSeeds.get(fixture.name)!.ids[0]!]);
|
||||
writes.push(performance.now() - start);
|
||||
start = performance.now();
|
||||
await monitor.$client.unsafe("SELECT id FROM issues WHERE company_id=$1 LIMIT 1", [fixture.companyId]);
|
||||
reads.push(performance.now() - start);
|
||||
await new Promise(resolve => setTimeout(resolve, 10));
|
||||
}
|
||||
})();
|
||||
const observe = (async () => {
|
||||
while (!completed) {
|
||||
const rows = await monitor.$client.unsafe("SELECT phase,blocks_done,blocks_total FROM pg_stat_progress_create_index WHERE datid=(SELECT oid FROM pg_database WHERE datname=current_database())");
|
||||
progress.push(...rows);
|
||||
await new Promise(resolve => setTimeout(resolve, 10));
|
||||
}
|
||||
})();
|
||||
let firstRunMs: number;
|
||||
try {
|
||||
const start = performance.now();
|
||||
await applyPendingMigrations(baselineConnectionString);
|
||||
firstRunMs = performance.now() - start;
|
||||
} finally {
|
||||
completed = true;
|
||||
await Promise.all([load, observe]);
|
||||
}
|
||||
assert.ok(reads.length > 0 && writes.length > 0, "Migration proof must overlap real reads and writes");
|
||||
const validity = async () => (await monitor.$client.unsafe("SELECT i.indisvalid valid,pg_relation_size(i.indexrelid)::text bytes FROM pg_index i JOIN pg_class c ON c.oid=i.indexrelid WHERE c.relname=$1", [indexName]))[0];
|
||||
assert.equal((await validity())?.valid, true);
|
||||
const rerunStart = performance.now();
|
||||
await applyPendingMigrations(baselineConnectionString);
|
||||
const rerunMs = performance.now() - rerunStart;
|
||||
await monitor.$client.unsafe(`DROP INDEX CONCURRENTLY "${indexName}"`);
|
||||
await monitor.$client.unsafe("DELETE FROM drizzle.__drizzle_migrations WHERE hash=$1", [migrationHash]);
|
||||
const blocker = await monitor.$client.reserve();
|
||||
let interruptionCode: string | undefined;
|
||||
try {
|
||||
await blocker.unsafe("BEGIN");
|
||||
await blocker.unsafe("INSERT INTO activity_log(company_id,entity_type,entity_id,actor_type,actor_id,action) VALUES($1,'issue',$2,'agent','synthetic-blocker','issue.updated')", [fixture.companyId, fixtureSeeds.get(fixture.name)!.ids[0]!]);
|
||||
const interrupted = applyPendingMigrations(baselineConnectionString).catch((error: { code?: string }) => { interruptionCode = error.code; });
|
||||
const deadline = Date.now() + 10_000;
|
||||
let cancelled = false;
|
||||
while (Date.now() < deadline) {
|
||||
const active = await monitor.$client.unsafe("SELECT p.pid FROM pg_stat_progress_create_index p JOIN pg_class c ON c.oid=p.index_relid WHERE p.datid=(SELECT oid FROM pg_database WHERE datname=current_database()) AND c.relname=$1 AND p.phase LIKE 'waiting for writers%'", [indexName]);
|
||||
if (active.length) { await monitor.$client.unsafe("SELECT pg_cancel_backend($1)", [active[0]!.pid]); cancelled = true; break; }
|
||||
await new Promise(resolve => setTimeout(resolve, 25));
|
||||
}
|
||||
await blocker.unsafe("ROLLBACK");
|
||||
await interrupted;
|
||||
assert.ok(cancelled, "Observe and cancel only this owned migration's blocked index build");
|
||||
} finally {
|
||||
await blocker.unsafe("ROLLBACK");
|
||||
blocker.release();
|
||||
}
|
||||
assert.equal(interruptionCode, "57014");
|
||||
const interruptedIndex = await validity();
|
||||
assert.equal(interruptedIndex?.valid, false);
|
||||
assert.deepEqual((await inspectMigrations(baselineConnectionString)).pendingMigrations, [migrationName]);
|
||||
const retryStart = performance.now();
|
||||
await applyPendingMigrations(baselineConnectionString);
|
||||
const retryMs = performance.now() - retryStart;
|
||||
const finalIndex = await validity();
|
||||
assert.equal(finalIndex?.valid, true);
|
||||
assert.equal((await inspectMigrations(baselineConnectionString)).status, "upToDate");
|
||||
await monitor.$client.end({ timeout: 0 });
|
||||
return { migrationName, migrationSha256: createHash("sha256").update(migrationSql).digest("hex"), statementCount: 1, firstRunMs, rerunMs, retryMs, size, reads: stats(reads), writes: stats(writes), progress, interruptionCode, interruptedIndex, finalIndex };
|
||||
}
|
||||
const results = [];
|
||||
const shapes = diagnostic ? [[diagnosticDense ? "dense" : "sparse", 1200, diagnosticDense ? 1200 : 12, 50, 20] as const] : [
|
||||
["small", 12, 12, 4, 2],
|
||||
["dense", 1200, 1200, 50, 20],
|
||||
["sparse", 1200, 12, 50, 20],
|
||||
] as const;
|
||||
const selectedShapes = shapes.filter(shape => fixtureFilter === undefined || shape[0] === fixtureFilter);
|
||||
assert.ok(selectedShapes.length, "Selected fixture must exist in this diagnostic mode");
|
||||
if (holdAfterReport) assert.equal(selectedShapes.length, 1, "Retain only one explicitly selected diagnostic fixture");
|
||||
for (const shape of selectedShapes) {
|
||||
const fixture = { ...await seed(...shape), baselineFixtureMs: (await seed(...shape, baselineDb)).fixtureMs };
|
||||
if (sampleGateDir) {
|
||||
await fs.mkdir(sampleGateDir, { recursive: true });
|
||||
const release = path.join(sampleGateDir, `${fixture.name}.release`);
|
||||
const scoped = await issueReadSqlCondition(db, { type: "board", source: "session", userId: fixture.userId, authorizingUserId: fixture.userId });
|
||||
const estimatedPlans = [];
|
||||
await db.$client.unsafe("set jit = on");
|
||||
await baselineDb.$client.unsafe("set jit = on");
|
||||
for (const [arm, factory] of [["baseline", baseline.issueService], ["treatment", issueService]] as const) {
|
||||
const queries: Query[] = [];
|
||||
try { await factory(captureDb(queries, true, arm === "baseline" ? baselineDb : db)).unreadInboxIssueIds(fixture.companyId, fixture.userId, scoped); }
|
||||
catch (error) { if (error !== captured) throw error; }
|
||||
assert.equal(queries.length, 1);
|
||||
const query = queries[0]!;
|
||||
const [plan] = await (arm === "baseline" ? baselineDb : db).$client.unsafe(`EXPLAIN (FORMAT JSON) ${query.sql}`, query.params as never[]);
|
||||
estimatedPlans.push({ arm, plan: plan["QUERY PLAN"] });
|
||||
}
|
||||
await fs.writeFile(path.join(sampleGateDir, `${fixture.name}.ready.json`), JSON.stringify({ fixture, settings, estimatedPlans }));
|
||||
console.error(`Ready for paired samples: ${fixture.name}`);
|
||||
const deadline = Date.now() + 600_000;
|
||||
while (true) {
|
||||
try { await fs.access(release); break; } catch {}
|
||||
assert.ok(Date.now() < deadline, `Timed out waiting for ${release}`);
|
||||
await new Promise(resolve => setTimeout(resolve, 250));
|
||||
}
|
||||
await fs.rm(release);
|
||||
}
|
||||
const cases = [];
|
||||
for (const readMode of ["unrestricted", "scoped"]) {
|
||||
const readCondition = readMode === "unrestricted" ? sql<boolean>`true` : await issueReadSqlCondition(db, {
|
||||
type: "board", source: "session", userId: fixture.userId, authorizingUserId: fixture.userId,
|
||||
});
|
||||
for (const jit of ["on", "off"]) {
|
||||
await db.$client.unsafe(`set jit = ${jit}`);
|
||||
await baselineDb.$client.unsafe(`set jit = ${jit}`);
|
||||
const estimatedPlans = [];
|
||||
if (diagnostic) {
|
||||
for (const [arm, factory] of [["baseline", baseline.issueService], ["treatment", issueService]] as const) {
|
||||
const queries: Query[] = [];
|
||||
try { await factory(captureDb(queries, true, arm === "baseline" ? baselineDb : db)).unreadInboxIssueIds(fixture.companyId, fixture.userId, readCondition); }
|
||||
catch (error) { if (error !== captured) throw error; }
|
||||
assert.equal(queries.length, 1);
|
||||
const query = queries[0]!;
|
||||
const [plan] = await (arm === "baseline" ? baselineDb : db).$client.unsafe(`EXPLAIN (FORMAT JSON) ${query.sql}`, query.params as never[]);
|
||||
estimatedPlans.push({ arm, plan: plan["QUERY PLAN"] });
|
||||
console.error(JSON.stringify({ diagnostic: fixture.name, jit, arm, estimatedCost: plan["QUERY PLAN"][0].Plan["Total Cost"] }));
|
||||
}
|
||||
}
|
||||
const beforeTimes: number[] = [], afterTimes: number[] = [];
|
||||
let beforeResult: string[] | undefined, afterResult: string[] | undefined;
|
||||
for (let iteration = diagnostic ? 0 : -2; iteration < (diagnostic ? 1 : 5); iteration++) {
|
||||
const order = iteration % 2 === 0 ? ["before", "after"] : ["after", "before"];
|
||||
for (const arm of order) {
|
||||
const sourceDb = arm === "before" ? baselineDb : db;
|
||||
const factory = arm === "before" || treatmentIsBaseline ? baseline.issueService : issueService;
|
||||
const start = performance.now();
|
||||
const current = await factory(sourceDb).unreadInboxIssueIds(fixture.companyId, fixture.userId, readCondition);
|
||||
const elapsedMs = performance.now() - start;
|
||||
if (arm === "before") { beforeResult = current; if (iteration >= 0) beforeTimes.push(elapsedMs); }
|
||||
else { afterResult = current; if (iteration >= 0) afterTimes.push(elapsedMs); }
|
||||
}
|
||||
assert.deepEqual(afterResult, beforeResult, `${fixture.name}/${jit}: complete returned ID order`);
|
||||
}
|
||||
const before = await run(baseline.issueService, fixture, readCondition, baselineDb);
|
||||
const after = await run(treatmentIsBaseline ? baseline.issueService : issueService, fixture, readCondition, db);
|
||||
assert.deepEqual(before.result, beforeResult, "SQL capture must preserve the measured baseline result");
|
||||
assert.deepEqual(after.result, afterResult, "SQL capture must preserve the measured treatment result");
|
||||
assert.equal(before!.queries.length, 2);
|
||||
assert.equal(after!.queries.length, 2);
|
||||
for (let index = 0; index < 2; index++) {
|
||||
const oldQuery = before!.queries[index]!, newQuery = after!.queries[index]!;
|
||||
const oldRows = await baselineDb.$client.unsafe(oldQuery.sql, oldQuery.params as never[]).values();
|
||||
const newRows = await db.$client.unsafe(newQuery.sql, newQuery.params as never[]).values();
|
||||
assert.deepEqual(newRows, oldRows, `${fixture.name}/${jit}: full positional SQL results phase ${index + 1}`);
|
||||
}
|
||||
const plans = [];
|
||||
for (const [arm, current] of [["baseline", before!], ["treatment", after!]] as const) {
|
||||
const queries = [];
|
||||
for (const query of current.queries) {
|
||||
const [plan] = await (arm === "baseline" ? baselineDb : db).$client.unsafe(`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${query.sql}`, query.params as never[]);
|
||||
queries.push({ sql: query.sql, params: query.params, rows: query.rows.length, plan: plan["QUERY PLAN"] });
|
||||
}
|
||||
plans.push({ arm, queries });
|
||||
}
|
||||
const baselineStats = stats(beforeTimes), treatmentStats = stats(afterTimes);
|
||||
cases.push({ readMode, jit, baseline: baselineStats, treatment: treatmentStats, speedup: baselineStats.medianMs / treatmentStats.medianMs, equalIds: after!.result.length, estimatedPlans, plans });
|
||||
}
|
||||
}
|
||||
results.push({ fixture, cases });
|
||||
if (sampleGateDir) await fs.writeFile(path.join(sampleGateDir, `${fixture.name}.done.json`), JSON.stringify({ fixture, cases }));
|
||||
if (verifyMigration && shape === selectedShapes[selectedShapes.length - 1]) migrationProof = await measureMigration(fixture);
|
||||
if (!holdAfterReport) await db.$client.unsafe("TRUNCATE activity_log, issues, companies CASCADE");
|
||||
if (!holdAfterReport) await baselineDb.$client.unsafe("TRUNCATE activity_log, issues, companies CASCADE");
|
||||
}
|
||||
const report = { baselineRevision, treatmentRevision: gitOutput(["rev-parse", "HEAD"]).trim(),
|
||||
treatmentIsBaseline, diagnostic, migrationProof, baselineIndexes, treatmentIndexes, sourceHashes: Object.fromEntries(await Promise.all(proofPaths.map(async file => [file, createHash("sha256").update(await fs.readFile(repositoryPath(file))).digest("hex")]))),
|
||||
checkedInLockSha256: createHash("sha256").update(await fs.readFile(repositoryPath("pnpm-lock.yaml"))).digest("hex"), nodeVersion: process.version, dependencyVersions: Object.fromEntries(await Promise.all(["postgres", "drizzle-orm"].map(async name => [name, JSON.parse(await fs.readFile(repositoryPath(`packages/db/node_modules/${name}/package.json`), "utf8")).version]))), settings, setupMs, totalMs: performance.now() - started, results };
|
||||
const json = JSON.stringify(report, null, 2);
|
||||
if (evidenceDir) { await fs.mkdir(evidenceDir, { recursive: true }); await fs.writeFile(path.join(evidenceDir, "benchmark.json"), json); }
|
||||
console.log(json);
|
||||
if (holdAfterReport) {
|
||||
await fs.mkdir(holdAfterReport, { recursive: true });
|
||||
await fs.writeFile(path.join(holdAfterReport, "ready.json"), JSON.stringify({ baselineConnectionString, treatmentConnectionString: database.connectionString, baselineDir, fixture: results[0]!.fixture }), { mode: 0o600 });
|
||||
const deadline = Date.now() + 600_000;
|
||||
while (true) {
|
||||
try { await fs.access(path.join(holdAfterReport, "release")); await fs.rm(path.join(holdAfterReport, "release")); break; } catch {}
|
||||
assert.ok(Date.now() < deadline, "Timed out waiting for retained diagnostic release");
|
||||
await new Promise(resolve => setTimeout(resolve, 250));
|
||||
}
|
||||
}
|
||||
if (enforceBudget && !diagnostic) assertPerformance(results);
|
||||
} finally {
|
||||
if (holdAfterReport) await fs.rm(path.join(holdAfterReport, "ready.json"), { force: true });
|
||||
await baselineDb?.$client.end({ timeout: 0 });
|
||||
await db?.$client.end({ timeout: 0 });
|
||||
await database?.cleanup();
|
||||
await fs.rm(baselineDir, { recursive: true, force: true });
|
||||
clearInterval(keepAlive);
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { createDb, startEmbeddedPostgresTestDatabase, type Db } from "../packages/db/src/index.js";
|
||||
import { taskInterruptionPresentations } from "../server/src/services/task-interruption-presentation.js";
|
||||
import { captureInterruptionQuery, seedInterruptionFixture, type CapturedQuery } from "../server/src/__tests__/helpers/task-interruption-fixture.js";
|
||||
|
||||
const startedAt = performance.now();
|
||||
const baselineRef = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15);
|
||||
assert.ok(baselineRef, "Pass an explicit pre-fix --baseline-ref");
|
||||
const baselineRevision = execFileSync("git", ["rev-parse", "--verify", `${baselineRef}^{commit}`], { encoding: "utf8" }).trim();
|
||||
const modulePath = "server/src/services/task-interruption-presentation.ts";
|
||||
const changedRuntime = execFileSync("git", ["diff", "--name-only", baselineRevision, "--", "server/src/services", "packages/db/src", "packages/shared/src", "pnpm-lock.yaml"], { encoding: "utf8" }).trim().split("\n").filter(Boolean);
|
||||
assert.deepEqual(changedRuntime.filter(file => file !== modulePath), [], "Changed baseline dependency requires explicit isolation");
|
||||
const keepAlive = setInterval(() => {}, 1000);
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: Db | undefined, baselineDir: string | undefined;
|
||||
try {
|
||||
const oldSource = execFileSync("git", ["show", `${baselineRevision}:${modulePath}`], { encoding: "utf8" });
|
||||
const moduleUrl = new URL(`../${modulePath}`, import.meta.url);
|
||||
const source = oldSource.replace(/from "(\.{1,2}\/[^\"]+)"/g, (_match, specifier: string) =>
|
||||
`from ${JSON.stringify(fileURLToPath(new URL(specifier, moduleUrl)))}`);
|
||||
baselineDir = await fs.mkdtemp(path.join(fileURLToPath(new URL("../server/scripts/", import.meta.url)), ".interruption-baseline-"));
|
||||
const baselineFile = path.join(baselineDir, "task-interruption.ts");
|
||||
await fs.writeFile(baselineFile, source);
|
||||
const baseline = await import(pathToFileURL(baselineFile).href) as typeof import("../server/src/services/task-interruption-presentation.js");
|
||||
database = await startEmbeddedPostgresTestDatabase("interruption-benchmark-");
|
||||
db = createDb(database.connectionString);
|
||||
const settings = await db.$client.unsafe("select version(), current_setting('shared_buffers') shared_buffers, current_setting('work_mem') work_mem, current_setting('jit') jit");
|
||||
const setupMs = performance.now() - startedAt;
|
||||
const results = [];
|
||||
for (const fixture of [
|
||||
{ selected: 1, history: 1, unrelated: 0, payloadBytes: 128 },
|
||||
{ selected: 100, history: 1, unrelated: 1200, payloadBytes: 24 * 1024 },
|
||||
{ selected: 100, history: 12, unrelated: 1200, payloadBytes: 24 * 1024 },
|
||||
{ selected: 1, history: 120, unrelated: 1200, payloadBytes: 24 * 1024 },
|
||||
{ selected: 100, history: 40, unrelated: 1200, payloadBytes: 12 * 1024 },
|
||||
{ selected: 100, history: 40, unrelated: 1200, payloadBytes: 12 * 1024, preserveLegacyBindings: false },
|
||||
{ selected: 1000, history: 1, unrelated: 1200, payloadBytes: 12 * 1024, preserveLegacyBindings: false },
|
||||
]) {
|
||||
await db.$client.begin(async client => {
|
||||
await client.unsafe("set local client_min_messages = warning");
|
||||
await client.unsafe("truncate heartbeat_runs cascade");
|
||||
});
|
||||
const seedStart = performance.now();
|
||||
const f = await seedInterruptionFixture(db, fixture);
|
||||
const seedMs = performance.now() - seedStart;
|
||||
const before = await captureInterruptionQuery(db, baseline.taskInterruptionPresentations, f.companyId, f.input);
|
||||
const after = await captureInterruptionQuery(db, taskInterruptionPresentations, f.companyId, f.input);
|
||||
const execute = async (query: CapturedQuery) => db!.$client.unsafe(query.sql, query.params as never[]).values();
|
||||
const beforeRows = await execute(before), afterRows = await execute(after);
|
||||
assert.deepEqual(afterRows, beforeRows, "Every positional SQL field and row order must match");
|
||||
assert.deepEqual(await taskInterruptionPresentations(db, f.companyId, f.input),
|
||||
await baseline.taskInterruptionPresentations(db, f.companyId, f.input), "Full service output must match");
|
||||
const samples = { baseline: [] as number[], treatment: [] as number[] };
|
||||
for (let round = 0; round < 7; round++) {
|
||||
for (const arm of round % 2 ? ["treatment", "baseline"] as const : ["baseline", "treatment"] as const) {
|
||||
const start = performance.now();
|
||||
await execute(arm === "baseline" ? before : after);
|
||||
if (round >= 2) samples[arm].push(performance.now() - start);
|
||||
}
|
||||
}
|
||||
const explain = async (query: CapturedQuery) => (await db!.$client.unsafe(`explain (analyze, buffers, format json) ${query.sql}`, query.params as never[]))[0]!["QUERY PLAN"];
|
||||
const summarize = (values: number[]) => { const sorted = [...values].sort((a, b) => a - b); return { medianMs: sorted[2], minMs: sorted[0], maxMs: sorted[4], samplesMs: values }; };
|
||||
const baselineTiming = summarize(samples.baseline), treatmentTiming = summarize(samples.treatment);
|
||||
results.push({ fixture, seedMs, returnedRows: afterRows.length, positionalEquality: true, serviceEquality: true,
|
||||
baseline: { ...baselineTiming, query: before, plan: await explain(before) },
|
||||
treatment: { ...treatmentTiming, query: after, plan: await explain(after) },
|
||||
speedup: baselineTiming.medianMs! / treatmentTiming.medianMs! });
|
||||
}
|
||||
console.log(JSON.stringify({ baselineRevision, treatmentRevision: execFileSync("git", ["rev-parse", "HEAD"], { encoding: "utf8" }).trim(),
|
||||
treatmentSourceSha256: createHash("sha256").update(await fs.readFile(fileURLToPath(new URL(`../${modulePath}`, import.meta.url)))).digest("hex"),
|
||||
changedRuntime, settings, node: process.version, setupMs, totalMs: performance.now() - startedAt, warmupsPerArm: 2, measuredSamplesPerArm: 5, results }, null, 2));
|
||||
} finally {
|
||||
try { await db?.$client.end({ timeout: 0 }); await database?.cleanup(); if (baselineDir) await fs.rm(baselineDir, { recursive: true, force: true }); }
|
||||
finally { clearInterval(keepAlive); }
|
||||
}
|
||||
@@ -3,7 +3,7 @@
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="description" content="Paperclip evaluation results, run histories, and guides for Runner Evals and Product E2E Evals.">
|
||||
<meta name="description" content="Paperclip E2E tests and Runner protocol evals: suites, deployment targets, results, screenshots, and run history.">
|
||||
<title>Paperclip Evals</title>
|
||||
<style>
|
||||
:root { color-scheme: light dark; --bg:#f8f8f5; --card:#fff; --text:#20221e; --muted:#60645c; --line:#d9ddd3; --accent:#306340; --tint:#edf3ec; }
|
||||
@@ -49,23 +49,23 @@
|
||||
<p class="intro">Two ways to test Paperclip. Follow each program from a test case to its evidence, results, and run history.</p>
|
||||
<div class="grid">
|
||||
<article>
|
||||
<div class="number">01 / RUNNER</div>
|
||||
<h2>Runner Evals</h2>
|
||||
<div class="number">01 / E2E</div>
|
||||
<h2>E2E tests</h2>
|
||||
<p>Can a user complete work through Paperclip, from the first request to a usable result?</p>
|
||||
<div class="boundary">Real app journeys, including browser-only and agent workflows<br>Filter by suite, deployment target, agent execution, and model</div>
|
||||
<div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$product_counts</div><div class="meta">$product_date · $product_coverage</div><div class="id">$product_id</div></div>
|
||||
<div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history">Browse suites and runs →</a><a href="https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history&preset=qa-smoke">QA smoke runs</a><a href="$product_url">Latest report</a></div>
|
||||
<a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/tests/runner-e2e/README.md">E2E operations guide</a>
|
||||
</article>
|
||||
<article>
|
||||
<div class="number">02 / RUNNER PROTOCOL</div>
|
||||
<h2>Runner protocol evals</h2>
|
||||
<p>Does the runner use tools correctly, respect permissions, and preserve its execution contract?</p>
|
||||
<div class="boundary">Real runner + model provider<br>Controlled mock Paperclip control plane</div>
|
||||
<div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$runner_counts</div><div class="meta">$runner_date · $runner_coverage</div><div class="id">$runner_id</div></div>
|
||||
<div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html">Browse run history →</a><a href="$runner_url">Open report</a></div>
|
||||
<a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/packages/paperclip-runner/docs/runner-protocol-live-evals.md">Runner operations guide</a>
|
||||
</article>
|
||||
<article>
|
||||
<div class="number">02 / PRODUCT</div>
|
||||
<h2>Product E2E Evals</h2>
|
||||
<p>Can a user complete work through Paperclip, from the first request to a usable result?</p>
|
||||
<div class="boundary">Real browser + server + database + runner + model provider<br>Local and Daytona environments, as selected</div>
|
||||
<div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$product_counts</div><div class="meta">$product_date · $product_coverage</div><div class="id">$product_id</div></div>
|
||||
<div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-e2e/">Browse run history →</a><a href="$product_url">Open report</a></div>
|
||||
<a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/tests/runner-e2e/README.md">Product E2E operations guide</a>
|
||||
</article>
|
||||
</div>
|
||||
<p class="note">Snapshot refreshed $refreshed. History pages may contain newer runs. A partial campaign covers only its selected cases. These programs measure different boundaries; their pass rates are not combined.</p>
|
||||
<section class="start"><div><h2>Choose a test. Make the result inspectable.</h2><p>The eval guide explains which program to use, how to add a case, and how to distinguish a product defect from model behavior, a grading error, or a setup failure. Evalbook is a report framework used to present evidence; it is not an execution boundary.</p></div><a class="button" href="$guide_url">Read the eval guide →</a></section>
|
||||
|
||||
@@ -11,6 +11,12 @@ export const GROK_PUBLIC_INSTALL_LIFECYCLE = [
|
||||
'npm', 'rebuild', '--offline', '--ignore-scripts=false', '--dangerously-allow-all-scripts',
|
||||
];
|
||||
|
||||
export function grokPublicInstallVersion(sourceRevision) {
|
||||
assert.match(sourceRevision, /^[a-f0-9]{40}$/);
|
||||
// An all-numeric hash with a leading zero is not a valid numeric prerelease.
|
||||
return `0.0.0-grok-verify.g${sourceRevision.slice(0, 12)}`;
|
||||
}
|
||||
|
||||
export function runMacInitialInstallWithDiagnostics(install, { sourceRevision, platform, architecture, cacheDirectory, receiptPath }) {
|
||||
const started = performance.now();
|
||||
try {
|
||||
|
||||
@@ -14,7 +14,7 @@ import { fileURLToPath } from 'node:url';
|
||||
import { packConcurrency, packStagedPackages, packTimings } from './pack-staged-packages.mjs';
|
||||
import { materializePublishManifest } from './prepare-bundled-package.mjs';
|
||||
import { publicInstallClosure } from './public-install-closure.mjs';
|
||||
import { GROK_PUBLIC_INSTALL_IMAGE, GROK_PUBLIC_INSTALL_LIFECYCLE, assertInstalledNativeConsumerContinuity, assertNoBundledCodexPayloads, grokConsumerDockerArgs, installedClaudeProbeSource, installedCodexProbeSource, installedDaemonProbeSource, macNodeHeadersDirectory, macPublicInstallLifecycleProfile, macInstalledStartupProbeSource, runMacInitialInstallWithDiagnostics } from './grok-public-install-sandbox.mjs';
|
||||
import { GROK_PUBLIC_INSTALL_IMAGE, GROK_PUBLIC_INSTALL_LIFECYCLE, assertInstalledNativeConsumerContinuity, assertNoBundledCodexPayloads, grokConsumerDockerArgs, grokPublicInstallVersion, installedClaudeProbeSource, installedCodexProbeSource, installedDaemonProbeSource, macNodeHeadersDirectory, macPublicInstallLifecycleProfile, macInstalledStartupProbeSource, runMacInitialInstallWithDiagnostics } from './grok-public-install-sandbox.mjs';
|
||||
import { startPrefetch, stopPrefetches } from './prefetch-child.mjs';
|
||||
import { retainRunnerQualificationPackages } from './retain-runner-qualification-packages.mjs';
|
||||
import { bundledInstallTimings, startBundledInstalls } from './stage-bundled-packages.mjs';
|
||||
@@ -204,7 +204,7 @@ mkdirSync(cache, { recursive: true });
|
||||
const env = { ...process.env, NODE_PATH: '', ...(process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE ? { npm_config_cache: cache } : {}), PAPERCLIP_RELEASE_REUSE_UI_DIST: '1', npm_config_ignore_scripts: 'false', npm_config_audit: 'false', npm_config_fund: 'false' };
|
||||
const run = (cmd, args, cwd = root) => execFileSync(cmd, args, { cwd, env, stdio: 'pipe', maxBuffer: 32 * 1024 * 1024 });
|
||||
const sourceRevision = run('git', ['rev-parse', 'HEAD'], repo).toString().trim();
|
||||
const releaseVersion = `0.0.0-grok-verify.${sourceRevision.slice(0, 12)}`;
|
||||
const releaseVersion = grokPublicInstallVersion(sourceRevision);
|
||||
// The two network downloads the probes need do not depend on anything packed
|
||||
// below, so they overlap the CPU-bound staging instead of running after it.
|
||||
// The consumer image is pinned by digest and the Grok binary is checked against
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { connect, createServer, type Socket } from "node:net";
|
||||
import express from "express";
|
||||
import { get as httpGet } from "node:http";
|
||||
import { context, trace } from "@opentelemetry/api";
|
||||
import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";
|
||||
import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { authUsers, companies, companyMemberships, companySkills, companySkillSources,
|
||||
projectSkillSources, projects, applyPendingMigrations, closeRegisteredClients, createDb,
|
||||
startEmbeddedPostgresTestDatabase } from "../../packages/db/src/index.js";
|
||||
import * as schema from "../../packages/db/src/schema/index.js";
|
||||
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "../../packages/db/src/response-capture.js";
|
||||
import { withRunExecutionTiming } from "../src/services/execution-preparation-timing.js";
|
||||
|
||||
const argument = (name: string) => process.argv.find(value => value.startsWith(`${name}=`))?.slice(name.length + 1);
|
||||
const sampleCount = Number(argument("--samples") ?? 5);
|
||||
const baselineRef = argument("--baseline-ref");
|
||||
const delayMs = Number(argument("--network-delay-ms") ?? 3.5);
|
||||
const captureArgument = argument("--capture-values") ?? "true";
|
||||
assert.ok(["true", "false"].includes(captureArgument));
|
||||
const captureValues = captureArgument === "true";
|
||||
assert.ok(Number.isInteger(sampleCount) && sampleCount >= 3 && sampleCount <= 10);
|
||||
assert.ok(Number.isFinite(delayMs) && delayMs >= 0 && delayMs <= 20);
|
||||
const previous = { home: process.env.PAPERCLIP_HOME, instance: process.env.PAPERCLIP_INSTANCE_ID,
|
||||
endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT };
|
||||
delete process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
|
||||
const { companySkillService } = await import("../src/services/company-skills.js");
|
||||
const { createAgentLifecycle } = await import("../src/services/agent-lifecycle.js");
|
||||
const { companySkillRoutes } = await import("../src/routes/company-skills.js");
|
||||
const { skillAccessRequestContext, withSkillActor } = await import("../src/services/skill-access.js");
|
||||
const home = await fs.mkdtemp(path.join(os.tmpdir(), "skill-list-benchmark-"));
|
||||
const sourceFiles = [fileURLToPath(import.meta.url), fileURLToPath(new URL("../src/services/company-skills.ts", import.meta.url))];
|
||||
const sourceHash = async () => {
|
||||
const hash = createHash("sha256");
|
||||
for (const file of sourceFiles) hash.update(await fs.readFile(file));
|
||||
return hash.digest("hex");
|
||||
};
|
||||
const sourceRevision = execFileSync("git", ["rev-parse", "HEAD"], { cwd: fileURLToPath(new URL("../../", import.meta.url)), encoding: "utf8" }).trim();
|
||||
const measuredSourceHash = await sourceHash();
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: ReturnType<typeof createDb> | undefined;
|
||||
let proxy: ReturnType<typeof createServer> | undefined;
|
||||
let server: ReturnType<ReturnType<typeof express>["listen"]> | undefined;
|
||||
let baselineFile: string | undefined;
|
||||
let baselineRevision: string | undefined;
|
||||
let baselineSourceHash: string | undefined;
|
||||
let provider: TracerProvider | undefined;
|
||||
let manager: AsyncLocalStorageContextManager | undefined;
|
||||
const sockets = new Set<Socket>();
|
||||
try {
|
||||
process.env.PAPERCLIP_HOME = home;
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "default";
|
||||
const port = argument("--postgres-port");
|
||||
if (port) {
|
||||
assert.match(port, /^\d{1,5}$/);
|
||||
assert.ok(Number(port) > 0 && Number(port) < 65536);
|
||||
const user = encodeURIComponent(argument("--postgres-user") ?? os.userInfo().username);
|
||||
const name = `skill_list_${randomUUID().replaceAll("-", "")}_test`;
|
||||
const adminUrl = `postgres://${user}@127.0.0.1:${port}/postgres`;
|
||||
const connectionString = `postgres://${user}@127.0.0.1:${port}/${name}`;
|
||||
const admin = createDb(adminUrl, { maxConnections: 1 }).$client;
|
||||
try { await admin.unsafe(`CREATE DATABASE "${name}"`); } finally { await admin.end({ timeout: 1 }); }
|
||||
const cleanup = async () => {
|
||||
await closeRegisteredClients(connectionString);
|
||||
const client = createDb(adminUrl, { maxConnections: 1 }).$client;
|
||||
try { await client.unsafe(`DROP DATABASE "${name}"`); } finally { await client.end({ timeout: 1 }); }
|
||||
};
|
||||
database = { connectionString, cleanup };
|
||||
await applyPendingMigrations(connectionString);
|
||||
} else database = await startEmbeddedPostgresTestDatabase("skill-list-benchmark-");
|
||||
|
||||
const target = new URL(database.connectionString);
|
||||
const upstreamPort = Number(target.port);
|
||||
if (delayMs) {
|
||||
proxy = createServer(client => {
|
||||
const upstream = connect({ host: "127.0.0.1", port: upstreamPort });
|
||||
for (const socket of [client, upstream]) {
|
||||
socket.setNoDelay(true); sockets.add(socket);
|
||||
socket.on("error", () => { client.destroy(); upstream.destroy(); });
|
||||
socket.on("close", () => sockets.delete(socket));
|
||||
}
|
||||
const relay = (from: Socket, to: Socket) => {
|
||||
from.on("data", chunk => { from.pause(); setTimeout(() => {
|
||||
if (to.destroyed) return;
|
||||
if (to.write(chunk)) from.resume();
|
||||
else to.once("drain", () => from.resume());
|
||||
}, delayMs); });
|
||||
from.on("end", () => setTimeout(() => to.end(), delayMs));
|
||||
};
|
||||
relay(client, upstream); relay(upstream, client);
|
||||
});
|
||||
await new Promise<void>(resolve => proxy!.listen(0, "127.0.0.1", resolve));
|
||||
const address = proxy.address(); assert.ok(address && typeof address !== "string");
|
||||
target.port = String(address.port);
|
||||
}
|
||||
db = createDb(target.href);
|
||||
const companyId = randomUUID(), userId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Skill list benchmark", issuePrefix: "SLB" });
|
||||
await db.insert(authUsers).values({ id: userId, name: "Fixture owner", email: `${userId}@example.invalid`, createdAt: new Date(), updatedAt: new Date() });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "owner" });
|
||||
const projectRows = await db.insert(projects).values(Array.from({ length: 14 }, (_, i) => ({ companyId, name: `Fixture ${i}` }))).returning();
|
||||
const sources = await db.insert(companySkillSources).values(Array.from({ length: 10 }, (_, i) => ({ companyId,
|
||||
repositoryUrl: `https://example.invalid/fixture-${i}`, fullName: `fixture/${i}`, trackingRef: "main",
|
||||
accessScope: i < 4 ? "projects" as const : "company" as const }))).returning();
|
||||
await db.insert(projectSkillSources).values(sources.slice(0, 8).map((source, i) => ({ companyId, sourceId: source.id,
|
||||
projectId: projectRows[i % 5]!.id, repositoryUrl: source.repositoryUrl, trackingRef: "main", status: "ready" as const })));
|
||||
const skills = await db.insert(companySkills).values(Array.from({ length: 155 }, (_, i) => ({ companyId,
|
||||
key: `fixture/skill-${i}`, slug: `skill-${i}`, name: `Skill ${String(i).padStart(3, "0")}`,
|
||||
markdown: "Synthetic unused markdown. ".repeat(280), sourceType: "github",
|
||||
sourceLocator: "https://example.invalid/immutable-fixture", accessSourceId: sources[i % 10]!.id,
|
||||
metadata: { unused: "metadata ".repeat(1000) }, fileInventory: [{ path: "SKILL.md", kind: "skill" as const }],
|
||||
}))).returning();
|
||||
const lifecycle = createAgentLifecycle(db);
|
||||
for (let i = 0; i < 40; i++) await lifecycle.requestHire(companyId, {
|
||||
name: `Reader ${i}`, adapterType: "process", status: i === 39 ? "terminated" : "paused",
|
||||
adapterConfig: { instructions: "Synthetic agent context. ".repeat(2600),
|
||||
paperclipSkillSync: { desiredSkills: [skills[i % 155]!.key, { key: skills[(i + 1) % 155]!.id, versionId: null }] } },
|
||||
runtimeConfig: { unused: "runtime ".repeat(1000) }, metadata: { unused: "metadata ".repeat(1000) },
|
||||
}, { runnerResolved: true });
|
||||
await companySkillService(db).list(companyId);
|
||||
|
||||
const actor = { type: "board" as const, source: "session" as const, userId, companyIds: [companyId],
|
||||
memberships: [{ companyId, membershipRole: "owner" }] };
|
||||
let baselineService: ReturnType<typeof companySkillService> | undefined;
|
||||
if (baselineRef) {
|
||||
assert.equal(captureValues, false, "Paired acceptance uses --capture-values=false");
|
||||
baselineRevision = execFileSync("git", ["rev-parse", "--verify", "--end-of-options", `${baselineRef}^{commit}`], { encoding: "utf8" }).trim();
|
||||
const source = execFileSync("git", ["show", `${baselineRevision}:server/src/services/company-skills.ts`], { encoding: "utf8", maxBuffer: 2 * 1024 * 1024 });
|
||||
baselineSourceHash = createHash("sha256").update(source).digest("hex");
|
||||
baselineFile = fileURLToPath(new URL(`../src/services/.company-skills-benchmark-${randomUUID()}.ts`, import.meta.url));
|
||||
await fs.writeFile(baselineFile, source, { flag: "wx", mode: 0o600 });
|
||||
const baselineModule = await import(pathToFileURL(baselineFile).href) as { companySkillService: typeof companySkillService };
|
||||
baselineService = baselineModule.companySkillService(db);
|
||||
await withSkillActor(actor, () => baselineService!.list(companyId));
|
||||
}
|
||||
|
||||
const exporter = new InMemorySpanExporter();
|
||||
provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] });
|
||||
manager = new AsyncLocalStorageContextManager().enable();
|
||||
assert.ok(context.setGlobalContextManager(manager)); assert.ok(trace.setGlobalTracerProvider(provider));
|
||||
process.env.OTEL_EXPORTER_OTLP_ENDPOINT = "http://127.0.0.1:4318";
|
||||
const app = express();
|
||||
app.use((req, res, next) => {
|
||||
req.actor = actor;
|
||||
void withRunExecutionTiming(randomUUID(), () => new Promise<void>(resolve => { res.once("finish", resolve); next(); })).catch(next);
|
||||
});
|
||||
const candidateService = companySkillService(db);
|
||||
if (baselineService) app.get("/paired/:variant", skillAccessRequestContext, async (req, res) => {
|
||||
const selected = req.params.variant === "baseline" ? baselineService! : candidateService;
|
||||
res.json(await selected.list(companyId));
|
||||
});
|
||||
app.use("/api", companySkillRoutes(db));
|
||||
app.use((error: Error, _req: express.Request, res: express.Response, _next: express.NextFunction) => res.status(500).json({ error: error.message }));
|
||||
server = app.listen(0, "127.0.0.1");
|
||||
await new Promise<void>(resolve => server!.once("listening", resolve));
|
||||
const address = server.address(); assert.ok(address && typeof address !== "string");
|
||||
const url = `http://127.0.0.1:${address.port}/api/companies/${companyId}/skills`;
|
||||
let queryCount = 0;
|
||||
db.$client.options.debug = () => { queryCount++; };
|
||||
const request = async (etag?: string, requestUrl = url) => {
|
||||
queryCount = 0; exporter.reset();
|
||||
const started = performance.now();
|
||||
const response = await new Promise<{ status: number; body: string; etag: string | undefined }>((resolve, reject) => {
|
||||
const request = httpGet(requestUrl, { headers: etag ? { "if-none-match": etag } : {} }, response => {
|
||||
const chunks: Buffer[] = [];
|
||||
response.on("data", chunk => chunks.push(Buffer.from(chunk)));
|
||||
response.on("error", reject);
|
||||
response.on("end", () => resolve({ status: response.statusCode!, body: Buffer.concat(chunks).toString("utf8"), etag: response.headers.etag }));
|
||||
});
|
||||
request.setTimeout(20_000, () => request.destroy(new Error("Local benchmark request timed out")));
|
||||
request.on("error", reject);
|
||||
});
|
||||
const body = response.body;
|
||||
const elapsedMs = performance.now() - started;
|
||||
assert.equal(response.status, etag ? 304 : 200, "Unexpected local skills-list HTTP status");
|
||||
if (!etag) {
|
||||
const items = JSON.parse(body) as Array<{ key: string; attachedAgentCount: number }>;
|
||||
assert.ok(items.length >= 155);
|
||||
assert.equal(items.find(item => item.key === skills[0]!.key)?.attachedAgentCount, 1);
|
||||
assert.equal(items.find(item => item.key === skills[39]!.key)?.attachedAgentCount, 1);
|
||||
assert.equal(items.find(item => item.key === skills[40]!.key)?.attachedAgentCount, 0);
|
||||
}
|
||||
await provider!.forceFlush();
|
||||
return { elapsedMs, queryCount, status: response.status, etag: response.etag,
|
||||
bodyHash: createHash("sha256").update(body).digest("hex"),
|
||||
phases: exporter.getFinishedSpans().filter(span => span.name.startsWith("skills.")).map(span => ({ name: span.name,
|
||||
durationMs: span.duration[0] * 1000 + span.duration[1] / 1e6 })) };
|
||||
};
|
||||
const first = await request(); assert.ok(first.etag);
|
||||
const events: Array<Record<string, unknown>> = [], now = Date.now();
|
||||
const capture = captureValues ? createDatabaseResponseCapture({ start: now - 1, end: now + 59_000 }, event => events.push(event)) : undefined;
|
||||
if (capture) installDatabaseResponseCapture(db, schema, capture);
|
||||
const samples = [];
|
||||
try {
|
||||
for (let i = 0; i < sampleCount; i++) {
|
||||
const variants = baselineService ? (i % 2 ? ["candidate", "baseline"] : ["baseline", "candidate"]) : ["candidate"];
|
||||
for (const variant of variants) {
|
||||
const requestUrl = baselineService ? `http://127.0.0.1:${address.port}/paired/${variant}` : url;
|
||||
const result = await request(i % 2 ? first.etag! : undefined, requestUrl);
|
||||
if (result.status === 200) assert.equal(result.bodyHash, first.bodyHash, "Baseline and candidate must return the complete same list");
|
||||
samples.push({ variant, pair: i, ...result });
|
||||
console.error(JSON.stringify({ sample: i, variant, elapsedMs: result.elapsedMs, queries: result.queryCount, status: result.status }));
|
||||
}
|
||||
}
|
||||
if (capture) assert.ok(Date.now() < now + 59_000, "Capture expired; repeat with fewer samples or less injected delay");
|
||||
} finally { capture?.flush(); }
|
||||
assert.equal(await sourceHash(), measuredSourceHash, "Source changed during measurement");
|
||||
const timings = samples.map(sample => sample.elapsedMs).sort((a, b) => a - b);
|
||||
const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1");
|
||||
if (capture) assert.ok(summary && summary.inFlightAtEnd === 0 && summary.overflowCalls === 0);
|
||||
const rowCount = (await db.select().from(companySkills).where(eq(companySkills.companyId, companyId))).length;
|
||||
console.log(JSON.stringify({ schema: "paperclip.skill-list-benchmark.v1", sourceRevision, measuredSourceHash,
|
||||
fixture: { remoteSkills: 155, totalSkills: rowCount, agents: 40, nonterminatedAgents: 39, projects: 14, sources: 10 },
|
||||
captureValues, networkDelayMsPerDirection: delayMs, postgresVersion: await db.$client`select version()`,
|
||||
boundary: "production router and real service; synthetic authenticated board actor; postgres-js response values",
|
||||
medianMs: timings[Math.floor(timings.length / 2)], minMs: timings[0], maxMs: timings.at(-1),
|
||||
baselineRevision, baselineSourceHash,
|
||||
baselineDependencyScope: baselineService ? "Only company-skills.ts differs; imported helpers and runtime dependencies use this checkout for both versions" : null,
|
||||
variantResults: ["baseline", "candidate"].flatMap(variant => {
|
||||
const selected = samples.filter(sample => sample.variant === variant);
|
||||
if (!selected.length) return [];
|
||||
const durations = selected.map(sample => sample.elapsedMs).sort((a, b) => a - b);
|
||||
return [{ variant, samples: selected.length, medianMs: durations[Math.floor(durations.length / 2)], minMs: durations[0], maxMs: durations.at(-1) }];
|
||||
}),
|
||||
samples, responseCapture: events }, null, 2));
|
||||
} finally {
|
||||
const failures: unknown[] = [];
|
||||
const clean = async (operation: () => Promise<unknown>) => { try { await operation(); } catch (error) { failures.push(error); } };
|
||||
await clean(async () => { if (baselineFile) await fs.rm(baselineFile, { force: true }); });
|
||||
await clean(async () => { if (server) { server.closeAllConnections(); await new Promise<void>(resolve => server!.close(() => resolve())); } });
|
||||
await clean(async () => db?.$client.end({ timeout: 1 }));
|
||||
for (const socket of sockets) socket.destroy();
|
||||
await clean(async () => { if (proxy) await new Promise<void>(resolve => proxy!.close(() => resolve())); });
|
||||
await clean(async () => database?.cleanup());
|
||||
await clean(async () => provider?.shutdown());
|
||||
trace.disable(); context.disable(); manager?.disable();
|
||||
for (const [name, value] of Object.entries({ PAPERCLIP_HOME: previous.home, PAPERCLIP_INSTANCE_ID: previous.instance, OTEL_EXPORTER_OTLP_ENDPOINT: previous.endpoint })) {
|
||||
if (value === undefined) delete process.env[name]; else process.env[name] = value;
|
||||
}
|
||||
await clean(() => fs.rm(home, { recursive: true, force: true }));
|
||||
if (failures.length) throw new AggregateError(failures, "Local benchmark cleanup failed");
|
||||
}
|
||||
@@ -9,7 +9,7 @@ import express from "express";
|
||||
import request from "supertest";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { activityLog, agents, authUsers, companies, companyMemberships, createDb, closeRegisteredClients, heartbeatRuns, issues, plugins, principalPermissionGrants, toolConnectionInstalls } from "@paperclipai/db";
|
||||
import { activityLog, agents, authUsers, companies, companyMemberships, createDb, closeRegisteredClients, environments, heartbeatRuns, issues, plugins, principalPermissionGrants, toolConnectionInstalls } from "@paperclipai/db";
|
||||
import { type AiConnectionBinding, type AiConnectionPoolMember, type PaperclipPluginManifestV1 } from "@paperclipai/shared";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { agentRoutes } from "../routes/agents.js";
|
||||
@@ -209,6 +209,78 @@ describe("agent-created hires use managed AI connections", () => {
|
||||
expect(saved.runtimeConfig.aiConnection).toEqual(f.binding);
|
||||
});
|
||||
|
||||
it.each(["shared", "delegated", "responsible_user", "router"] as const)("saves unrelated edits without probing an unchanged %s connection read from PostgreSQL", async (mode) => {
|
||||
const f = await fixture("openai");
|
||||
await db.insert(principalPermissionGrants).values({ companyId: f.companyId, principalType: "user", principalId: f.userId, permissionKey: "agents:configure" });
|
||||
const account = mode === "shared"
|
||||
? await aiConnectionService(db).save(f.companyId, f.userId, {
|
||||
provider: "openai", method: "api_key", ownership: "shared", name: "Shared save fixture",
|
||||
apiKey: "fixture", agentIds: [f.agentId], allAgents: false,
|
||||
}, "fixture")
|
||||
: f.account;
|
||||
const binding = mode === "router" ? (await poolFixture(f)).binding
|
||||
: mode === "responsible_user" ? f.binding
|
||||
: { provider: "openai", method: "api_key", mode, ...account };
|
||||
const adapterType = mode === "shared" ? "paperclip_runner" : f.adapterType;
|
||||
await db.update(agents).set({ adapterType,
|
||||
...(adapterType === "paperclip_runner" ? { adapterConfig: { provider: "codex", model: "gpt-5.6-sol" } } : {}),
|
||||
runtimeConfig: { aiConnection: binding },
|
||||
}).where(eq(agents.id, f.agentId));
|
||||
const [before] = await db.select().from(agents).where(eq(agents.id, f.agentId));
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
req.actor = { type: "board", userId: f.userId, companyIds: [f.companyId], source: "session" };
|
||||
next();
|
||||
});
|
||||
app.use("/api", agentRoutes(db));
|
||||
app.use(errorHandler);
|
||||
const original = getServerAdapter(adapterType);
|
||||
const probe = vi.fn(async () => { throw new Error("The runtime is unavailable; unrelated edits must still save"); });
|
||||
registerServerAdapter({ ...original, testEnvironment: probe });
|
||||
try {
|
||||
for (const patch of [
|
||||
{ title: "Updated title" },
|
||||
{ adapterConfig: { reasoningEffort: "high" } },
|
||||
{ defaultEnvironmentId: null },
|
||||
{ runtimeConfig: { ...before.runtimeConfig, heartbeat: { enabled: false } } },
|
||||
]) {
|
||||
const response = await request(app).patch(`/api/agents/${f.agentId}`).send(patch);
|
||||
expect(response.status, JSON.stringify(response.body)).toBe(200);
|
||||
expect(response.body.runtimeConfig.aiConnection).toEqual(binding);
|
||||
}
|
||||
expect(probe).not.toHaveBeenCalled();
|
||||
const [saved] = await db.select().from(agents).where(eq(agents.id, f.agentId));
|
||||
expect(saved).toMatchObject({ title: "Updated title", adapterConfig: { reasoningEffort: "high" },
|
||||
runtimeConfig: { aiConnection: binding, heartbeat: { enabled: false } } });
|
||||
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, f.agentId), eq(activityLog.action, "agent.updated")))).toHaveLength(4);
|
||||
} finally { unregisterServerAdapter(adapterType); }
|
||||
});
|
||||
|
||||
it.each(["model", "engine", "environment", "harness", "connection"] as const)("probes a changed %s selection and preserves the saved configuration on failure", async (selection) => {
|
||||
const f = await fixture("openai");
|
||||
await db.insert(principalPermissionGrants).values({ companyId: f.companyId, principalType: "user", principalId: f.userId, permissionKey: "agents:configure" });
|
||||
const [environment] = await db.select().from(environments).where(eq(environments.driver, "local"));
|
||||
const adapterType = selection === "harness" ? "paperclip_runner" : f.adapterType;
|
||||
const original = getServerAdapter(adapterType);
|
||||
const probe = vi.fn(async () => ({ adapterType, status: "fail" as const, testedAt: new Date().toISOString(),
|
||||
checks: [{ code: "fixture_runtime_unavailable", level: "error" as const, message: "The selected runtime is unavailable" }],
|
||||
}));
|
||||
registerServerAdapter({ ...original, testEnvironment: probe });
|
||||
const patch = selection === "environment" ? { defaultEnvironmentId: environment.id }
|
||||
: selection === "harness" ? { adapterType, adapterConfig: { provider: "codex", model: "gpt-5.6-sol" } }
|
||||
: selection === "connection" ? { runtimeConfig: { aiConnection: { ...f.binding, mode: "delegated", ...f.account } } }
|
||||
: { adapterConfig: { [selection]: selection === "model" ? "changed-model" : "cli" } };
|
||||
try {
|
||||
const response = await request(f.app).patch(`/api/agents/${f.agentId}`).send(patch);
|
||||
expect(response.status, JSON.stringify(response.body)).toBe(422);
|
||||
expect(probe).toHaveBeenCalledOnce();
|
||||
const [saved] = await db.select().from(agents).where(eq(agents.id, f.agentId));
|
||||
expect(saved).toMatchObject({ adapterType: f.adapterType, adapterConfig: {}, defaultEnvironmentId: null,
|
||||
runtimeConfig: { aiConnection: f.binding } });
|
||||
} finally { unregisterServerAdapter(adapterType); }
|
||||
});
|
||||
|
||||
it("does not install a restricted shared connection through pool membership", async () => {
|
||||
const f = await fixture("openai");
|
||||
const otherOwner = `other-${f.companyId}`;
|
||||
|
||||
@@ -25,8 +25,6 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
processFailedGitHubWebhookDeliveries,
|
||||
processPendingDeliveries,
|
||||
flushPublications,
|
||||
reconcileProviderRuntimes: async () => undefined,
|
||||
projectRunMilestones: async () => 0,
|
||||
processPendingSlackFileUploadReceipts: async () => undefined,
|
||||
processPendingSlackSessionSyncs: async () => undefined,
|
||||
onError,
|
||||
@@ -74,6 +72,34 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it("keeps receiver and voice inventory out of periodic chat maintenance", () => {
|
||||
const source = readFileSync(new URL("../app.ts", import.meta.url), "utf8");
|
||||
const maintenance = source.slice(source.indexOf("const chatReconciliation ="), source.indexOf("const unsubscribeChatPublicationSignals"));
|
||||
expect(maintenance).not.toContain("chatChannels.reconcileProviderRuntimes()");
|
||||
expect(maintenance).not.toContain("chatChannels.voice.");
|
||||
expect(maintenance).not.toContain("enqueueSafeNativeChatProgress");
|
||||
expect(source).toContain("nativeChatProgressWork(db).register(deliveryWork,");
|
||||
expect(source).toContain("chatChannels.registerReceiverWork(deliveryWork,");
|
||||
expect(source).toContain("chatChannels.registerTeamsConsentExpiryWork(deliveryWork,");
|
||||
const teams = readFileSync(new URL("../services/chat-channels/microsoft-teams/file-delivery.ts", import.meta.url), "utf8");
|
||||
expect(source).toContain("chatChannels.registerTeamsAttemptRecoveryWork(deliveryWork,");
|
||||
expect(teams).not.toContain("scheduleTeamsFileMaintenance");
|
||||
expect(teams).not.toContain("selectDistinct");
|
||||
const chat = readFileSync(new URL("../services/chat-channels.ts", import.meta.url), "utf8");
|
||||
expect(chat).not.toContain("scheduleTeamsFileMaintenance");
|
||||
});
|
||||
|
||||
it("separates live voice recovery from the remaining periodic maintenance", () => {
|
||||
const source = readFileSync(new URL("../app.ts", import.meta.url), "utf8");
|
||||
expect(source).toContain("chatChannels.voice.registerRecoveryWork(deliveryWork,");
|
||||
expect(source).toContain("chatChannels.voice.registerReplyWork(deliveryWork,");
|
||||
expect(source).toContain("chatChannels.voice.inbound.registerCleanupWork(deliveryWork,");
|
||||
expect(source).toContain("chatChannels.voice.inbound.registerRetentionWork(deliveryWork,");
|
||||
expect(source).toContain("chatChannels.voice.registerReportWork(deliveryWork,");
|
||||
expect(source).not.toContain("reconcileVoiceSessions");
|
||||
expect(source).not.toContain("chatChannels.voice.reconcile()");
|
||||
});
|
||||
|
||||
it("wires GitHub recovery into its independent lane", () => {
|
||||
const source = readFileSync(new URL("../app.ts", import.meta.url), "utf8");
|
||||
expect(source).toContain(
|
||||
@@ -116,119 +142,6 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
expect(actionMaintenance).not.toContain("processPendingGitHubWebhookIngress(");
|
||||
});
|
||||
|
||||
it.each([false, true])(
|
||||
"isolates a blocked milestone projector and joins its completion (rejects: %s)",
|
||||
async (rejects) => {
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const failure = new Error("milestone projection failed");
|
||||
const projectRunMilestones = vi
|
||||
.fn(async () => 0)
|
||||
.mockImplementationOnce(async () => {
|
||||
await held;
|
||||
if (rejects) throw failure;
|
||||
return 1;
|
||||
});
|
||||
const flushPublications = vi.fn(async () => undefined);
|
||||
const onError = vi.fn();
|
||||
const coordinator = createChatReconciliationCoordinator({
|
||||
projectRunMilestones,
|
||||
reconcileProviderRuntimes: async () => undefined,
|
||||
processPendingDeliveries: async () => undefined,
|
||||
flushPublications,
|
||||
processPendingSlackFileUploadReceipts: async () => undefined,
|
||||
processPendingSlackSessionSyncs: async () => undefined,
|
||||
onError,
|
||||
});
|
||||
try {
|
||||
coordinator.reconcile();
|
||||
await vi.waitFor(() => {
|
||||
expect(projectRunMilestones).toHaveBeenCalledTimes(1);
|
||||
expect(flushPublications).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
// A newly committed final/question must dispatch even while a different
|
||||
// issue's projection is blocked. Repeated signals leave one dirty retry.
|
||||
coordinator.notifyPublications();
|
||||
coordinator.notifyPublications();
|
||||
await vi.waitFor(() =>
|
||||
expect(flushPublications).toHaveBeenCalledTimes(2),
|
||||
);
|
||||
expect(projectRunMilestones).toHaveBeenCalledTimes(1);
|
||||
let drained = false;
|
||||
const draining = coordinator.drain().then(() => {
|
||||
drained = true;
|
||||
});
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
expect(drained).toBe(false);
|
||||
release();
|
||||
await draining;
|
||||
expect(projectRunMilestones).toHaveBeenCalledTimes(2);
|
||||
expect(onError).toHaveBeenCalledTimes(rejects ? 1 : 0);
|
||||
if (rejects)
|
||||
expect(onError).toHaveBeenCalledWith("run milestones", failure);
|
||||
const completedCounts = [
|
||||
projectRunMilestones.mock.calls.length,
|
||||
flushPublications.mock.calls.length,
|
||||
];
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
expect([
|
||||
projectRunMilestones.mock.calls.length,
|
||||
flushPublications.mock.calls.length,
|
||||
]).toEqual(completedCounts);
|
||||
} finally {
|
||||
release();
|
||||
coordinator.stop();
|
||||
await coordinator.drain();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("joins a blocked milestone projector at shutdown without starting dirty follow-ups", async () => {
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const projectRunMilestones = vi.fn(async () => {
|
||||
await held;
|
||||
return 1;
|
||||
});
|
||||
const flushPublications = vi.fn(async () => undefined);
|
||||
const coordinator = createChatReconciliationCoordinator({
|
||||
projectRunMilestones,
|
||||
flushPublications,
|
||||
reconcileProviderRuntimes: async () => undefined,
|
||||
processPendingDeliveries: async () => undefined,
|
||||
processPendingSlackFileUploadReceipts: async () => undefined,
|
||||
processPendingSlackSessionSyncs: async () => undefined,
|
||||
onError: vi.fn(),
|
||||
});
|
||||
try {
|
||||
coordinator.reconcile();
|
||||
await vi.waitFor(() => {
|
||||
expect(projectRunMilestones).toHaveBeenCalledOnce();
|
||||
expect(flushPublications).toHaveBeenCalledOnce();
|
||||
});
|
||||
coordinator.notifyPublications();
|
||||
coordinator.stop();
|
||||
let drained = false;
|
||||
const draining = coordinator.drain().then(() => {
|
||||
drained = true;
|
||||
});
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
expect(drained).toBe(false);
|
||||
release();
|
||||
await draining;
|
||||
expect(projectRunMilestones).toHaveBeenCalledOnce();
|
||||
expect(flushPublications).toHaveBeenCalledOnce();
|
||||
} finally {
|
||||
release();
|
||||
coordinator.stop();
|
||||
await coordinator.drain();
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps slow optional recovery from suppressing later publication sweeps", async () => {
|
||||
let releaseDelivery!: () => void;
|
||||
let releaseSlackStatus!: () => void;
|
||||
@@ -238,7 +151,6 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
const slackStatusReleased = new Promise<void>((resolve) => {
|
||||
releaseSlackStatus = resolve;
|
||||
});
|
||||
const reconcileProviderRuntimes = vi.fn(async () => undefined);
|
||||
const processPendingDeliveries = vi.fn(async () => deliveryReleased);
|
||||
const flushPublications = vi.fn(async () => undefined);
|
||||
const processPendingSlackFileUploadReceipts = vi.fn(async () => undefined);
|
||||
@@ -247,8 +159,6 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
);
|
||||
const onError = vi.fn();
|
||||
const coordinator = createChatReconciliationCoordinator({
|
||||
projectRunMilestones: async () => 0,
|
||||
reconcileProviderRuntimes,
|
||||
processPendingDeliveries,
|
||||
flushPublications,
|
||||
processPendingSlackFileUploadReceipts,
|
||||
@@ -258,7 +168,6 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
|
||||
coordinator.reconcile();
|
||||
await vi.waitFor(() => {
|
||||
expect(reconcileProviderRuntimes).toHaveBeenCalledTimes(1);
|
||||
expect(processPendingDeliveries).toHaveBeenCalledTimes(1);
|
||||
expect(flushPublications).toHaveBeenCalledTimes(1);
|
||||
expect(processPendingSlackFileUploadReceipts).toHaveBeenCalledTimes(1);
|
||||
@@ -266,7 +175,6 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
});
|
||||
coordinator.reconcile();
|
||||
await vi.waitFor(() => {
|
||||
expect(reconcileProviderRuntimes).toHaveBeenCalledTimes(2);
|
||||
expect(flushPublications).toHaveBeenCalledTimes(2);
|
||||
expect(processPendingSlackFileUploadReceipts).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
@@ -293,14 +201,11 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
await receiptReleased;
|
||||
if (rejects) throw lookupError;
|
||||
});
|
||||
const reconcileProviderRuntimes = vi.fn(async () => undefined);
|
||||
const processPendingDeliveries = vi.fn(async () => undefined);
|
||||
const flushPublications = vi.fn(async () => undefined);
|
||||
const processPendingSlackSessionSyncs = vi.fn(async () => undefined);
|
||||
const onError = vi.fn();
|
||||
const coordinator = createChatReconciliationCoordinator({
|
||||
projectRunMilestones: async () => 0,
|
||||
reconcileProviderRuntimes,
|
||||
processPendingDeliveries,
|
||||
flushPublications,
|
||||
processPendingSlackFileUploadReceipts,
|
||||
@@ -318,7 +223,6 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
});
|
||||
coordinator.reconcile();
|
||||
await vi.waitFor(() => {
|
||||
expect(reconcileProviderRuntimes).toHaveBeenCalledTimes(2);
|
||||
expect(processPendingDeliveries).toHaveBeenCalledTimes(2);
|
||||
expect(flushPublications).toHaveBeenCalledTimes(2);
|
||||
expect(processPendingSlackSessionSyncs).toHaveBeenCalledTimes(2);
|
||||
|
||||
@@ -10,12 +10,16 @@ import {
|
||||
import { errorHandler } from "../middleware/error-handler.js";
|
||||
import { assertCompanyAccess } from "../routes/authz.js";
|
||||
|
||||
function queryResult<T>(rows: T) {
|
||||
return Object.assign(Promise.resolve(rows), { for: () => Promise.resolve(rows) });
|
||||
}
|
||||
|
||||
function createSelectChain(rows: unknown[]) {
|
||||
return {
|
||||
from() {
|
||||
return {
|
||||
where() {
|
||||
return Promise.resolve(rows);
|
||||
return queryResult(rows);
|
||||
},
|
||||
};
|
||||
},
|
||||
@@ -102,6 +106,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
delete: vi.fn(() => ({ where: () => Promise.resolve(undefined) })),
|
||||
select: vi.fn(() => createSelectChain([])),
|
||||
} as any;
|
||||
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<unknown>) => run(db));
|
||||
const app = express();
|
||||
app.use(
|
||||
actorMiddleware(db, {
|
||||
@@ -176,7 +181,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
select: vi.fn(() => ({
|
||||
from: (table: unknown) => ({
|
||||
where: () =>
|
||||
Promise.resolve(
|
||||
queryResult(
|
||||
table === companyMemberships
|
||||
? [{ companyId: importedCompanyId, membershipRole: "member", status: "active" }]
|
||||
: [],
|
||||
@@ -186,6 +191,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
insert: vi.fn(() => insertChain),
|
||||
delete: vi.fn(() => ({ where: () => Promise.resolve(undefined) })),
|
||||
} as any;
|
||||
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<unknown>) => run(db));
|
||||
const app = express();
|
||||
app.use(
|
||||
actorMiddleware(db, {
|
||||
@@ -252,7 +258,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
select: vi.fn(() => ({
|
||||
from: (table: unknown) => ({
|
||||
where: () =>
|
||||
Promise.resolve(
|
||||
queryResult(
|
||||
table === companies
|
||||
? [{ name: "paperclip-stack-purple-rain" }]
|
||||
: [],
|
||||
@@ -280,7 +286,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
})),
|
||||
delete: vi.fn(() => ({ where: () => Promise.resolve(undefined) })),
|
||||
} as any;
|
||||
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<void>) => run(db));
|
||||
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<unknown>) => run(db));
|
||||
const app = express();
|
||||
app.use(
|
||||
actorMiddleware(db, {
|
||||
@@ -349,7 +355,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
select: vi.fn(() => ({
|
||||
from: (table: unknown) => ({
|
||||
where: () =>
|
||||
Promise.resolve(
|
||||
queryResult(
|
||||
table === instanceUserRoles && state.staleInstanceAdminRow ? [{ id: "stale-role-row" }] : [],
|
||||
),
|
||||
}),
|
||||
@@ -362,6 +368,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
},
|
||||
})),
|
||||
} as any;
|
||||
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<unknown>) => run(db));
|
||||
const app = express();
|
||||
app.use(
|
||||
actorMiddleware(db, {
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
@@ -8,6 +10,7 @@ import {
|
||||
createDb,
|
||||
instanceUserRoles,
|
||||
principalPermissionGrants,
|
||||
subscribeDatabaseWork,
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
claimBoardOwnership,
|
||||
@@ -20,7 +23,8 @@ import {
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_BOARD_CLAIM_TEST_DATABASE_URL");
|
||||
const embeddedPostgresSupport = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
describeEmbeddedPostgres("board claim", () => {
|
||||
@@ -28,8 +32,8 @@ describeEmbeddedPostgres("board claim", () => {
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-board-claim-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-board-claim-");
|
||||
db = createDb(external ?? tempDb!.connectionString);
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
@@ -42,10 +46,11 @@ describeEmbeddedPostgres("board claim", () => {
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
it("lets a signed-in user claim a local-board-only authenticated instance", async () => {
|
||||
it.each(["new", "suspended", "active"] as const)("claims a local-board-only instance with %s membership and wakes only restored authority", async prior => {
|
||||
const now = new Date();
|
||||
const userId = `claim-user-${randomUUID()}`;
|
||||
const company = await db
|
||||
@@ -70,6 +75,10 @@ describeEmbeddedPostgres("board claim", () => {
|
||||
role: "instance_admin",
|
||||
});
|
||||
|
||||
if (prior !== "new") await db.insert(companyMemberships).values({
|
||||
companyId: company.id, principalType: "user", principalId: userId,
|
||||
status: prior, membershipRole: "owner",
|
||||
});
|
||||
await initializeBoardClaimChallenge(db, { deploymentMode: "authenticated" });
|
||||
const warningUrl = getBoardClaimWarningUrl("127.0.0.1", 3197);
|
||||
expect(warningUrl).toBeTruthy();
|
||||
@@ -84,12 +93,19 @@ describeEmbeddedPostgres("board claim", () => {
|
||||
claimedByUserId: null,
|
||||
});
|
||||
|
||||
await expect(
|
||||
claimBoardOwnership(db, { token, code, userId }),
|
||||
).resolves.toEqual({
|
||||
status: "claimed",
|
||||
claimedByUserId: userId,
|
||||
});
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => events.push(event));
|
||||
try {
|
||||
await db.transaction(async tx => {
|
||||
await expect(claimBoardOwnership(tx as unknown as typeof db, { token, code, userId }))
|
||||
.resolves.toEqual({ status: "claimed", claimedByUserId: userId });
|
||||
expect(events).toEqual(prior === "active" ? [] : ["intent"]);
|
||||
// A nested claim must not publish a commit before its caller settles.
|
||||
const outside = await db.select().from(companyMemberships).where(eq(companyMemberships.companyId, company.id));
|
||||
expect(outside.map(row => row.status)).toEqual(prior === "new" ? [] : [prior]);
|
||||
});
|
||||
expect(events).toEqual(prior === "active" ? [] : ["intent", "settled"]);
|
||||
} finally { unsubscribe(); }
|
||||
|
||||
await expect(
|
||||
db
|
||||
|
||||
@@ -47,7 +47,7 @@ describe("chat action scheduling", () => {
|
||||
const work = s.start();
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(s.service.processTelegramSubscriptionDiscovery).toHaveBeenCalledTimes(1);
|
||||
await s.owner.transaction(tx => notifyChatEndpointWork(tx, "connection-fixture"));
|
||||
await s.owner.transaction(tx => notifyChatEndpointWork(tx, "connection-fixture", "company-fixture"));
|
||||
expect(s.service.scheduleTelegramSubscriptionDiscovery).toHaveBeenCalledWith("connection-fixture");
|
||||
s.service.processTelegramSubscriptionDiscovery.mockRejectedValueOnce(new Error("discovery unavailable"));
|
||||
release();
|
||||
@@ -87,7 +87,7 @@ describe("chat action scheduling", () => {
|
||||
for (const run of s.runs) expect(run).toHaveBeenCalledTimes(2);
|
||||
// Queue claims/settlements do not rediscover every configured endpoint.
|
||||
expect(s.service.processTelegramSubscriptionDiscovery).toHaveBeenCalledTimes(1);
|
||||
await s.owner.transaction(tx => notifyChatEndpointWork(tx, "connection-fixture"));
|
||||
await s.owner.transaction(tx => notifyChatEndpointWork(tx, "connection-fixture", "company-fixture"));
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
for (const run of s.runs) expect(run).toHaveBeenCalledTimes(3);
|
||||
expect(s.service.processTelegramSubscriptionDiscovery).toHaveBeenCalledTimes(2);
|
||||
|
||||
@@ -1,3 +1,10 @@
|
||||
import { nativeChatProgressWork } from "../services/chat-native-progress-work.js";
|
||||
import { cloudActorHeaderSourceFromHeaders, resolveCloudTenantActor } from "../middleware/auth.js";
|
||||
import { cloudTenantPrimaryCompanyId } from "../services/cloud-instance.js";
|
||||
import { accessService } from "../services/access.js";
|
||||
import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
import { restoreNativeChatReviewPresentationInTransaction } from "../services/native-runtime/native-chat-review-presentation.js";
|
||||
import { notifyChatEndpointConfiguration, notifyChatCompanyConfiguration } from "../services/chat-channels/configuration-work-notifications.js";
|
||||
import { deleteCompany } from "../services/company-deletion.js";
|
||||
import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js";
|
||||
import { chatRunSourceMessageId } from "../services/chat-channels/run-origin.js";
|
||||
@@ -22,6 +29,7 @@ import { chatSlackManagerGrants, chatSlackRegistrations, toolOauthStates, enviro
|
||||
import type { PaperclipCloudConnector, SealedConnectorCredentials } from "../services/paperclip-cloud-connector.js";
|
||||
import { buildSlackAppManifest, SLACK_MANAGER_SCOPES } from "@paperclipai/shared";
|
||||
import { registerChatActionWork, dueChatAction } from "../services/chat-action-work.js";
|
||||
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { registerChatDeliveryWork } from "../services/chat-delivery-work.js";
|
||||
import { notifyChatPublicationWork, notifyChatActionWork, CHAT_ACTION_QUEUES } from "../services/chat-work-notifications.js";
|
||||
@@ -37,6 +45,7 @@ import { secretService } from "../services/secrets.js";
|
||||
import { githubChatWizardService } from "../services/chat-github-wizard.js";
|
||||
import { dispatchGitHubBotCloudEvent, registerGitHubBotCloudSetupPoller } from "../services/chat-github-cloud-ingress.js";
|
||||
import { githubChatManagementService } from "../services/chat-github-management.js";
|
||||
import { GITHUB_INLINE_RETRY_WINDOW_MS } from "../services/chat-channels/github/retry.js";
|
||||
import { githubChatReviewService } from "../services/chat-github-reviews.js";
|
||||
import { resolveGitHubReviewInstructions } from "../services/chat-channels/github/review-instructions.js";
|
||||
import { githubResponseCommentService } from "../services/chat-github-response-comments.js";
|
||||
@@ -200,6 +209,8 @@ import {
|
||||
} from "../services/chat-interaction-publications.js";
|
||||
import {
|
||||
enqueueChatRunMilestones,
|
||||
enqueueChatRunLifecycleMilestones,
|
||||
enqueueSafeNativeChatProgress,
|
||||
githubRunReplyState,
|
||||
resolveChatRunPresentationAuthorizationReason,
|
||||
CHAT_RUN_PRESENTATION_AUTHORIZATION_REASON,
|
||||
@@ -1142,8 +1153,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
const companyPrefixAttempts = 8;
|
||||
const freshCompanyPrefix = () => `C${randomUUID().replaceAll("-", "").slice(0, 7).toUpperCase()}`;
|
||||
|
||||
async function seedCompany(nextIssuePrefix = freshCompanyPrefix) {
|
||||
const companyId = randomUUID();
|
||||
async function seedCompany(nextIssuePrefix = freshCompanyPrefix, companyId = randomUUID()) {
|
||||
const assignedAgentId = randomUUID();
|
||||
const replacementAgentId = randomUUID();
|
||||
let inserted = false;
|
||||
@@ -1217,6 +1227,81 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
return { companyId, assignedAgentId, replacementAgentId };
|
||||
}
|
||||
|
||||
it("tracks the actual Board Teams claim and retires recovery when its receipt commits", async () => {
|
||||
const context = await teamsFileAuthorityFixture();
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
let release!: () => void, entered!: () => void;
|
||||
const hold = new Promise<void>(resolve => { release = resolve; }), ready = new Promise<void>(resolve => { entered = resolve; });
|
||||
context.controls.beforeConsentReceipt = async () => { entered(); await hold; };
|
||||
let send: ReturnType<typeof context.service.publishBoardMessage> | undefined;
|
||||
try {
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
await context.service.registerTeamsAttemptRecoveryWork(coordinator, () => true).ready;
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
const attachment = await context.createFile();
|
||||
send = context.service.publishBoardMessage(context.endpoint.id, context.conversation.id,
|
||||
"File with an active claim", `teams-attempt-${randomUUID()}`, "owner-user", [attachment.id]);
|
||||
await ready;
|
||||
const [claim] = await db.select().from(chatTeamsFileTransfers).where(eq(chatTeamsFileTransfers.endpointId, context.endpoint.id));
|
||||
expect(claim.phase).toBe("consent_sending"); expect(context.consentCards).toHaveLength(1);
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000); expect(coordinator.nextWakeAt()).toBe(claim.attemptExpiresAt!.getTime());
|
||||
}, { timeout: 5000 });
|
||||
release(); await send;
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000); expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 5000 });
|
||||
const [settled] = await db.select().from(chatTeamsFileTransfers).where(eq(chatTeamsFileTransfers.id, claim.id));
|
||||
expect(settled.phase).toBe("awaiting_consent"); expect(settled.attemptId).toBeNull();
|
||||
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
|
||||
try { await vi.advanceTimersByTimeAsync(120_000); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
|
||||
finally { reads.mockRestore(); writes.mockRestore(); }
|
||||
expect(context.consentCards).toHaveLength(1); expect(context.fileCards).toEqual([]);
|
||||
expect(context.uploadRequest).not.toHaveBeenCalled(); expect(errors).toEqual([]);
|
||||
} finally {
|
||||
release(); await send;
|
||||
await coordinator.stop(); vi.useRealTimers();
|
||||
await retirePublicationFixture(context.service, context.endpoint.id);
|
||||
// This test intentionally leaves unused consent; do not feed the next global expiry fixture.
|
||||
await db.delete(chatTeamsFileTransfers).where(eq(chatTeamsFileTransfers.endpointId, context.endpoint.id));
|
||||
}
|
||||
});
|
||||
|
||||
it("expires Teams consent from the committed Board transfer without periodic maintenance", async () => {
|
||||
const context = await teamsFileAuthorityFixture();
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const settle = () => vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000); expect(idleWorkSnapshot().active).toBe(0);
|
||||
}, { timeout: 5000 });
|
||||
try {
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
// Register before the ordinary Board producer, with no transfer rows yet.
|
||||
await context.service.registerTeamsConsentExpiryWork(coordinator, () => true).ready;
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
const attachment = await context.createFile();
|
||||
const publication = await context.service.publishBoardMessage(context.endpoint.id, context.conversation.id,
|
||||
"A file awaiting consent", `teams-expiry-${randomUUID()}`, "owner-user", [attachment.id]);
|
||||
const [transfer] = await db.select().from(chatTeamsFileTransfers).where(eq(chatTeamsFileTransfers.publicationId, publication.id));
|
||||
expect(transfer.phase).toBe("awaiting_consent"); expect(context.consentCards).toHaveLength(1);
|
||||
await settle(); expect(coordinator.nextWakeAt()).toBe(transfer.expiresAt.getTime());
|
||||
const reads = vi.spyOn(db, "select");
|
||||
try { await vi.advanceTimersByTimeAsync(10_000); expect(reads).not.toHaveBeenCalled(); }
|
||||
finally { reads.mockRestore(); }
|
||||
await vi.advanceTimersByTimeAsync(transfer.expiresAt.getTime() - Date.now() + 1000); await settle();
|
||||
const [expired] = await db.select().from(chatTeamsFileTransfers).where(eq(chatTeamsFileTransfers.id, transfer.id));
|
||||
const [projected] = await db.select().from(chatPublications).where(eq(chatPublications.id, publication.id));
|
||||
expect(expired.phase).toBe("expired"); expect(projected.state).toBe("cancelled");
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
expect(context.consentCards).toHaveLength(1); expect(context.fileCards).toEqual([]);
|
||||
expect(context.uploadRequest).not.toHaveBeenCalled(); expect(errors).toEqual([]);
|
||||
} finally {
|
||||
await coordinator.stop(); vi.useRealTimers();
|
||||
await retirePublicationFixture(context.service, context.endpoint.id);
|
||||
}
|
||||
});
|
||||
|
||||
it("retries a colliding company fixture prefix and creates a distinct complete fixture", async () => {
|
||||
const existing = await seedCompany();
|
||||
const [original] = await db.select().from(companies).where(eq(companies.id, existing.companyId));
|
||||
@@ -3860,7 +3945,11 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
const retry = gateway.executeTool(input);
|
||||
release();
|
||||
expect(await retry).toMatchObject({ result: { data: { actionId: expect.any(String) } } });
|
||||
await expect.poll(async () => (await f.action())?.status).toBe("processed");
|
||||
// A replay may return the durable receipt while the timed-out request
|
||||
// still publishes. Observe the owner's bounded retry window, not the
|
||||
// assertion library's one-second default.
|
||||
await expect.poll(async () => (await f.action())?.status,
|
||||
{ timeout: GITHUB_INLINE_RETRY_WINDOW_MS }).toBe("processed");
|
||||
expect(await gateway.executeTool(input)).toMatchObject({ result: { data: { status: "processed" } } });
|
||||
const actions = await db.select().from(chatActions).where(and(eq(chatActions.endpointId, f.endpoint.id),
|
||||
eq(chatActions.kind, "github_review_publication")));
|
||||
@@ -3868,7 +3957,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
expect(actions[0].status).toBe("processed");
|
||||
expect(f.responseComments.size).toBe(1);
|
||||
expect(f.responseWrites.map(write => write.method)).toEqual(["POST", "PATCH"]);
|
||||
});
|
||||
}, 60_000);
|
||||
it("rejects changed requests and revoked access before repeating an unstaged review", async () => {
|
||||
const f = await publicationRetryFixture();
|
||||
let metadataReads = 0;
|
||||
@@ -5948,6 +6037,68 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
.where(eq(chatEndpointLeases.endpointId, f.endpoint.id)),
|
||||
).toHaveLength(0);
|
||||
});
|
||||
it("wakes GitHub milestone identity restoration only after outer commit and stays quiet on reconfirmation", async () => {
|
||||
const f = await reviewBotFixture();
|
||||
await f.service.revokeLink(f.endpoint.id, f.principal.id);
|
||||
const verified = { githubUserId: "42", login: "octocat" };
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, (event, scopes) => {
|
||||
if ([...scopes ?? []].some(scope => JSON.parse(scope).companyId === f.companyId)) events.push(event);
|
||||
});
|
||||
const link = () => db.select().from(chatIdentityLinks).where(and(
|
||||
eq(chatIdentityLinks.endpointId, f.endpoint.id), eq(chatIdentityLinks.principalId, f.principal.id),
|
||||
)).then(rows => rows[0]);
|
||||
try {
|
||||
await expect(db.transaction(async tx => {
|
||||
await githubChatManagementService(tx as unknown as typeof db, f.providerFetch)
|
||||
.linkObservedIdentity(f.endpoint.id, "owner-user", verified);
|
||||
expect(events).toEqual(["intent"]);
|
||||
expect((await link()).status).toBe("revoked");
|
||||
throw new Error("rollback identity restoration");
|
||||
})).rejects.toThrow("rollback identity restoration");
|
||||
expect((await link()).status).toBe("revoked");
|
||||
events.length = 0;
|
||||
await db.transaction(async tx => {
|
||||
await githubChatManagementService(tx as unknown as typeof db, f.providerFetch)
|
||||
.linkObservedIdentity(f.endpoint.id, "owner-user", verified);
|
||||
expect(events).toEqual(["intent"]);
|
||||
expect((await link()).status).toBe("revoked");
|
||||
});
|
||||
expect(events).toEqual(["intent", "settled"]);
|
||||
expect((await link()).status).toBe("linked");
|
||||
events.length = 0;
|
||||
await f.management.linkObservedIdentity(f.endpoint.id, "owner-user", verified);
|
||||
expect(events).toEqual([]);
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
|
||||
it("wakes milestone authority only when GitHub guest settings change, after outer commit", async () => {
|
||||
const f = await reviewBotFixture();
|
||||
const initial = await f.management.configuration(f.endpoint.id, "owner-user");
|
||||
const configuration = { ...initial.configuration, people: [{
|
||||
kind: "guest" as const, githubUserId: "84", login: "contributor",
|
||||
sponsorUserId: "owner-user", permissionProfile: "restricted" as const, automaticReviews: false,
|
||||
}] };
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, (event, scopes) => {
|
||||
if ([...scopes ?? []].some(scope => JSON.parse(scope).companyId === f.companyId)) events.push(event);
|
||||
});
|
||||
try {
|
||||
await db.transaction(async tx => {
|
||||
await githubChatManagementService(tx as unknown as typeof db, f.providerFetch)
|
||||
.saveConfiguration(f.endpoint.id, { expectedRevision: initial.revision, configuration }, "owner-user");
|
||||
expect(events).toEqual(["intent"]);
|
||||
expect((await f.service.get(f.endpoint.id)).allowUnlinkedPeople).toBe(false);
|
||||
});
|
||||
expect(events).toEqual(["intent", "settled"]);
|
||||
expect((await f.service.get(f.endpoint.id)).allowUnlinkedPeople).toBe(true);
|
||||
events.length = 0;
|
||||
const current = await f.management.configuration(f.endpoint.id, "owner-user");
|
||||
await f.management.saveConfiguration(f.endpoint.id, { expectedRevision: current.revision, configuration }, "owner-user");
|
||||
expect(events).toEqual([]);
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
|
||||
it("scopes guest tasks and rechecks their sponsor before every bot tool", async () => {
|
||||
const f = await reviewBotFixture();
|
||||
const initial = await f.management.configuration(
|
||||
@@ -7133,6 +7284,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
|
||||
async function registrationFixture(
|
||||
mode: "success" | "unavailable" | "unknown" = "success",
|
||||
receiverOptions: Partial<Pick<ChatChannelServiceOptions, "discordGatewayLeaseTtlMs" | "discordGatewayLeaseRenewalIntervalMs">> = {},
|
||||
) {
|
||||
const fixture = await seedCompany();
|
||||
const applicationId = uniqueDiscordApplicationId();
|
||||
@@ -7190,7 +7342,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
return Response.json(command);
|
||||
},
|
||||
) as unknown as typeof globalThis.fetch;
|
||||
const context = createService(new FakeChatSdkRuntime(), providerFetch);
|
||||
const context = createService(new FakeChatSdkRuntime(), providerFetch, receiverOptions);
|
||||
const endpoint = await context.service.create(
|
||||
fixture.companyId,
|
||||
{
|
||||
@@ -7243,6 +7395,125 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
};
|
||||
}
|
||||
|
||||
it("drives receiver discovery through real configure, rotation, pause, and resume producers", async () => {
|
||||
const f = await registrationFixture("success", { discordGatewayLeaseTtlMs: 600_000, discordGatewayLeaseRenewalIntervalMs: 300_000 });
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true });
|
||||
const onError = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
|
||||
const settle = async () => vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 5000 });
|
||||
const quiet = async () => {
|
||||
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(20_000);
|
||||
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
|
||||
} finally { reads.mockRestore(); writes.mockRestore(); }
|
||||
};
|
||||
try {
|
||||
await f.service.registerReceiverWork(coordinator, () => true).ready;
|
||||
await settle(); await quiet();
|
||||
await f.configure(); await settle();
|
||||
const first = f.runtime.get(f.endpoint.id); expect(first).not.toBeNull();
|
||||
await quiet();
|
||||
await f.service.configure(f.endpoint.id, { action: "reconnect", credentials: { ...f.credentials, botToken: "rotated-receiver-token" } }, "owner-user");
|
||||
await settle(); expect(f.runtime.get(f.endpoint.id)).not.toBe(first); expect(first!.shutdown).toHaveBeenCalledOnce();
|
||||
// Pause requires completed onboarding; configure/reconnect intentionally
|
||||
// leave this fixture waiting for its first test conversation.
|
||||
const configured = await f.service.get(f.endpoint.id);
|
||||
await db.update(chatEndpoints).set({ status: "active", setup: { ...configured.setup, step: "complete" } })
|
||||
.where(eq(chatEndpoints.id, f.endpoint.id));
|
||||
await f.service.configure(f.endpoint.id, { action: "pause" }, "owner-user");
|
||||
await settle(); expect(f.runtime.get(f.endpoint.id)).toBeNull(); await quiet();
|
||||
await f.service.configure(f.endpoint.id, { action: "resume" }, "owner-user");
|
||||
await settle(); expect(f.runtime.get(f.endpoint.id)).not.toBeNull(); await quiet();
|
||||
await f.service.configure(f.endpoint.id, { action: "remove" }, "owner-user");
|
||||
await settle(); expect(f.runtime.get(f.endpoint.id)).toBeNull(); await quiet();
|
||||
expect(onError).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
await coordinator.stop(); await f.service.shutdown(); vi.useRealTimers();
|
||||
await retireRegistrationFixture(f.service, f.endpoint.id);
|
||||
}
|
||||
});
|
||||
|
||||
it("does not rediscover a receiver stopped by a company removal refresh", async () => {
|
||||
const f = await registrationFixture("success", { discordGatewayLeaseTtlMs: 600_000, discordGatewayLeaseRenewalIntervalMs: 300_000 });
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true });
|
||||
const onError = vi.fn(), coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
|
||||
let reads: ReturnType<typeof vi.spyOn> | undefined;
|
||||
try {
|
||||
await f.configure(); await f.service.registerReceiverWork(coordinator, () => true).ready;
|
||||
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(1000); expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull(); });
|
||||
const owned = f.runtime.get(f.endpoint.id)!;
|
||||
reads = vi.spyOn(db, "select");
|
||||
await db.transaction(async tx => {
|
||||
await notifyChatCompanyConfiguration(tx, f.fixture.companyId);
|
||||
await tx.delete(chatEndpoints).where(eq(chatEndpoints.id, f.endpoint.id));
|
||||
});
|
||||
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(1000); expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull(); });
|
||||
expect(owned.shutdown).toHaveBeenCalledOnce();
|
||||
expect(reads).toHaveBeenCalledOnce(); expect(onError).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
reads?.mockRestore(); await coordinator.stop(); vi.useRealTimers(); await f.service.shutdown();
|
||||
await db.update(chatEndpoints).set({ status: "archived" }).where(eq(chatEndpoints.id, f.endpoint.id));
|
||||
}
|
||||
});
|
||||
|
||||
it("takes over a persisted foreign receiver lease at its deadline through the worker", async () => {
|
||||
const f = await registrationFixture();
|
||||
await f.configure(); await f.service.shutdown();
|
||||
const token = randomUUID(), expiresAt = new Date(Date.now() + 500);
|
||||
await db.insert(chatEndpointLeases).values({ companyId: f.fixture.companyId, endpointId: f.endpoint.id,
|
||||
leaseKey: "discord_gateway_runtime", token, expiresAt });
|
||||
const standby = createService(new FakeChatSdkRuntime(), f.providerFetch);
|
||||
const onError = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
|
||||
try {
|
||||
await standby.service.registerReceiverWork(coordinator, () => true).ready;
|
||||
expect(standby.runtime.get(f.endpoint.id)).toBeNull();
|
||||
expect(coordinator.nextWakeAt()).toBeGreaterThanOrEqual(expiresAt.getTime());
|
||||
await vi.waitFor(() => expect(standby.runtime.get(f.endpoint.id)).not.toBeNull(), { timeout: 5000 });
|
||||
const [lease] = await db.select().from(chatEndpointLeases).where(eq(chatEndpointLeases.endpointId, f.endpoint.id));
|
||||
expect(lease.token).not.toBe(token);
|
||||
expect(lease.expiresAt.getTime()).toBeGreaterThan(expiresAt.getTime());
|
||||
expect(onError).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
await coordinator.stop(); await retireRegistrationFixture(standby.service, f.endpoint.id);
|
||||
}
|
||||
});
|
||||
|
||||
it("restarts a receiver after token loss only after the old listener has stopped", async () => {
|
||||
const f = await registrationFixture("success", { discordGatewayLeaseTtlMs: 60_000, discordGatewayLeaseRenewalIntervalMs: 50 });
|
||||
const onError = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
|
||||
let release!: () => void;
|
||||
const stopped = new Promise<void>(resolve => { release = resolve; });
|
||||
try {
|
||||
await f.configure();
|
||||
await f.service.registerReceiverWork(coordinator, () => true).ready;
|
||||
const old = f.runtime.get(f.endpoint.id)!;
|
||||
old.shutdown.mockImplementation(async () => { await stopped; });
|
||||
await db.delete(chatEndpointLeases).where(eq(chatEndpointLeases.endpointId, f.endpoint.id));
|
||||
await vi.waitFor(() => expect(old.shutdown).toHaveBeenCalledOnce());
|
||||
await db.transaction(tx => notifyChatEndpointConfiguration(tx, f.endpoint));
|
||||
await new Promise(resolve => setTimeout(resolve, 1500));
|
||||
expect(f.runtime.get(f.endpoint.id)).toBeNull();
|
||||
release();
|
||||
await vi.waitFor(async () => {
|
||||
expect(f.runtime.get(f.endpoint.id)).not.toBeNull();
|
||||
expect(f.runtime.get(f.endpoint.id)).not.toBe(old);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
});
|
||||
expect(onError).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
release(); await coordinator.stop(); vi.useRealTimers();
|
||||
await retireRegistrationFixture(f.service, f.endpoint.id);
|
||||
}
|
||||
});
|
||||
|
||||
it("learns the first configured command and cancels its deadline when the endpoint is removed", async () => {
|
||||
const f = await registrationFixture();
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
@@ -8094,11 +8365,62 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
expect(posts(f)).toHaveLength(1);
|
||||
expect(new URLSearchParams(String(posts(f)[0]![1]!.body)).get("channel")).toBe(f.installerId);
|
||||
});
|
||||
it("links the actor's own manager identity without requiring a connect command", async () => {
|
||||
it("signals milestone eligibility around the real Slack account link and stays quiet on reconfirmation", async () => {
|
||||
const f = await managedFixture(); await f.provision();
|
||||
await db.delete(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id));
|
||||
expect(await f.service.sendSlackTestMessage(f.endpoint.id, actor, { requestId: randomUUID() })).toMatchObject({ status: "sent" });
|
||||
expect(await db.select().from(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id))).toEqual([expect.objectContaining({ paperclipUserId: actor.userId, status: "linked" })]);
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, (event, scopes) => {
|
||||
if ([...scopes ?? []].some(scope => JSON.parse(scope).companyId === f.companyId)) events.push(event);
|
||||
});
|
||||
let release!: () => void, acquired!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const locked = new Promise<void>(resolve => { acquired = resolve; });
|
||||
const lock = db.transaction(async tx => {
|
||||
await tx.execute(sql`lock table chat_identity_links in share mode`);
|
||||
acquired(); await gate;
|
||||
});
|
||||
await locked;
|
||||
const linking = f.service.sendSlackTestMessage(f.endpoint.id, actor, { requestId: randomUUID() });
|
||||
void linking.catch(() => undefined);
|
||||
try {
|
||||
try {
|
||||
await vi.waitFor(async () => {
|
||||
const [waiting] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity
|
||||
where datname = current_database() and state = 'active' and wait_event_type = 'Lock'
|
||||
and query like 'insert into "chat_identity_links"%'`);
|
||||
expect(waiting.count).toBeGreaterThan(0);
|
||||
});
|
||||
expect(events).toEqual(["intent"]);
|
||||
expect(await db.select().from(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id))).toEqual([]);
|
||||
} finally { release(); await lock; await linking; }
|
||||
expect(events).toEqual(["intent", "settled"]);
|
||||
expect(await db.select().from(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id)))
|
||||
.toEqual([expect.objectContaining({ paperclipUserId: actor.userId, status: "linked" })]);
|
||||
events.length = 0;
|
||||
expect(await f.service.sendSlackTestMessage(f.endpoint.id, actor, { requestId: randomUUID() })).toMatchObject({ status: "sent" });
|
||||
expect(events).toEqual([]);
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
it("settles a failed Slack account-link intent without committing identity or sending a test message", async () => {
|
||||
const f = await managedFixture(); await f.provision();
|
||||
await db.delete(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id));
|
||||
f.provider.mockClear();
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, (event, scopes) => {
|
||||
if ([...scopes ?? []].some(scope => JSON.parse(scope).companyId === f.companyId)) events.push(event);
|
||||
});
|
||||
await db.execute(sql`create function sleep_slack_link_failure() returns trigger language plpgsql as $$ begin raise exception 'link write unavailable'; end $$`);
|
||||
await db.execute(sql`create trigger sleep_slack_link_failure before insert on chat_identity_links for each row execute function sleep_slack_link_failure()`);
|
||||
try {
|
||||
await expect(f.service.sendSlackTestMessage(f.endpoint.id, actor, { requestId: randomUUID() })).rejects.toThrow();
|
||||
expect(events).toEqual(["intent", "settled"]);
|
||||
expect(await db.select().from(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id))).toEqual([]);
|
||||
expect(posts(f)).toHaveLength(0);
|
||||
} finally {
|
||||
unsubscribe();
|
||||
await db.execute(sql`drop trigger sleep_slack_link_failure on chat_identity_links`);
|
||||
await db.execute(sql`drop function sleep_slack_link_failure()`);
|
||||
}
|
||||
});
|
||||
it("requires a private Slack challenge for an unknown account before linking and sending the test", async () => {
|
||||
const f = await managedFixture(); await f.provision();
|
||||
@@ -34732,7 +35054,44 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it("drains bounded Slack chat-origin milestones without admitting an internal issue run", async () => {
|
||||
it("wakes an existing chat milestone when ordinary ingress reactivates its conversation", async () => {
|
||||
const fixture = await seedCompany();
|
||||
const { callbacks, endpoint, wakeup } = await configuredSlackEndpoint(fixture);
|
||||
const thread = makeThread({ channelId: "C-REOPEN-MILESTONE", id: "slack:C-REOPEN-MILESTONE:4100.1", name: "reopen-milestone" });
|
||||
const send = (id: string) => deliverMessage({ callbacks, endpointId: endpoint.id, thread: thread.thread,
|
||||
message: makeMessage({ id, text: "@maya continue this conversation", mentioned: true }), trigger: "mention" });
|
||||
await send("4100.1");
|
||||
const [conversation] = await readChatConversationsByEndpointId(endpoint.id);
|
||||
const [link] = await db.select().from(chatMessageLinks).where(and(
|
||||
eq(chatMessageLinks.conversationId, conversation.id), eq(chatMessageLinks.providerMessageId, "4100.1")));
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId: fixture.companyId, agentId: fixture.assignedAgentId,
|
||||
status: "failed", errorCode: "low_trust_isolation_unavailable", updatedAt: new Date(),
|
||||
contextSnapshot: { issueId: conversation.issueId, source: "chat:slack", endpointId: endpoint.id, wakeCommentId: link.commentId },
|
||||
});
|
||||
await db.update(chatConversations).set({ state: "unavailable" }).where(eq(chatConversations.id, conversation.id));
|
||||
const onError = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
|
||||
const notice = () => db.select().from(chatPublications).where(and(eq(chatPublications.issueId, conversation.issueId),
|
||||
eq(chatPublications.idempotencyKey, `run:${runId}:failed:${endpoint.id}`)));
|
||||
try {
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
|
||||
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
|
||||
expect(await notice()).toEqual([]);
|
||||
const wakesBefore = wakeup.mock.calls.length;
|
||||
await send("4100.2");
|
||||
expect(wakeup.mock.calls.length).toBe(wakesBefore + 1);
|
||||
// The harness accepts the new wake without creating a run. Existing
|
||||
// projection debt must wake from ingress, not a later run transition.
|
||||
await expect.poll(notice, { timeout: 5000 }).toEqual([
|
||||
expect.objectContaining({ idempotencyKey: `run:${runId}:failed:${endpoint.id}` }),
|
||||
]);
|
||||
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
|
||||
expect(onError).not.toHaveBeenCalled();
|
||||
} finally { await coordinator.stop(); }
|
||||
});
|
||||
|
||||
it.each(["milliseconds", "microseconds"])("drains bounded Slack chat-origin milestones without admitting an internal issue run (%s)", async (precision) => {
|
||||
const fixture = await seedCompany();
|
||||
const { callbacks, endpoint } = await configuredSlackEndpoint(fixture);
|
||||
const thread = makeThread({
|
||||
@@ -34827,15 +35186,32 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
})),
|
||||
]);
|
||||
|
||||
const inserted: number[] = [];
|
||||
for (let index = 0; index < 4; index += 1) {
|
||||
inserted.push(
|
||||
await enqueueChatRunMilestones(db, {
|
||||
since: new Date("2026-09-05T14:00:00.000Z"),
|
||||
limit: 1,
|
||||
}),
|
||||
);
|
||||
if (precision === "microseconds") {
|
||||
await db.update(heartbeatRuns).set({
|
||||
updatedAt: sql`'2026-09-05T14:59:00.000123Z'::timestamptz`,
|
||||
}).where(inArray(heartbeatRuns.id, resolverRejectedRuns.map(run => run.id)));
|
||||
}
|
||||
// Fail a non-advancing pagination loop explicitly instead of leaving a
|
||||
// background query chain running after the test timeout.
|
||||
const select = db.select.bind(db);
|
||||
let reads = 0;
|
||||
const readGuard = vi.spyOn(db, "select").mockImplementation((...args) => {
|
||||
if (args[0] && "runUpdatedAt" in args[0] && ++reads > 100) {
|
||||
throw new Error("Milestone projection did not advance past rejected candidates");
|
||||
}
|
||||
return select(...args);
|
||||
});
|
||||
const inserted: number[] = [];
|
||||
try {
|
||||
for (let index = 0; index < 4; index += 1) {
|
||||
inserted.push(
|
||||
await enqueueChatRunLifecycleMilestones(db, {
|
||||
since: new Date("2026-09-05T14:00:00.000Z"),
|
||||
limit: 1,
|
||||
}),
|
||||
);
|
||||
}
|
||||
} finally { readGuard.mockRestore(); }
|
||||
expect(inserted).toEqual([1, 1, 1, 0]);
|
||||
const publications = await db
|
||||
.select()
|
||||
@@ -41526,13 +41902,18 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("publishes a closed-choice question, settles its Slack card, and delivers its exact continuation response", async () => {
|
||||
it("publishes a closed-choice question, settles its Slack card, and delivers its exact continuation response", async ({ onTestFinished }) => {
|
||||
let receiptWakeObserved = false;
|
||||
const fixture = await seedCompany();
|
||||
const continuationRunId = randomUUID();
|
||||
const resolveNativeQuestion = vi.fn(async () => "not_native" as const);
|
||||
const { callbacks, endpoint, runtime, service } =
|
||||
await configuredSlackEndpoint(fixture, {
|
||||
resolveNativeQuestion,
|
||||
questionResolutionPersistBarrier: async () => {
|
||||
// Receipt commit must wake projection before the separate answer write.
|
||||
expect(receiptWakeObserved).toBe(true);
|
||||
},
|
||||
wakeup: async (agentId, options) => {
|
||||
if (options.contextSnapshot?.source !== "issue.interaction.respond") {
|
||||
return { accepted: true };
|
||||
@@ -41855,6 +42236,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
.set({ state: "delivery_unknown", providerMessageId: null })
|
||||
.where(eq(chatPublications.id, publication.id));
|
||||
|
||||
const unsubscribeReceipt = subscribeDatabaseWork(db, "chat-run-milestones", (event, hints) => {
|
||||
if (event === "settled" && hints?.has(JSON.stringify({ companyId: fixture.companyId, issueId: conversation.issueId }))) {
|
||||
receiptWakeObserved = true;
|
||||
}
|
||||
});
|
||||
onTestFinished(unsubscribeReceipt);
|
||||
const providerAcknowledgement = callbacks.onAction(actionEvent());
|
||||
await expect(
|
||||
Promise.race([
|
||||
@@ -51601,8 +51988,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
const registeredCommandId = "888888888888888819";
|
||||
let interactionSequence = 0n;
|
||||
|
||||
async function commandFixture() {
|
||||
const fixture = await seedCompany();
|
||||
async function commandFixture(companyId?: string) {
|
||||
const fixture = await seedCompany(undefined, companyId);
|
||||
const applicationId = uniqueDiscordApplicationId();
|
||||
let holdWork = false;
|
||||
const queued: Array<() => void> = [];
|
||||
@@ -52696,8 +53083,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
: lastFreshDiscordMessageId + 1n;
|
||||
return lastFreshDiscordMessageId.toString();
|
||||
};
|
||||
async function reopenedFixture(includeDuringConfirmationSource = false) {
|
||||
const f = await commandFixture();
|
||||
async function reopenedFixture(includeDuringConfirmationSource = false, companyId?: string) {
|
||||
const f = await commandFixture(companyId);
|
||||
const threadId = `discord:${guildId}:${channelId}:${rootMessageId}`;
|
||||
const admitMessage = async (message: ReturnType<typeof makeMessage>) => {
|
||||
await deliverMessage({
|
||||
@@ -52845,6 +53232,193 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
};
|
||||
}
|
||||
|
||||
it.each(["action", "delivery"] as const)("wakes a post-close milestone when interrupted inbound acceptance settles its %s receipt", async receipt => {
|
||||
const f = await reopenedFixture();
|
||||
const onError = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
|
||||
const notice = () => db.select().from(chatPublications).where(and(
|
||||
eq(chatPublications.issueId, f.conversation.issueId),
|
||||
eq(chatPublications.idempotencyKey, `run:${f.fresh.runId}:working:${f.endpoint.id}`),
|
||||
));
|
||||
try {
|
||||
// Crash boundary: the run/wake is durable, but its inbound action has
|
||||
// not yet recorded settlement. Recovery must not dispatch it again.
|
||||
if (receipt === "action") {
|
||||
await db.update(chatActions).set({ status: "issued", result: {}, updatedAt: new Date() })
|
||||
.where(eq(chatActions.id, f.fresh.action.id));
|
||||
} else {
|
||||
// Operator repair of a failed ledger keeps the already-admitted
|
||||
// action/run. Only delivery acceptance needs to commit again.
|
||||
await db.update(chatDeliveries).set({ state: "retry", processedAt: null, nextAttemptAt: null })
|
||||
.where(eq(chatDeliveries.id, f.delivery.id));
|
||||
}
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
|
||||
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
|
||||
expect(await notice()).toEqual([]);
|
||||
const wakesBefore = f.wakeup.mock.calls.length;
|
||||
await expect.poll(async () => {
|
||||
await f.service.processPendingDeliveries(25, f.delivery.id);
|
||||
const [action] = await db.select().from(chatActions).where(eq(chatActions.id, f.fresh.action.id));
|
||||
const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.id, f.delivery.id));
|
||||
return [action.status, delivery.state];
|
||||
}, { timeout: 10_000 }).toEqual(["processed", "processed"]);
|
||||
expect(f.wakeup.mock.calls.length).toBe(wakesBefore);
|
||||
await expect.poll(notice, { timeout: 5000 }).toEqual([
|
||||
expect.objectContaining({ idempotencyKey: `run:${f.fresh.runId}:working:${f.endpoint.id}` }),
|
||||
]);
|
||||
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
|
||||
expect(onError).not.toHaveBeenCalled();
|
||||
} finally { await coordinator.stop(); await f.close(); }
|
||||
});
|
||||
|
||||
it.each(["connection", "resource", "unlinked", "identity", "member", "member_permissions", "ensure_membership", "company_access", "visibility", "cloud_sync"] as const)(
|
||||
"wakes a post-close milestone when %s authority is restored", async owner => {
|
||||
const stackId = randomUUID();
|
||||
const f = await reopenedFixture(false, owner === "cloud_sync" ? cloudTenantPrimaryCompanyId(stackId) : undefined);
|
||||
const previousToken = process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
|
||||
const [previousUser] = owner === "cloud_sync"
|
||||
? await db.select().from(authUsers).where(eq(authUsers.id, "owner-user"))
|
||||
: [];
|
||||
const onError = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
|
||||
const notice = () => db.select().from(chatPublications).where(and(
|
||||
eq(chatPublications.issueId, f.conversation.issueId),
|
||||
eq(chatPublications.idempotencyKey, `run:${f.fresh.runId}:working:${f.endpoint.id}`),
|
||||
));
|
||||
const access = accessService(db);
|
||||
const membership = (await db.select().from(companyMemberships).where(and(
|
||||
eq(companyMemberships.companyId, f.fixture.companyId), eq(companyMemberships.principalId, "owner-user"),
|
||||
)))[0];
|
||||
try {
|
||||
let restore: () => Promise<unknown>;
|
||||
if (owner === "connection") {
|
||||
await toolAccessService(db).updateConnection(f.endpoint.connectionId, { enabled: false });
|
||||
restore = () => toolAccessService(db).updateConnection(f.endpoint.connectionId, { enabled: true });
|
||||
} else if (owner === "resource") {
|
||||
const [conversation] = await db.select().from(chatConversations).where(eq(chatConversations.id, f.conversation.id));
|
||||
expect(conversation.resourceId).not.toBeNull();
|
||||
await f.service.replaceResources(f.endpoint.id, [{ id: conversation.resourceId!, enabled: false }], "owner-user");
|
||||
restore = () => f.service.replaceResources(f.endpoint.id, [{ id: conversation.resourceId!, enabled: true }], "owner-user");
|
||||
} else if (owner === "identity" || owner === "unlinked") {
|
||||
await f.service.update(f.endpoint.id, { allowUnlinkedPeople: false }, "owner-user");
|
||||
await f.service.revokeLink(f.endpoint.id, f.fresh.action.principalId!);
|
||||
if (owner === "unlinked") {
|
||||
restore = () => f.service.update(f.endpoint.id, { allowUnlinkedPeople: true }, "owner-user");
|
||||
} else {
|
||||
const intent = await f.service.createLinkIntent(f.endpoint.id, f.fresh.action.principalId!, 1800);
|
||||
restore = () => f.service.confirmIdentityLink(new URL(intent.confirmationUrl).searchParams.get("token")!, "owner-user");
|
||||
}
|
||||
} else if (owner === "visibility") {
|
||||
await issueService(db).update(f.conversation.issueId, { hiddenAt: new Date() });
|
||||
restore = () => issueService(db).update(f.conversation.issueId, { hiddenAt: null });
|
||||
} else {
|
||||
// Model existing denied authority without exercising revocation's
|
||||
// unrelated credential cleanup; restoration uses its actual owner.
|
||||
await db.update(companyMemberships).set({ status: "suspended" }).where(eq(companyMemberships.id, membership.id));
|
||||
if (owner === "member") restore = () => access.updateMember(f.fixture.companyId, membership.id, { status: "active" });
|
||||
else if (owner === "member_permissions") restore = () => access.updateMemberAndPermissions(f.fixture.companyId, membership.id, { status: "active", grants: [] }, "owner-user");
|
||||
else if (owner === "cloud_sync") {
|
||||
process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = "milestone-cloud-sync-test";
|
||||
let request = 0;
|
||||
restore = () => resolveCloudTenantActor(db, cloudActorHeaderSourceFromHeaders({
|
||||
"x-paperclip-cloud-tenant-token": "milestone-cloud-sync-test",
|
||||
"x-paperclip-cloud-user-id": "owner-user",
|
||||
"x-paperclip-cloud-user-email": "owner@example.test",
|
||||
// Change the profile to force an actual sync on replay. The
|
||||
// debounce must not hide repeated unchanged membership wakes.
|
||||
"x-paperclip-cloud-user-name": `Owner ${request++}`,
|
||||
"x-paperclip-cloud-stack-id": stackId,
|
||||
"x-paperclip-cloud-stack-role": "owner",
|
||||
}));
|
||||
}
|
||||
else if (owner === "ensure_membership") restore = () => access.ensureMembership(f.fixture.companyId, "user", "owner-user", membership.membershipRole, "active");
|
||||
else {
|
||||
const existing = await db.select().from(companyMemberships).where(and(
|
||||
eq(companyMemberships.principalType, "user"), eq(companyMemberships.principalId, "owner-user"),
|
||||
));
|
||||
restore = () => access.setUserCompanyAccess("owner-user", existing.map(row => row.companyId));
|
||||
}
|
||||
}
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
|
||||
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
|
||||
expect(await notice()).toEqual([]);
|
||||
await restore();
|
||||
await expect.poll(notice, { timeout: 5000 }).toEqual([
|
||||
expect.objectContaining({ idempotencyKey: `run:${f.fresh.runId}:working:${f.endpoint.id}` }),
|
||||
]);
|
||||
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
|
||||
if (owner !== "identity" && owner !== "connection") {
|
||||
const notices: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => notices.push(event));
|
||||
try { await restore(); expect(notices).toEqual([]); }
|
||||
finally { unsubscribe(); }
|
||||
expect(await notice()).toHaveLength(1);
|
||||
}
|
||||
expect(onError).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
if (previousToken === undefined) delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
|
||||
else process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = previousToken;
|
||||
await coordinator.stop();
|
||||
if (previousUser) await db.update(authUsers).set({
|
||||
name: previousUser.name, email: previousUser.email,
|
||||
emailVerified: previousUser.emailVerified, updatedAt: previousUser.updatedAt,
|
||||
}).where(eq(authUsers.id, previousUser.id));
|
||||
await f.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["identity", "resource", "visibility"] as const)(
|
||||
"wakes saved post-close native progress when %s authority returns", async owner => {
|
||||
const f = await reopenedFixture();
|
||||
const onError = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
|
||||
const progress = () => db.select().from(chatPublications).where(and(
|
||||
eq(chatPublications.issueId, f.conversation.issueId), like(chatPublications.idempotencyKey, "%:native:%"),
|
||||
));
|
||||
try {
|
||||
await db.insert(chatPublications).values({
|
||||
companyId: f.fixture.companyId, endpointId: f.endpoint.id,
|
||||
conversationId: f.conversation.id, issueId: f.conversation.issueId,
|
||||
idempotencyKey: `run:${f.fresh.runId}:working:${f.endpoint.id}`,
|
||||
payload: { text: "Maya is working…", progressState: "working" }, state: "published",
|
||||
createdAt: new Date(Date.now() - 60_000),
|
||||
});
|
||||
await db.insert(heartbeatRunEvents).values({
|
||||
companyId: f.fixture.companyId, agentId: f.fixture.assignedAgentId,
|
||||
runId: f.fresh.runId, seq: 1, eventType: "item.completed",
|
||||
});
|
||||
let restore: () => Promise<unknown>;
|
||||
if (owner === "identity") {
|
||||
await f.service.update(f.endpoint.id, { allowUnlinkedPeople: false }, "owner-user");
|
||||
await f.service.revokeLink(f.endpoint.id, f.fresh.action.principalId!);
|
||||
const intent = await f.service.createLinkIntent(f.endpoint.id, f.fresh.action.principalId!, 1800);
|
||||
restore = () => f.service.confirmIdentityLink(new URL(intent.confirmationUrl).searchParams.get("token")!, "owner-user");
|
||||
} else if (owner === "resource") {
|
||||
const [conversation] = await db.select().from(chatConversations).where(eq(chatConversations.id, f.conversation.id));
|
||||
expect(conversation.resourceId).not.toBeNull();
|
||||
await f.service.replaceResources(f.endpoint.id, [{ id: conversation.resourceId!, enabled: false }], "owner-user");
|
||||
restore = () => f.service.replaceResources(f.endpoint.id, [{ id: conversation.resourceId!, enabled: true }], "owner-user");
|
||||
} else {
|
||||
await issueService(db).update(f.conversation.issueId, { hiddenAt: new Date() });
|
||||
restore = () => issueService(db).update(f.conversation.issueId, { hiddenAt: null });
|
||||
}
|
||||
await nativeChatProgressWork(db).register(coordinator, () => true).ready;
|
||||
await vi.waitFor(() => { expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0); });
|
||||
expect(await progress()).toEqual([]);
|
||||
await restore();
|
||||
await expect.poll(progress, { timeout: 5000 }).toEqual([
|
||||
expect.objectContaining({ idempotencyKey: `run:${f.fresh.runId}:working:${f.endpoint.id}:native:making_progress:1` }),
|
||||
]);
|
||||
await vi.waitFor(() => { expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0); });
|
||||
const select = vi.spyOn(db, "select");
|
||||
try { await new Promise(resolve => setTimeout(resolve, 1200)); expect(select).not.toHaveBeenCalled(); }
|
||||
finally { select.mockRestore(); }
|
||||
expect(onError).not.toHaveBeenCalled();
|
||||
} finally { await coordinator.stop(); await f.close(); }
|
||||
},
|
||||
);
|
||||
|
||||
it("publishes working and the exact fresh final after pinned close without resurrecting the old source", async () => {
|
||||
const f = await reopenedFixture();
|
||||
try {
|
||||
@@ -65124,6 +65698,121 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
};
|
||||
}
|
||||
|
||||
it.each(["replay", "operator retry"] as const)(
|
||||
"wakes saved native progress when %s restores eligibility", async owner => {
|
||||
const context = await safeNativeProgressFixture("telegram", `12${["replay", "operator retry"].indexOf(owner)}`, "channel", "42", true);
|
||||
const onError = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
|
||||
const progress = () => db.select().from(chatPublications).where(and(
|
||||
eq(chatPublications.issueId, context.conversation.issueId),
|
||||
like(chatPublications.idempotencyKey, "%:native:%"),
|
||||
));
|
||||
const settled = () => vi.waitFor(() => {
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
}, { timeout: 5000 });
|
||||
try {
|
||||
const run = await context.createRun("saved progress awaiting eligibility");
|
||||
await context.addEvent(run);
|
||||
const [base] = await db.update(chatPublications).set({
|
||||
state: owner === "replay" ? "failed" : "delivery_unknown",
|
||||
}).where(eq(chatPublications.idempotencyKey, `run:${run.runId}:working:${context.endpoint.id}`)).returning();
|
||||
const restore = () => owner === "replay"
|
||||
? context.service.replayPublication(context.endpoint.id, base.id)
|
||||
: context.service.resolvePublication(context.endpoint.id, base.id, "retry_anyway", "owner-user");
|
||||
await nativeChatProgressWork(db).register(coordinator, () => true).ready;
|
||||
await settled(); expect(await progress()).toEqual([]);
|
||||
await restore();
|
||||
await expect.poll(progress, { timeout: 5000 }).toEqual([
|
||||
expect.objectContaining({ idempotencyKey: `run:${run.runId}:working:${context.endpoint.id}:native:making_progress:1` }),
|
||||
]);
|
||||
await settled();
|
||||
const select = vi.spyOn(db, "select"), transaction = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await new Promise(resolve => setTimeout(resolve, 1200));
|
||||
expect(select).not.toHaveBeenCalled(); expect(transaction).not.toHaveBeenCalled();
|
||||
} finally { select.mockRestore(); transaction.mockRestore(); }
|
||||
expect(onError).not.toHaveBeenCalled();
|
||||
} finally { await coordinator.stop(); await retirePublicationFixture(context.service, context.endpoint.id); }
|
||||
},
|
||||
);
|
||||
|
||||
it("advances native progress past suppressed candidates with microsecond timestamps", async () => {
|
||||
const context = await safeNativeProgressFixture("telegram", "119");
|
||||
try {
|
||||
const valid = await context.createRun("eligible after rejected page");
|
||||
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, valid.runId));
|
||||
const rejectedIds = Array.from({ length: 30 }, () => randomUUID());
|
||||
await db.insert(heartbeatRuns).values(rejectedIds.map(id => ({
|
||||
id, companyId: source.companyId, agentId: source.agentId,
|
||||
runtimeMode: "native", status: "running",
|
||||
contextSnapshot: { ...source.contextSnapshot, endpointId: randomUUID() },
|
||||
})));
|
||||
await db.insert(chatPublications).values(rejectedIds.map(id => ({
|
||||
companyId: context.fixture.companyId, endpointId: context.endpoint.id,
|
||||
conversationId: context.conversation.id, issueId: context.conversation.issueId,
|
||||
idempotencyKey: `run:${id}:working:${context.endpoint.id}`,
|
||||
payload: { text: "Maya is working…", progressState: "working" },
|
||||
state: "published", createdAt: valid.baseCreatedAt,
|
||||
})));
|
||||
const eventAt = new Date(valid.baseCreatedAt.getTime() + 30_000);
|
||||
await db.insert(heartbeatRunEvents).values(rejectedIds.map(runId => ({
|
||||
companyId: source.companyId, runId, agentId: source.agentId,
|
||||
seq: 1, eventType: "item.completed",
|
||||
createdAt: sql`${eventAt.toISOString()}::timestamptz + interval '0.000123 seconds'`,
|
||||
})));
|
||||
await context.addEvent(valid, "item.completed", 1, new Date(eventAt.getTime() + 1));
|
||||
const select = db.select.bind(db);
|
||||
let pages = 0;
|
||||
const guard = vi.spyOn(db, "select").mockImplementation((...args) => {
|
||||
if (args[0] && "eventCreatedAt" in args[0] && ++pages > 10) {
|
||||
throw new Error("Native progress pagination did not advance past suppressed candidates");
|
||||
}
|
||||
return select(...args);
|
||||
});
|
||||
try {
|
||||
await expect(enqueueSafeNativeChatProgress(db, { since: eventAt, limit: 1 })).resolves.toBe(1);
|
||||
} finally { guard.mockRestore(); }
|
||||
const progress = await db.select().from(chatPublications).where(and(
|
||||
eq(chatPublications.conversationId, context.conversation.id),
|
||||
like(chatPublications.idempotencyKey, "%:native:%"),
|
||||
));
|
||||
expect(progress).toEqual([expect.objectContaining({
|
||||
idempotencyKey: `run:${valid.runId}:working:${context.endpoint.id}:native:making_progress:1`,
|
||||
})]);
|
||||
} finally { await retirePublicationFixture(context.service, context.endpoint.id); }
|
||||
});
|
||||
|
||||
it.each(["separate owners", "combined budget"] as const)(
|
||||
"keeps lifecycle and native progress projection independent with %s",
|
||||
async (mode) => {
|
||||
await enqueueChatRunMilestones(db, { since: new Date(0) });
|
||||
const context = await safeNativeProgressFixture("telegram", mode === "separate owners" ? "111" : "112");
|
||||
try {
|
||||
const progressRun = await context.createRun("ongoing work");
|
||||
await context.addEvent(progressRun);
|
||||
const failedRun = await context.createRun("failed work");
|
||||
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, failedRun.runId));
|
||||
const input = { since: new Date(0), limit: 1 };
|
||||
await expect((mode === "separate owners" ? enqueueChatRunLifecycleMilestones : enqueueChatRunMilestones)(db, input)).resolves.toBe(1);
|
||||
const rows = () => db.select({ key: chatPublications.idempotencyKey }).from(chatPublications)
|
||||
.where(eq(chatPublications.conversationId, context.conversation.id));
|
||||
expect((await rows()).map(row => row.key)).toContain(`run:${failedRun.runId}:failed:${context.endpoint.id}`);
|
||||
expect((await rows()).some(row => row.key.includes(":native:"))).toBe(false);
|
||||
if (mode === "separate owners") {
|
||||
await expect(enqueueChatRunLifecycleMilestones(db, input)).resolves.toBe(0);
|
||||
}
|
||||
await expect((mode === "separate owners" ? enqueueSafeNativeChatProgress : enqueueChatRunMilestones)(db, input)).resolves.toBe(1);
|
||||
expect((await rows()).filter(row => row.key.includes(":native:"))).toEqual([
|
||||
{ key: `run:${progressRun.runId}:working:${context.endpoint.id}:native:making_progress:1` },
|
||||
]);
|
||||
await expect(enqueueChatRunMilestones(db, input)).resolves.toBe(0);
|
||||
} finally {
|
||||
await retirePublicationFixture(context.service, context.endpoint.id);
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["issue", "run"] as const)(
|
||||
"skips a locked native progress %s while unrelated final and question publications deliver",
|
||||
async (lockTarget) => {
|
||||
@@ -72178,6 +72867,48 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
};
|
||||
}
|
||||
|
||||
it("restores changed native review metadata once and signals only its committed change", async () => {
|
||||
const context = await committedNativeReviewPublicationFixture("milestone-repair");
|
||||
// Earlier failure cases can leave intent awaiting XID reconciliation on the
|
||||
// suite's shared owner. Subscribing to that owner immediately replays intent.
|
||||
await expect(db.transaction(async tx => {
|
||||
await signalDatabaseWork(tx, "chat-run-milestones");
|
||||
throw new Error("unrelated rolled-back producer");
|
||||
})).rejects.toThrow("unrelated rolled-back producer");
|
||||
const repairDb = createDb(externalTestDatabaseUrl ?? tempDb!.connectionString);
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(repairDb, "chat-run-milestones", event => events.push(event));
|
||||
try {
|
||||
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, context.runId));
|
||||
const input = { companyId: context.fixture.companyId, issueId: coordinator.issueId, runId: context.runId,
|
||||
decisionId: coordinator.decisionId!, resultId: coordinator.resultId!, assessmentId: coordinator.assessmentId! };
|
||||
const savedAt = new Date(Date.now() - 60_000);
|
||||
await db.update(heartbeatRuns).set({ updatedAt: savedAt }).where(eq(heartbeatRuns.id, context.runId));
|
||||
// Already-restored metadata still authorizes materialization by the caller,
|
||||
// but must neither rewrite the run nor wake projection again.
|
||||
await expect(repairDb.transaction(tx => restoreNativeChatReviewPresentationInTransaction(tx as unknown as typeof db, input))).resolves.toBe(true);
|
||||
expect((await readHeartbeatRunsById(context.runId))[0].updatedAt).toEqual(savedAt);
|
||||
expect(events).toEqual([]);
|
||||
await db.update(heartbeatRuns).set({ resultJson: sql`${heartbeatRuns.resultJson} - 'externalChatReviewPresentation'` })
|
||||
.where(eq(heartbeatRuns.id, context.runId));
|
||||
await repairDb.transaction(async tx => {
|
||||
expect(await restoreNativeChatReviewPresentationInTransaction(tx as unknown as typeof db, input)).toBe(true);
|
||||
expect(events).toEqual(["intent"]);
|
||||
});
|
||||
expect(events).toEqual(["intent", "settled"]);
|
||||
expect((await readHeartbeatRunsById(context.runId))[0].resultJson).toMatchObject({
|
||||
externalChatReviewPresentation: { decisionId: coordinator.decisionId, runId: context.runId },
|
||||
});
|
||||
events.length = 0;
|
||||
await repairDb.transaction(tx => restoreNativeChatReviewPresentationInTransaction(tx as unknown as typeof db, input));
|
||||
expect(events).toEqual([]);
|
||||
} finally {
|
||||
unsubscribe();
|
||||
await repairDb.$client.end({ timeout: 1 });
|
||||
await context.service.shutdown();
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["principal_revoked", "gate_changed"] as const)(
|
||||
"blocks a committed native review response and selected files at transport after %s",
|
||||
async (kind) => {
|
||||
|
||||
@@ -0,0 +1,350 @@
|
||||
import { enqueueChatRunLifecycleMilestones } from "../services/chat-run-publications.js";
|
||||
import { persistReprojectedWorkspaceDiffs, projectCodexWorkspaceDiffsFromTrace } from "../services/provider-trace-workspace-diff-reprojection.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq, like, and } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { chatPublications, createDb, heartbeatRunEvents, heartbeatRuns, issues, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { nativeChatProgressWork } from "../services/chat-native-progress-work.js";
|
||||
import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js";
|
||||
import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js";
|
||||
import { PaperclipControlPlanePort } from "../services/native-runtime/paperclip-control-plane-port.js";
|
||||
import type { PrpEvent } from "../vendor/paperclip-runner/index.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_PROGRESS_TEST_DATABASE_URL");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("native chat progress work", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let fixtures: ReturnType<typeof createChatMilestoneFixtures>;
|
||||
let f: Awaited<ReturnType<typeof fixtures.create>>;
|
||||
let runId: string;
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
let canRun = true;
|
||||
const errors: unknown[] = [];
|
||||
beforeAll(async () => {
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("native-chat-progress-");
|
||||
db = createDb(external ?? temporary!.connectionString, { idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
||||
fixtures = createChatMilestoneFixtures(db);
|
||||
}, 60_000);
|
||||
beforeEach(async () => {
|
||||
errors.length = 0; canRun = true; f = await fixtures.create(); runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
|
||||
nativeIssueId: f.issueId, runtimeMode: "native", status: "running", responsibleUserId: f.userId,
|
||||
contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: f.endpointId, wakeCommentId: f.commentId },
|
||||
});
|
||||
await db.insert(chatPublications).values({ companyId: f.companyId, endpointId: f.endpointId,
|
||||
conversationId: f.conversationId, issueId: f.issueId, idempotencyKey: `run:${runId}:working:${f.endpointId}`,
|
||||
payload: { text: "Maya is working…", progressState: "working" }, state: "published",
|
||||
createdAt: new Date(Date.now() - 60_000),
|
||||
});
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
});
|
||||
afterEach(async () => {
|
||||
await coordinator?.stop(); coordinator = undefined;
|
||||
vi.useRealTimers(); vi.restoreAllMocks(); await fixtures.cleanup();
|
||||
});
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
|
||||
const publications = () => db.select().from(chatPublications).where(and(
|
||||
eq(chatPublications.issueId, f.issueId), like(chatPublications.idempotencyKey, "%:native:%"),
|
||||
));
|
||||
async function start() {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun, canReconcile: () => true,
|
||||
onError: error => errors.push(error) });
|
||||
await nativeChatProgressWork(db).register(coordinator, () => canRun).ready;
|
||||
}
|
||||
async function settle() {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
}
|
||||
async function quiet() {
|
||||
const select = vi.spyOn(db, "select"), transaction = vi.spyOn(db, "transaction");
|
||||
try { await vi.advanceTimersByTimeAsync(120_000); expect(select).not.toHaveBeenCalled(); expect(transaction).not.toHaveBeenCalled(); }
|
||||
finally { select.mockRestore(); transaction.mockRestore(); }
|
||||
}
|
||||
function event(nativeSource = false, eventType = "item.completed") {
|
||||
return { companyId: f.companyId, agentId: f.agentId, runId, eventType,
|
||||
message: "PRIVATE event text", payload: { token: "PRIVATE token" },
|
||||
...(nativeSource ? { nativeSource: { sourceInstanceId: runId, sourceEventId: "event-1", sourceSeq: 1,
|
||||
protocolSchemaVersion: 1, canonicalPayload: { eventType } } } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
it.each([false, true])("wakes from a real committed event after idle (native source: %s)", async nativeSource => {
|
||||
await start(); await settle(); await quiet(); expect(await publications()).toEqual([]);
|
||||
const input = event(nativeSource);
|
||||
const save = (rollback: boolean) => db.transaction(async tx => {
|
||||
await appendHeartbeatRunEvent(tx as unknown as typeof db, input);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await publications()).toEqual([]);
|
||||
if (rollback) throw new Error("outer rollback");
|
||||
});
|
||||
await expect(save(true)).rejects.toThrow("outer rollback");
|
||||
await settle(); expect(await publications()).toEqual([]); await quiet();
|
||||
await save(false);
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await publications()).toEqual([expect.objectContaining({ payload: expect.objectContaining({ text: "Maya is making progress…" }) })]);
|
||||
}, { timeout: 5000 });
|
||||
await settle(); await quiet(); expect(errors).toEqual([]);
|
||||
if (nativeSource) {
|
||||
const signals: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatNativeProgress, kind => signals.push(kind));
|
||||
try { expect((await appendHeartbeatRunEvent(db, input)).disposition).toBe("duplicate"); expect(signals).toEqual([]); }
|
||||
finally { unsubscribe(); }
|
||||
await quiet();
|
||||
}
|
||||
});
|
||||
|
||||
it("wakes a saved event when its first working publication commits after event selection", async () => {
|
||||
await db.delete(chatPublications).where(eq(chatPublications.issueId, f.issueId));
|
||||
await appendHeartbeatRunEvent(db, event(true));
|
||||
await start(); await settle(); expect(await publications()).toEqual([]); await quiet();
|
||||
await db.transaction(async tx => {
|
||||
expect(await enqueueChatRunLifecycleMilestones(tx as unknown as typeof db, {
|
||||
scope: { companyId: f.companyId, issueId: f.issueId },
|
||||
})).toBe(1);
|
||||
// Model a base transaction that began before the event but committed
|
||||
// afterwards, without a wall-clock delay in the suite. PostgreSQL's
|
||||
// default now() is transaction-start time, not commit time.
|
||||
await tx.update(chatPublications).set({ createdAt: new Date(Date.now() - 300_000) })
|
||||
.where(eq(chatPublications.idempotencyKey, `run:${runId}:working:${f.endpointId}`));
|
||||
await vi.advanceTimersByTimeAsync(1000); expect(await publications()).toEqual([]);
|
||||
});
|
||||
await settle(); expect(await publications()).toHaveLength(1); await quiet(); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it("reconstructs saved progress after restart and becomes quiet with published history", async () => {
|
||||
await appendHeartbeatRunEvent(db, event(true));
|
||||
await start(); await settle(); expect(await publications()).toHaveLength(1); await quiet();
|
||||
await coordinator!.stop(); await start(); await settle();
|
||||
expect(await publications()).toHaveLength(1); await quiet(); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it.each(["issue", "run"] as const)("retries a contended %s without another event", async owner => {
|
||||
await appendHeartbeatRunEvent(db, event(true));
|
||||
let release!: () => void, entered!: () => void;
|
||||
const hold = new Promise<void>(resolve => { release = resolve; });
|
||||
const ready = new Promise<void>(resolve => { entered = resolve; });
|
||||
const lock = db.transaction(async tx => {
|
||||
if (owner === "issue") await tx.select().from(issues).where(eq(issues.id, f.issueId)).for("update");
|
||||
else await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
|
||||
entered(); await hold;
|
||||
});
|
||||
try {
|
||||
await ready; await start(); expect(await publications()).toEqual([]);
|
||||
expect(coordinator!.nextWakeAt()).not.toBeNull();
|
||||
release(); await lock;
|
||||
await settle(); expect(await publications()).toHaveLength(1); await quiet(); expect(errors).toEqual([]);
|
||||
} finally { release(); await lock; }
|
||||
});
|
||||
|
||||
it.each(["port", "coordinator"] as const)("wakes from the real %s event owner after outer commit, with quiet replay", async kind => {
|
||||
const sessionId = randomUUID(), runnerId = randomUUID();
|
||||
await db.update(heartbeatRuns).set({ nativeSessionId: sessionId }).where(eq(heartbeatRuns.id, runId));
|
||||
const binding = { companyId: f.companyId, issueId: f.issueId, runId, agentId: f.agentId,
|
||||
sessionId, completionContractId: randomUUID(), completionContractSha256: "fixture",
|
||||
sourceInstanceId: runnerId, controlPlaneSourceInstanceId: "control",
|
||||
normalizedSessionId: sessionId, runnerSourceInstanceId: runnerId,
|
||||
completionContractRevision: "1", completionContractCriterionIds: [],
|
||||
};
|
||||
const source: PrpEvent = { schema: "paperclip.prp.event.v1", schemaVersion: 1,
|
||||
sourceEventId: "event-1", sourceSeq: 1, sourceInstanceId: runnerId, sourceKind: "runner",
|
||||
runId, normalizedSessionId: sessionId, turnId: "turn-1", itemId: "item-1",
|
||||
eventType: "item.completed", priority: 1, emittedAt: new Date().toISOString(), payload: {},
|
||||
};
|
||||
const append = (client: typeof db) => (kind === "port"
|
||||
? new PaperclipControlPlanePort(client, binding)
|
||||
: new NativeRunCoordinatorStore(client, binding)).appendEvent(source);
|
||||
await start(); await settle(); await quiet();
|
||||
await expect(db.transaction(async tx => {
|
||||
await append(tx as unknown as typeof db); throw new Error("rollback owner");
|
||||
})).rejects.toThrow("rollback owner");
|
||||
await settle(); expect(await publications()).toEqual([]); await quiet();
|
||||
await db.transaction(async tx => {
|
||||
await append(tx as unknown as typeof db); await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await publications()).toEqual([]);
|
||||
});
|
||||
await settle(); expect(await publications()).toHaveLength(1); await quiet();
|
||||
const signals: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatNativeProgress, kind => signals.push(kind));
|
||||
try { expect((await append(db)).disposition).toBe("duplicate"); expect(signals).toEqual([]); }
|
||||
finally { unsubscribe(); }
|
||||
await quiet(); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it.each(["issue", "run"] as const)("disarms after terminalization wins during a contended %s", async owner => {
|
||||
await appendHeartbeatRunEvent(db, event(true));
|
||||
let release!: () => void, entered!: () => void;
|
||||
const hold = new Promise<void>(resolve => { release = resolve; });
|
||||
const ready = new Promise<void>(resolve => { entered = resolve; });
|
||||
const lock = db.transaction(async tx => {
|
||||
if (owner === "issue") await tx.select().from(issues).where(eq(issues.id, f.issueId)).for("update");
|
||||
else await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
|
||||
entered(); await hold;
|
||||
await tx.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, runId));
|
||||
});
|
||||
try {
|
||||
await ready; await start(); expect(coordinator!.nextWakeAt()).not.toBeNull();
|
||||
release(); await lock; await settle(); expect(await publications()).toEqual([]); await quiet();
|
||||
expect(errors).toEqual([]);
|
||||
} finally { release(); await lock; }
|
||||
});
|
||||
|
||||
it.each(["first write", "startup"] as const)("retains %s through drain without passive SQL", async kind => {
|
||||
if (kind === "first write") { await start(); await settle(); }
|
||||
canRun = false;
|
||||
await appendHeartbeatRunEvent(db, event(true));
|
||||
if (kind === "startup") await start();
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
|
||||
expect(await publications()).toEqual([]); await quiet();
|
||||
canRun = true; await vi.advanceTimersByTimeAsync(30_000); await settle();
|
||||
expect(await publications()).toHaveLength(1); await quiet(); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it("retains failed startup selection through drain and retries after resume", async () => {
|
||||
await appendHeartbeatRunEvent(db, event(true));
|
||||
const select = vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("temporary read failure"); });
|
||||
try { await start(); } finally { select.mockRestore(); }
|
||||
expect(errors.splice(0)).toHaveLength(1); canRun = false;
|
||||
await vi.advanceTimersByTimeAsync(30_000);
|
||||
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
|
||||
await quiet(); expect(await publications()).toEqual([]);
|
||||
canRun = true; await vi.advanceTimersByTimeAsync(30_000); await settle();
|
||||
expect(await publications()).toHaveLength(1); await quiet(); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it("leaves unrelated direct log events on their atomic statement path without projection intent", async () => {
|
||||
await start(); await settle();
|
||||
const signals: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatNativeProgress, kind => signals.push(kind));
|
||||
const transaction = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await appendHeartbeatRunEvent(db, event(false, "item.delta"));
|
||||
expect(transaction).not.toHaveBeenCalled(); expect(signals).toEqual([]);
|
||||
} finally { transaction.mockRestore(); unsubscribe(); }
|
||||
await quiet(); expect(await publications()).toEqual([]); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it("wakes saved trace reprojection without live runner ingress and leaves duplicate recovery quiet", async () => {
|
||||
await start(); await settle(); await quiet();
|
||||
const projection = projectCodexWorkspaceDiffsFromTrace([{
|
||||
kind: "frame", direction: "provider_to_client", frameId: 1,
|
||||
rawBase64: Buffer.from(JSON.stringify({ method: "turn/diff/updated", params: {
|
||||
turnId: "turn-1", diff: "diff --git a/private.txt b/private.txt\n--- a/private.txt\n+++ b/private.txt\n@@ -1 +1 @@\n-private\n+PRIVATE replacement\n",
|
||||
} })).toString("base64"),
|
||||
}]);
|
||||
expect(projection.turns).toHaveLength(1);
|
||||
const input = { traceId: randomUUID(), companyId: f.companyId, agentId: f.agentId, runId, projection };
|
||||
expect(await persistReprojectedWorkspaceDiffs(db, input)).toMatchObject({ created: 1 });
|
||||
await settle();
|
||||
expect(await publications()).toEqual([expect.objectContaining({ payload: expect.objectContaining({ text: "Maya is working with files…" }) })]);
|
||||
await quiet();
|
||||
const signals: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatNativeProgress, kind => signals.push(kind));
|
||||
try { expect(await persistReprojectedWorkspaceDiffs(db, input)).toMatchObject({ created: 0 }); expect(signals).toEqual([]); }
|
||||
finally { unsubscribe(); }
|
||||
await quiet(); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it("retains a committed event that arrives before the current projection pass returns", async () => {
|
||||
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
const secondRunId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: secondRunId, companyId: f.companyId, agentId: f.agentId, nativeIssueId: f.issueId,
|
||||
runtimeMode: "native", status: "running", responsibleUserId: f.userId, contextSnapshot: source.contextSnapshot,
|
||||
});
|
||||
await db.insert(chatPublications).values({
|
||||
companyId: f.companyId, endpointId: f.endpointId, conversationId: f.conversationId,
|
||||
issueId: f.issueId, idempotencyKey: `run:${secondRunId}:working:${f.endpointId}`,
|
||||
payload: { text: "Maya is working…", progressState: "working" }, state: "published",
|
||||
createdAt: new Date(Date.now() - 60_000),
|
||||
});
|
||||
await start(); await settle();
|
||||
let held = false, release!: () => void;
|
||||
const pending = new Promise<void>(resolve => { release = resolve; });
|
||||
const transaction = db.transaction.bind(db);
|
||||
const barrier = vi.spyOn(db, "transaction").mockImplementation(async (callback, options) => {
|
||||
const result = await transaction(callback, options);
|
||||
// Pause after the first projection commits, before its pass can disarm.
|
||||
// Producer transactions return event records, not an inserted row count.
|
||||
if (result === 1 && !held) { held = true; await pending; }
|
||||
return result;
|
||||
});
|
||||
try {
|
||||
await appendHeartbeatRunEvent(db, event(true));
|
||||
await vi.waitFor(() => expect(held).toBe(true), { timeout: 5000 });
|
||||
await appendHeartbeatRunEvent(db, { ...event(false), runId: secondRunId });
|
||||
release(); await settle();
|
||||
expect(await publications()).toHaveLength(2); barrier.mockRestore();
|
||||
await quiet(); expect(errors).toEqual([]);
|
||||
} finally { release(); barrier.mockRestore(); }
|
||||
});
|
||||
|
||||
it("limits an event wake to its issue and reconstructs other saved scopes on restart", async () => {
|
||||
await start(); await settle();
|
||||
// Seed rows without signals to expose an accidentally global event wake.
|
||||
const others = [await fixtures.create(f.companyId), await fixtures.create()];
|
||||
for (const other of others) {
|
||||
const id = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id, companyId: other.companyId, agentId: other.agentId, nativeIssueId: other.issueId,
|
||||
runtimeMode: "native", status: "running", responsibleUserId: other.userId,
|
||||
contextSnapshot: { issueId: other.issueId, source: "chat:slack", endpointId: other.endpointId, wakeCommentId: other.commentId },
|
||||
});
|
||||
await db.insert(chatPublications).values({
|
||||
companyId: other.companyId, endpointId: other.endpointId, conversationId: other.conversationId,
|
||||
issueId: other.issueId, idempotencyKey: `run:${id}:working:${other.endpointId}`,
|
||||
payload: { text: "Maya is working…", progressState: "working" }, state: "published",
|
||||
createdAt: new Date(Date.now() - 60_000),
|
||||
});
|
||||
await db.insert(heartbeatRunEvents).values({ companyId: other.companyId, agentId: other.agentId,
|
||||
runId: id, seq: 1, eventType: "item.completed" });
|
||||
}
|
||||
const saved = (issueId: string) => db.select().from(chatPublications).where(and(
|
||||
eq(chatPublications.issueId, issueId), like(chatPublications.idempotencyKey, "%:native:%"),
|
||||
));
|
||||
await appendHeartbeatRunEvent(db, event(true)); await settle();
|
||||
expect(await publications()).toHaveLength(1);
|
||||
for (const other of others) expect(await saved(other.issueId)).toEqual([]);
|
||||
await quiet(); await coordinator!.stop(); await start(); await settle();
|
||||
for (const other of others) expect(await saved(other.issueId)).toHaveLength(1);
|
||||
await quiet(); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it("drains more than one issue batch and becomes quiet with all progress retained", async () => {
|
||||
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
const ids = Array.from({ length: 105 }, () => randomUUID());
|
||||
await db.insert(heartbeatRuns).values(ids.map(id => ({
|
||||
id, companyId: f.companyId, agentId: f.agentId, nativeIssueId: f.issueId,
|
||||
runtimeMode: "native", status: "running", responsibleUserId: f.userId, contextSnapshot: source.contextSnapshot,
|
||||
})));
|
||||
await db.insert(chatPublications).values(ids.map(id => ({
|
||||
companyId: f.companyId, endpointId: f.endpointId, conversationId: f.conversationId,
|
||||
issueId: f.issueId, idempotencyKey: `run:${id}:working:${f.endpointId}`,
|
||||
payload: { text: "Maya is working…", progressState: "working" }, state: "published",
|
||||
createdAt: new Date(Date.now() - 60_000),
|
||||
})));
|
||||
await db.insert(heartbeatRunEvents).values(ids.map(id => ({
|
||||
companyId: f.companyId, agentId: f.agentId, runId: id, seq: 1, eventType: "item.completed",
|
||||
})));
|
||||
await start(); await settle();
|
||||
expect(await publications()).toHaveLength(ids.length); await quiet(); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it("unsubscribes at shutdown and reconstructs a later event on restart", async () => {
|
||||
await start(); await settle(); await coordinator!.stop();
|
||||
await appendHeartbeatRunEvent(db, event(true));
|
||||
await quiet(); expect(await publications()).toEqual([]);
|
||||
await start(); await settle(); expect(await publications()).toHaveLength(1); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
});
|
||||
Loaded 100 of 308 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user