mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 12:59:13 +01:00
fix(server): recover captured instruction saves from committed deadlines (#17670)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A stopped run can leave captured instruction edits waiting for a persistent save. > - Existing working-copy records retain the candidate bytes, attempt count, and retry deadline. > - The orphan reaper scans those records on every heartbeat, even when no save is due. > - This PR wakes the existing save owner after committed changes and at saved deadlines. > - The worker preserves permissions, conflicts, and bounded retries, then becomes quiet. ## Linked Issues or Issue Description Refs #17667, the merged preceding PR in this recurring-server-work stack. This PR targets master. Related warm-file ownership work: #14695. Its process-stop and retained-directory guarantees remain with their existing owner. The dedup search found no equivalent captured-save scheduling migration. **What happened?** The commit portion of `recoverCaptured` runs inside the periodic orphan sweep. Prepared copies, completed history, conflicts, and exhausted candidates do not need repeated save discovery. A saved candidate does need recovery after restart or a failed foreground save. **Expected behavior** Committed candidate changes reactivate recovery. The worker honors the saved deadline and existing three-attempt bound. It stops querying after settlement or deliberate conflict/exhaustion. Permission checks and canonical save semantics remain in the revision service. **Steps to reproduce** 1. Start a server with heartbeat scheduling enabled and no captured save pending. 2. Observe repeated pending-commit queries from the orphan reaper. 3. Capture edited instructions and fail their first canonical file replacement. The durable candidate must be saved after its retry deadline without starting a provider. **Paperclip version or commit** Reviewed parent `405e533394b26215e22d0452e9b0207d3f784a79`. Actual parent squash and current master baseline `27270e9bf8b7a294cf03f8b84f4857f1b1ff05c7`. **Deployment mode** Self-hosted server with the existing single-process transaction signal model. Local verification uses PostgreSQL 17 and actual managed instruction files. ## What Changed - Preserve foreground instruction-save ownership and its failure deadline before recovery spends another attempt. - Add a bounded captured-commit worker inside the instruction-copy owner. Existing rows and deadlines supply its state. No new table, migration, or external scheduler is added. - Signal writes entering or leaving pending commit from the existing copy mutation boundary. Only these writes gain a short transaction. File collection, file replacement, and provider operations stay outside that new transaction. - Signal company/agent deletion and worktree execution setting changes. Preserve drain, standby, startup reconstruction, shutdown join, and uncertain-transaction handling through the shared coordinator. - Guard attempt claims with current copy ownership and a still-due deadline. A concurrent writer that postpones the retry wins without consuming an attempt. Manual/startup recovery uses the same guarded owner. - Remove only captured commits from periodic reaping. Stopped collection and the separate post-save directory cleanup sweep remain for later slices. - Add real producer, file, transaction, concurrency, and lifecycle tests. Add guarded external test databases to the two filesystem suites so they can run where embedded PostgreSQL is unavailable. ## Verification Published head: `915ef6f20a68fffe89711cc97a6196521ea9885d`. Source acceptance: `9fb75e508f06b05b740099403ce021a61e7c6603`. The actual #17667 squash ancestry is recorded by a normal merge. Its entire result tree equals the prior accepted descendant tree; the final commit changes the audit only. - All 268 cases in six complete files pass, without skips, in 41.89 seconds on the updated parent. Coverage includes captured saves, directory consumers, startup, instruction-copy lifecycle boundaries, recovery and run lifecycle. - Four unused fresh PostgreSQL 17 databases migrated through parent 0406 isolate the destructive fixtures. Three use supported external URL paths. The embedded-only lifecycle fixture temporarily uses its own external database on this IPC-exhausted host, then is restored byte for byte. Assertions and original timing limits are unchanged. CI must still pass with the original embedded fixture. - Actual database build, server typecheck/build, module boundaries and exact-current migration order pass. Commands select each operation by name. The earlier local no-projects filter error remains recorded and is not treated as a build pass. - The normal parent merge preserves both complete added/removed patches and every unrelated upstream blob. The entire database subtree is parent-identical, with no owned schema or migration changes. New parent worker behavior has current consumer/startup acceptance here and separate owner acceptance in #17667. - Earlier foreground ownership, postponed deadline, attempt cap, drain hold, fixture and skipped-test findings remain in the audit. The prior head's Greptile 5/5, Ponytail 5/5 and green CI are historical after this new publication. - Current-head Greptile and Ponytail both give 5/5. All 58 checks pass. Prior stacked-head scores are historical; the new published SHA requires fresh gates. These checks do not establish deployed host/database suspension or external wake delivery. ## Risks - Signals cover supported in-process writers. Independent SQL writers and other replicas require a separate ingress contract. Startup reconstructs persisted obligations. - Exhausted candidates stay preserved under the existing policy; the worker does not increase retry limits or silently discard saved edits. Database reconstruction failures retain shared backoff. - Inventory pages scalar fields but retains candidate identifiers in memory. The query-plan result covers completed history, not a constant-cost promise for arbitrary exhausted-pending history. - File saves can outlive a stop request once admitted. Shutdown joins them. Exact-byte replay and current authorization remain with the revision owner. - Other instruction recovery duties and other stack blockers remain. This PR does not prove host suspension, database suspension, or external wake delivery. An ordinary protected merge is authorized only after current review and CI gates pass. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and local PostgreSQL/filesystem tests. The runtime does not expose a more precise model identifier or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Review follow-up: remove seven newly added comment lines to satisfy CONTRIBUTING.md, preserving the explanations in the owning audit. Both changed files retain the exact non-comment TypeScript token stream; other source bytes are unchanged. The original cffe server-shard-9 run ended after a runner shutdown signal and cancellation, without a final assertion verdict; that failure is retained and not counted as a pass. No old-head retry is requested. Fresh current-head CI and Greptile are required. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
27270e9bf8
commit
b294de546b
14 files changed
+879
-25
No files matched your search
@@ -2535,3 +2535,37 @@ real worker and owner acceptance passed, but startup wiring coverage was missing
|
||||
from the local selection. The double now exposes the method, and new assertions
|
||||
verify scheduler enablement and periodic-reaper exclusion. All 48 startup tests
|
||||
pass in 5.34 seconds. This is a test-only repair; fresh head CI/review is required.
|
||||
|
||||
Slice 39 is draft [#17667](https://github.com/paperclipai/paperclip/pull/17667),
|
||||
initial head `bd490ee50b303187f0ae6f8fe2aa95477d66a40f`, targeting slice 38.
|
||||
Its initial review was 5/5. The startup repair is now pushed at `ecef7776a4`;
|
||||
fresh CI and review are required. No merge or full-stack acceptance is claimed.
|
||||
|
||||
The instruction-copy audit splits the apparent two reaper calls into three
|
||||
reasons: stopped collection, captured-byte commit, and post-save directory release.
|
||||
Slice 40 will own captured-byte commit only. See
|
||||
[instruction recovery audit](2026-10-25-instruction-recovery-sleep-audit.md).
|
||||
Existing instruction-copy and directory suites pass 20 and 87 tests respectively
|
||||
on separate external PostgreSQL databases before any production behavior changes.
|
||||
|
||||
|
||||
## Slice 39 accepted; slice 40 captured instruction commits
|
||||
|
||||
[#17667](https://github.com/paperclipai/paperclip/pull/17667), head
|
||||
`ecef7776a4b38e0c7e0b86d7009fcd01ed7658e5`, has all 53 checks passing,
|
||||
fresh Greptile 5/5, and no unresolved threads. The startup fixture failure and
|
||||
repair remain recorded. This PR is still draft and unmerged.
|
||||
|
||||
Slice 40 migrates only captured instruction commits. Existing saved bytes,
|
||||
attempt counts, and retry deadlines drive a bounded worker. Local writes signal
|
||||
the existing transaction boundary; the revision service still owns authorization,
|
||||
conflicts, file replacement, and exact-byte replay. A race test exposed a newer
|
||||
retry deadline being ignored; the attempt claim now checks that it is still due.
|
||||
Manual/startup recovery shares the guarded owner. Stopped collection and post-save
|
||||
directory release remain separate duties. No schema migration is needed.
|
||||
|
||||
Final instruction acceptance passes 33 tests, directory/startup passes 137, and
|
||||
startup/instruction-release wiring passes 64 (184 distinct tests across these
|
||||
selections). Workspace types/build pass. Fresh remote gates remain required.
|
||||
See the instruction recovery audit for producer coverage and all failed attempts.
|
||||
Earlier stack CI/review findings and final master reconciliation remain outstanding.
|
||||
@@ -0,0 +1,364 @@
|
||||
# Instruction recovery sleep audit
|
||||
|
||||
Inspected from slice 39 head bd490ee50b303187f0ae6f8fe2aa95477d66a40f.
|
||||
Master inspected 5a28f7acf8. The following is the initial audit; implementation
|
||||
and acceptance evidence for slice 40 are recorded below.
|
||||
|
||||
The apparent two calls in heartbeat/recovery.ts hide at least three reasons:
|
||||
1. recoverStopped: collect/dispose registered instruction copies after terminal run
|
||||
AND independently established local/remote process stop. Never infer stop from
|
||||
terminal status. Temporary staging failure has its own pre-launch guarantee.
|
||||
2. recoverCaptured pending loop: retry saving durable candidate bytes, state
|
||||
pending_commit, attempts <=2, nextAttemptAt null or due. Canonical authorization
|
||||
and base comparison remain in agentInstructionRevisionService.commit.
|
||||
3. recoverCaptured cleanup loop: release temporary agent directories after a
|
||||
terminal save receipt plus processStoppedAt, baseline/cleanupPending marker,
|
||||
and due nextAttemptAt. Preserve remote identity and cleanup authority.
|
||||
|
||||
Keep one reason per PR. Captured commit is the smallest independent next slice;
|
||||
stopped collection and post-save release must remain on the existing reaper until
|
||||
separately migrated. Do not retire recoverCaptured wholesale with only save covered.
|
||||
|
||||
Durability already exists in agent_instruction_working_copies, defined in
|
||||
packages/db/src/schema/agent_instruction_revisions.ts. Existing index
|
||||
agent_instruction_copies_pending_idx(state,next_attempt_at) may suffice; inspect
|
||||
EXPLAIN before adding an index. No new table needed. Attempts and nextAttemptAt
|
||||
are business recovery state; do not reset the three-attempt bound.
|
||||
|
||||
Producer map:
|
||||
- agent-instruction-working-copies.ts::patch owns CAS writes with state, attempts,
|
||||
baseHash, retainedByRunId predicates. Add before-write signal in short existing/
|
||||
new DB transaction, without enclosing file/network work.
|
||||
- collectStopped captures bytes via patch(pending_commit) before calling commit.
|
||||
commitCandidate updates saved/conflict/retry state via patch; resolve also patch.
|
||||
- prepare inserts preparing rows; no captured work until patch. acknowledgeExplicitSave
|
||||
only advances prepared baseline, not pending work.
|
||||
- agent-directory-working-copies.ts receives patch for ordinary mutations. Its
|
||||
direct inserts/adoption/deletes need explicit audit but currently create prepared
|
||||
or staging rows, not pending_commit. Do not assume legacy candidate payloads
|
||||
and agent-directory receipts have the same semantics.
|
||||
- Run/company/agent deletion cascades can remove obligations: signal exact scope
|
||||
before supported deletion to disarm stale entries.
|
||||
|
||||
Concurrency risk to test before relying on events:
|
||||
A pending_commit write can wake recovery while foreground commit is still active.
|
||||
Revision commit serializes through its owner and explicitly deduplicates exact
|
||||
candidate bytes before testing baseline equality. Copy CAS can lose to a competing
|
||||
attempt counter update; a later worker must recover the receipt without falsely
|
||||
reporting conflict or saving twice. Test actual producer plus held foreground
|
||||
commit and verify canonical bytes, one durable activity receipt, and final state.
|
||||
No transaction should contain process/file collection just to emit a signal.
|
||||
|
||||
Stopped-copy producer audit is wider:
|
||||
- terminal run transitions (including direct recovery writers)
|
||||
- copies.processStoppedAt and retention/handoff state
|
||||
- native.local_process_stopped via recordNativeLocalProcessStop in heartbeat,
|
||||
native restart recovery, and native-local-process-stop itself
|
||||
- later process-start/identity events invalidate local stop receipts
|
||||
- environment lease remoteExecutionTermination receipts and removal/reacquisition
|
||||
via environment-runtime, environments, task runtime, workspace recovery/export
|
||||
- stalled unconfirmed owners are real pending obligations; preserve safe backoff
|
||||
or a complete notification replacement. Never claim quiet by dropping them.
|
||||
|
||||
Testing planned:
|
||||
Real managed instruction file and authorization, actual collectStopped producer;
|
||||
no-config/prepared/history/exhausted/conflict quiet; exact saved deadline; transient
|
||||
save error; rollback/outer commit; foreground race; restart; drain/standby; shutdown;
|
||||
more than one page and company fairness. Assert file bytes and save receipts, not
|
||||
only callback invocation. Existing test harnesses currently require embedded DB;
|
||||
use guarded external disposable URLs here as needed, maintaining embedded default.
|
||||
|
||||
Remaining acceptance of 39 and older stack gates stays outstanding. No merge
|
||||
is authorized. No claim about host suspension or expanded Cloud alarm coverage.
|
||||
|
||||
Baseline acceptance before changing production behavior:
|
||||
- 20 instruction working-copy tests passed in 6.83 seconds.
|
||||
- 87 persistent agent-directory tests passed in 18.27 seconds.
|
||||
- Suites used separate dedicated PostgreSQL databases through the guarded external
|
||||
test URL helper. Their embedded default and temporary filesystem isolation remain.
|
||||
- This only establishes a baseline. No periodic instruction duty is migrated yet.
|
||||
|
||||
## Slice 40 implementation: captured commit only
|
||||
|
||||
Parent: slice 39 at `ecef7776a4b38e0c7e0b86d7009fcd01ed7658e5`.
|
||||
Remote master remains `5a28f7acf8a8c86536c4f5c5369bfeb49a46c41a`.
|
||||
|
||||
The instruction service owns eligibility, saved retry times, and save attempts.
|
||||
The existing scoped delivery worker owns coalescing, company-fair batches of 20,
|
||||
reconstruction, pause/resume, failure backoff, and shutdown. Inventory uses scalar
|
||||
fields and keyset pages of 100. Candidate bytes are read only for an actual attempt.
|
||||
No table, migration, provider-specific scheduler, or Cloud alarm contract is added.
|
||||
|
||||
The pending predicate remains `state = pending_commit` and `attempts < 3`.
|
||||
The actual `nextAttemptAt` is used; null means due now. Completed copies,
|
||||
prepared/live copies, permission conflicts, and exhausted candidates are quiet.
|
||||
The three-attempt bound and preserved-candidate policy are unchanged. A bounded
|
||||
business retry is distinct from a database/inventory failure: the latter retains
|
||||
shared recovery backoff rather than discarding an unconfirmed obligation.
|
||||
|
||||
The existing copy mutation boundary emits intent before writes that enter or
|
||||
leave pending commit. Other copy mutations keep their existing query boundary.
|
||||
The notification and update share a short transaction, including a caller-owned
|
||||
outer transaction. Filesystem reads, file replacement, directory merge locks, and
|
||||
provider transport operations are not enclosed in that new transaction. Company
|
||||
and agent deletion signal company scope before cascading run deletion. Worktree
|
||||
execution settings signal reconstruction; the heartbeat facade reads that policy
|
||||
fresh and gates drain/standby. Local process signals do not cover independent SQL
|
||||
writers or another replica; this remains the existing supported writer model.
|
||||
|
||||
A worker rereads the candidate and atomically claims an attempt using the existing
|
||||
state, attempt, baseline hash, and retained-owner guards. It also requires the
|
||||
saved deadline still to be due at claim time. A newer future deadline wins without
|
||||
consuming an attempt. Manual/startup recovery delegates to this same owner. The
|
||||
existing revision service retains permission checks, canonical-file locking,
|
||||
exact-byte replay receipts, and baseline conflicts. Shutdown joins a save already
|
||||
admitted; it does not discard the persisted candidate to become quiet.
|
||||
|
||||
Only the commit portion of `recoverCaptured` is excluded from periodic reaping.
|
||||
Stopped-copy collection and post-save directory release keep their prior recovery
|
||||
path for separate slices. The existing directory release after a successful save
|
||||
remains part of that foreground settlement; its independent crash-recovery sweep
|
||||
is not claimed migrated.
|
||||
|
||||
## Evidence and failure record
|
||||
|
||||
- Final instruction suite: 33 passed, including 13 worker cases, in 10.23 seconds.
|
||||
Covers quiet prepared/completed/conflict/exhausted history, real capture after a
|
||||
filesystem failure, actual saved deadline, restart, drain, standby, outer commit
|
||||
and rollback, existing attempt cap, permission conflict, foreground save race,
|
||||
shutdown join, newer-deadline race, worktree setting reactivation, and 103 saved
|
||||
candidates across multiple inventory pages. Tests verify canonical bytes and
|
||||
the save receipt, including one activity receipt after concurrent exact saves.
|
||||
- Existing directory and startup acceptance: 137 passed on fresh fixtures.
|
||||
- Final startup and heartbeat instruction-release wiring: 64 passed. Startup is
|
||||
counted in both selections; the three final suite groups cover 184 distinct tests.
|
||||
- Workspace typecheck/build passed. Final server typecheck passed again
|
||||
after the manual recovery path adopted the common owner.
|
||||
- Existing pending-copy index plan: with 10,000 synthetic completed rows and one
|
||||
eligible candidate, PostgreSQL used `agent_instruction_copies_pending_idx`, read
|
||||
one candidate and filtered two exhausted rows, with five shared buffer hits and
|
||||
0.011 ms execution. The fixture transaction rolled back. This only demonstrates
|
||||
that completed history need not be scanned; it is not a fleet benchmark or a
|
||||
constant-cost claim for arbitrary exhausted-pending history.
|
||||
- Initial worker run: 26 passed, one drain fixture failed. The fixture disabled
|
||||
transaction reconciliation as well as work admission, so the intent hold correctly
|
||||
remained active. Drain now permits reconciliation; a separate standby case keeps
|
||||
it disabled and verifies no passive SQL while retaining the hold.
|
||||
- First repeated directory run: 106 passed, 29 failed because its reused external
|
||||
database retained rows naming the prior run's removed temporary directories.
|
||||
Fresh database plus fresh filesystem fixtures passed all 137 tests. The production
|
||||
source was not changed to hide those failures. These filesystem suites require
|
||||
a fresh external database for each full run, as embedded mode already provides.
|
||||
- A new race regression failed with the candidate incorrectly saved before a
|
||||
newly postponed deadline. Adding the due-time predicate to the attempt claim
|
||||
made the focused regression pass, followed by all 33 instruction cases.
|
||||
|
||||
## Boundaries and remaining acceptance
|
||||
|
||||
Related warm-directory ownership work: #14695. Its stop/retention guarantees stay
|
||||
with the directory owner. No terminal status is promoted into process-stop proof.
|
||||
No provider is launched to recover a captured legacy instruction candidate.
|
||||
|
||||
Current PR CI and fresh review are still required. Host suspension, database
|
||||
suspension, external wake delivery, the two remaining instruction duties, the
|
||||
other run-reaper duties, and the complete original stack inventory remain open.
|
||||
The later stack campaign authorizes an ordinary protected merge only after exact-current-head Greptile 5/5, clean CI, and required review findings are resolved.
|
||||
|
||||
## Foreground ownership correction (2026-10-27)
|
||||
|
||||
Review of PR #17670 found that the original foreground race assertion accepted
|
||||
an extra recovery attempt while the first save still held the canonical-file
|
||||
operation. Exact-byte deduplication prevented a duplicate write but did not protect
|
||||
the copy receipt: an incremented attempt invalidated the foreground failure CAS
|
||||
and could lose its 30-second retry deadline.
|
||||
|
||||
The instruction-copy service now serializes collection and captured recovery per
|
||||
company/run using the existing database work scope. Separate service instances and
|
||||
dedicated connections belonging to that scope share the same in-flight owner.
|
||||
Recovery joins the live operation, then rereads durable state and its deadline
|
||||
before claiming an attempt. Rejected operations release ownership; no elapsed-time
|
||||
lease, new table, polling loop, or transaction around filesystem work is added.
|
||||
This is process-local ownership within the existing supported writer model, not a
|
||||
cross-replica lock. Persisted candidates still reconstruct after process restart.
|
||||
|
||||
Regression evidence against owner head d1a7c64c8b03ada0741c7c5ebb8ff9cb5190479b:
|
||||
- The six-case real-save barrier regression failed in five cases before the fix:
|
||||
attempts advanced from one to two while foreground file replacement was held.
|
||||
The shared-service success case passed; the result is not described as six failures.
|
||||
- With the fix, all 38 instruction-copy tests and 14 heartbeat helper tests pass.
|
||||
Tests cover successful and failed foreground saves across shared, separate, and
|
||||
dedicated-connection services, a two-minute held operation, the preserved retry
|
||||
deadline, eventual canonical bytes, and exactly one file-update activity receipt.
|
||||
- All 87 persistent agent-directory tests pass on a separate fresh PostgreSQL
|
||||
database, covering the existing directory consumer of the collection boundary.
|
||||
- Server typechecking passes with the expanded regression tests.
|
||||
- Four embedded-only heartbeat integration cases were skipped by their existing
|
||||
support gate; they are not included in the 139 passing cases above. This is not
|
||||
evidence of full repository CI or deployed host/database suspension.
|
||||
|
||||
## Retry fixture timing correction (2026-10-27)
|
||||
|
||||
A later CI run exposed a test race, not a different production retry policy.
|
||||
The attempt counter is written before filesystem replacement completes. The old
|
||||
attempt-cap test waited for that counter, advanced a fixed 31 seconds, and could
|
||||
move the clock before the pending save recorded its next retry deadline. It then
|
||||
expected attempt three before that newly recorded deadline was due.
|
||||
|
||||
Holding the second filesystem replacement across that clock advance reproduced
|
||||
the exact CI assertion: expected three attempts, received two. The test now waits
|
||||
for the completed attempt's persisted deadline and advances to that deadline.
|
||||
It retains the deliberate slow-save barrier, the three-attempt cap, the original
|
||||
canonical bytes, the captured candidate, and the absence of later retries. The
|
||||
captured-commit group also retires its own company fixture during teardown, so a
|
||||
failed test cannot leave retry debt in the next test's global reconstruction.
|
||||
|
||||
All 38 instruction-copy cases pass on a fresh PostgreSQL 17 database with the
|
||||
owner branch's 396 migrations. The controlled reproduction fails without the
|
||||
fixture correction. No production retry code, timeout, or assertion is weakened.
|
||||
|
||||
|
||||
## Current parent composition and acceptance (2026-10-28)
|
||||
|
||||
The existing slice was integrated with the updated cancellation parent
|
||||
`d9da519b32bd7e5ecaf2156c3de217fc47187f50` by a normal merge, yielding
|
||||
`77cd93fa08310ecdc834964e9a4326f794c2be87`. All substantive owned edits remain
|
||||
identical after excluding unchanged context and hunk coordinates. The literal
|
||||
patch comparison initially rejected the updated context; zero-context comparison
|
||||
and inspection establish that the edits themselves are unchanged. Every unrelated
|
||||
upstream blob is preserved. The entire database source tree is identical to the
|
||||
parent: this slice has no migration or table.
|
||||
|
||||
All six complete files passed, **268 cases with no skips**, in 46.26 seconds:
|
||||
|
||||
| Complete file | Passing cases |
|
||||
| --- | ---: |
|
||||
| Instruction working copies and captured commits | 38 |
|
||||
| Persistent agent directory working copies | 87 |
|
||||
| Server startup and feedback export | 53 |
|
||||
| Heartbeat instruction-copy boundary | 14 |
|
||||
| Heartbeat recovery | 45 |
|
||||
| Heartbeat run lifecycle | 31 |
|
||||
|
||||
Instruction copies, directory copies, and recovery each used a separate newly
|
||||
created PostgreSQL database migrated through the current 0404 parent history
|
||||
before fixture setup. The lifecycle suite retained its real embedded database.
|
||||
No reused filesystem database, skipped suite, increased timeout, or narrowed
|
||||
case-name selection is counted as acceptance. The earlier failures above remain
|
||||
part of the record.
|
||||
|
||||
Database build, production server typecheck and build, module boundaries, target
|
||||
migration-order validation, and all three fresh migrations passed. File collection
|
||||
and replacement stay outside the short pending-state transaction. Foreground
|
||||
ownership, the persisted next deadline, attempt cap, authorization, exact-byte
|
||||
receipt and drain/shutdown contracts retain their owners. No provider turn is
|
||||
started to replay captured bytes.
|
||||
|
||||
This is local worker and integration evidence. Final publication must carry the
|
||||
actual landed parent ancestry, followed by fresh current-head CI and review.
|
||||
It does not establish host or database suspension or a durable Cloud alarm.
|
||||
|
||||
|
||||
## Review style correction and interrupted CI
|
||||
|
||||
Greptile reviewed `cffe70b42b7b41f0b69c548dc47b8f1d5b753de5` at 5/5 but left
|
||||
an actionable P2: CONTRIBUTING.md prohibits new code comments. Seven new
|
||||
comment lines in the instruction-copy owner and tests are removed. The
|
||||
foreground ownership, attempt-cap barrier and fixture isolation explanations
|
||||
remain in this audit. TypeScript parsing verifies that both affected files have
|
||||
exactly the same non-comment token stream before and after removal. All other
|
||||
source bytes remain unchanged; this does not reinterpret the 268-case source
|
||||
acceptance as a new whole-suite run. Fresh head CI and Greptile are required.
|
||||
|
||||
The first cffe CI server shard 9 stopped after receiving a runner shutdown
|
||||
signal at 23:50:45Z, then reported cancellation at 23:50:57Z. It has no final
|
||||
assertion verdict and is not counted as passing. The log is retained; no old-head
|
||||
retry is requested because the required style correction creates a new head.
|
||||
The full new-head CI must run independently.
|
||||
|
||||
## Current master and append-only parent adoption
|
||||
|
||||
The parent retry-cancellation slice now carries actual #17666 squash
|
||||
`7afee8f0622e273d10506b0febfb68f143d168b1`, the new Coordinator and Runner
|
||||
changes, and the regenerated append-only migration 0406. A clean normal merge
|
||||
preserves both complete added/removed patches and every unrelated upstream blob.
|
||||
The complete database subtree equals parent
|
||||
`48b3fcb83f305616df02a4337a42ec0867bb4b80`; this slice adds no migration.
|
||||
The parent audit follow-up changes documentation only.
|
||||
|
||||
Source `654c4694d5fad8ebfcc1219bc4a76ec94f4516b8` passes all **268 cases in six
|
||||
complete files**, with no skipped cases, in 45.38 seconds. Four separate fresh
|
||||
PostgreSQL 17 databases migrated through 0406 prevent fixture interference.
|
||||
Instruction, directory, and recovery suites use their supported external paths.
|
||||
The embedded-only lifecycle suite temporarily used its own external fixture
|
||||
because this host cannot start more embedded clusters. Its fixture import was
|
||||
restored byte for byte afterward. All assertions and original timing limits were
|
||||
unchanged. This is qualified external-database acceptance; fresh CI still must
|
||||
pass with the original embedded fixture. Database build, server typecheck/build,
|
||||
module boundaries, target migration order, and all four fresh migrations pass.
|
||||
No new code comments remain relative to this parent.
|
||||
|
||||
Prior head `a1a0ae5b02e8a650408bf4c55ae99071c3bff59c` had passing CI, Greptile
|
||||
5/5 and Ponytail 5/5. Those results become historical when the new head is
|
||||
published. Keep the old runner interruption and all earlier failures in the
|
||||
ledger. The current parent must merge first, and the new published head needs
|
||||
fresh exact-head reviews and CI. These results do not establish deployed sleep.
|
||||
|
||||
### Local validation command correction before publication
|
||||
|
||||
A cloned local runner selected its first command by position, but this slice's
|
||||
first command was the database build rather than migration order. It replaced
|
||||
the build with a pnpm filter that matched no projects and exited zero. That was
|
||||
not evidence of a database build. The migration-order command also retained old
|
||||
revision arguments. The next unpublished slice exposed the error when its first
|
||||
fresh migration could not find compiled migration assets. Publication stopped.
|
||||
The original command receipts and logs remain; no product pass is inferred from
|
||||
the no-match command or the failed migration.
|
||||
|
||||
The runner now selects commands by their names. It executed the actual database
|
||||
build and migration check against the exact current parent and head. The full
|
||||
six-file acceptance then ran again at audit-only descendant
|
||||
`b25636e6261f98f1e53ff28a1f259ea03b9c41a8`: **268 cases pass without skips in
|
||||
40.97 seconds**, using four new disposable databases migrated through 0406.
|
||||
Types, server build, boundaries, and the actual current migration-order check
|
||||
also pass. The lifecycle fixture qualification above still applies; the file was
|
||||
restored exactly. The already-published parent used a different, correct command
|
||||
layout and was not affected by this runner error.
|
||||
|
||||
## Bounded retry-cancellation parent acceptance
|
||||
|
||||
A clean normal merge adopts published parent
|
||||
`405e533394b26215e22d0452e9b0207d3f784a79`, including its bounded reconstruction
|
||||
and durable browser reply check. Both complete added/removed patches and every
|
||||
unrelated upstream blob are preserved. The entire database subtree still equals
|
||||
that parent: this slice has no schema or migration change.
|
||||
|
||||
Source `9fb75e508f06b05b740099403ce021a61e7c6603` passes all **268 cases in six
|
||||
complete files**, without skips, in 41.89 seconds. Four unused fresh PostgreSQL 17
|
||||
databases migrated through 0406 isolate the fixtures. The same qualified temporary
|
||||
external lifecycle fixture is restored byte for byte. Actual database build,
|
||||
server types/build, module boundaries and exact-current migration order pass.
|
||||
The earlier local runner failure and corrected evidence remain in the preceding
|
||||
section; this run uses commands selected by name.
|
||||
|
||||
Stacked publication may run this non-migration slice's independent CI while the
|
||||
parent finishes. It does not allow merging into an unmerged parent. At merge,
|
||||
the exact reviewed parent must have landed, the target must be master, and the
|
||||
actual combined tree must preserve the complete owned patch and upstream changes.
|
||||
The subsequent index-bearing child still needs the actual parent migration squash
|
||||
ancestry before publication. Old reviews on this slice's prior head do not count
|
||||
for the new published head.
|
||||
|
||||
## Landed retry-cancellation squash ancestry
|
||||
|
||||
Parent #17667 merged as `27270e9bf8b7a294cf03f8b84f4857f1b1ff05c7` after
|
||||
all 58 checks and both current-head reviews passed. Its entire tree equals the
|
||||
already-adopted reviewed parent `405e533394b26215e22d0452e9b0207d3f784a79`.
|
||||
The ordinary merge probe still found nine overlapping text conflicts from the
|
||||
squashed ancestry. A normal merge records that actual ancestry. The exact parent
|
||||
was already an ancestor of the accepted descendant; restoring the accepted
|
||||
versions of the inspected conflicts leaves the **entire result tree identical**
|
||||
to the previously published `fc5cee9fa00923a2d6fba27386ad7a6ab0ef101e`.
|
||||
This proves all automatic and manual resolutions preserve tested code and parent
|
||||
changes. The follow-up here changes documentation only. The 268-case source
|
||||
acceptance and database proofs remain applicable, but the new SHA needs fresh
|
||||
CI and exact-head reviews. Prior stacked-head results are not reused as passes.
|
||||
@@ -1,3 +1,4 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import fs from "node:fs/promises";
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
@@ -22,6 +23,8 @@ import { AgentDirectoryReuseInvalidatedError, agentDirectoryWorkingCopyService }
|
||||
import { withDirectoryMergeLock } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
import { heartbeatRunEvents } from "@paperclipai/db";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_AGENT_DIRECTORY_TEST_DATABASE_URL");
|
||||
|
||||
describe("persistent agent directories", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
@@ -42,13 +45,14 @@ describe("persistent agent directories", () => {
|
||||
beforeAll(async () => {
|
||||
home = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "instruction-working-copies-")));
|
||||
process.env.PAPERCLIP_HOME = home;
|
||||
database = await startEmbeddedPostgresTestDatabase("instruction-copies-db-");
|
||||
db = createDb(database.connectionString);
|
||||
if (!external) database = await startEmbeddedPostgresTestDatabase("instruction-copies-db-");
|
||||
db = createDb(external ?? database.connectionString);
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
revisions = agentInstructionRevisionService(db);
|
||||
}, 90_000);
|
||||
afterAll(async () => {
|
||||
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await database?.cleanup();
|
||||
if (home) {
|
||||
const writable = async (dir: string) => {
|
||||
|
||||
@@ -1,3 +1,9 @@
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { instanceSettingsService } from "../services/instance-settings.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { notifyInstructionCommit } from "../services/instruction-commit-notifications.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import fs from "node:fs/promises";
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
@@ -8,9 +14,9 @@ const execFile = promisify(execFileCallback);
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { agents, companies, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, agentInstructionWorkingCopies, createDb } from "@paperclipai/db";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { activityLog, agents, companies, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, agentInstructionWorkingCopies, createDb, withDedicatedDbConnection } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js";
|
||||
import { agentInstructionWorkingCopyService } from "../services/agent-instruction-working-copies.js";
|
||||
@@ -18,6 +24,8 @@ import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js";
|
||||
import { resolveManagedInstructionsRoot } from "../services/agent-instructions.js";
|
||||
import { buildNativeRuntimeContext } from "../services/native-runtime/runtime-context.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_INSTRUCTION_COPY_TEST_DATABASE_URL");
|
||||
|
||||
describe("registered run instruction copies", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
@@ -38,13 +46,14 @@ describe("registered run instruction copies", () => {
|
||||
beforeAll(async () => {
|
||||
home = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "instruction-working-copies-")));
|
||||
process.env.PAPERCLIP_HOME = home;
|
||||
database = await startEmbeddedPostgresTestDatabase("instruction-copies-db-");
|
||||
db = createDb(database.connectionString);
|
||||
if (!external) database = await startEmbeddedPostgresTestDatabase("instruction-copies-db-");
|
||||
db = createDb(external ?? database.connectionString);
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
revisions = agentInstructionRevisionService(db);
|
||||
}, 90_000);
|
||||
afterAll(async () => {
|
||||
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await database?.cleanup();
|
||||
if (home) {
|
||||
const writable = async (dir: string) => {
|
||||
@@ -70,6 +79,334 @@ describe("registered run instruction copies", () => {
|
||||
await fs.writeFile(path.join(root, entryFile), initial);
|
||||
});
|
||||
|
||||
describe("captured instruction commit worker", () => {
|
||||
afterEach(async () => {
|
||||
vi.useRealTimers(); vi.restoreAllMocks();
|
||||
|
||||
await db.update(agentInstructionWorkingCopies).set({ state: "saved", nextAttemptAt: null })
|
||||
.where(eq(agentInstructionWorkingCopies.companyId, companyId));
|
||||
});
|
||||
const clock = () => vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
async function captured(copy: NonNullable<Awaited<ReturnType<typeof copies.get>>>, dueAt = new Date(), owner = db) {
|
||||
await owner.transaction(async tx => {
|
||||
await notifyInstructionCommit(tx, { companyId: copy.companyId, runId: copy.runId });
|
||||
await tx.update(agentInstructionWorkingCopies).set({ state: "pending_commit", attempts: 1,
|
||||
candidateBase64: Buffer.from("recovered bytes").toString("base64"), nextAttemptAt: dueAt,
|
||||
processStoppedAt: new Date(),
|
||||
}).where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
});
|
||||
}
|
||||
async function saved(runId: string) {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
expect((await copies.get(companyId, runId))?.state).toBe("saved");
|
||||
}, { timeout: 5000 });
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe("recovered bytes");
|
||||
}
|
||||
function coordinator(canRun = () => true, canReconcile = () => true) {
|
||||
return createDeliveryWorkCoordinator({ owner: db, canRun, canReconcile, onError: error => { throw error; } });
|
||||
}
|
||||
it("stays quiet for prepared copies, completed history, conflicts, and exhausted candidates", async () => {
|
||||
for (const state of ["prepared", "saved", "conflict", "pending_commit"]) {
|
||||
const copy = await run();
|
||||
await db.update(agentInstructionWorkingCopies).set({ state, attempts: 3 }).where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
}
|
||||
clock(); const worker = coordinator();
|
||||
try {
|
||||
await copies.registerCapturedCommitWork(worker, () => true, async () => true).ready;
|
||||
await expect.poll(() => idleWorkSnapshot().active).toBe(0);
|
||||
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "update");
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
|
||||
expect(worker.nextWakeAt()).toBeNull();
|
||||
} finally { await worker.stop(); }
|
||||
});
|
||||
it("recovers a real capture at its saved deadline after a transient save failure", async () => {
|
||||
const copy = await run();
|
||||
await fs.writeFile(path.join(copy.localRoot, entryFile), "recovered bytes");
|
||||
clock(); const worker = coordinator();
|
||||
try {
|
||||
await copies.registerCapturedCommitWork(worker, () => true, async () => true).ready;
|
||||
const rename = fs.rename.bind(fs);
|
||||
let fail = true;
|
||||
const renameSpy = vi.spyOn(fs, "rename").mockImplementation(async (from, to) => {
|
||||
if (fail && to === path.join(root, entryFile)) { fail = false; throw new Error("temporary file replacement failure"); }
|
||||
return rename(from, to);
|
||||
});
|
||||
const pending = await copies.collectStopped({ companyId, runId: copy.runId });
|
||||
expect(pending).toMatchObject({ state: "pending_commit", attempts: 1, nextAttemptAt: expect.any(Date) });
|
||||
renameSpy.mockRestore();
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await expect.poll(() => idleWorkSnapshot().active).toBe(0);
|
||||
expect(worker.nextWakeAt()).toBe(pending!.nextAttemptAt!.getTime());
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
await vi.advanceTimersByTimeAsync(30_000);
|
||||
await saved(copy.runId);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await expect.poll(() => idleWorkSnapshot().active).toBe(0);
|
||||
expect(worker.nextWakeAt()).toBeNull();
|
||||
} finally { await worker.stop(); }
|
||||
});
|
||||
it.each(["restart", "drain", "standby"] as const)("reconstructs a saved candidate across %s", async mode => {
|
||||
const copy = await run();
|
||||
if (mode === "restart") await captured(copy);
|
||||
let admitted = mode === "restart";
|
||||
clock(); const worker = coordinator(() => admitted, () => mode !== "standby");
|
||||
try {
|
||||
await agentInstructionWorkingCopyService(db).registerCapturedCommitWork(worker, () => admitted, async () => true).ready;
|
||||
if (mode !== "restart") {
|
||||
await captured(copy);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
if (mode !== "standby") await expect.poll(() => idleWorkSnapshot().active).toBe(0);
|
||||
const reads = vi.spyOn(db, "select");
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect(reads).not.toHaveBeenCalled(); reads.mockRestore();
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
admitted = true; await vi.advanceTimersByTimeAsync(30_000);
|
||||
}
|
||||
await saved(copy.runId);
|
||||
} finally { await worker.stop(); }
|
||||
});
|
||||
it("preserves the existing attempt cap and becomes quiet with the candidate retained", async () => {
|
||||
const copy = await run(); await fs.writeFile(path.join(copy.localRoot, entryFile), "recovered bytes");
|
||||
clock(); const worker = coordinator();
|
||||
const rename = fs.rename.bind(fs);
|
||||
let release!: () => void, entered!: () => void;
|
||||
const held = new Promise<void>(resolve => { release = resolve; });
|
||||
const saving = new Promise<void>(resolve => { entered = resolve; });
|
||||
let destinationWrites = 0;
|
||||
const writes = vi.spyOn(fs, "rename").mockImplementation(async (from, to) => {
|
||||
if (to === path.join(root, entryFile)) {
|
||||
if (++destinationWrites === 2) { entered(); await held; }
|
||||
throw new Error("file replacement still unavailable");
|
||||
}
|
||||
return rename(from, to);
|
||||
});
|
||||
try {
|
||||
await copies.registerCapturedCommitWork(worker, () => true, async () => true).ready;
|
||||
expect((await copies.collectStopped({ companyId, runId: copy.runId }))?.attempts).toBe(1);
|
||||
for (const attempts of [2, 3]) {
|
||||
const previous = await copies.get(companyId, copy.runId);
|
||||
expect(previous?.nextAttemptAt).toBeInstanceOf(Date);
|
||||
await vi.advanceTimersByTimeAsync(Math.max(0, previous!.nextAttemptAt!.getTime() - Date.now()) + 1);
|
||||
if (attempts === 2) {
|
||||
await saving;
|
||||
|
||||
|
||||
expect((await copies.get(companyId, copy.runId))?.attempts).toBe(2);
|
||||
await vi.advanceTimersByTimeAsync(31_000);
|
||||
release();
|
||||
}
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
const row = await copies.get(companyId, copy.runId);
|
||||
expect(row?.attempts).toBe(attempts);
|
||||
if (attempts < 3) expect(row!.nextAttemptAt!.getTime()).toBeGreaterThan(Date.now());
|
||||
else expect(row?.nextAttemptAt).toBeNull();
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
});
|
||||
}
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await expect.poll(() => idleWorkSnapshot().active).toBe(0);
|
||||
const count = writes.mock.calls.length;
|
||||
await vi.advanceTimersByTimeAsync(300_000);
|
||||
expect(writes.mock.calls.length).toBe(count);
|
||||
expect(worker.nextWakeAt()).toBeNull();
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
expect(await copies.get(companyId, copy.runId)).toMatchObject({ state: "pending_commit", attempts: 3,
|
||||
candidateBase64: Buffer.from("recovered bytes").toString("base64"), nextAttemptAt: null });
|
||||
} finally { release(); writes.mockRestore(); await worker.stop(); }
|
||||
});
|
||||
it("joins an active save during shutdown and reconstructs its remaining receipt", async () => {
|
||||
const copy = await run(); await captured(copy);
|
||||
clock(); const worker = coordinator();
|
||||
const rename = fs.rename.bind(fs);
|
||||
let release!: () => void, entered!: () => void;
|
||||
const held = new Promise<void>(resolve => { release = resolve; });
|
||||
const reached = new Promise<void>(resolve => { entered = resolve; });
|
||||
const renameSpy = vi.spyOn(fs, "rename").mockImplementation(async (from, to) => {
|
||||
if (to === path.join(root, entryFile)) { entered(); await held; }
|
||||
return rename(from, to);
|
||||
});
|
||||
let stopped: Promise<void> | undefined;
|
||||
try {
|
||||
const registration = copies.registerCapturedCommitWork(worker, () => true, async () => true);
|
||||
await reached;
|
||||
let complete = false;
|
||||
stopped = worker.stop().then(() => { complete = true; });
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(complete).toBe(false);
|
||||
release(); await stopped; await registration.ready;
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe("recovered bytes");
|
||||
expect((await copies.get(companyId, copy.runId))?.state).toBe("saved");
|
||||
const restarted = coordinator();
|
||||
try {
|
||||
await copies.registerCapturedCommitWork(restarted, () => true, async () => true).ready;
|
||||
expect(restarted.nextWakeAt()).toBeNull();
|
||||
} finally { await restarted.stop(); }
|
||||
} finally { release(); renameSpy.mockRestore(); await stopped; await worker.stop(); }
|
||||
});
|
||||
it("honors a newer retry deadline written before its attempt claim", async () => {
|
||||
const copy = await run(); await captured(copy);
|
||||
clock(); const worker = coordinator();
|
||||
const later = new Date(Date.now() + 60_000);
|
||||
const transaction = db.transaction.bind(db);
|
||||
let changed = false;
|
||||
const transactions = vi.spyOn(db, "transaction").mockImplementation((async (callback, options) => {
|
||||
if (!changed) {
|
||||
changed = true;
|
||||
await transaction(async tx => {
|
||||
await notifyInstructionCommit(tx, { companyId, runId: copy.runId });
|
||||
await tx.update(agentInstructionWorkingCopies).set({ nextAttemptAt: later })
|
||||
.where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
});
|
||||
}
|
||||
return transaction(callback, options);
|
||||
}) as typeof db.transaction);
|
||||
try {
|
||||
await copies.registerCapturedCommitWork(worker, () => true, async () => true).ready;
|
||||
expect(changed).toBe(true);
|
||||
expect(await copies.get(companyId, copy.runId)).toMatchObject({ state: "pending_commit", attempts: 1, nextAttemptAt: later });
|
||||
transactions.mockRestore();
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await expect.poll(() => idleWorkSnapshot().active).toBe(0);
|
||||
expect(worker.nextWakeAt()).toBe(later.getTime());
|
||||
await vi.advanceTimersByTimeAsync(60_000); await saved(copy.runId);
|
||||
} finally { transactions.mockRestore(); await worker.stop(); }
|
||||
});
|
||||
it("reactivates after a real worktree setting change without polling disabled settings", async () => {
|
||||
const copy = await run(); await captured(copy);
|
||||
const runtimeEnv = { PAPERCLIP_IN_WORKTREE: "true", PAPERCLIP_INSTANCE_ID: "sleep40-instruction-activation" };
|
||||
const settings = instanceSettingsService(db, { runtimeEnv });
|
||||
await settings.updateExperimental({ enableWorktreeRunExecution: false });
|
||||
const service = heartbeatService(db, { runtimeEnv });
|
||||
clock(); const worker = coordinator();
|
||||
try {
|
||||
await service.registerCapturedInstructionCommitWork(worker, () => true).ready;
|
||||
expect(worker.nextWakeAt()).toBeNull();
|
||||
const reads = vi.spyOn(db, "select");
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect(reads).not.toHaveBeenCalled(); reads.mockRestore();
|
||||
await service.resolveSchedulingSuppression();
|
||||
await settings.updateExperimental({ enableWorktreeRunExecution: true });
|
||||
await saved(copy.runId);
|
||||
} finally { await worker.stop(); await settings.updateExperimental({ enableWorktreeRunExecution: false }); }
|
||||
});
|
||||
it("reconstructs more than one page and drains the saved candidates in bounded batches", async () => {
|
||||
const copy = await run();
|
||||
const ids = Array.from({ length: 103 }, () => randomUUID());
|
||||
await db.insert(heartbeatRuns).values(ids.map(id => ({ id, companyId, agentId,
|
||||
invocationSource: "on_demand", responsibleUserId: userId })));
|
||||
await db.insert(agentInstructionWorkingCopies).values(ids.map(runId => ({ ...copy, runId,
|
||||
state: "pending_commit", attempts: 1, nextAttemptAt: new Date(), processStoppedAt: new Date(),
|
||||
candidateBase64: Buffer.from("recovered bytes").toString("base64"),
|
||||
})));
|
||||
clock(); const worker = coordinator();
|
||||
try {
|
||||
await copies.registerCapturedCommitWork(worker, () => true, async () => true).ready;
|
||||
const rows = () => db.select({ state: agentInstructionWorkingCopies.state }).from(agentInstructionWorkingCopies)
|
||||
.where(eq(agentInstructionWorkingCopies.companyId, companyId));
|
||||
expect((await rows()).filter(row => row.state === "saved").length).toBeLessThanOrEqual(20);
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect((await rows()).filter(row => row.state === "saved")).toHaveLength(103);
|
||||
}, { timeout: 15_000 });
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe("recovered bytes");
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await expect.poll(() => idleWorkSnapshot().active).toBe(0);
|
||||
expect(worker.nextWakeAt()).toBeNull();
|
||||
} finally { await worker.stop(); }
|
||||
}, 20_000);
|
||||
it("requires outer commit and ignores a nested rollback", async () => {
|
||||
const copy = await run(); clock(); const worker = coordinator();
|
||||
try {
|
||||
await copies.registerCapturedCommitWork(worker, () => true, async () => true).ready;
|
||||
await expect(db.transaction(async tx => {
|
||||
await captured(copy, new Date(), tx as unknown as typeof db);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
throw new Error("outer rollback");
|
||||
})).rejects.toThrow("outer rollback");
|
||||
expect((await copies.get(companyId, copy.runId))?.state).toBe("prepared");
|
||||
await db.transaction(async tx => {
|
||||
await captured(copy, new Date(), tx as unknown as typeof db);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
});
|
||||
await saved(copy.runId);
|
||||
} finally { await worker.stop(); }
|
||||
});
|
||||
it("preserves a permission conflict instead of retrying it forever", async () => {
|
||||
const copy = await run(); await captured(copy);
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.companyId, companyId));
|
||||
clock(); const worker = coordinator();
|
||||
try {
|
||||
await copies.registerCapturedCommitWork(worker, () => true, async () => true).ready;
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect((await copies.get(companyId, copy.runId))?.state).toBe("conflict");
|
||||
});
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
expect((await copies.get(companyId, copy.runId))?.candidateBase64).toBe(Buffer.from("recovered bytes").toString("base64"));
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await expect.poll(() => idleWorkSnapshot().active).toBe(0);
|
||||
expect(worker.nextWakeAt()).toBeNull();
|
||||
} finally { await worker.stop(); }
|
||||
});
|
||||
it.each((["shared", "separate", "dedicated"] as const).flatMap(mode =>
|
||||
(["saved", "retry"] as const).map(outcome => ({ mode, outcome }))))("joins foreground ownership without consuming a recovery attempt ($mode, $outcome)", async ({ mode, outcome }) => {
|
||||
const copy = await run(); await fs.writeFile(path.join(copy.localRoot, entryFile), "recovered bytes");
|
||||
const exercise = async (workerDb: typeof db) => {
|
||||
const recoveryCopies = mode === "shared" ? copies : agentInstructionWorkingCopyService(workerDb);
|
||||
clock(); const worker = coordinator();
|
||||
let release!: () => void, entered!: () => void;
|
||||
const held = new Promise<void>(resolve => { release = resolve; });
|
||||
const reached = new Promise<void>(resolve => { entered = resolve; });
|
||||
let foreground: ReturnType<typeof copies.collectStopped> | undefined;
|
||||
const rename = fs.rename.bind(fs);
|
||||
let holding = true;
|
||||
const renameSpy = vi.spyOn(fs, "rename").mockImplementation(async (from, to) => {
|
||||
if (holding && to === path.join(root, entryFile)) {
|
||||
holding = false; entered(); await held;
|
||||
if (outcome === "retry") throw new Error("temporary foreground replacement failure");
|
||||
}
|
||||
return rename(from, to);
|
||||
});
|
||||
try {
|
||||
await recoveryCopies.registerCapturedCommitWork(worker, () => true, async () => true).ready;
|
||||
foreground = copies.collectStopped({ companyId, runId: copy.runId });
|
||||
await reached;
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(await copies.get(companyId, copy.runId)).toMatchObject({ state: "pending_commit", attempts: 1 });
|
||||
release();
|
||||
const result = await foreground;
|
||||
if (outcome === "retry") {
|
||||
expect(result).toMatchObject({ state: "pending_commit", attempts: 1, nextAttemptAt: expect.any(Date) });
|
||||
expect(result!.nextAttemptAt!.getTime()).toBeGreaterThan(Date.now() + 29_000);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await expect.poll(() => idleWorkSnapshot().active).toBe(0);
|
||||
expect(worker.nextWakeAt()).toBe(result!.nextAttemptAt!.getTime());
|
||||
await vi.advanceTimersByTimeAsync(Math.max(0, result!.nextAttemptAt!.getTime() - Date.now() - 5000));
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
expect((await copies.get(companyId, copy.runId))?.attempts).toBe(1);
|
||||
await vi.advanceTimersByTimeAsync(5000);
|
||||
} else expect(result).toMatchObject({ state: "saved", attempts: 1 });
|
||||
await saved(copy.runId);
|
||||
const events = await db.select().from(activityLog).where(and(eq(activityLog.companyId, companyId), eq(activityLog.action, "agent.files_updated")));
|
||||
expect(events).toHaveLength(1);
|
||||
} finally { release(); await foreground; renameSpy.mockRestore(); await worker.stop(); }
|
||||
};
|
||||
if (mode === "dedicated") await withDedicatedDbConnection(db, exercise);
|
||||
else await exercise(db);
|
||||
});
|
||||
});
|
||||
|
||||
it("persists an atomic editor replacement, ignores repository instructions, and gives the next run exact bytes", async () => {
|
||||
const copy = await run();
|
||||
await fs.writeFile(path.join(home, "AGENTS.md"), "Repository text must not become agent instructions");
|
||||
|
||||
@@ -115,6 +115,7 @@ const {
|
||||
reconcileHotRestartAdoption: vi.fn(async () => ({ mode: "none" })),
|
||||
reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })),
|
||||
registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerCapturedInstructionCommitWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
promoteDueScheduledRetries: vi.fn(async () => ({ promoted: 0, runIds: [] })),
|
||||
resumeQueuedRuns: vi.fn(async () => undefined),
|
||||
recoverPendingSessionGoalActions: vi.fn(async () => ({ scanned: 0, enqueued: 0, alreadyQueued: 0, invalid: 0 })),
|
||||
@@ -550,6 +551,13 @@ describe("startServer feedback export wiring", () => {
|
||||
process.env.BETTER_AUTH_SECRET = "test-secret";
|
||||
});
|
||||
|
||||
it.each([false, true])("registers captured instruction recovery only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
expect(heartbeatServiceMock.registerCapturedInstructionCommitWork).toHaveBeenCalledTimes(enabled ? 1 : 0);
|
||||
if (enabled) expect(heartbeatServiceMock.registerCapturedInstructionCommitWork.mock.calls[0][1]()).toBe(true);
|
||||
});
|
||||
|
||||
it.each([false, true])("registers retry cancellation recovery only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
@@ -941,7 +949,7 @@ describe("startServer feedback export wiring", () => {
|
||||
intervalCallback?.();
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2);
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true });
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true });
|
||||
expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1);
|
||||
expect(retiredDetector).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
|
||||
+4
-1
@@ -1820,6 +1820,9 @@ async function startServerWithDatabaseTeardown(
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerScheduledRetryCancellationWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerCapturedInstructionCommitWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
|
||||
const runRetentionSweep = async () => {
|
||||
const activeCompanies = await db.select({ id: companies.id }).from(companies).where(eq(companies.status, "active"));
|
||||
let archived = 0;
|
||||
@@ -1947,7 +1950,7 @@ async function startServerWithDatabaseTeardown(
|
||||
// Periodically reap orphaned runs (5-min staleness threshold) and make sure
|
||||
// persisted queued work is still being driven forward.
|
||||
runPeriodicRecovery(() => heartbeat
|
||||
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true })
|
||||
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true })
|
||||
.then(() => heartbeat.promoteDueScheduledRetries())
|
||||
.then(async (promotion) => {
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyInstructionCommit } from "./instruction-commit-notifications.js";
|
||||
import { notifyScheduledRetryCancellation } from "./scheduled-retry-cancellation-notifications.js";
|
||||
import { and, eq, or, sql } from "drizzle-orm";
|
||||
import { agents, agentApiKeys, agentRuntimeState, agentTaskSessions, agentWakeupRequests, activityLog,
|
||||
@@ -44,6 +45,7 @@ export async function deleteAgentDependencies(tx: Db, companyId: string, id: str
|
||||
await tx.delete(issueExecutionDecisions).where(eq(issueExecutionDecisions.actorAgentId, id));
|
||||
await tx.delete(issueComments).where(eq(issueComments.authorAgentId, id));
|
||||
await notifyScheduledRetryCancellation(tx, { companyId: companyId });
|
||||
await notifyInstructionCommit(tx, { companyId: companyId });
|
||||
await tx.delete(heartbeatRuns).where(eq(heartbeatRuns.agentId, id));
|
||||
await tx.delete(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, id));
|
||||
await tx.delete(agentApiKeys).where(eq(agentApiKeys.agentId, id));
|
||||
|
||||
@@ -1,11 +1,15 @@
|
||||
import { notifyInstructionCommit, type InstructionCommitScope } from "./instruction-commit-notifications.js";
|
||||
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
||||
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
||||
import { registerScopedDeadlineWork } from "./scoped-deadline-work.js";
|
||||
import { agentDirectoryWorkingCopyService, isAgentDirectoryCopy } from "./agent-directory-working-copies.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { and, asc, desc, eq, inArray, lte, or, isNull, isNotNull, sql } from "drizzle-orm";
|
||||
import { agents, heartbeatRuns, agentInstructionWorkingCopies as copies, type Db } from "@paperclipai/db";
|
||||
import { and, asc, desc, eq, gt, inArray, lte, or, isNull, isNotNull, sql } from "drizzle-orm";
|
||||
import { agents, heartbeatRuns, agentInstructionWorkingCopies as copies, databaseWorkScopeKey, type Db } from "@paperclipai/db";
|
||||
import {
|
||||
prepareAdapterExecutionTargetRuntime,
|
||||
runAdapterExecutionTargetShellCommand,
|
||||
@@ -32,6 +36,23 @@ const MAX_COLLECTION_ATTEMPTS = 3;
|
||||
// supply one. These handles are optional: durable captured bytes survive restart.
|
||||
const liveTargets = new Map<string, AdapterExecutionTarget>();
|
||||
const targetKey = (companyId: string, runId: string) => `${companyId}:${runId}`;
|
||||
const copySaves = new WeakMap<object, Map<string, Promise<void>>>();
|
||||
|
||||
function serializeCopySave<T>(db: Db, companyId: string, runId: string, save: () => Promise<T>): Promise<T> {
|
||||
const scope = databaseWorkScopeKey(db);
|
||||
let active = copySaves.get(scope);
|
||||
if (!active) copySaves.set(scope, active = new Map());
|
||||
const key = targetKey(companyId, runId);
|
||||
|
||||
|
||||
const operation = (active.get(key) ?? Promise.resolve()).then(save);
|
||||
const settled = operation.then(() => undefined, () => undefined);
|
||||
active.set(key, settled);
|
||||
return operation.finally(() => {
|
||||
if (active.get(key) === settled) active.delete(key);
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
/** Finish only bounded immediate collection work; deferred ownership remains pending. */
|
||||
export async function collectStoppedInstructionCopyWithRetries<T extends Pick<Copy, "state" | "attempts" | "nextAttemptAt">>(collect: () => Promise<T | null>): Promise<T | null> {
|
||||
@@ -73,11 +94,22 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
||||
const [row] = await db.select().from(copies).where(scope(companyId, runId));
|
||||
return row ?? null;
|
||||
}
|
||||
async function tryPatch(row: Copy, values: Partial<typeof copies.$inferInsert>, requireDue = false) {
|
||||
const signalsCommit = row.state === "pending_commit" || values.state === "pending_commit";
|
||||
const write = async (tx: Db) => {
|
||||
if (signalsCommit)
|
||||
await notifyInstructionCommit(tx, { companyId: row.companyId, runId: row.runId });
|
||||
const [updated] = await tx.update(copies).set({ ...values, updatedAt: new Date() }).where(and(
|
||||
scope(row.companyId, row.runId), eq(copies.state, row.state), eq(copies.attempts, row.attempts), eq(copies.baseHash, row.baseHash),
|
||||
sql`${copies.receipt}->>'retainedByRunId' IS NOT DISTINCT FROM ${typeof row.receipt?.retainedByRunId === "string" ? row.receipt.retainedByRunId : null}`,
|
||||
requireDue ? or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())) : undefined,
|
||||
)).returning();
|
||||
return updated;
|
||||
};
|
||||
return signalsCommit ? db.transaction(tx => write(tx as unknown as Db)) : write(db);
|
||||
}
|
||||
async function patch(row: Copy, values: Partial<typeof copies.$inferInsert>) {
|
||||
const [updated] = await db.update(copies).set({ ...values, updatedAt: new Date() }).where(and(
|
||||
scope(row.companyId, row.runId), eq(copies.state, row.state), eq(copies.attempts, row.attempts), eq(copies.baseHash, row.baseHash),
|
||||
sql`${copies.receipt}->>'retainedByRunId' IS NOT DISTINCT FROM ${typeof row.receipt?.retainedByRunId === "string" ? row.receipt.retainedByRunId : null}`,
|
||||
)).returning();
|
||||
const updated = await tryPatch(row, values);
|
||||
// A late cleanup must not overwrite an explicit resolution or a newer
|
||||
// baseline. Canonical content has its own independent head CAS.
|
||||
return updated ?? (await get(row.companyId, row.runId))!;
|
||||
@@ -277,7 +309,10 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
||||
}
|
||||
|
||||
/** Caller must have joined the owned provider process before granting this proof. */
|
||||
async function collectStopped(input: { companyId: string; runId: string; target?: AdapterExecutionTarget | null }) {
|
||||
function collectStopped(input: { companyId: string; runId: string; target?: AdapterExecutionTarget | null }): Promise<Copy | null> {
|
||||
return serializeCopySave(db, input.companyId, input.runId, () => collectStoppedOnce(input));
|
||||
}
|
||||
async function collectStoppedOnce(input: Parameters<typeof collectStopped>[0]) {
|
||||
let row = await get(input.companyId, input.runId);
|
||||
if (!row) return row;
|
||||
if (isAgentDirectoryCopy(row)) return directories.collectStopped(row, input.target);
|
||||
@@ -300,16 +335,13 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
||||
}
|
||||
|
||||
/** Restarts can retry persisted bytes without touching a live filesystem or launching a model. */
|
||||
async function recoverCaptured() {
|
||||
const pending = await db.select().from(copies).where(and(
|
||||
async function recoverCaptured(options: { skipCommit?: boolean } = {}) {
|
||||
const pending = options.skipCommit ? [] : await db.select().from(copies).where(and(
|
||||
eq(copies.state, "pending_commit"),
|
||||
or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())),
|
||||
lte(copies.attempts, MAX_COLLECTION_ATTEMPTS - 1),
|
||||
)).limit(20);
|
||||
for (const row of pending) {
|
||||
const result = await commitCandidate(await patch(row, { attempts: row.attempts + 1 }));
|
||||
if (isAgentDirectoryCopy(result) && completed.has(result.state)) await directories.release(result);
|
||||
}
|
||||
for (const row of pending) await recoverCommit(commitEntry(row)!, () => true);
|
||||
// A crash between a durable save and release must not retain run snapshots
|
||||
// forever. Only discard copies with recorded stop proof and a terminal receipt.
|
||||
const cleanup = await db.select().from(copies).where(and(
|
||||
@@ -374,6 +406,56 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
||||
return pending.length;
|
||||
}
|
||||
|
||||
type CommitEntry = { id: string; companyId: string; dueAt: number };
|
||||
function commitEntry(row: Pick<Copy, "runId" | "companyId" | "state" | "attempts" | "nextAttemptAt"> | null): CommitEntry | null {
|
||||
return row?.state === "pending_commit" && row.attempts < MAX_COLLECTION_ATTEMPTS
|
||||
? { id: row.runId, companyId: row.companyId, dueAt: row.nextAttemptAt?.getTime() ?? 0 } : null;
|
||||
}
|
||||
async function loadCommits(scope: InstructionCommitScope | undefined, admitted: () => boolean) {
|
||||
const entries: CommitEntry[] = [];
|
||||
let cursor: string | undefined;
|
||||
while (admitted()) {
|
||||
const rows = await db.select({ runId: copies.runId, companyId: copies.companyId, state: copies.state,
|
||||
attempts: copies.attempts, nextAttemptAt: copies.nextAttemptAt }).from(copies).where(and(
|
||||
eq(copies.state, "pending_commit"), lte(copies.attempts, MAX_COLLECTION_ATTEMPTS - 1),
|
||||
scope ? eq(copies.companyId, scope.companyId) : undefined,
|
||||
scope?.runId ? eq(copies.runId, scope.runId) : undefined,
|
||||
cursor ? gt(copies.runId, cursor) : undefined,
|
||||
)).orderBy(asc(copies.runId)).limit(100);
|
||||
for (const row of rows) entries.push(commitEntry(row)!);
|
||||
if (rows.length < 100) return entries;
|
||||
cursor = rows[rows.length - 1].runId;
|
||||
}
|
||||
throw new Error("Instruction commit reconstruction stopped");
|
||||
}
|
||||
function recoverCommit(entry: CommitEntry, admitted: () => boolean): Promise<CommitEntry | null> {
|
||||
return serializeCopySave(db, entry.companyId, entry.id, () => recoverCommitOnce(entry, admitted));
|
||||
}
|
||||
async function recoverCommitOnce(entry: CommitEntry, admitted: () => boolean) {
|
||||
const row = await get(entry.companyId, entry.id);
|
||||
const current = commitEntry(row);
|
||||
if (!row || !current || current.dueAt > Date.now()) return current;
|
||||
if (!admitted()) throw new Error("Instruction commit recovery stopped");
|
||||
const claimed = await tryPatch(row, { attempts: row.attempts + 1 }, true);
|
||||
if (claimed) {
|
||||
const result = await commitCandidate(claimed);
|
||||
if (isAgentDirectoryCopy(result) && completed.has(result.state)) await directories.release(result);
|
||||
}
|
||||
return commitEntry(await get(entry.companyId, entry.id));
|
||||
}
|
||||
function registerCapturedCommitWork(coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
|
||||
canRun: () => boolean, canRecover: () => Promise<boolean>) {
|
||||
return registerScopedDeadlineWork(coordinator, {
|
||||
queue: DELIVERY_QUEUES.instructionCommit,
|
||||
batchSize: 20,
|
||||
decodeScope: (hint: string): InstructionCommitScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
|
||||
matches: (entry: CommitEntry, scope?: InstructionCommitScope) => !scope ||
|
||||
entry.companyId === scope.companyId && (!scope.runId || entry.id === scope.runId),
|
||||
load: async (scope, signal) => await canRecover() ? loadCommits(scope, () => canRun() && !signal?.aborted) : [],
|
||||
check: async (entry, signal) => await canRecover() ? recoverCommit(entry, () => canRun() && !signal?.aborted) : null,
|
||||
}, canRun);
|
||||
}
|
||||
|
||||
async function list(companyId: string, agentId: string, actor: AuthorizationActor) {
|
||||
await revisions.readCurrent({ companyId, agentId }, actor);
|
||||
const rows = await db.select().from(copies).where(and(eq(copies.companyId, companyId), eq(copies.agentId, agentId), inArray(copies.state, ["conflict", "pending_collection", "pending_commit", "unavailable"]))).orderBy(desc(copies.createdAt)).limit(50);
|
||||
@@ -422,5 +504,5 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
||||
}
|
||||
return row;
|
||||
}
|
||||
return { prepare, checkpointWarm, canReuseWarm: directories.canReuse, adopt, reportRetirementUnconfirmed, get, hasChanges, acknowledgeExplicitSave, collectStopped, recoverCaptured, recoverStopped, list, resolve, reportUnavailable, release };
|
||||
return { registerCapturedCommitWork, prepare, checkpointWarm, canReuseWarm: directories.canReuse, adopt, reportRetirementUnconfirmed, get, hasChanges, acknowledgeExplicitSave, collectStopped, recoverCaptured, recoverStopped, list, resolve, reportUnavailable, release };
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyInstructionCommit } from "./instruction-commit-notifications.js";
|
||||
import { notifyScheduledRetryCancellation } from "./scheduled-retry-cancellation-notifications.js";
|
||||
import { notifyChatCompanyConfiguration } from "./chat-channels/configuration-work-notifications.js";
|
||||
import { chatCompanyDeletion } from "./chat-channels/company-deletion.js";
|
||||
@@ -98,6 +99,7 @@ export async function deleteCompany(db: Db, id: string) {
|
||||
await tx.delete(activityLog).where(eq(activityLog.companyId, id));
|
||||
await tx.delete(runIdentityContexts).where(eq(runIdentityContexts.companyId, id));
|
||||
await notifyScheduledRetryCancellation(tx, { companyId: id });
|
||||
await notifyInstructionCommit(tx, { companyId: id });
|
||||
await tx.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, id));
|
||||
await tx.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, id));
|
||||
await tx.delete(agentApiKeys).where(eq(agentApiKeys.companyId, id));
|
||||
|
||||
@@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
|
||||
/** Topics for existing durable queues; register intent before their writes. */
|
||||
export const DELIVERY_QUEUES = {
|
||||
instructionCommit: "instruction-commit",
|
||||
scheduledRetryCancellation: "scheduled-retry-cancellation",
|
||||
executionFinalization: "execution-finalization",
|
||||
workspaceRecovery: "workspace-recovery",
|
||||
|
||||
@@ -4084,6 +4084,14 @@ export function heartbeatService(
|
||||
triggerIssueMonitor,
|
||||
|
||||
reportRunActivity: clearDetachedRunWarning,
|
||||
registerCapturedInstructionCommitWork: (
|
||||
coordinator: Parameters<typeof instructionCopies.registerCapturedCommitWork>[0], canRun: () => boolean,
|
||||
) => instructionCopies.registerCapturedCommitWork(coordinator,
|
||||
() => canRun() && readTaskDrain(new Date()) === null,
|
||||
async () => {
|
||||
const state = await getSchedulingSuppression(true);
|
||||
return !state.suppressed || state.reason === "task_drain";
|
||||
}),
|
||||
registerScheduledRetryCancellationWork: (
|
||||
coordinator: Parameters<typeof registerScheduledRetryCancellationWork>[0], canRun: () => boolean,
|
||||
) => registerScheduledRetryCancellationWork(coordinator,
|
||||
|
||||
@@ -1947,7 +1947,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
|
||||
let settledProcessesDiscovered = false;
|
||||
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean }) {
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean }) {
|
||||
await preserveCancelledNativeWorkspaces(db);
|
||||
const staleThresholdMs = opts?.staleThresholdMs ?? 0;
|
||||
const now = new Date();
|
||||
@@ -1957,7 +1957,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
await instructionCopies.recoverStopped().catch(error => {
|
||||
logger.warn({ err: error }, "failed to recover stopped instruction copies");
|
||||
});
|
||||
await instructionCopies.recoverCaptured().catch(error => {
|
||||
await instructionCopies.recoverCaptured({ skipCommit: opts?.skipCapturedInstructionCommit }).catch(error => {
|
||||
logger.warn({ err: error }, "failed to retry captured instruction revisions");
|
||||
});
|
||||
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyInstructionCommit } from "./instruction-commit-notifications.js";
|
||||
import { notifyScheduledRetryCancellation } from "./scheduled-retry-cancellation-notifications.js";
|
||||
import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js";
|
||||
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
|
||||
@@ -617,6 +618,7 @@ export function instanceSettingsService(db: Db, options: InstanceSettingsService
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.dotEvents, "*");
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.grokDispatch);
|
||||
if (patch.enableWorktreeRunExecution !== undefined) await notifyScheduledRetryCancellation(tx);
|
||||
if (patch.enableWorktreeRunExecution !== undefined) await notifyInstructionCommit(tx);
|
||||
if (patch.enableChatConnectors !== undefined) await notifyVoiceChange(tx);
|
||||
await notifyToolConnectionHealth(tx, "*");
|
||||
return tx
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type InstructionCommitScope = { companyId: string; runId?: string };
|
||||
|
||||
export async function notifyInstructionCommit(transaction: object, scope?: InstructionCommitScope) {
|
||||
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.instructionCommit, scope ? JSON.stringify(scope) : "*");
|
||||
}
|
||||
Reference in new issue
Block a user