mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
test(restart-chaos): count staging-mix completions only from DONE, and attribute runs after DONE (#21226)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness grades staging-mix tasks across server
swaps, and reports duplicated side effects separately from completion
> - `completion_once` counted every comment that mentions `DONE-<nonce>`
as a completion
> - In wave `w-20261112T223803-s7101`, task t001's blocked run fcc55827
wrote that it "will … post `DONE-<nonce>`" later, and the harness
counted that as a second completion next to the real one from run
37724b59
> - Runs that start after a task is done are also hard to attribute
without a manual trace through the wake evidence
> - This pull request counts only real completions, and lists each run
started after DONE with its wake reason and actor
## Linked Issues or Issue Description
Refs #21168, #21173.
**What existing behavior does this improve?**
The staging-mix `completion_once` check and the wave summary.
**Subsystem affected**
Test harness only: `tests/restart-chaos/`. No product code changes.
**Current behavior**
Any agent comment that mentions the DONE marker counts as a completion.
A blocked or progress comment that promises to post the marker later
becomes a false "duplicate completion". The summary does not say why a
run started after the task was done.
**Proposed behavior**
A run completes the task only when it posts the bare `DONE-<nonce>`
comment (the rule `isDoneComment` already uses), or posts a comment that
mentions the marker at or after the first DONE comment. The summary gets
a "Runs started after the task's DONE" section. It lists each such run
with its wake reason, source, trigger and requesting actor, taken from
the saved wake evidence.
**Reason and benefit**
The duplicate count stays honest, and extra turns can be attributed
straight from the wave results.
## What Changed
- `staging-mix.ts`:
- Comments keep their creation time.
- The new `completionRunIds` counts a run as completing the task when it
posts the bare DONE comment, or posts a later comment that mentions the
marker.
- `done_comment_posted` and `done_comment_once` are unchanged.
- `staging-mix-retry.ts`:
- `firstDoneAt` anchors DONE on the first bare DONE comment, else on the
issue's completion time.
- `runsAfterDone` joins each later run to its wake row in
`wake-evidence/<task>.json`.
- The results JSON gains `extraTurns`, and the summary prints one line
per run, for example `run 91f0c3d2 started … after DONE; wake
issue_recovery_action_restored (automation/system) by system sweep`.
- Runs after DONE are filtered by when they started, not when they were
queued. A run queued after DONE that never started is listed separately
as "queued after DONE, never started" (`queuedAfterDone`).
- `wake-evidence.ts` keeps only issue status transitions: `issue.*`
entries whose status is an issue status. In t001 of
`w-20261112T223803-s7101`, a `runner.api_called` entry with HTTP status
201 was being listed as a status change.
- README: the `completion_once` row, the results paragraph and the
wake-evidence status note.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
68 files, 774 tests pass.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- A regression test copies t001's shape from `w-20261112T223803-s7101`:
the blocked comment from run fcc55827 at 22:56 mentions the marker as a
future step, and the bare DONE comment comes from run 37724b59 at 23:12.
The test checks `completion_once` passes with `runs=1`. It fails on
master, which reports two runs.
- The earlier duplicate case still fails `completion_once`: a different
run's completion summary posted after the DONE comment.
- Tests cover the run-after-DONE join, the DONE anchor, and the summary
line.
## Risks
- Low risk. Test harness only.
- A comment from a run that was cut off before it could post the bare
marker, but that mentioned the marker before the first DONE comment, is
no longer counted as a completion. That is the intended change.
- Known master CI failures, not touched here: the storage S3 multipart
throttle test and the Runner native-distribution-integrity test. Recent
cancelled-runner CI failures ("The runner has received a shutdown
signal") are infrastructure, not tests.
- `ci / General tests (server (11/12))` failed on head 94c4751803 with a
real server test failure that this harness-only diff cannot reach:
`server/src/services/fresh-continuation-fallback.test.ts >
fresh-continuation fallback > keeps the continuation pending while the
agent's daily run limit declines it, then delivers it` (`expected [ {
…(81) } ] to have a length of +0 but got 1`). It failed on another PR at
the same time.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits). Extended reasoning was available.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
45523d21d0
commit
c1cf43e666
6 files changed
+338
-33
No files matched your search
@@ -1555,6 +1555,21 @@ worked when the clicked task meets the same completion rule. The banner list fol
|
||||
`GET /api/companies/:id/assigned-work-invariant` at the end of the wave: `violationCount`, `violations` by reason,
|
||||
`agentsWithStalledSlots` and `openTasks`.
|
||||
|
||||
The summary also lists, by start time, every run that started after the task completed, under "Runs started after
|
||||
the task completed" (`extraTurns`, with counts by wake reason in `extraTurnsByReason`). A run queued after completion
|
||||
that never started is listed on its own (`queuedAfterDone`).
|
||||
|
||||
The completion time comes from the product's own record, in this order:
|
||||
|
||||
1. the earliest accepted run result whose `reportedWorkDisposition` is `done`;
|
||||
2. the issue's `completedAt`;
|
||||
3. the first bare `DONE-<nonce>` comment, only when neither exists.
|
||||
|
||||
Each line prints the DONE comment time too when it differs, and gives the run's wake reason, source, trigger and
|
||||
requesting actor from the saved wake evidence. A bare DONE comment posted by a run whose accepted result was not `done`
|
||||
(for example `blocked` while a parent waits on its subtasks) is reported on its own, under "DONE comment posted before
|
||||
the task completed" (`doneBeforeCompletion`).
|
||||
|
||||
**Grade.** The usual R1–R6, U1–U5 and T rules apply. A needed Retry still fails R3. The T-oracle checks the result,
|
||||
not the status:
|
||||
|
||||
@@ -1565,7 +1580,7 @@ not the status:
|
||||
| `rounds_once` | a round line appears twice (a tool ran again) |
|
||||
| `done_comment_posted` | no comment's whole body is `DONE-<nonce>`; a summary that quotes the marker does not count |
|
||||
| `done_comment_once` | that comment was posted more than once |
|
||||
| `completion_once` | more than one run posted a comment that mentions `DONE-<nonce>` (a duplicate completion), or the comments could not be read |
|
||||
| `completion_once` | more than one run completed the task (a duplicate completion), or the comments could not be read. A run completes the task when its accepted result has disposition `done`, or when it posts a comment that mentions `DONE-<nonce>` at or after the completion time. Without accepted results in the wave evidence, the bare DONE comment counts instead. A progress or blocked comment that says it will post the marker later is not a completion |
|
||||
| `follow_up_posted` | a planned follow-up was never posted (the detail lists the post errors) |
|
||||
| `follow_up_served` | a follow-up was posted during a run and the ledger lacks its line |
|
||||
| `subtask_<k>_done`, `parent_read_subtask_<k>` | a subtask is missing, not done or its ledger fails, or the parent did not report its last hash |
|
||||
@@ -1978,7 +1993,8 @@ re-arms the task's own triggered faults instead of using their offsets.
|
||||
- each run's `context_snapshot`, invocation source and trigger;
|
||||
- the `agent_wakeup_requests` row behind each run and every wake for the issue: reason, source, trigger detail,
|
||||
requesting actor, idempotency key, payload and times;
|
||||
- the issue's status changes from the activity log.
|
||||
- the issue's status changes from the activity log: only `issue.*` entries whose status is an issue status, so a
|
||||
`runner.api_called` entry with an HTTP status is not mistaken for one.
|
||||
|
||||
Each wake and run also names the server generations alive when it was created (`servingAtCreate`): the pid, boot
|
||||
id, epoch and process space from `controller_instances`, and the harness's own primary or secondary role during an
|
||||
|
||||
@@ -1,9 +1,12 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { parseCommand } from "./config.js";
|
||||
import { collectStagingMixArtifacts, doneMarker, expectedRoundHash, gradeStagingMix, planStagingMixWave, ROUND_SECONDS, type StagingMixTask } from "./staging-mix.js";
|
||||
import {
|
||||
bannerSink, issueView, readAssignedWorkInvariant, retryAfterSettleStage, shouldClickRetryAfterSettle, stagingMixHeadline, summarizeStagingMix,
|
||||
type RetryStageObserver,
|
||||
collectStagingMixArtifacts, completedAtAnchor, doneCommentsBeforeCompletion, doneMarker, expectedRoundHash, gradeStagingMix, planStagingMixWave, ROUND_SECONDS,
|
||||
stagingMixDescription, type CompletionEvidence, type StagingMixTask,
|
||||
} from "./staging-mix.js";
|
||||
import {
|
||||
bannerSink, commentEvidence, evidenceCompletion, extraTurnLine, extraTurnsByReason, issueView, readAssignedWorkInvariant, retryAfterSettleStage, runsAfterDone, shouldClickRetryAfterSettle,
|
||||
stagingMixHeadline, summarizeStagingMix, type RetryStageObserver,
|
||||
} from "./staging-mix-retry.js";
|
||||
import type { OracleArtifacts, TaskPlan, TaskRecord, TaskSample } from "./types.js";
|
||||
|
||||
@@ -158,9 +161,10 @@ describe("staging-mix Retry policy", () => {
|
||||
|
||||
describe("staging-mix completion comments", () => {
|
||||
const nonce = "b3d70e77";
|
||||
const base = (comments: Array<{ body: string; runId: string | null }>): OracleArtifacts => ({ issueStatus: "done", documents: { ledger: ledger(nonce, 4) },
|
||||
const base = (comments: Array<{ body: string; runId: string | null; at?: string }>): OracleArtifacts => ({ issueStatus: "done", documents: { ledger: ledger(nonce, 4) },
|
||||
agentComments: comments.map(comment => comment.body), files: {}, commands: {}, workspacePath: null,
|
||||
stagingMix: collectStagingMixArtifacts({} as TaskRecord, comments.map(comment => ({ body: comment.body, authorAgentId: "a", createdByRunId: comment.runId }))) });
|
||||
stagingMix: collectStagingMixArtifacts({} as TaskRecord, comments.map(comment => ({ body: comment.body, authorAgentId: "a", createdByRunId: comment.runId,
|
||||
...(comment.at ? { createdAt: comment.at } : {}) }))) });
|
||||
const check = (artifacts: OracleArtifacts, name: string) => gradeStagingMix(nonce, task(), artifacts).checks.find(entry => entry.name === name)!;
|
||||
|
||||
it("counts only the DONE comment itself, not a closing summary that quotes it", () => {
|
||||
@@ -182,13 +186,114 @@ describe("staging-mix completion comments", () => {
|
||||
expect(check(unread, "done_comment_posted").ok).toBe(true);
|
||||
});
|
||||
|
||||
it("does not count a blocked run's promise to post DONE later as a completion (w-20261112T223803-s7101 t001)", () => {
|
||||
const comments = [
|
||||
{ body: "Prepared Parent ledger log for this response.", runId: "37724b59-4e6a", at: "2026-11-12T23:12:54.696Z" },
|
||||
{ body: doneMarker(nonce), runId: "37724b59-4e6a", at: "2026-11-12T23:12:44.587Z" },
|
||||
{ body: "Blocked report accepted. Summary of progress so far:\n\n- Completed all 4 parent rounds.\n\nParent now waits on both subtasks to finish. "
|
||||
+ `When they complete, I'll read each subtask's ledger hash, save this task's \`ledger\` document, post \`${doneMarker(nonce)}\`, and mark the task done.`,
|
||||
runId: "fcc55827-ab78", at: "2026-11-12T22:56:04.145Z" },
|
||||
];
|
||||
expect(check(base(comments), "completion_once")).toMatchObject({ ok: true, detail: "runs=1" });
|
||||
expect(check(base(comments), "done_comment_once")).toMatchObject({ ok: true, detail: "comments=1" });
|
||||
expect(check(base(comments), "done_comment_posted").ok).toBe(true);
|
||||
});
|
||||
|
||||
it("flags a completion summary repeated by a different run as a duplicate completion", () => {
|
||||
const comments = [
|
||||
{ body: `CHA-1 complete. Posted exactly one comment ${doneMarker(nonce)} and marked the task done.`, runId: "413ddcf2-ffce" },
|
||||
{ body: `CHA-1 complete. Posted \`${doneMarker(nonce)}\`.`, runId: "13de31be-fb13" },
|
||||
{ body: doneMarker(nonce), runId: "13de31be-fb13" },
|
||||
{ body: `CHA-1 complete. Posted exactly one comment ${doneMarker(nonce)} and marked the task done.`, runId: "413ddcf2-ffce", at: "2026-11-12T18:10:00.000Z" },
|
||||
{ body: `CHA-1 complete. Posted \`${doneMarker(nonce)}\`.`, runId: "13de31be-fb13", at: "2026-11-12T18:05:01.000Z" },
|
||||
{ body: doneMarker(nonce), runId: "13de31be-fb13", at: "2026-11-12T18:05:00.000Z" },
|
||||
];
|
||||
expect(check(base(comments), "done_comment_once").ok).toBe(true);
|
||||
expect(check(base(comments), "completion_once")).toMatchObject({ ok: false, detail: "duplicate completion: runs 413ddcf2, 13de31be each reported completion" });
|
||||
expect(check(base(comments), "completion_once")).toMatchObject({ ok: false, detail: "duplicate completion: runs 13de31be, 413ddcf2 each reported completion" });
|
||||
expect(check(base([...comments.slice(1), { body: doneMarker(nonce), runId: "run-z", at: "2026-11-12T18:20:00.000Z" }]), "completion_once").ok).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("staging-mix runs after the task completed", () => {
|
||||
const evidence = {
|
||||
completedAt: 8_000,
|
||||
runs: [{ runId: "6b825e48-aaaa", createdAt: 1_000, startedAt: 1_100, invocationSource: "assignment", triggerDetail: null, wakeupRequestId: "w1", acceptedDisposition: "blocked", acceptedAt: 2_000 },
|
||||
{ runId: "37724b59-bbbb", createdAt: 5_000, startedAt: 5_100, invocationSource: "automation", triggerDetail: null, wakeupRequestId: "w2", acceptedDisposition: "done", acceptedAt: 7_900 },
|
||||
{ runId: "5e5e5e5e-eeee", createdAt: 7_000, startedAt: 8_500, invocationSource: "automation", triggerDetail: null, wakeupRequestId: "w2", acceptedDisposition: null, acceptedAt: null },
|
||||
{ runId: "91f0c3d2-cccc", createdAt: 9_000, startedAt: 9_100, invocationSource: "automation", triggerDetail: "system", wakeupRequestId: "w3", acceptedDisposition: null, acceptedAt: null },
|
||||
{ runId: "aa11bb22-dddd", createdAt: 9_500, startedAt: null, invocationSource: "on_demand", triggerDetail: "manual", wakeupRequestId: null },
|
||||
{ runId: "0f0f0f0f-ffff", createdAt: 7_500, startedAt: null, invocationSource: "on_demand", triggerDetail: null, wakeupRequestId: null }],
|
||||
wakes: [{ id: "w1", reason: "issue_assigned", source: "assignment", triggerDetail: null, requestedByActorType: "user", requestedByActorId: "u1" },
|
||||
{ id: "w2", reason: "issue_children_completed", source: "automation", triggerDetail: null, requestedByActorType: "system", requestedByActorId: null },
|
||||
{ id: "w3", reason: "issue_recovery_action_restored", source: "automation", triggerDetail: "system", requestedByActorType: "system", requestedByActorId: "sweep" }],
|
||||
};
|
||||
const comments = [{ body: doneMarker("n1"), runId: "6b825e48-aaaa", createdAt: 3_000 }];
|
||||
|
||||
it("anchors on the accepted done result, then completedAt, then the DONE comment", () => {
|
||||
expect(completedAtAnchor(evidenceCompletion(evidence), comments, "n1")).toEqual({ at: 7_900, source: "accepted_result" });
|
||||
expect(completedAtAnchor({ completedAt: 8_000, runs: [] }, comments, "n1")).toEqual({ at: 8_000, source: "completed_at" });
|
||||
expect(completedAtAnchor(null, comments, "n1")).toEqual({ at: 3_000, source: "done_comment" });
|
||||
expect(completedAtAnchor(evidenceCompletion(null, "2026-11-13T00:06:13.120Z"), [], "n1")).toEqual({ at: Date.parse("2026-11-13T00:06:13.120Z"), source: "completed_at" });
|
||||
});
|
||||
|
||||
it("lists runs that started after completion, by start time, and prints the DONE comment time when it differs", () => {
|
||||
const { started, neverStarted } = runsAfterDone("t001", evidence, completedAtAnchor(evidenceCompletion(evidence), comments, "n1"), 3_000);
|
||||
expect(started.map(turn => [turn.runId, turn.wakeReason, turn.actorType])).toEqual([["5e5e5e5e-eeee", "issue_children_completed", "system"],
|
||||
["91f0c3d2-cccc", "issue_recovery_action_restored", "system"]]);
|
||||
expect(extraTurnLine(started[1]!)).toBe(`- t001: run 91f0c3d2 started ${new Date(9_100).toISOString()} after completion ${new Date(7_900).toISOString()}, `
|
||||
+ `DONE comment ${new Date(3_000).toISOString()}; wake issue_recovery_action_restored (automation/system) by system sweep`);
|
||||
expect(neverStarted.map(turn => turn.runId)).toEqual(["aa11bb22-dddd"]);
|
||||
expect(extraTurnLine(neverStarted[0]!)).toContain("never started");
|
||||
expect(extraTurnsByReason(started)).toEqual({ issue_children_completed: 1, issue_recovery_action_restored: 1 });
|
||||
expect(runsAfterDone("t001", evidence, { at: null, source: null })).toEqual({ started: [], neverStarted: [] });
|
||||
expect(runsAfterDone("t001", null, { at: 1, source: "completed_at" })).toEqual({ started: [], neverStarted: [] });
|
||||
});
|
||||
|
||||
it("does not count the run that completed the task as an extra turn (w-20261112T233630-s7201 t005)", () => {
|
||||
const t005 = { completedAt: Date.parse("2026-11-13T00:06:13.120Z"),
|
||||
runs: [{ runId: "2936e0b3-6a40", createdAt: 1_794_526_637_260, startedAt: 1_794_526_637_356, invocationSource: "assignment", triggerDetail: null, wakeupRequestId: "f61202d1",
|
||||
acceptedDisposition: "blocked", acceptedAt: Date.parse("2026-11-12T23:53:29.905Z") },
|
||||
{ runId: "94066b4e-aacd", createdAt: 1_794_527_610_556, startedAt: 1_794_528_342_051, invocationSource: "automation", triggerDetail: null, wakeupRequestId: "0645a2dc",
|
||||
acceptedDisposition: "done", acceptedAt: Date.parse("2026-11-13T00:06:12.706Z") }],
|
||||
wakes: [{ id: "0645a2dc", reason: "issue_execution_promoted", source: "automation", triggerDetail: null, requestedByActorType: "user", requestedByActorId: "local-board" }] };
|
||||
const posted = commentEvidence([{ body: doneMarker("c9b1da0f"), authorAgentId: "a", createdByRunId: "94066b4e-aacd", createdAt: "2026-11-13T00:06:07.168Z" },
|
||||
{ body: `Blocked on child completion. I will post ${doneMarker("c9b1da0f")} later.`, authorAgentId: "a", createdByRunId: "2936e0b3-6a40", createdAt: "2026-11-12T23:53:30.499Z" }]);
|
||||
const completion = evidenceCompletion(t005);
|
||||
expect(runsAfterDone("t005", t005, completedAtAnchor(completion, posted, "c9b1da0f")).started).toEqual([]);
|
||||
const grade = gradeStagingMix("c9b1da0f", { turnMinutes: 15, rounds: 10, subtasks: 0, followUpAfterMs: null, arriveAtMs: 0, holdMs: 0 },
|
||||
{ issueStatus: "done", documents: { ledger: ledger("c9b1da0f", 10) }, agentComments: [], files: {}, commands: {}, workspacePath: null,
|
||||
stagingMix: { followUp: null, followUpErrors: [], retries: [], comments: posted, completion: completion! } });
|
||||
expect(grade.checks.find(check => check.name === "completion_once")).toMatchObject({ ok: true, detail: "runs=1" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("staging-mix completion from accepted results", () => {
|
||||
const nonce = "abcd1234";
|
||||
const completion = (runs: CompletionEvidence["runs"]): CompletionEvidence => ({ completedAt: 10_000, runs });
|
||||
|
||||
it("counts runs whose accepted result was done, plus a completion summary posted after completion by another run", () => {
|
||||
const runs = completion([{ runId: "r1", startedAt: 1, acceptedDisposition: "blocked", acceptedAt: 5_000 }, { runId: "r2", startedAt: 6_000, acceptedDisposition: "done", acceptedAt: 9_000 }]);
|
||||
const posted = [{ body: doneMarker(nonce), runId: "r2", createdAt: 8_900 }];
|
||||
const grade = (extra: Array<{ body: string; runId: string; createdAt: number }>) => gradeStagingMix(nonce, { turnMinutes: 5, rounds: 4, subtasks: 0, followUpAfterMs: null, arriveAtMs: 0, holdMs: 0 },
|
||||
{ issueStatus: "done", documents: { ledger: ledger(nonce, 4) }, agentComments: [], files: {}, commands: {}, workspacePath: null,
|
||||
stagingMix: { followUp: null, followUpErrors: [], retries: [], comments: [...posted, ...extra], completion: runs } }).checks.find(check => check.name === "completion_once")!;
|
||||
expect(grade([])).toMatchObject({ ok: true, detail: "runs=1" });
|
||||
expect(grade([{ body: `Will post ${doneMarker(nonce)} when the subtasks finish.`, runId: "r1", createdAt: 4_000 }]).ok).toBe(true);
|
||||
expect(grade([{ body: `Task complete; posted ${doneMarker(nonce)}.`, runId: "r3", createdAt: 9_500 }])).toMatchObject({ ok: false });
|
||||
const twice = completion([{ runId: "r2", startedAt: 6_000, acceptedDisposition: "done", acceptedAt: 9_000 }, { runId: "r4", startedAt: 9_600, acceptedDisposition: "done", acceptedAt: 9_900 }]);
|
||||
expect(gradeStagingMix(nonce, { turnMinutes: 5, rounds: 4, subtasks: 0, followUpAfterMs: null, arriveAtMs: 0, holdMs: 0 },
|
||||
{ issueStatus: "done", documents: { ledger: ledger(nonce, 4) }, agentComments: [], files: {}, commands: {}, workspacePath: null,
|
||||
stagingMix: { followUp: null, followUpErrors: [], retries: [], comments: posted, completion: twice } }).checks.find(check => check.name === "completion_once")!.ok).toBe(false);
|
||||
});
|
||||
|
||||
it("reports a DONE comment from a run whose accepted result was not done as its own finding", () => {
|
||||
const runs = completion([{ runId: "r1", startedAt: 1, acceptedDisposition: "blocked", acceptedAt: 5_000 }, { runId: "r2", startedAt: 6_000, acceptedDisposition: "done", acceptedAt: 9_000 }]);
|
||||
expect(doneCommentsBeforeCompletion([{ body: doneMarker(nonce), runId: "r1", createdAt: 4_900 }, { body: doneMarker(nonce), runId: "r2", createdAt: 8_900 }], nonce, runs))
|
||||
.toEqual([{ runId: "r1", disposition: "blocked" }]);
|
||||
expect(doneCommentsBeforeCompletion([{ body: doneMarker(nonce), runId: "r1", createdAt: 4_900 }], nonce, { completedAt: 1, runs: [] })).toEqual([]);
|
||||
});
|
||||
|
||||
it("tells a parent not to post DONE until both subtasks are done and their hashes are recorded", () => {
|
||||
const text = stagingMixDescription(nonce, { turnMinutes: 5, rounds: 4, subtasks: 2, followUpAfterMs: null, arriveAtMs: 0, holdMs: 0 });
|
||||
expect(text).toContain(`end your turn now: do not save the \`ledger\` document, do not post the \`${doneMarker(nonce)}\` comment and do not mark this task done`);
|
||||
expect(text).toContain("Only after both subtasks are done and both CHILD lines are in the `ledger` document, post exactly one comment");
|
||||
expect(stagingMixDescription(nonce, { turnMinutes: 5, rounds: 4, subtasks: 0, followUpAfterMs: null, arriveAtMs: 0, holdMs: 0 })).toContain("Post exactly one comment");
|
||||
});
|
||||
});
|
||||
@@ -2,7 +2,9 @@ import { readFile, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import type { ChaosApi } from "./api.js";
|
||||
import { bannerShownMs, DUPLICATE_CHECKS, noteRetryBanner } from "./staging-mix.js";
|
||||
import {
|
||||
bannerShownMs, completedAtAnchor, doneCommentsBeforeCompletion, DUPLICATE_CHECKS, firstDoneCommentAt, noteRetryBanner, type CompletionEvidence,
|
||||
} from "./staging-mix.js";
|
||||
import type { InterruptionSample, OracleResult, TaskPlan, TaskRecord, TaskSample } from "./types.js";
|
||||
|
||||
type Json = Record<string, any>;
|
||||
@@ -170,6 +172,81 @@ export function stagingMixHeadline(result: StagingMixResult): string[] {
|
||||
];
|
||||
}
|
||||
|
||||
export interface ExtraTurn {
|
||||
taskId: string;
|
||||
runId: string | null;
|
||||
createdAt: number | null;
|
||||
startedAt: number | null;
|
||||
completedAt: number | null;
|
||||
completedAtSource: string | null;
|
||||
doneCommentAt: number | null;
|
||||
invocationSource: string | null;
|
||||
wakeReason: string | null;
|
||||
wakeSource: string | null;
|
||||
triggerDetail: string | null;
|
||||
actorType: string | null;
|
||||
actorId: string | null;
|
||||
}
|
||||
|
||||
export interface WakeEvidenceFile {
|
||||
completedAt?: number | null;
|
||||
runs?: Array<{ runId: string | null; createdAt: number | null; startedAt?: number | null; invocationSource: string | null; triggerDetail: string | null;
|
||||
wakeupRequestId: string | null; acceptedDisposition?: string | null; acceptedAt?: number | null }>;
|
||||
wakes?: Array<{ id: string | null; reason: string | null; source: string | null; triggerDetail: string | null; requestedByActorType: string | null; requestedByActorId: string | null }>;
|
||||
}
|
||||
|
||||
export type CommentRow = { body?: unknown; authorAgentId?: unknown; createdByRunId?: unknown; createdAt?: unknown };
|
||||
|
||||
export function commentEvidence(rows: readonly CommentRow[] | null): Array<{ body: string; runId: string | null; createdAt: number | null }> {
|
||||
return (rows ?? []).filter(row => row.authorAgentId).map(row => {
|
||||
const at = typeof row.createdAt === "string" ? Date.parse(row.createdAt) : NaN;
|
||||
return { body: String(row.body ?? ""), runId: typeof row.createdByRunId === "string" ? row.createdByRunId : null, createdAt: Number.isFinite(at) ? at : null };
|
||||
});
|
||||
}
|
||||
|
||||
export function evidenceCompletion(evidence: WakeEvidenceFile | null, issueCompletedAt?: unknown): CompletionEvidence | null {
|
||||
const fromIssue = typeof issueCompletedAt === "string" && Number.isFinite(Date.parse(issueCompletedAt)) ? Date.parse(issueCompletedAt) : null;
|
||||
if (!evidence && fromIssue === null) return null;
|
||||
return { completedAt: evidence?.completedAt ?? fromIssue, runs: (evidence?.runs ?? []).map(run => ({ runId: run.runId, startedAt: run.startedAt ?? null,
|
||||
acceptedDisposition: run.acceptedDisposition ?? null, acceptedAt: run.acceptedAt ?? null })) };
|
||||
}
|
||||
|
||||
export function runsAfterDone(taskId: string, evidence: WakeEvidenceFile | null, anchor: { at: number | null; source: string | null }, doneCommentAt: number | null = null):
|
||||
{ started: ExtraTurn[]; neverStarted: ExtraTurn[] } {
|
||||
const doneAt = anchor.at;
|
||||
if (!evidence || doneAt === null) return { started: [], neverStarted: [] };
|
||||
const wakes = new Map((evidence.wakes ?? []).map(wake => [wake.id, wake]));
|
||||
const turn = (run: NonNullable<WakeEvidenceFile["runs"]>[number]): ExtraTurn => {
|
||||
const wake = run.wakeupRequestId ? wakes.get(run.wakeupRequestId) : undefined;
|
||||
return { taskId, runId: run.runId, createdAt: run.createdAt, startedAt: run.startedAt ?? null, completedAt: doneAt, completedAtSource: anchor.source, doneCommentAt,
|
||||
invocationSource: run.invocationSource, wakeReason: wake?.reason ?? null, wakeSource: wake?.source ?? null, triggerDetail: wake?.triggerDetail ?? run.triggerDetail,
|
||||
actorType: wake?.requestedByActorType ?? null, actorId: wake?.requestedByActorId ?? null };
|
||||
};
|
||||
const runs = evidence.runs ?? [];
|
||||
return {
|
||||
started: runs.filter(run => run.startedAt != null && run.startedAt > doneAt).map(turn),
|
||||
neverStarted: runs.filter(run => run.startedAt == null && run.createdAt !== null && run.createdAt > doneAt).map(turn),
|
||||
};
|
||||
}
|
||||
|
||||
const iso = (at: number | null) => at === null ? "?" : new Date(at).toISOString();
|
||||
|
||||
export function extraTurnLine(turn: ExtraTurn): string {
|
||||
const reference = `completion ${iso(turn.completedAt)}${turn.completedAtSource && turn.completedAtSource !== "accepted_result" ? ` (${turn.completedAtSource})` : ""}`
|
||||
+ `${turn.doneCommentAt !== null && turn.doneCommentAt !== turn.completedAt ? `, DONE comment ${iso(turn.doneCommentAt)}` : ""}`;
|
||||
const when = turn.startedAt !== null ? `started ${iso(turn.startedAt)} after ${reference}`
|
||||
: `queued ${iso(turn.createdAt)} after ${reference}, never started`;
|
||||
return `- ${turn.taskId}: run ${turn.runId?.slice(0, 8) ?? "?"} ${when}; `
|
||||
+ `wake ${turn.wakeReason ?? "unknown"} (${turn.wakeSource ?? turn.invocationSource ?? "?"}${turn.triggerDetail ? `/${turn.triggerDetail}` : ""}) `
|
||||
+ `by ${turn.actorType ?? "unknown"}${turn.actorId ? ` ${turn.actorId}` : ""}`;
|
||||
}
|
||||
|
||||
export function extraTurnsByReason(turns: readonly ExtraTurn[]): Record<string, number> {
|
||||
const counts: Record<string, number> = {};
|
||||
for (const turn of turns) counts[turn.wakeReason ?? "unknown"] = (counts[turn.wakeReason ?? "unknown"] ?? 0) + 1;
|
||||
return counts;
|
||||
}
|
||||
|
||||
export async function readAssignedWorkInvariant(api: Pick<ChaosApi, "get">, companyId: string): Promise<StagingMixResult["invariant"]> {
|
||||
try {
|
||||
const report = await api.get<Json>(`/api/companies/${companyId}/assigned-work-invariant`, 30_000);
|
||||
@@ -185,8 +262,20 @@ export async function writeStagingMixResults(input: { waveDir: string; api: Pick
|
||||
tasks: ReadonlyArray<{ plan: TaskPlan; record: TaskRecord }>; timeline: ReadonlyArray<{ kind: string; [key: string]: unknown }> }): Promise<StagingMixResult> {
|
||||
const extraClicks = new Set(input.timeline.filter(entry => entry.kind === "release_swap_retry_clicked" && typeof entry.issueId === "string").map(entry => String(entry.issueId)));
|
||||
const tasks: ResultTask[] = [];
|
||||
const extraTurns: ExtraTurn[] = [];
|
||||
const queuedAfterDone: ExtraTurn[] = [];
|
||||
const doneBeforeCompletion: Array<{ taskId: string; runId: string; disposition: string }> = [];
|
||||
for (const task of input.tasks) {
|
||||
const issue = task.record.issueId ? await input.api.get<Json>(`/api/issues/${task.record.issueId}`, 10_000).catch(() => null) : null;
|
||||
const comments = task.record.issueId ? await input.api.get<Json[]>(`/api/issues/${task.record.issueId}/comments`, 10_000).catch(() => null) : null;
|
||||
const evidence = await readFile(path.join(input.waveDir, "wake-evidence", `${task.plan.taskId}.json`), "utf8")
|
||||
.then(text => JSON.parse(text) as WakeEvidenceFile).catch(() => null);
|
||||
const posted = commentEvidence(comments);
|
||||
const completion = evidenceCompletion(evidence, issue?.completedAt);
|
||||
const afterDone = runsAfterDone(task.plan.taskId, evidence, completedAtAnchor(completion, posted, task.plan.nonce), firstDoneCommentAt(posted, task.plan.nonce));
|
||||
doneBeforeCompletion.push(...doneCommentsBeforeCompletion(posted, task.plan.nonce, completion).map(entry => ({ taskId: task.plan.taskId, ...entry })));
|
||||
extraTurns.push(...afterDone.started);
|
||||
queuedAfterDone.push(...afterDone.neverStarted);
|
||||
const grade = await readFile(path.join(input.waveDir, `${task.plan.taskId}.grade.json`), "utf8")
|
||||
.then(text => JSON.parse(text) as { grade?: { T?: OracleResult } }).catch(() => null);
|
||||
tasks.push({ taskId: task.plan.taskId, issueId: task.record.issueId, record: task.record, finalStatus: issue?.status ?? task.record.finalStatus,
|
||||
@@ -194,13 +283,17 @@ export async function writeStagingMixResults(input: { waveDir: string; api: Pick
|
||||
}
|
||||
const result = summarizeStagingMix(tasks, extraClicks, await readAssignedWorkInvariant(input.api, input.companyId));
|
||||
await writeFile(path.join(input.waveDir, "staging-mix-results.json"), `${JSON.stringify({ schema: "paperclip.restart-chaos.staging-mix/v1",
|
||||
headline: stagingMixHeadline(result), ...result }, null, 2)}\n`, { mode: 0o600 });
|
||||
headline: stagingMixHeadline(result), ...result, extraTurns, extraTurnsByReason: extraTurnsByReason(extraTurns), queuedAfterDone, doneBeforeCompletion }, null, 2)}\n`, { mode: 0o600 });
|
||||
const invariant = result.invariant.violationCount === null ? `unreadable${result.invariant.error ? ` (${result.invariant.error})` : ""}`
|
||||
: `${result.invariant.violationCount} violations, ${result.invariant.agentsWithStalledSlots} agents with stalled slots, ${result.invariant.openTasks} open tasks`;
|
||||
await writeFile(path.join(input.waveDir, "staging-mix-summary.md"), [...stagingMixHeadline(result), `Assigned-work invariant: ${invariant}`, "",
|
||||
...result.duplicatedSideEffects.map(entry => `- ${entry.taskId}: duplicated ${entry.checks.join(", ")}`),
|
||||
...result.evidenceIncomplete.map(entry => `- ${entry.taskId}: evidence incomplete for ${entry.checks.join(", ")}`),
|
||||
...result.bannerShown.map(entry => `- ${entry.taskId}: Retry banner from ${entry.firstShownAt} for ${Math.round(entry.shownMs / 1000)} s`
|
||||
+ `${entry.errorCode ? ` (${entry.errorCode})` : ""}${entry.completedAnyway ? ", completed without a click" : ""}`), ""].join("\n"), { mode: 0o600 });
|
||||
+ `${entry.errorCode ? ` (${entry.errorCode})` : ""}${entry.completedAnyway ? ", completed without a click" : ""}`),
|
||||
...(extraTurns.length ? ["", `Runs started after the task completed: ${extraTurns.length}`, ...extraTurns.map(extraTurnLine)] : []),
|
||||
...(queuedAfterDone.length ? ["", `Runs queued after the task completed, never started: ${queuedAfterDone.length}`, ...queuedAfterDone.map(extraTurnLine)] : []),
|
||||
...(doneBeforeCompletion.length ? ["", `DONE comment posted before the task completed: ${doneBeforeCompletion.length}`,
|
||||
...doneBeforeCompletion.map(entry => `- ${entry.taskId}: run ${entry.runId.slice(0, 8)} posted DONE but its accepted result was ${entry.disposition}`)] : []), ""].join("\n"), { mode: 0o600 });
|
||||
return result;
|
||||
}
|
||||
@@ -54,6 +54,12 @@ export interface StagingMixRecord {
|
||||
retries?: Array<{ at: number; inOverlap: boolean }>;
|
||||
retryBanner?: RetryBanner;
|
||||
retryAfterSettle?: { at: number; accepted: boolean | null; issueStatusAtClick: string | null };
|
||||
completion?: CompletionEvidence;
|
||||
}
|
||||
|
||||
export interface CompletionEvidence {
|
||||
completedAt: number | null;
|
||||
runs: Array<{ runId: string | null; startedAt: number | null; acceptedDisposition: string | null; acceptedAt: number | null }>;
|
||||
}
|
||||
|
||||
export interface RetryBanner { firstShownAt: number; lastShownAt: number; shownMs: number; visibleSince: number | null; errorCode: string | null }
|
||||
@@ -62,8 +68,9 @@ export interface StagingMixArtifacts {
|
||||
followUp: StagingMixRecord["followUp"] | null;
|
||||
followUpErrors: string[];
|
||||
retries: NonNullable<StagingMixRecord["retries"]>;
|
||||
comments?: Array<{ body: string; runId: string | null }>;
|
||||
comments?: Array<{ body: string; runId: string | null; createdAt?: number | null }>;
|
||||
commentsUnreadable?: boolean;
|
||||
completion?: CompletionEvidence;
|
||||
}
|
||||
|
||||
export const DUPLICATE_CHECKS: ReadonlySet<string> = new Set(["rounds_once", "done_comment_once", "completion_once", "subtasks_once", "subtask_rounds_once"]);
|
||||
@@ -362,17 +369,58 @@ export function stagingMixDescription(nonce: string, task: StagingMixTask, agent
|
||||
lines.push(`${step++}. ${roundsSteps(nonce, task.rounds).join("\n ")}`);
|
||||
lines.push(`${step++}. If the board comments while you work and asks you to include a line, keep working and include that line on its own line in the document below.`);
|
||||
if (parent) {
|
||||
lines.push(`${step++}. Wait until both subtasks are done. If they are not done yet, end your turn without marking this task done; you will be woken when they finish.`,
|
||||
lines.push(`${step++}. Wait until both subtasks are done. If they are not done yet, end your turn now: do not save the \`ledger\` document, do not post the`
|
||||
+ ` \`${doneMarker(nonce)}\` comment and do not mark this task done. You will be woken when they finish.`,
|
||||
` Then read each subtask's \`ledger\` document and take the hash at the end of its last ROUND line.`);
|
||||
}
|
||||
lines.push(`${step++}. Save this issue's document with key \`ledger\` (title "Ledger"): the full contents of \`${ledgerFile(nonce)}\` exactly as the file reads,`
|
||||
+ ` then every line the board asked for${parent ? `, then one line \`${childLine(1, "<hash>")}\` per subtask (CHILD-2 for subtask 2), with <hash> from that subtask's ledger` : ""}.`);
|
||||
lines.push(`${step++}. Post exactly one comment on this task with the body \`${doneMarker(nonce)}\`. Then mark this task done.`);
|
||||
lines.push(`${step++}. ${parent ? "Only after both subtasks are done and both CHILD lines are in the `ledger` document, post" : "Post"} exactly one comment on this task`
|
||||
+ ` with the body \`${doneMarker(nonce)}\`. Then mark this task done.`);
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
export const followUpRequestId = (issueId: string, nonce: string) => stableRequestId(`followup-${issueId}-${nonce}`);
|
||||
|
||||
type CommentEvidence = { body: string; runId: string | null; createdAt?: number | null };
|
||||
|
||||
export function firstDoneCommentAt(comments: readonly CommentEvidence[], nonce: string): number | null {
|
||||
const times = comments.filter(comment => isDoneComment(comment.body, nonce) && typeof comment.createdAt === "number").map(comment => comment.createdAt!);
|
||||
return times.length ? Math.min(...times) : null;
|
||||
}
|
||||
|
||||
export function hasDispositions(completion: CompletionEvidence | null | undefined): completion is CompletionEvidence {
|
||||
return Boolean(completion?.runs.some(run => run.acceptedDisposition !== null));
|
||||
}
|
||||
|
||||
export function completedAtAnchor(completion: CompletionEvidence | null | undefined, comments: readonly CommentEvidence[], nonce: string):
|
||||
{ at: number | null; source: "accepted_result" | "completed_at" | "done_comment" | null } {
|
||||
const accepted = (completion?.runs ?? []).filter(run => run.acceptedDisposition === "done" && run.acceptedAt !== null).map(run => run.acceptedAt!);
|
||||
if (accepted.length) return { at: Math.min(...accepted), source: "accepted_result" };
|
||||
if (completion?.completedAt != null) return { at: completion.completedAt, source: "completed_at" };
|
||||
const comment = firstDoneCommentAt(comments, nonce);
|
||||
return { at: comment, source: comment === null ? null : "done_comment" };
|
||||
}
|
||||
|
||||
export function completionRunIds(comments: readonly CommentEvidence[], nonce: string, completion?: CompletionEvidence | null): string[] {
|
||||
const anchor = completedAtAnchor(completion, comments, nonce).at;
|
||||
const summaries = comments.filter(comment => anchor !== null && typeof comment.createdAt === "number" && comment.createdAt >= anchor
|
||||
&& comment.body.includes(doneMarker(nonce)) && comment.runId).map(comment => comment.runId!);
|
||||
if (hasDispositions(completion)) {
|
||||
const accepted = completion.runs.filter(run => run.acceptedDisposition === "done" && run.runId).map(run => run.runId!);
|
||||
return [...new Set([...accepted, ...summaries])];
|
||||
}
|
||||
return [...new Set([...comments.filter(comment => isDoneComment(comment.body, nonce) && comment.runId).map(comment => comment.runId!), ...summaries])];
|
||||
}
|
||||
|
||||
export function doneCommentsBeforeCompletion(comments: readonly CommentEvidence[], nonce: string, completion?: CompletionEvidence | null):
|
||||
Array<{ runId: string; disposition: string }> {
|
||||
if (!hasDispositions(completion)) return [];
|
||||
const dispositions = new Map(completion.runs.flatMap(run => run.runId ? [[run.runId, run.acceptedDisposition] as const] : []));
|
||||
return [...new Set(comments.filter(comment => isDoneComment(comment.body, nonce) && comment.runId).map(comment => comment.runId!))]
|
||||
.flatMap(runId => { const disposition = dispositions.get(runId); return disposition && disposition !== "done" ? [{ runId, disposition }] : []; });
|
||||
}
|
||||
|
||||
export function isDoneComment(body: string, nonce: string): boolean {
|
||||
return body.trim().replace(/\.$/, "").replace(/^`+|`+$/g, "").trim() === doneMarker(nonce);
|
||||
}
|
||||
@@ -412,7 +460,7 @@ export function gradeStagingMix(nonce: string, task: StagingMixTask, artifacts:
|
||||
const own = verifyLedger(nonce, task.rounds, ledger);
|
||||
const comments = artifacts.stagingMix?.comments ?? artifacts.agentComments.map(body => ({ body, runId: null }));
|
||||
const doneComments = comments.filter(comment => isDoneComment(comment.body, nonce)).length;
|
||||
const completionRuns = [...new Set(comments.filter(comment => comment.body.includes(doneMarker(nonce)) && comment.runId).map(comment => comment.runId!))];
|
||||
const completionRuns = completionRunIds(comments, nonce, artifacts.stagingMix?.completion);
|
||||
const followUp = artifacts.stagingMix?.followUp ?? null;
|
||||
const children = (artifacts.children ?? []).filter(child => (child.title ?? "").includes(nonce));
|
||||
const checks: OracleResult["checks"] = [
|
||||
@@ -573,8 +621,12 @@ export function bannerShownMs(banner: RetryBanner, now: number): number {
|
||||
return banner.shownMs + (banner.visibleSince === null ? 0 : now - banner.visibleSince);
|
||||
}
|
||||
|
||||
export function collectStagingMixArtifacts(record: TaskRecord, rows: ReadonlyArray<{ body?: unknown; authorAgentId?: unknown; createdByRunId?: unknown }> | null): StagingMixArtifacts {
|
||||
export function collectStagingMixArtifacts(record: TaskRecord,
|
||||
rows: ReadonlyArray<{ body?: unknown; authorAgentId?: unknown; createdByRunId?: unknown; createdAt?: unknown }> | null): StagingMixArtifacts {
|
||||
const time = (value: unknown) => typeof value === "string" && Number.isFinite(Date.parse(value)) ? Date.parse(value) : null;
|
||||
return { followUp: record.stagingMix?.followUp ?? null, followUpErrors: record.stagingMix?.followUpErrors ?? [], retries: record.stagingMix?.retries ?? [],
|
||||
...(rows ? { comments: rows.filter(row => row.authorAgentId).map(row => ({ body: String(row.body ?? ""), runId: typeof row.createdByRunId === "string" ? row.createdByRunId : null })) }
|
||||
...(record.stagingMix?.completion ? { completion: record.stagingMix.completion } : {}),
|
||||
...(rows ? { comments: rows.filter(row => row.authorAgentId).map(row => ({ body: String(row.body ?? ""),
|
||||
runId: typeof row.createdByRunId === "string" ? row.createdByRunId : null, createdAt: time(row.createdAt) })) }
|
||||
: { commentsUnreadable: true }) };
|
||||
}
|
||||
@@ -62,6 +62,16 @@ describe("wake evidence", () => {
|
||||
expect(notes).toEqual([["wake_evidence_failed", { error: "Error: database is gone" }]]);
|
||||
});
|
||||
|
||||
it("keeps only issue status transitions, not other actions' status fields (w-20261112T223803-s7101 t001)", () => {
|
||||
const evidence = wakeEvidenceForIssue("i1", { runs: [], wakes: [], controllers: [], windows: [], statusChanges: [
|
||||
{ issueId: "i1", action: "issue.created", actorType: "user", actorId: "local-board", fromStatus: null, toStatus: "todo", createdAt: 1_794_523_115_500 },
|
||||
{ issueId: "i1", action: "runner.api_called", actorType: "agent", runId: "37724b59", fromStatus: null, toStatus: "201", createdAt: 1_794_525_164_631 },
|
||||
{ issueId: "i1", action: "issue.updated", actorType: "agent", runId: "37724b59", fromStatus: "in_progress", toStatus: "done", createdAt: 1_794_525_165_000 },
|
||||
{ issueId: "i1", action: "issue.updated", actorType: "agent", fromStatus: null, toStatus: "200", createdAt: 1_794_525_166_000 },
|
||||
] });
|
||||
expect(evidence.statusChanges.map(change => [change.action, change.from, change.to])).toEqual([["issue.created", null, "todo"], ["issue.updated", "in_progress", "done"]]);
|
||||
});
|
||||
|
||||
it("joins each run to its wake, owner and status changes for one issue", () => {
|
||||
const evidence = wakeEvidenceForIssue("i1", {
|
||||
runs: [{ id: "r1", issueId: "i1", status: "succeeded", wakeupRequestId: "w2", controllerBootId: "boot-b", createdAt: 550_000, contextSnapshot: { wakeReason: "retry" } },
|
||||
|
||||
@@ -2,6 +2,8 @@ import { mkdir, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import type { ServerProcessWindow } from "./multi-server.js";
|
||||
import { unshiftJson, unshiftMs, type AgingShift } from "./time-faults.js";
|
||||
import { ISSUE_STATUSES } from "../../packages/shared/src/constants.js";
|
||||
import type { CompletionEvidence } from "./staging-mix.js";
|
||||
import { createDb, type Db } from "../../packages/db/src/index.js";
|
||||
|
||||
type Row = Record<string, unknown>;
|
||||
@@ -14,6 +16,12 @@ const text = (value: unknown) => value === null || value === undefined ? null :
|
||||
|
||||
export const PROCESS_MATCH_SLACK_MS = 60_000;
|
||||
|
||||
const STATUSES: ReadonlySet<string> = new Set(ISSUE_STATUSES);
|
||||
|
||||
export function isIssueStatusTransition(row: Row): boolean {
|
||||
return String(row.action ?? "").startsWith("issue.") && STATUSES.has(String(row.toStatus ?? "")) && (row.fromStatus == null || STATUSES.has(String(row.fromStatus)));
|
||||
}
|
||||
|
||||
export function boundByProcessExit(controllers: readonly ControllerRow[], windows: readonly ServerProcessWindow[]): ControllerRow[] {
|
||||
return controllers.map(controller => {
|
||||
const window = windows.filter(candidate => candidate.pid === controller.pid && candidate.spawnedAt <= controller.startedAt + PROCESS_MATCH_SLACK_MS
|
||||
@@ -51,6 +59,7 @@ export interface WakeEvidenceInput {
|
||||
statusChanges: readonly Row[];
|
||||
controllers: readonly ControllerRow[];
|
||||
windows: readonly ServerProcessWindow[];
|
||||
issues?: readonly Row[];
|
||||
}
|
||||
|
||||
export function wakeEvidenceForIssue(issueId: string, input: WakeEvidenceInput) {
|
||||
@@ -61,11 +70,12 @@ export function wakeEvidenceForIssue(issueId: string, input: WakeEvidenceInput)
|
||||
const controllerByBoot = new Map(controllers.map(controller => [controller.bootId, controller]));
|
||||
return {
|
||||
issueId,
|
||||
completedAt: ms(input.issues?.find(issue => issue.id === issueId)?.completedAt),
|
||||
runs: runs.map(run => {
|
||||
const owner = controllerByBoot.get(String(run.controllerBootId ?? ""));
|
||||
return { runId: text(run.id), status: text(run.status), invocationSource: text(run.invocationSource), triggerDetail: text(run.triggerDetail),
|
||||
wakeupRequestId: text(run.wakeupRequestId), createdAt: ms(run.createdAt), startedAt: ms(run.startedAt), finishedAt: ms(run.finishedAt),
|
||||
contextSnapshot: run.contextSnapshot ?? null,
|
||||
acceptedDisposition: text(run.acceptedDisposition), acceptedAt: ms(run.acceptedAt), contextSnapshot: run.contextSnapshot ?? null,
|
||||
owner: owner ? { pid: owner.pid, bootId: owner.bootId, epoch: owner.epoch, processSpace: owner.processSpace } : null,
|
||||
servingAtCreate: servingAt(ms(run.createdAt), controllers, input.windows) };
|
||||
}),
|
||||
@@ -74,7 +84,7 @@ export function wakeEvidenceForIssue(issueId: string, input: WakeEvidenceInput)
|
||||
status: text(wake.status), runId: text(wake.runId), payload: wake.payload ?? null, error: text(wake.error),
|
||||
createdAt: ms(wake.createdAt), requestedAt: ms(wake.requestedAt), claimedAt: ms(wake.claimedAt), finishedAt: ms(wake.finishedAt),
|
||||
servingAtCreate: servingAt(ms(wake.createdAt), controllers, input.windows) })),
|
||||
statusChanges: input.statusChanges.filter(change => change.issueId === issueId).map(change => ({ at: ms(change.createdAt), action: text(change.action),
|
||||
statusChanges: input.statusChanges.filter(change => change.issueId === issueId && isIssueStatusTransition(change)).map(change => ({ at: ms(change.createdAt), action: text(change.action),
|
||||
actorType: text(change.actorType), actorId: text(change.actorId), runId: text(change.runId), from: text(change.fromStatus), to: text(change.toStatus) })),
|
||||
};
|
||||
}
|
||||
@@ -82,11 +92,18 @@ export function wakeEvidenceForIssue(issueId: string, input: WakeEvidenceInput)
|
||||
async function readRows(db: Pick<Db, "$client">, companyId: string, shifts: readonly AgingShift[]): Promise<Omit<WakeEvidenceInput, "windows">> {
|
||||
const client = db.$client;
|
||||
const runs = await client.unsafe(
|
||||
`select id::text as id, issue_id::text as "issueId", status, invocation_source as "invocationSource", trigger_detail as "triggerDetail",
|
||||
wakeup_request_id::text as "wakeupRequestId", context_snapshot as "contextSnapshot", controller_boot_id::text as "controllerBootId",
|
||||
floor(extract(epoch from created_at) * 1000) as "createdAt", floor(extract(epoch from started_at) * 1000) as "startedAt",
|
||||
floor(extract(epoch from finished_at) * 1000) as "finishedAt"
|
||||
from heartbeat_runs where company_id::text = $1 order by created_at`, [companyId]) as Row[];
|
||||
`select r.id::text as id, r.issue_id::text as "issueId", r.status, r.invocation_source as "invocationSource", r.trigger_detail as "triggerDetail",
|
||||
r.wakeup_request_id::text as "wakeupRequestId", r.context_snapshot as "contextSnapshot", r.controller_boot_id::text as "controllerBootId",
|
||||
floor(extract(epoch from r.created_at) * 1000) as "createdAt", floor(extract(epoch from r.started_at) * 1000) as "startedAt",
|
||||
floor(extract(epoch from r.finished_at) * 1000) as "finishedAt", accepted.disposition as "acceptedDisposition",
|
||||
floor(extract(epoch from accepted.accepted_at) * 1000) as "acceptedAt"
|
||||
from heartbeat_runs r
|
||||
left join lateral (select coalesce(e.payload->'prpEvent'->'payload'->'result'->>'reportedWorkDisposition', e.payload->'result'->>'reportedWorkDisposition') as disposition, e.created_at as accepted_at
|
||||
from heartbeat_run_events e where e.run_id = r.id and e.event_type = 'run.result.accepted'
|
||||
order by e.seq desc limit 1) accepted on true
|
||||
where r.company_id::text = $1 order by r.created_at`, [companyId]) as Row[];
|
||||
const issues = await client.unsafe(
|
||||
`select id::text as id, floor(extract(epoch from completed_at) * 1000) as "completedAt" from issues where company_id::text = $1`, [companyId]) as Row[];
|
||||
const wakes = await client.unsafe(
|
||||
`select id::text as id, payload->>'issueId' as "issueId", source, trigger_detail as "triggerDetail", reason, payload, status,
|
||||
requested_by_actor_type as "requestedByActorType", requested_by_actor_id as "requestedByActorId", idempotency_key as "idempotencyKey",
|
||||
@@ -98,15 +115,16 @@ async function readRows(db: Pick<Db, "$client">, companyId: string, shifts: read
|
||||
`select entity_id as "issueId", action, actor_type as "actorType", actor_id as "actorId", run_id::text as "runId",
|
||||
coalesce(details->'_previous'->>'status', details->>'previousStatus') as "fromStatus", details->>'status' as "toStatus",
|
||||
floor(extract(epoch from created_at) * 1000) as "createdAt"
|
||||
from activity_log where company_id::text = $1 and entity_type = 'issue'
|
||||
and (details ? 'status' or details->'_previous' ? 'status' or details ? 'previousStatus') order by created_at`, [companyId]) as Row[];
|
||||
from activity_log where company_id::text = $1 and entity_type = 'issue' and action like 'issue.%'
|
||||
and details ? 'status' order by created_at`, [companyId]) as Row[];
|
||||
const controllers = (await client.unsafe(
|
||||
`select boot_id as "bootId", epoch, process_space as "processSpace", pid, floor(extract(epoch from started_at) * 1000) as "startedAt",
|
||||
floor(extract(epoch from coalesce(stopped_at, fenced_at)) * 1000) as "endedAt" from controller_instances order by epoch`) as Row[])
|
||||
.map(row => ({ bootId: String(row.bootId), epoch: Number(row.epoch), processSpace: String(row.processSpace), pid: Number(row.pid),
|
||||
startedAt: Number(row.startedAt), endedAt: ms(row.endedAt) }));
|
||||
return {
|
||||
runs: unshiftRows(runs, ["createdAt", "startedAt", "finishedAt"], ["contextSnapshot"], shifts),
|
||||
runs: unshiftRows(runs, ["createdAt", "startedAt", "finishedAt", "acceptedAt"], ["contextSnapshot"], shifts),
|
||||
issues: unshiftRows(issues, ["completedAt"], [], shifts),
|
||||
wakes: unshiftRows(wakes, ["createdAt", "requestedAt", "claimedAt", "finishedAt"], ["payload"], shifts),
|
||||
statusChanges: unshiftRows(statusChanges, ["createdAt"], [], shifts),
|
||||
controllers: controllers.map(controller => ({ ...controller, startedAt: unshiftMs(controller.startedAt, shifts),
|
||||
@@ -115,10 +133,10 @@ async function readRows(db: Pick<Db, "$client">, companyId: string, shifts: read
|
||||
}
|
||||
|
||||
export async function captureWakeEvidence(input: { db: Pick<Db, "$client">; companyId: string; waveDir: string;
|
||||
observers: ReadonlyArray<{ plan: { taskId: string }; record: { issueId: string | null } }> | null; windows: readonly ServerProcessWindow[];
|
||||
observers: ReadonlyArray<EvidenceObserver> | null; windows: readonly ServerProcessWindow[];
|
||||
shifts?: readonly AgingShift[]; redact: (text: string) => string }): Promise<number> {
|
||||
const rows = await readRows(input.db, input.companyId, input.shifts ?? []);
|
||||
const observers = input.observers ?? [...new Set(rows.runs.map(run => text(run.issueId)).filter((id): id is string => id !== null))]
|
||||
const observers: readonly EvidenceObserver[] = input.observers ?? [...new Set(rows.runs.map(run => text(run.issueId)).filter((id): id is string => id !== null))]
|
||||
.map(issueId => ({ plan: { taskId: `issue-${issueId}` }, record: { issueId } }));
|
||||
const directory = path.join(input.waveDir, "wake-evidence");
|
||||
await mkdir(directory, { recursive: true, mode: 0o700 });
|
||||
@@ -127,6 +145,7 @@ export async function captureWakeEvidence(input: { db: Pick<Db, "$client">; comp
|
||||
if (!observer.record.issueId) continue;
|
||||
const evidence = { schema: "paperclip.restart-chaos.wake-evidence/v1", taskId: observer.plan.taskId,
|
||||
...wakeEvidenceForIssue(observer.record.issueId, { ...rows, windows: input.windows }) };
|
||||
if (observer.plan.stagingMix) (observer.record.stagingMix ??= {}).completion = completionEvidence(evidence);
|
||||
await writeFile(path.join(directory, `${observer.plan.taskId}.json`), input.redact(`${JSON.stringify(evidence, null, 2)}\n`), { mode: 0o600 });
|
||||
written += 1;
|
||||
}
|
||||
@@ -134,8 +153,18 @@ export async function captureWakeEvidence(input: { db: Pick<Db, "$client">; comp
|
||||
return written;
|
||||
}
|
||||
|
||||
export interface EvidenceObserver {
|
||||
plan: { taskId: string; stagingMix?: unknown };
|
||||
record: { issueId: string | null; stagingMix?: { completion?: CompletionEvidence } };
|
||||
}
|
||||
|
||||
export function completionEvidence(evidence: ReturnType<typeof wakeEvidenceForIssue>): CompletionEvidence {
|
||||
return { completedAt: evidence.completedAt, runs: evidence.runs.map(run => ({ runId: run.runId, startedAt: run.startedAt,
|
||||
acceptedDisposition: run.acceptedDisposition, acceptedAt: run.acceptedAt })) };
|
||||
}
|
||||
|
||||
export function wakeEvidenceOnce(input: { databaseUrl: () => string; companyId: string; waveDir: string;
|
||||
observers: () => ReadonlyArray<{ plan: { taskId: string }; record: { issueId: string | null } }>; windows: readonly ServerProcessWindow[];
|
||||
observers: () => ReadonlyArray<EvidenceObserver>; windows: readonly ServerProcessWindow[];
|
||||
shifts: readonly AgingShift[]; redact: (text: string) => string; note: (kind: string, data: Record<string, unknown>) => void;
|
||||
open?: (url: string) => Pick<Db, "$client"> }): () => Promise<void> {
|
||||
let done: Promise<void> | null = null;
|
||||
|
||||
Reference in new issue
Block a user