mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-06 05:56:19 +01:00
Merge current master lifecycle and CI fixes into Claude behavior
Co-Authored-By: Paperclip <noreply@paperclip.ing> * codex/claude-runner-protocol: fix(grok): avoid routine wakes for proactive admission (#16422) feat(ci): profile full CODEOWNER PR suites in Honeycomb (#16378) test: bound command snapshot forwarding checks (#16421) fix(agents): report actionable lifecycle failure steps (#16418)
This commit is contained in:
DottaandPaperclip committed
commit
bd3ceab2b2
50 files changed
+1630
-60
No files matched your search
@@ -75,6 +75,10 @@ When changing the workflow path or branch, authorize the new reference before
|
||||
updating the caller. Retain older authorized SHA references while queued runs or
|
||||
supported reruns still use them.
|
||||
|
||||
For a full wall-clock profile of all main PR CI jobs, add `profile-ci` to the PR.
|
||||
See [CI profiling](ci-profiling.md) for the Honeycomb waterfall, coverage,
|
||||
operator setup, and activation requirements.
|
||||
|
||||
## Start Dev
|
||||
|
||||
From repo root:
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
# Profile the complete PR suite
|
||||
|
||||
Add the `profile-ci` label to a PR authored by a CI CODEOWNER. Adding the label starts a fresh `PR` workflow
|
||||
run. Further pushes while the label remains also run with profiling enabled.
|
||||
The run is named **CI profile · PR #number**. It runs all lanes of the main PR
|
||||
suite, including middle-of-stack and release-notes-only PRs that normally skip
|
||||
most lanes.
|
||||
|
||||
After the run finishes, open its Summary and follow **Open the Honeycomb
|
||||
waterfall**. The separate **CI profile** workflow exports the trace after all
|
||||
jobs finish. Its summary has the exact time range, job/span counts, and coverage
|
||||
warnings. Keep the label on the PR and its head unchanged until export finishes.
|
||||
Remove the label to stop profiling subsequent pushes. To profile the same
|
||||
revision again, remove and re-add the label or rerun all jobs.
|
||||
|
||||
The PR author, original workflow actor, and rerun actor must all be individual
|
||||
users in the trusted `master` branch's `.github/**` CODEOWNERS rule. GitHub user
|
||||
IDs are resolved from that rule; PR edits to CODEOWNERS cannot grant access.
|
||||
A label on an outside contributor's PR does not enable profiling, even when a
|
||||
maintainer applies it. Failed identity lookups disable profiling.
|
||||
|
||||
Ordinary PRs do not write timing artifacts or export traces. The exporter job is
|
||||
skipped for ordinary run names. Adding another label does not run the CI matrix.
|
||||
|
||||
Profiling preserves the existing trusted AWS runner routing. Eligible PR jobs
|
||||
use `paperclip-public-pr-x64`; the routing gate uses `paperclip-ci-routing-x64`.
|
||||
The existing fail-closed GitHub-hosted fallback still applies if AWS routing
|
||||
cannot be authorized. Only the separate post-run Honeycomb exporter uses
|
||||
`ubuntu-latest` directly; it is outside the measured PR suite.
|
||||
|
||||
## Read the waterfall
|
||||
|
||||
The workflow root measures elapsed time from run creation to the last job's
|
||||
completion. For reruns, it starts at that attempt's `run_started_at` instead of
|
||||
the original creation time. Every job is a child on the same wall clock, with
|
||||
its steps and available test details beneath it. All spans use a sample rate of
|
||||
one. There is no sampling within the selected attempt. For a compact Gantt view,
|
||||
open a job row's menu and choose **Collapse spans at this depth**. Expand an
|
||||
individual job to inspect its steps and test files.
|
||||
|
||||
The expandable hierarchy is **workflow → job → step → test filename → tests**.
|
||||
Vitest, Playwright, and Node file rows use repository-relative filenames, also
|
||||
available in `test.file`. Separate files retain their own children even when
|
||||
individual test names repeat across files.
|
||||
|
||||
- Compare the jobs that finish last (`ci.finishes_last`) with other overlapping
|
||||
work. A long job that finishes early is not necessarily delaying CI.
|
||||
- Expand setup, dependency installation, cache restore, Rust compilation, builds,
|
||||
and tests to find repeated or slow work.
|
||||
- `ci.execution_ms` measures a job's execution. `ci.start_offset_ms` shows when
|
||||
it started relative to the whole run. Do not add parallel job durations to
|
||||
calculate elapsed time.
|
||||
- Waiting spans cover job creation to runner start. Dependency waits can occur
|
||||
before GitHub creates a job, so they appear as gaps from the workflow start.
|
||||
GitHub does not expose a separate ready timestamp; do not interpret every gap
|
||||
as runner queue time or claim this is a computed dependency critical path.
|
||||
|
||||
GitHub job/step timestamps have second precision. Vitest and Playwright record
|
||||
test timestamps at the runner. Node's test reporter records completion receipt
|
||||
time and duration (`test.clock=reporter-receipt`); buffered IPC delivery can shift
|
||||
those small test spans. Vitest file spans include its test/hook execution and
|
||||
carry collection/setup durations as fields. Playwright file spans cover the
|
||||
first through last test attempt in that file. Rust and standalone script checks
|
||||
retain their full job/step timings; stable libtest does not expose per-test wall
|
||||
timestamps. The trace is CI timing data, not application tracing.
|
||||
|
||||
Scope includes all main PR lanes: policy, typecheck/release registry, general and
|
||||
serialized tests, Runner checks, Docker context checks, build, canary dry run,
|
||||
browser E2E, and aggregate checks. Separate Storybook, image, deployment, release,
|
||||
and live-provider eval workflows are outside this trace.
|
||||
|
||||
## Failure, coverage, and reruns
|
||||
|
||||
The exporter uses GitHub's attempt-specific job API and paginates jobs and
|
||||
artifacts. It includes failed and cancelled jobs and uploads available test
|
||||
timings with `always()`. A force-cancelled or failed setup job may leave no
|
||||
artifact. Coverage warnings and `ci.profile_complete=false` make missing
|
||||
artifacts, interrupted reporters, rejected data, and absent test details visible.
|
||||
Test launchers write an expected-invocation record before starting each Vitest,
|
||||
Playwright, or Node process. Each reporter must write matching start/end records,
|
||||
so a missing reporter cannot hide behind another successful invocation in the
|
||||
same job. Compare `ci.expected_invocations` with `ci.reported_invocations`.
|
||||
Failed or cancelled workflows are conservatively incomplete because a failed
|
||||
command can prevent later tests or steps from running.
|
||||
|
||||
Each attempt gets a different trace ID. Rerunning failed jobs alone produces an
|
||||
attempt trace for those jobs; it does not replay completed jobs. Reruns are marked
|
||||
`ci.partial_rerun=true` conservatively. Use a fresh label-triggered run for an
|
||||
unambiguous full-suite baseline. GitHub may carry earlier successful jobs into
|
||||
the new attempt's job inventory. Jobs completed before the new attempt started
|
||||
are excluded and counted in `ci.carried_job_count`.
|
||||
|
||||
If export fails, inspect **CI profile** and rerun that exporter after fixing the
|
||||
cause. A partial HTTP export may already have sent some spans; retries use the
|
||||
same IDs but the Events API does not promise deduplication. Timing artifacts
|
||||
remain available for 14 days. Invalid/oversized artifacts stop export instead of
|
||||
silently truncating the suite. No logs, environment dumps, error messages, stack
|
||||
traces, or credentials are included in the exported field allowlist.
|
||||
|
||||
## Operator setup and activation
|
||||
|
||||
Use the existing Honeycomb `paperclip` team, `test` environment, and a separate
|
||||
service dataset named `paperclip-ci`. Create GitHub environment `ci-profiling` with **Selected branches and tags**:
|
||||
allow only the branch `master`, with no PR-ref or tag patterns. Store environment
|
||||
secret `HONEYCOMB_CI_API_KEY` with a dedicated ingest-only key for the Honeycomb
|
||||
`test` environment. Do not create a repository- or organization-level secret of
|
||||
that name: the environment restriction prevents in-repository PR workflows from
|
||||
requesting the key. Tests still execute the PR branch; only the separate exporter
|
||||
runs from `master`. It needs event
|
||||
ingest permission; dataset creation permission is needed only if the dataset
|
||||
has not been created yet. Do not store the environment configuration key in CI.
|
||||
Create the repository label `profile-ci` if it does not exist.
|
||||
|
||||
`ci-profile.yml` runs from the trusted default-branch revision. It checks out only
|
||||
the exporter, installs no dependencies, and parses PR artifacts as bounded JSON
|
||||
without extracting or executing their contents. The PR jobs never receive the
|
||||
Honeycomb key. The exporter independently verifies CI CODEOWNER identities,
|
||||
the current label, repository, and exact tested PR revision before reading
|
||||
artifacts or sending data. Its credential is sent only in an HTTPS header to
|
||||
`api.honeycomb.io`; redirects are rejected. Exported fields are allowlisted.
|
||||
`CODEOWNERS` also covers `scripts/ci-profile/**`, so changes to the privileged
|
||||
exporter require infrastructure-owner review.
|
||||
|
||||
The new trusted workflow and routing definitions must merge to master before
|
||||
the label-triggered journey is active. A PR uses `pr-trusted.yml@master` and
|
||||
`ci-routing.yml@master`; do not change those trust references just to test this
|
||||
feature before merge.
|
||||
|
||||
## Local qualification
|
||||
|
||||
```sh
|
||||
node --test .github/scripts/tests/ci-profile.test.mjs .github/scripts/tests/ci-routing.test.mjs
|
||||
node --test scripts/ci-profile/reporters.test.mjs
|
||||
actionlint .github/workflows/{pr,pr-trusted,ci-routing,ci-profile}.yml
|
||||
node scripts/ci-profile/export.mjs --run-id=RUN_ID --attempt=1 --historical --dry-run --output=/tmp/ci-trace.json
|
||||
```
|
||||
|
||||
`--historical` is local-only and permits a deliberate export of an existing
|
||||
unlabeled run. It has complete GitHub job/step timing but cannot reconstruct
|
||||
test details that were never recorded. Omit `--dry-run` and supply
|
||||
`HONEYCOMB_CI_API_KEY` to send the selected baseline to Honeycomb. It is labeled
|
||||
as historical/incomplete test coverage; it is not proof of the label-triggered
|
||||
workflow.
|
||||
|
||||
References: [Honeycomb trace fields](https://docs.honeycomb.io/get-started/best-practices/organizing-data/),
|
||||
[Events API](https://docs.honeycomb.io/api/events/create-events),
|
||||
[GitHub workflow_run](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#workflow_run).
|
||||
@@ -29,8 +29,11 @@ paste it into Paperclip's authorization page. Keep it out of Bot chat.
|
||||
Assign ordinary tasks to the agent. Read its progress, answer its question cards,
|
||||
and review its documents and files in the task thread. The Bot can initiate a
|
||||
new task from its Grok conversation through the admitted work-session tools.
|
||||
An immediate proactive claim does not send the same assignment back through the
|
||||
routine.
|
||||
The originating conversation checks the same request while admission is queued.
|
||||
Neither immediate nor delayed Bot-initiated admission sends that assignment back
|
||||
through the routine. Grok keeps the request ID stable until admission succeeds
|
||||
or fails. Human-assigned work and continuations after an answered question still
|
||||
use their normal assignment wakes.
|
||||
|
||||
## Files and completion
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Grok Bot production readiness
|
||||
|
||||
Status as of October 15: a real assigned task now reaches canonical completion. Qualification found an active-session wake hazard; its correction is implemented and passing local checks. Production acceptance is incomplete. The current release goal and matrix below supersede the earlier dated campaign entries. Historical successes and failures remain evidence for the revisions actually tested.
|
||||
Status as of October 15: real assigned and Bot-initiated tasks reach canonical completion. The active-session wake correction is merged. A real proactive test found a delayed-admission echo; its correction is under verification. Production acceptance is incomplete. The current release goal and matrix below supersede the earlier dated campaign entries. Historical successes and failures remain evidence for the revisions actually tested.
|
||||
|
||||
## Current production finish line — October 14
|
||||
|
||||
@@ -12,12 +12,12 @@ There is no testing call, case-count, token or spending cap. The former six-case
|
||||
| --- | --- | --- |
|
||||
| Ordinary onboarding works | Actual Grok UI, fixed agent consent, saved setup, wrong-key behavior, authenticated challenge, refresh, reconnect and revocation | Resumed real Grok authorization, saved setup Done, authenticated routine challenge and automatic refresh passed on the October 15 serving revision. Runtime reconnect copy fix is merged; verify its published UI and fresh production setup next. |
|
||||
| Assigned work reaches a usable result | One actual managed workflow: progress, question and answer, document, downloaded file/hash, canonical completion or review and cleanup | A preliminary comparison task on serving Core ff0 completed progress → question → answer → report/CSV → canonical Done, with both cleanup receipts. The cited CSV downloads with the exact publication hash. Two distinct CSV uploads accompanied an active-session result wake; correct that wake behavior and requalify the final candidate. |
|
||||
| Bot initiates governed work | Real-client task creation/claim/completion with stable receipts and no outbound echo or duplicate effects | Unverified. Run after the assigned-work gate. |
|
||||
| Bot initiates governed work | Real-client task creation/claim/completion with stable receipts and no outbound echo or duplicate effects | A real idle-conversation request created one task and one managed run, saved a 100-word report, reached accepted Done and cleaned up its lease. A delayed-admission wake and two acceptance attempts were also recorded. Functional completion passed; no-echo acceptance failed. Requalify the correction. |
|
||||
| Recovery and authority are reliable | Healthy work past the prior five-minute failure window; actual controller restart preserving assignment/session/process and no duplicate wake; Stop, pause, reassignment, revocation, stale calls, queues and cleanup | The old hold was inspected and cleared. The successful comparison continuation ended below five minutes. Healthy work beyond that window, an actual restart and active Stop remain unverified; protocol tests are separate evidence. |
|
||||
| One identifiable reviewed candidate passes source gates | Focused protocol/auth/Runner/UI tests, full CI/typecheck/tests/build/token/Storybook evidence and exact artifacts | The active-session wake correction follows origin/master. Initial local checks used d6e4b36c8c049b85813caf3e3cbf1f8d87e4e068; CI and review must verify the final PR head. Three regressions failed before the fix. Focused tests, repository typecheck/build, token gates and Storybook checks pass. Full CI, review and publication of this correction remain required; prior candidate receipts are historical. |
|
||||
| One identifiable reviewed candidate passes source gates | Focused protocol/auth/Runner/UI tests, full CI/typecheck/tests/build/token/Storybook evidence and exact artifacts | Active-session wake correction #16394 merged with 55 passing checks, two optional skips and review 5/5. The proactive echo correction started from master 7fc2dc8b3236e1149ae32756b6153a503623404b and was rebased onto freshly fetched master 250429942e27d568f20d3bc7dddcfc84a666480a before submission. Two regression cases reproduce the real delayed-admission wake before the fix. All 214 focused protocol/auth/question checks pass, including 80 Grok and Dot checks. Repository typecheck/build, token gates, the 2,731-story structure and Storybook build pass. Fresh CI, review, publication and the real no-echo repeat remain required. |
|
||||
| Production opt-in is delivered and smoke-tested | Staging acceptance, correct served Core/distribution/Runner, normal editable opt-in controls/defaults, actual production Bot workflow and inspectable task/output links | Cloud opt-in policy and release-guide corrections are merged. Staging Cloud provenance and minimal feature values were verified, with Dot disabled and defaults unchanged. The corrected wake candidate, remaining real-client cases, ordinary production opt-in controls and production smoke are outstanding. |
|
||||
|
||||
The critical path is to review and publish the active-session wake correction, verify its served identity, then complete the remaining real-client acceptance cases. The selected staging workspace remains the test destination. Preserve one source manifest per case; do not combine incompatible runs into one green release.
|
||||
The critical path is to review and publish the proactive-admission echo correction, verify its served identity, then complete the remaining real-client acceptance cases. The selected staging workspace remains the test destination. Preserve one source manifest per case; do not combine incompatible runs into one green release.
|
||||
|
||||
Real Grok reconnect restored the same agent, plugin and webhook-only routine. Saved setup reached Done after both delivery and authenticated MCP acknowledgement. After the initial access token expired, Grok refreshed it and acknowledged another challenge without reauthorization. Grok/MCP/Runner were enabled with Dot disabled; setup allocated no managed Runner. The operator used the existing saved-invitation path to get the secure pairing-code copy control before the Runtime copy correction published. Record that as assisted recovery, not proof of unassisted fresh onboarding. The copy correction is merged in #16373 after 55 passing CI checks, two optional skips and review 5/5.
|
||||
|
||||
@@ -27,7 +27,11 @@ An unread asynchronous tool result also caused a webhook during accepted work. T
|
||||
|
||||
Core d73d7a8651db0897cb50adbca42731bcb8aed550 and its canonical private Cloud composition passed publication. One single-workspace request passed preparation but was blocked by an existing shared rollout; it created no campaign. The existing rollout later released the reservation. That prepared source is now superseded by the wake correction. Keep its receipts as history and do not resume its older deployment request. Select a new immutable reviewed source after this fix passes CI and review. Production opt-in policy and release-guide changes are merged, but production rollout and smoke remain outstanding.
|
||||
|
||||
Desktop control stopped responding after the successful exchanges. Resetting the tool did not restore it, and the user has been asked to reopen Codex. Browser control and webhook/MCP work remain usable; native Bot-initiated prompts and fresh production setup require restored desktop control or ordinary human setup actions. The dedicated production workspace has no qualification agent grant or smoke task.
|
||||
Desktop control recovered on October 15. One ordinary message in the actual Grok conversation initiated the proactive source investigation; the operator created no preliminary task and sent no extra nudge. Serving Core 11d69d04c7531cf7eb37f7994b7112797e145837 includes #16394. The Bot created one task and one run, saved a 100-word report, submitted an accepted canonical result and reached Done. Its managed lease has a successful cleanup receipt. The exact public PR marker matches the saved control value, was absent from the operator prompt and appears in the saved report with its source. No Paperclip proxy operation occurred. The report records native GitHub API use; a complete provider tool transcript is unavailable.
|
||||
|
||||
The proactive run also received an authenticated delayed-admission webhook while its original conversation was waiting. Two acceptance requests and two title writes were recorded. This fails the no-echo release condition despite successful task completion. Two local regressions reproduce the extra webhook for a delayed offer and a prepared intent recovered after restart. The correction keeps admission in the original caller's polling path and rejects old prepared proactive wakes. Human assignments and question continuations retain their ordinary dispatch paths. Source and real-client requalification are required.
|
||||
|
||||
Normal staging automation has advanced beyond the older prepared candidate. No shared CI routing switch was changed. The earlier temporary fallback request is no longer needed for the already served successor. The dedicated production workspace still has no qualification agent grant or smoke task.
|
||||
|
||||
Keep native providers and person-purpose MCP connections unchanged. No browser automation belongs in normal operation. Defer localhost relays, internal Grok APIs, marketplace distribution and Paperclip-connected third-party tools. Completion of a Paperclip work session does not assert remote Grok termination.
|
||||
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# Opt-in CI profiling
|
||||
|
||||
Goal: label a PR `profile-ci`, run the complete main PR suite, and open one
|
||||
Honeycomb waterfall showing the full elapsed timeline. Use that evidence to
|
||||
decide which CI work to speed up or remove. CI optimizations themselves are a
|
||||
follow-up, not part of this change.
|
||||
|
||||
## Acceptance
|
||||
|
||||
| Case | Required behavior | Evidence |
|
||||
| --- | --- | --- |
|
||||
| Unlabeled PR | Existing scope and commands; no profile export | Routing and exporter tests |
|
||||
| Labeled CI CODEOWNER PR, including stacked/release-notes-only | All main PR lanes run | Routing tests and full-run job inventory |
|
||||
| Completed profile | One trace per run attempt; every job and step uses actual timestamps | Honeycomb waterfall reconciled against GitHub job API |
|
||||
| Parallel work | Shared wall clock, with waiting and execution distinguishable | Overlapping lanes and root duration checked against GitHub |
|
||||
| Tests | Vitest and Playwright file/test timings under their job/step; Node test timings where supported | Real reporter smoke runs and hierarchy checks |
|
||||
| Failure/cancellation/rerun | Preserve completed timing data; mark missing coverage; attempts stay separate | Exporter regression cases |
|
||||
| Credentials | Master-only environment owns ingest secret; trusted exporter verifies CI CODEOWNER author and actors; PR jobs only write timing artifacts | Workflow review, hostile-identity cases, and environment settings |
|
||||
|
||||
Scope: the main `PR` workflow and every lane of `pr-trusted.yml`, including
|
||||
policy, build, typecheck/release-registry, general/serialized test shards,
|
||||
Runner checks, canary, and browser E2E. Separate release/deploy/live-provider
|
||||
workflows remain outside this first trace. Rust checks are visible as complete
|
||||
job/step timings; stable libtest does not provide per-test wall-clock reports.
|
||||
No sampling inside a selected run.
|
||||
|
||||
## State
|
||||
|
||||
- Branch: `codex/ci-honeycomb-profiling` in the existing `76aa` worktree.
|
||||
- PR: https://github.com/paperclipai/paperclip/pull/16378
|
||||
- Baseline: GitHub run `38671274294`, 48 jobs, 479 seconds from creation to last
|
||||
job completion. Raw GitHub evidence is in `/tmp/paperclip-ci-profile-evidence`.
|
||||
- Honeycomb: existing `paperclip` team / `test` environment; dedicated `paperclip-ci`
|
||||
dataset. Authorized local configuration key ingested the historical baseline;
|
||||
verified 731 spans and 7m 59s in the [live Honeycomb waterfall](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace/oaPMzw8zGwG).
|
||||
- Implemented label selection, full-suite override, test timing reporters, trusted
|
||||
post-run exporter, direct waterfall link, and missing-data diagnostics.
|
||||
- Verified: 48-job / 731-span historical reconstruction; 508 workflow/reporter/shard
|
||||
checks; real root-config probe with 13 tests and one file span; full Node 24
|
||||
typecheck and build. The local full Vitest run was stopped after a setup-token
|
||||
timeout and a distribution-plugin-catalog fixture failure. The setup-token
|
||||
suite passed on focused rerun; the directory-symlink removal failure reproduces
|
||||
with the unchanged base configuration and test/service files on this macOS
|
||||
Node 24.11.0 host. The PR body records the latest remote CI and review results.
|
||||
- Created and verified the repository `profile-ci` label. Dedicated Honeycomb
|
||||
ingest key `paperclip-ci-profile` has no dataset-creation permission. It is
|
||||
stored only as `HONEYCOMB_CI_API_KEY` in the `ci-profiling` GitHub environment,
|
||||
restricted to the `master` branch. Temporary key file removed; both credentials
|
||||
scanned absent from repository files and branch history.
|
||||
- Added CODEOWNER author/actor authorization and rejection tests, plus CODEOWNERS
|
||||
coverage of the privileged exporter.
|
||||
- Review fixes: retain `node --test` for older checked-out revisions; record expected
|
||||
invocations outside reporters to detect wholly missing reporters. The shard
|
||||
inventory now writes synchronously before exit after reproducing truncation
|
||||
on both the unchanged baseline and working branch.
|
||||
- Real partial-rerun evidence showed GitHub carrying old successful jobs into
|
||||
the new attempt's inventory with the new attempt number. Exclude jobs whose
|
||||
completion precedes the attempt start, and record their count explicitly.
|
||||
- Delivery: reviewable PR and live Honeycomb evidence. Activation of new trusted
|
||||
workflow definitions requires merge to master; do not claim pre-merge evidence
|
||||
proves the post-merge label-triggered journey.
|
||||
Reference in new issue
Block a user