Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-11 05:11:49 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into perf/remote-warm-default

Browse Source 
* origin/master: (323 commits)
  test(restart-chaos): measure TTFT in a real repository workspace, after an idle restart, and on Daytona (#20047)
  fix: stop prompts from encouraging subtasks (#20362)
  fix(runner): record pending calls at sidecar shutdown (#20272)
  fix(native): release a restart claim that lapsed during a failed preparation (#20529)
  fix(native): give run finalization its own retry budget (#20555)
  perf(chat): authorize the waits of an issue page with one read (#20551)
  docs(agents): shorten AGENTS.md and move data-path detail to a scoped rule (#20557)
  fix(ui): restore pinned sidebar links above More (#20369)
  perf(workspaces): migrate with one archive and no readback (#20554)
  test(restart-chaos): fire a planned restart at native start (#20538)
  fix(recovery): start a Retry after a restart cut off a relaunch whose earlier Runners were proven stopped (#20537)
  fix(runner): explain task scope on missing tool inputs (#19774)
  fix(ui): show parent tasks in responsive breadcrumbs (#20366)
  test(runner-e2e): resync the SDK public source pins with master (#20550)
  fix(recovery): keep a remote restart claim's lease through slow preparation (#20540)
  fix(scripts): build the plugin SDK in the agent worktree preflight (#20473)
  test(evals): land the chat catalog eval harness on master (#19614)
  fix(native): accept harness diagnostics in a remote session-opened bootstrap proof (#20543)
  perf(ui): open sidebar Recents tasks from cache (#20450)
  fix(heartbeat): leave a preparing run to the next controller after a graceful drain (#20514)
  ...
This commit is contained in:
Dotta committed 2026-11-10 10:02:58 -06:00
2 parents 261232a43c 10bc6431da
commit 061c5706d7
1994 files changed
+253999 -72672
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.agents/skills/company-creator/SKILL.md
+1 -1
View File
@@ -152,7 +152,7 @@ Add a concise execution contract to every generated working agent:
- Start actionable work in the same heartbeat and do not stop at a plan unless planning was requested.
- Leave durable progress in comments, documents, or work products with the next action.
- Use child issues for long or parallel delegated work instead of polling agents, sessions, or processes.
- Continue work on the current issue. Do not busy-poll agents, sessions, or processes.
- Mark blocked work with the unblock owner and action.
- Respect budget, pause/cancel, approval gates, and company boundaries.
.agents/skills/company-creator/references/example-company.md
+1 -1
View File
@@ -110,7 +110,7 @@ Execution contract:
- Start actionable implementation work in the same heartbeat; do not stop at a plan unless planning was requested.
- Leave durable progress with a clear next action.
- Use child issues for long or parallel delegated work instead of polling agents, sessions, or processes.
- Continue work on the current issue. Do not busy-poll agents, sessions, or processes.
- Mark blocked work with the unblock owner and action.
```
.agents/skills/create-agent-adapter/SKILL.md
+1 -1
View File
@@ -546,7 +546,7 @@ Import from `@paperclipai/adapter-utils/server-utils`:
### Prompt Templates
- Support `promptTemplate` for every run
- Use `renderTemplate()` with the standard variable set
- Default prompt should use `DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE` from `@paperclipai/adapter-utils/server-utils` so local adapters share Paperclip's execution contract: act in the same heartbeat, avoid planning-only exits unless requested, leave durable progress and a next action, use child issues instead of polling, mark blockers with owner/action, and respect governance boundaries.
- Default prompt should use `DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE` from `@paperclipai/adapter-utils/server-utils` so local adapters share Paperclip's execution contract: act in the same heartbeat, avoid planning-only exits unless requested, leave durable progress and a next action, avoid busy-polling, mark blockers with owner/action, and respect governance boundaries.
### Error Handling
- Differentiate timeout vs process error vs parse failure
.agents/skills/create-paperclip-bundled-skill/SKILL.md
+2 -4
View File
@@ -53,10 +53,8 @@ xc search '<topic keywords>' -n 30 # related discussion
```
If `xc` is not on PATH, is unauthenticated, or the account lacks read access
(the check above fails for any reason), delegate the
fetch to a teammate with X/Twitter access (e.g. the Content Strategist agent)
via a child issue: give them the URL and ask for full text of the post + thread
+ any linked content.
(the check above fails for any reason), use another authorized source or ask
the user for the post, thread, and linked content. Report any unavailable context.
**Other sources.** Fetch linked articles/READMEs directly. Record the source
URL — it goes in the skill body or PR description as attribution.
.agents/skills/pr-gardening/SKILL.md
+5 -3
View File
@@ -101,15 +101,15 @@ Every `needs_gardening` PR here was opened by this instance (Stage A guarantees
2. **Deduplicate.** Search open Paperclip issues for the PR number/branch. If an equivalent open prepare-PR task already exists, reuse it with a concise status comment instead of creating another (see the deduplication section above).
3. **Run the prepare skill.** Create one focused child task per PR assigned to a coder agent (prefer CodexCoder) instructing it to run `/prepare-paperclip-pr` for that PR — include the PR URL, branch, current head SHA, and the exact machine-detected `reasons[]` from `readiness.json`. If you are the gardener and already have the PR's branch checked out in a worktree, you may run `/prepare-paperclip-pr` directly instead of delegating. Either way, the prepare work must never merge, approve, or close the PR.
3. **Run the prepare skill.** Run `/prepare-paperclip-pr` for each PR, using the PR URL, branch, current head SHA, and the exact machine-detected `reasons[]` from `readiness.json`. If the user requests delegation, include those inputs in the assigned task. The prepare work must never merge, approve, or close the PR.
4. **Leave the marker comment.** Comment on the originating issue with the marker above, the current head SHA, the copied `reasons[]`, the round counter, and a link to the prepare task. Use `POST /api/issues/:issueId/comments` with `X-Paperclip-Run-Id`. Include `resume: true` when the issue is terminal so the comment creates a live continuation.
4. **Leave the marker comment.** Comment on the originating issue with the marker above, the current head SHA, the copied `reasons[]`, the round counter, and a link to the current task or recorded result. For delegated work, link the assigned prepare task instead. Use `POST /api/issues/:issueId/comments` with `X-Paperclip-Run-Id`. Include `resume: true` when the issue is terminal so the comment creates a live continuation.
Suggested body:
```markdown
<!-- pr-gardening:paperclipai/paperclip#1234 -->
Gardening: dispatched `/prepare-paperclip-pr` for https://github.com/paperclipai/paperclip/pull/1234 via PAP-XXXX.
Gardening: ran `/prepare-paperclip-pr` for https://github.com/paperclipai/paperclip/pull/1234 on [this task](/PAP/issues/PAP-XXXX).
Current-head verification at `abc123` found:
- failing check: test
@@ -118,6 +118,8 @@ Current-head verification at `abc123` found:
Gardening round 1/3. Re-verification is required after changes; do not merge based on this comment.
```
For delegated work, say preparation was dispatched and link the assigned task.
## Stage D — Optional Inbox Tidy-Up
Run this stage only when the caller explicitly supplied `--archive-inbox`. `--dry-run` always suppresses inbox mutations, even when `--archive-inbox` is also present. Without the flag, do not archive anything.
.agents/skills/terminal-bench-loop/SKILL.md
+13 -7
View File
@@ -10,7 +10,7 @@ description: >
A repeatable operating skill for driving one Terminal-Bench problem to a passing smoke through Paperclip, with explicit issue topology, bounded runs, board-gated product fixes, and worktree continuity.
This skill is **operational + diagnostic**, not engineering. It coordinates issues, artifacts, and approvals around a Terminal-Bench loop. It does not authorize code changes — every accepted product fix lands as a separate implementation child issue after a board confirmation.
This skill is **operational + diagnostic**, not engineering. It coordinates issues, artifacts, and approvals around a Terminal-Bench loop. It does not authorize code changes — product fixes require a board confirmation before implementation.
Canonical execution model: read `doc/execution-semantics.md` before starting a loop or moving any loop issue. Every loop issue must rest in a state the doc allows: terminal (`done`/`cancelled`), explicitly live (active run / queued wake), explicitly waiting (`in_review` with participant/interaction/approval), or explicit recovery/blocker (`blocked` with `blockedByIssueIds` and a named owner).
@@ -49,7 +49,7 @@ Collect these on the top-level loop issue before iteration 1. Any input that can
- **Source issue.** The Paperclip issue that asked for the loop. The loop parent links back to it.
- **Terminal-Bench task name.** Single-task identifier (e.g. `terminal-bench/fix-git`). Multi-task suites are out of scope for this skill.
- **Iteration budget.** Maximum number of iterations before the loop must stop without further fixes (typical: 3–5). Also record a per-iteration wall-clock cap.
- **Paperclip App worktree issue.** The implementation-side issue under the Paperclip App project whose execution workspace owns the isolated worktree. First iteration creates it; later iterations reuse it via `inheritExecutionWorkspaceFromIssueId` or equivalent.
- **Paperclip App worktree issue.** The implementation-side issue under the Paperclip App project whose execution workspace owns the isolated worktree. Use the current issue's isolated worktree for the default workflow; the optional task-tree procedure below records its owning issue.
- **Benchmark command.** The exact `paperclip-bench` invocation, including the `PAPERCLIPAI_CMD` (or equivalent) binding pinned to the Paperclip App worktree under test. Record verbatim on the loop issue.
- **Dispatch runner config.** The exact Harbor/Paperclip runner dispatch config required for the smoke to actually start a Paperclip heartbeat. For the current Harbor wrapper, record the `PAPERCLIP_HARBOR_RUNNER_CONFIG` JSON (or equivalent config file) verbatim enough to preserve: `assignee`, `heartbeat_strategy`, `agent_adapter` / `agent_adapters`, `reuse_host_home` when local credentials are intentionally needed, and the stop budget. A bare Harbor command that creates `BEN-1` as unassigned `todo` with zero heartbeat-enabled agents is a harness/setup failure, not a valid product diagnosis.
- **Latest artifact root.** Filesystem or storage path under which `paperclip-bench` writes run artifacts (manifest, `results.jsonl`, Harbor raw job folders, redacted telemetry). Each iteration appends; nothing is overwritten.
@@ -57,9 +57,15 @@ Collect these on the top-level loop issue before iteration 1. Any input that can
Record each input on the top-level loop issue (description or a dedicated `inputs` document). If any input changes mid-loop, note the change and the iteration it took effect.
## Issue topology
## Default workflow
The loop must be representable as a tree, not as prose in comments:
Run the smoke, diagnosis, approved fix, and rerun on the current issue. Record each iteration in its documents. Save a fix proposal as `plan`, read back its latest revision, and request board confirmation bound to that revision before implementation. Use the same isolated worktree for the fix and rerun. Keep the iteration budget, evidence requirements, and QA/CTO review gates below; use the current issue's review participants or approval interactions for those gates.
The task-tree procedure below applies only when the user explicitly requests separate tasks or when continuing an existing tree. Do not propose a task tree merely to run this workflow. For the default workflow, iteration records replace iteration children, and plan approval and review waits stay on the current issue.
## Optional issue topology
For an explicitly requested task tree:
- **Top-level loop issue.** Long-lived. Holds inputs, iteration counter, current state, links to every iteration child, and the product-rule history. Rests in `in_progress` while an iteration is running, `in_review` only when a typed waiter sits directly on the loop parent (execution-policy participant, `request_confirmation` / `ask_user_questions` / `suggest_tasks` interaction, approval, or named human owner), `blocked` with `blockedByIssueIds` while a child issue is the gating work (iteration child holding the fix-proposal `request_confirmation`, or implementation, QA, or CTO review children), `done` on pass, or `cancelled` on board-rejection / budget exhaustion.
- **Iteration child issues.** One per iteration. Each carries: a bounded run issue (smoke), a diagnosis issue (applies `/diagnose-why-work-stopped`), a fix-proposal document with a `request_confirmation` interaction, and — only after acceptance — implementation, QA, CTO review, and rerun children. Iteration children are blocked by their predecessors so the executor wakes them in order.
@@ -67,7 +73,7 @@ The loop must be representable as a tree, not as prose in comments:
Wire dependencies with `blockedByIssueIds`, never with prose like "blocked by X". When a dependent child is `done`, the executor auto-wakes the next.
## Procedure
## Procedure for an explicitly requested task tree
### 0. Read the current execution contract
@@ -165,7 +171,7 @@ A loop must never end on a prose comment alone. Every stop is a status transitio
The loop must not test whatever Paperclip checkout happens to be current for the heartbeat. It must test the same isolated Paperclip App worktree where proposed fixes are applied.
- The first iteration creates the Paperclip App implementation child; that project's git-worktree policy spawns a fresh worktree.
- The default workflow uses the current issue's isolated worktree. In the optional task-tree workflow, the first iteration records the implementation issue that owns the worktree.
- The loop issue records the worktree-owning issue id and the workspace path (or workspace id).
- Every later implementation, QA, and rerun child sets `inheritExecutionWorkspaceFromIssueId` to that worktree-owning issue, so all subsequent loop work shares one workspace.
- The benchmark command always sets `PAPERCLIPAI_CMD` (or the equivalent command binding) to the CLI entrypoint inside that worktree, and it carries the recorded dispatch runner config (`PAPERCLIP_HARBOR_RUNNER_CONFIG` or equivalent) needed to assign the benchmark issue and start the heartbeat. The benchmark command stored on the loop issue is the source of truth — if a heartbeat needs to run the smoke from a different shell, it copies the recorded command block verbatim, not only the Harbor invocation line.
@@ -210,7 +216,7 @@ If a loop issue does not fit one of these on exit, the heartbeat is not done. Fi
## Deterministic smoke
Run this smoke after installing or changing the skill, before treating it as operational for a live Terminal-Bench loop:
This smoke checks the optional task-tree lifecycle. Run it when changing that lifecycle; it does not test the default single-issue workflow:
```sh
pnpm smoke:terminal-bench-loop-skill
.github/scripts/tests/ci-plan.test.mjs
+10 -2
View File
@@ -194,8 +194,8 @@ test("required aggregates reject omitted, failed and cancelled selected work", (
const resultPlan = plan(["ui/src/components/Sidebar.tsx"]);
const results = Object.fromEntries(names.map((name) => [name, { result:
name === "gate" || name === "policy" || resultPlan.jobs[name]?.run ? "success" : "skipped" }]));
const run = () => spawnSync(process.execPath, ["--input-type=module", "-e", code], { encoding: "utf8",
env: { ...process.env, FULL_CI: "true", CI_PLAN: JSON.stringify(resultPlan), CI_RESULTS: JSON.stringify(results) } });
const run = (env = {}) => spawnSync(process.execPath, ["--input-type=module", "-e", code], { encoding: "utf8",
env: { ...process.env, FULL_CI: "true", CI_PLAN: JSON.stringify(resultPlan), CI_RESULTS: JSON.stringify(results), ...env } });
assert.equal(run().status, 0);
for (const name of names) {
// Optional analysis can fail or be skipped; the unchanged task plan
@@ -208,6 +208,14 @@ test("required aggregates reject omitted, failed and cancelled selected work", (
}
results[name].result = previous;
}
for (const result of ["failure", "cancelled", "skipped"]) {
results.gate.result = result;
const failedGate = run({ FULL_CI: "", CI_PLAN: "" });
assert.notEqual(failedGate.status, 0, `${id}: an absent routing output must not pass`);
assert.match(failedGate.stderr, /CI routing must succeed/);
}
assert.match(block, /^ {4}runs-on: \$\{\{ needs\.gate\.outputs\.runner \|\| 'ubuntu-latest' \}\}$/m,
`${id}: the always-run summary must have a runner when routing produces no output`);
}
});
.github/workflows/chat-catalog-testops-case.yml
+6 -1
View File
@@ -165,7 +165,12 @@ jobs:
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ inputs.target_sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: refs/heads/codex/chat-catalog-master-integration
PAPERCLIP_PLAYWRIGHT_CHANNEL: chrome
run: pnpm test:e2e:runner -- --id "${{ fromJSON(inputs.cell).executionId }}" --max-parallel 1 --max-automatic-retries 0 > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1
PAPERCLIP_RUNNER_E2E_SELECTED_ID: ${{ fromJSON(inputs.cell).executionId }}
run: |
# Agents can read any process's argv in /proc; keep the case id off the command line.
selector=(--id "$PAPERCLIP_RUNNER_E2E_SELECTED_ID")
if grep -q PAPERCLIP_RUNNER_E2E_SELECTED_ID tests/runner-e2e/selectors.ts 2>/dev/null; then selector=(); fi
pnpm test:e2e:runner -- "${selector[@]}" --max-parallel 1 --max-automatic-retries 0 > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1
- name: Retain original evidence after success or failure
if: always() && steps.cell_credentials.outcome == 'success'
env: *cell_session
.github/workflows/pr-trusted.yml
+2 -2
View File
@@ -798,7 +798,7 @@ jobs:
name: verify
if: ${{ always() }}
needs: [gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, sandbox_provider_tests, verify_serialized_server, canary_dry_run, verify_runner_e2e_support]
runs-on: ${{ needs.gate.outputs.runner }}
runs-on: ${{ needs.gate.outputs.runner || 'ubuntu-latest' }}
timeout-minutes: 5
steps:
- name: Validate every planned suite result
@@ -1781,7 +1781,7 @@ jobs:
name: e2e
if: ${{ always() }}
needs: [gate, policy, e2e_shards]
runs-on: ${{ needs.gate.outputs.runner }}
runs-on: ${{ needs.gate.outputs.runner || 'ubuntu-latest' }}
timeout-minutes: 5
steps:
- name: Validate every planned suite result
.github/workflows/runner-chaos-evals.yml
+3
View File
@@ -50,6 +50,9 @@ jobs:
pnpm --filter @paperclipai/paperclip-eval-kernel build
pnpm --filter @paperclipai/paperclip-runner report:runner-chaos-evals
- name: Build native Runner test binaries
run: pnpm --filter @paperclipai/paperclip-runner build:rust
- name: Run Runner fault and replay suites
run: |
pnpm --filter @paperclipai/paperclip-runner exec vitest run \
.github/workflows/runner-full-stack-e2e.yml
+7 -2
View File
@@ -1435,11 +1435,16 @@ jobs:
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ needs.authorize.outputs.target_ref }}
PAPERCLIP_PLAYWRIGHT_CHANNEL: ${{ needs.authorize.outputs.playwright_channel }}
PAPERCLIP_RUNNER_E2E_SELECTED_ID: ${{ matrix.executionId }}
run: |
# Agents can read any process's argv in /proc. A target that reads the
# selected cell from the environment keeps the case id off the command line.
selector=(-- --id "$PAPERCLIP_RUNNER_E2E_SELECTED_ID")
if grep -q PAPERCLIP_RUNNER_E2E_SELECTED_ID tests/runner-e2e/selectors.ts 2>/dev/null; then selector=(); fi
if [ -n "$PAPERCLIP_RUNNER_E2E_PRIVATE_PREFLIGHT" ]; then
pnpm test:e2e:runner -- --id "${{ matrix.executionId }}" > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1
pnpm test:e2e:runner "${selector[@]}" > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1
else
pnpm test:e2e:runner -- --id "${{ matrix.executionId }}"
pnpm test:e2e:runner "${selector[@]}"
fi
- name: Stage bounded campaign evidence
.greptile/files.json
+15
View File
@@ -120,6 +120,21 @@
"packages/db/src/schema/agent_task_sessions.ts",
"packages/db/src/schema/run_identity_contexts.ts"
]
},
{
"path": "rules/data-paths.mdc",
"description": "Separation of Telemetry, Observability, and the run log, with the review level each path requires.",
"scope": [
"packages/shared/src/telemetry/**",
"server/src/instrumentation.ts",
"doc/observability.md",
"packages/adapter-utils/src/duplex-observability.ts",
"server/src/services/duplex-observability-recorder.ts",
"packages/adapter-utils/src/acpx-engine/startup-timing.ts",
"doc/run-log-events.md",
"packages/db/src/schema/heartbeat_run_events.ts",
"server/src/services/heartbeat.ts"
]
}
]
}
AGENTS.md
+91 -128
View File
@@ -2,29 +2,26 @@
Guidance for human and AI contributors working in this repository.
This codebase is yours: what it's for and how it's built are your call, and you'll keep building on it.
The invariants and rules below record what has been decided so far. If you think one of them is wrong, say so and propose the change; do not work around it.
## 1. Purpose
Paperclip is a control plane for AI-agent companies.
The current implementation target is V1 and is defined in `doc/SPEC-implementation.md`.
The code and its tests are the source of truth for current behavior.
`doc/SPEC-implementation.md` and `doc/SPEC.md` are background. They can be out of date, so do not treat them as requirements when they disagree with the code or with the task.
## 2. Read This First
Before making changes, read in this order:
Read the doc that matches your change before you start:
1. `doc/GOAL.md`
2. `doc/PRODUCT.md`
3. `doc/SPEC-implementation.md`
4. `doc/DEVELOPING.md`
5. `doc/DATABASE.md`
`doc/SPEC.md` is long-horizon product context.
`doc/SPEC-implementation.md` is the concrete V1 build contract.
When adding or changing an Apps catalog connection, also follow
`doc/connections/CONNECTOR-PLAYBOOK.md`. It is the canonical connection
authoring runbook for provider research, supported transport/auth patterns,
credential handling, branding, implementation, testing, live proof, and PR
submission.
| Your change touches | Read |
| --- | --- |
| Product behavior or scope | `doc/GOAL.md`, `doc/PRODUCT.md` |
| Setup, commands, testing, worktrees | `doc/DEVELOPING.md` |
| Schema, migrations, DB clients | `doc/DATABASE.md` |
| An Apps catalog connection | `doc/connections/CONNECTOR-PLAYBOOK.md` (the canonical connection authoring runbook) |
| Anything under `ui/` | `DESIGN.md` |
### Scoped repository rules
@@ -35,21 +32,8 @@ means the rule applies throughout the repository. Read all rules whose scope
matches the affected files. These rules supplement this file and any applicable
nested `AGENTS.md` instructions.
For `server/src/services/chat-channels.ts` and everything under
`server/src/services/chat-channels/`, read
[`rules/chat-channel-platform-boundaries.mdc`](rules/chat-channel-platform-boundaries.mdc).
For `server/src/services/heartbeat.ts`, `server/src/services/heartbeat/`,
heartbeat-prefixed sibling services, and heartbeat regression tests, read
[`rules/heartbeat-execution-boundaries.mdc`](rules/heartbeat-execution-boundaries.mdc).
For ACPX provider profiles, their release attestations and pins, the provider-pack
contract registry, and ACPX driver and provider code under
`packages/paperclip-runner/`, read
[`rules/acpx-profile-identity.mdc`](rules/acpx-profile-identity.mdc).
For native runtime, recovery, session preparation, runner, ACPX engine, and
provider-pack or sandbox image code that persists or resumes run state, and for
the native run, checkpoint, and session schema in `packages/db`, read
[`rules/upgrade-safe-run-state.mdc`](rules/upgrade-safe-run-state.mdc).
See [`rules/README.md`](rules/README.md) for rule authoring and Greptile setup.
The frontmatter is the source of truth for scope, so this file does not list the
rules. See [`rules/README.md`](rules/README.md) for rule authoring and Greptile setup.
## 3. Repo Map
@@ -65,20 +49,30 @@ See [`rules/README.md`](rules/README.md) for rule authoring and Greptile setup.
- `cli/`: `paperclipai` CLI package (published bin, agent-facing commands)
- `skills/`: Paperclip runtime/operational skills (not part of the app catalog)
- `doc/`: operational and product docs
- `rules/`: scoped repository rules (see section 2)
## 4. Dev Setup (Auto DB)
Use embedded PGlite in dev by leaving `DATABASE_URL` unset.
Prepare a new or updated worktree with one command:
```sh
node scripts/agent-preflight.mjs
```
Use it instead of running `pnpm install`, the Runner build, and `build:rust` by
hand. It skips unchanged steps and reuses other worktrees' builds from a
content-keyed cache in `~/.cache/paperclip/agent-build`. Do not share
`CARGO_TARGET_DIR` between worktrees. See "Agent worktree preflight" in
`doc/DEVELOPING.md`.
Start the app. Leave `DATABASE_URL` unset to use embedded PGlite:
```sh
pnpm install
pnpm dev
```
This starts:
- API: `http://localhost:3100`
- UI: `http://localhost:3100` (served by API server in dev middleware mode)
This serves the API and the UI at `http://localhost:3100` (the API server serves
the UI in dev middleware mode).
Quick checks:
@@ -87,7 +81,7 @@ curl http://localhost:3100/api/health
curl http://localhost:3100/api/companies
```
Reset local dev DB:
Reset the local dev DB. This deletes all local dev data:
```sh
rm -rf data/pglite
@@ -123,7 +117,7 @@ If you change schema/API behavior, update all impacted layers:
- Activity logging for mutating actions
4. Do not replace strategic docs wholesale unless asked.
Prefer additive updates. Keep `doc/SPEC.md` and `doc/SPEC-implementation.md` aligned.
Prefer additive updates.
5. Keep repo plan docs dated and centralized.
When you are creating a plan file in the repository itself, new plan documents belong in `doc/plans/` and should use `YYYY-MM-DD-slug.md` filenames. This does not replace Paperclip issue planning: if a Paperclip issue asks for a plan, update the issue `plan` document per the `paperclip` skill instead of creating a repo markdown file.
@@ -134,28 +128,11 @@ When your task produces a user-inspectable deliverable file, follow the Papercli
7. Name the three data paths correctly.
This repo has three separate data paths. Do not confuse them. Match a change to a path by its file path, not by the word "observability" or "telemetry" alone.
- **Telemetry** is the Paperclip first-party event system. It is opt-out and it sends data to a Paperclip endpoint by default. Its paths are:
- `packages/shared/src/telemetry/`
- the generated contract `packages/shared/src/telemetry/generated/paperclip-telemetry.ts`
- each caller of `packages/shared/src/telemetry/events.ts` or `packages/shared/src/telemetry/client.ts`
- **Observability** is the OpenTelemetry trace path. An operator must set an OTLP endpoint. Until an operator sets the endpoint, the tracer is a no-operation. Its paths are:
- `server/src/instrumentation.ts`
- `doc/observability.md`
- `packages/adapter-utils/src/duplex-observability.ts`
- `server/src/services/duplex-observability-recorder.ts`
- the span attributes in `packages/adapter-utils/src/acpx-engine/startup-timing.ts`
- **The run log** holds rows in the local `heartbeat_run_events` table. The data stays in the instance database. Its paths are:
- `doc/run-log-events.md`
- `packages/db/src/schema/heartbeat_run_events.ts`
- the append path `appendRunEvent` in `server/src/services/heartbeat.ts`
- **Telemetry** is the Paperclip first-party event system under `packages/shared/src/telemetry/`. It sends data to a Paperclip endpoint by default, so a change needs strict review: generated contract first, README update, privacy review.
- **Observability** is the OpenTelemetry trace path. It is a no-operation until an operator sets an OTLP endpoint. A change needs lighter review while it keeps that gate and stays inside the span-attribute allowlist.
- **The run log** holds rows in the local `heartbeat_run_events` table. The data stays in the instance database, so a change needs no extra review.
Apply a review level that matches the path:
- **Telemetry change (strict review).** The author updates the generated contract first. The author updates `packages/shared/src/telemetry/README.md` in the same pull request. The author requests a privacy review. Reason: a Telemetry event goes to a Paperclip endpoint by default, so a mistake sends data immediately.
- **Observability change (lighter review).** The operator endpoint gate stays in place. The no-operation behaviour stays when no endpoint is set. A privacy review is not necessary while the change stays inside the closed span-attribute allowlist.
- **Run-log change (no extra review).** A run-log change needs neither review level above, because the data stays in the instance database.
**Exclusion.** The word "observability" in a file such as `server/src/services/recovery-observability.ts` names a different concept. Apply this rule by path, not by word match.
Before you change any of these paths or add a Telemetry caller, read [`rules/data-paths.mdc`](rules/data-paths.mdc) for the exact file paths and review requirements.
## 6. Database Change Workflow
@@ -181,35 +158,7 @@ Notes:
## 7. Verification Before Hand-off
[maps/feature-map/README.md](maps/feature-map/README.md) is an optional reference for user
entry points, verification recipes, and coverage gaps. The map records coverage
scope, not proof that a live journey passed.
The sibling [agent access map](maps/agent-access-map/README.md) records APIs,
agent entry points and permission boundaries using the same capability IDs.
Follow [the feature-map scope rule](rules/feature-map-scope.mdc): additions require
substantial new features. Routine fixes, polish, refactors, and test changes do
not require map updates. Follow [the map contract](maps/README.md) for minimal
corrections when existing entries become materially inaccurate, and run
`node scripts/check-maps.mjs` after map edits.
Its structural checks do not prove live behavior or authorization.
Storybook has a downstream [feature catalog](maps/storybook-map/README.md).
Use `maintain-storybook` for story/fixture drift and `review-storybook` for browser
evidence. Keep the feature-map skills independent of this catalog. After story or recipe changes, update the affected source annotations or
recipe and run `node scripts/storybook-catalog.mjs`. Indexes are generated and
ignored; never commit aggregate catalogs, refresh review fingerprints, or append
a shared review journal. Use Git diffs and keep review evidence in the PR or issue.
Structural checks are not production acceptance.
Prepare a new or updated worktree with `node scripts/agent-preflight.mjs` instead of running
`pnpm install`, the Runner build and `build:rust` by hand. It skips unchanged
steps and reuses other worktrees' Runner builds, debug binaries and third-party
crates from a content-keyed cache in `~/.cache/paperclip/agent-build`. Do not
share `CARGO_TARGET_DIR` between worktrees. See "Agent worktree preflight" in
`doc/DEVELOPING.md`.
Default local/agent verification path:
Run the smallest check that proves the change. The default local and agent path is:
```sh
pnpm verify
@@ -217,40 +166,38 @@ pnpm verify
This selects affected Vitest files with the same conservative dependency policy
as CI. It includes committed, staged, unstaged, and untracked changes against the
merge base with local `origin/master` (or `master`). Use `--base <ref>` for another
base and `--explain` to inspect the selection. Passing checks with complete input
graphs can be reused within this worktree. Opaque integration checks run fresh.
Source, test, fixture, configuration, dependency, environment, and toolchain changes
invalidate matching results. Failed or interrupted attempts are never passes.
merge base with local `origin/master` (or `master`), and it reuses passing checks
whose inputs are unchanged. Use `--base <ref>` for another base, `--explain` to
inspect the selection, and `--no-cache` for fresh execution. See "Incremental
local verification" in `doc/DEVELOPING.md` for how reuse and invalidation work.
Exit 2 means the change needs broader verification: run focused checks and let
required CI supply that coverage, or explicitly use `pnpm verify --full` for the
full local Vitest suite. Do not automatically follow exit 2 with a full local run.
Use `--no-cache` to request fresh execution without reading or saving passes.
`pnpm test` and `pnpm test:run` retain their full-suite behavior.
Follow these rules:
Browser suites stay opt-in:
- **Exit 2 means the change needs broader verification.** Run focused checks and
let required CI supply that coverage, or explicitly use `pnpm verify --full`
for the full local Vitest suite. Do not automatically follow exit 2 with a full
local run.
- **Do not default to repo-wide checks.** Do not run repo-wide typecheck, build,
or test on every heartbeat when a narrower check is enough to prove the change.
Use the broad checks below only when they resolve an actual remaining gap, such
as a local platform fault or explicit full-suite qualification.
- **Browser suites stay opt-in.** Run `pnpm test:e2e` and
`pnpm test:release-smoke` only when your change touches them or when you are
explicitly verifying CI/release flows.
- **CI covers the final revision.** For a PR-ready handoff, run the relevant local
checks and require the applicable CI typecheck, build, and test checks to pass
for the final tested revision. Record their scope, revision, environment, and
result links. Fresh CI can supply broad coverage; do not duplicate a full local
run merely to prepare a PR, resume a chat, or repeat unchanged passing
verification. A selected local pass does not replace required CI, release
qualification, browser acceptance, or a local/platform-specific regression.
- **Evidence expires.** After edits or a rebase, rerun affected checks and
invalidate evidence whose inputs or required coverage changed. Failed or
interrupted attempts are never passes.
- **Report gaps.** If anything cannot be run, explicitly report what was not run
and why.
```sh
pnpm test:e2e
pnpm test:release-smoke
```
Run the browser suites only when your change touches them or when you are explicitly verifying CI/release flows.
For normal issue work, run the smallest relevant verification first. Do not default to repo-wide typecheck/build/test on every heartbeat when a narrower check is enough to prove the change.
For a PR-ready handoff, run the relevant local checks and require the applicable
CI typecheck, build, and test checks to pass for the final tested revision. Record
their scope, revision, environment, and result links. Fresh CI can supply broad
coverage; do not duplicate a full local run merely to prepare a PR, resume a chat,
or repeat unchanged passing verification. A selected local pass does not replace
required CI, release qualification, browser acceptance, or a local/platform-specific
regression. After edits or a rebase, rerun affected checks and invalidate evidence
whose inputs or required coverage changed.
Use broad local checks when they resolve an actual remaining gap, such as a local
platform fault or explicit full-suite qualification:
Broad local checks:
```sh
pnpm -r typecheck
@@ -258,9 +205,25 @@ pnpm test:run
pnpm build
```
If memory is limited, use `node scripts/typecheck-low-memory.mjs` instead of `pnpm -r typecheck`. See `doc/DEVELOPING.md`.
`pnpm test` and `pnpm test:run` run the full suite. If memory is limited, use
`node scripts/typecheck-low-memory.mjs` instead of `pnpm -r typecheck`. See
`doc/DEVELOPING.md`.
If anything cannot be run, explicitly report what was not run and why.
### Maps and Storybook
The [feature map](maps/feature-map/README.md) and the sibling
[agent access map](maps/agent-access-map/README.md) are optional references for
user entry points, agent entry points, permission boundaries, verification
recipes, and coverage gaps. They record coverage scope, not proof that a live
journey passed. Routine fixes, polish, refactors, and test changes do not require
map updates. When you do edit a map, follow
[the feature-map scope rule](rules/feature-map-scope.mdc) and
[the map contract](maps/README.md), then run `node scripts/check-maps.mjs`.
After story or recipe changes, update the affected source annotations or recipe
and run `node scripts/storybook-catalog.mjs`. Indexes are generated and ignored;
never commit them. See the [Storybook feature catalog](maps/storybook-map/README.md)
for the full workflow. Structural checks are not production acceptance.
## 8. API and Auth Expectations
@@ -283,6 +246,10 @@ When adding endpoints:
- Surface failures clearly; do not silently ignore API errors
- Form and wizard footers: keep Save & exit (or Cancel/Back) left and the primary action right in the same vertically aligned row. Each step owns the entire footer; never append Save & exit as a separate row. See `DESIGN.md`.
### Design system
`DESIGN.md` at the repo root is the source of truth for UI design decisions. The token-only rule applies to all `ui/` changes: every color, spacing, radius, type, shadow, and motion value in `ui/src/components/**` and `ui/src/pages/**` comes from the token layer in `ui/src/index.css` — no hex, raw px, arbitrary Tailwind bracket values, or raw `font-size`/`fontSize` declarations in components, outside the documented allowlist in `ui/src/index.css`. Run `pnpm check:token-gates` (`scripts/check-token-gates.mjs`) before committing UI changes — it fails on any violation not covered by that allowlist.
## 10. Pull Request Requirements
When creating a pull request (via `gh pr create` or any other method), you **must** read and fill in every section of [`.github/PULL_REQUEST_TEMPLATE.md`](.github/PULL_REQUEST_TEMPLATE.md). Do not craft ad-hoc PR bodies — use the template as the structure for your PR description. Required sections:
@@ -298,12 +265,8 @@ When creating a pull request (via `gh pr create` or any other method), you **mus
A change is done when all are true:
1. Behavior matches `doc/SPEC-implementation.md`
2. Relevant local verification and required CI typecheck, test, and build checks pass for the final revision, with any acceptance or platform-specific gaps resolved
1. Behavior matches what the task asked for and preserves the invariants in section 5
2. Relevant local verification and required CI typecheck, test, and build checks pass for the final revision, with any acceptance or platform-specific gaps resolved (section 7)
3. Contracts are synced across db/shared/server/ui
4. Docs updated when behavior or commands change
5. PR description follows the [PR template](.github/PULL_REQUEST_TEMPLATE.md) with all sections filled in (including Model Used)
## Design system
`DESIGN.md` at the repo root is the source of truth for UI design decisions. The token-only rule applies to all `ui/` changes: every color, spacing, radius, type, shadow, and motion value in `ui/src/components/**` and `ui/src/pages/**` comes from the token layer in `ui/src/index.css` — no hex, raw px, arbitrary Tailwind bracket values, or raw `font-size`/`fontSize` declarations in components, outside the documented allowlist in `ui/src/index.css`. Run `pnpm check:token-gates` (`scripts/check-token-gates.mjs`) before committing UI changes — it fails on any violation not covered by that allowlist.
CONTRIBUTING.md
+1 -1
View File
@@ -120,7 +120,7 @@ and coverage requirements.
### Telemetry Changes
This repo has three separate data paths: Telemetry, Observability, and the run log. See rule 7 in `AGENTS.md` for the full definitions and the review level each path needs.
This repo has three separate data paths: Telemetry, Observability, and the run log. See [`rules/data-paths.mdc`](rules/data-paths.mdc) (summarized in rule 7 of `AGENTS.md`) for the full definitions and the review level each path needs.
If your change adds, removes, or modifies emitted telemetry events, update the [Telemetry Data Contract](packages/shared/src/telemetry/README.md) in the same PR. Keep clients emitting raw dimension values and avoid documenting or relying on private delivery details. If your change adds, removes, or modifies an OpenTelemetry span or span attribute, keep the change inside the closed span-attribute allowlist in `packages/adapter-utils/src/acpx-engine/startup-timing.ts`. If your change adds or modifies a run-log event, update `doc/run-log-events.md` in the same PR.
DESIGN.md
+23 -1
View File
@@ -111,6 +111,8 @@ Keep the runner turn's avatar, name, status header and current-activity slot
in place from message submission through queued admission and native startup.
Saving the human comment does not end startup feedback. Keep the identity mounted
until its run appears or a newer execution outcome ends admission.
An older intent resolving cannot acknowledge a follow-up. When Send reopens a
completed task, retain startup feedback while the previous task status catches up.
Use the sent message's selected recipient through reassignment. Choosing the
recipient for a later draft must not change the response already starting.
Runtime discovery must not replace the turn with a generic spinner or a
@@ -122,6 +124,13 @@ starts this queued turn stays in history, including when it reopens an older
task. This placement is not a consumption receipt. Later follow-ups remain in
the composer queue with their existing edit, order and discard controls.
Queued messages render Markdown and show compact local attachment thumbnails.
Keep paragraph boundaries and wrap thumbnails and controls within narrow rows.
Omit inline images from the compact text preview. On
submit, move the text and attachment previews out of the composer together.
Retain upload receipts while saving so a rejected or uncertain send restores
the draft without discarding files added for the next message.
Stream the provider's final reply as the conversational answer. Completion-tool
reports remain inspectable run data while work continues. Use their summary as
an answer fallback only after the run ends and its final transcript fetch
@@ -832,6 +841,13 @@ status picker with its monospaced task ID, project and people. The title remains
editable and the existing task-actions menu stays at the upper right. Typography
and spacing use the shared tokens; toggling off restores the ordinary header.
Task breadcrumbs link the immediate parent before the current task. Each task
title displays at most 48 Unicode code points, including an ellipsis; saved
titles, hover text and accessible names retain the full title. The parent uses
at most 40% of the breadcrumb row, and both titles truncate when space runs out.
Mobile omits the collection link and parent identifier to preserve both titles
and the current task identifier. Private parents show a noninteractive label.
The local task detail also offers a **Thread preview** selector. DES-1 opens a
complete synthetic conversation matching the staging transcript layout; other
tasks start with their saved thread. Synthetic runner events pass through
@@ -1028,7 +1044,13 @@ or certify backend behavior or keyboard accessibility.
Agents and Connectors compose the same `ToolbarSearch`. Connectors also use canonical Standard filled tabs. `AgentsToolbar` is shared by the roster page and
toolbar references; `ConnectorsToolbar` is shared by Browse and toolbar references. Keep search
first, source selection next where present, and view/actions last. The agent roster uses a harness filter icon without a status selector.
first, source selection next where present, and view/actions last. The agent roster filter icon opens model, status, harness and connection selectors.
Each selector includes an All option. Selected filters intersect with name, role
and title search. Clear filters resets metadata selectors and keeps the search.
Filter choices use the same metadata as the roster cells, including harness defaults,
local credentials, connection pools, and agents that still need setup. Filters stay
active across list and org-chart views, retain matching nodes' ancestors, and preserve
bulk selection. The count reports matching agents, without ancestor context.
Connector source choices use radio semantics and preserve Browse's existing
automatic search expansion; they do not switch separate content panels.
Legacy agent status and built-in URLs redirect to the full roster at `/agents/all`.
cli/src/__tests__/agent-lifecycle.test.ts
+11
View File
@@ -131,6 +131,17 @@ describe("agent lifecycle commands", () => {
]);
});
it("grants and revokes one permission without replacing the grant list", async () => {
const fetchMock = vi.fn().mockImplementation(() => Promise.resolve(jsonResponse()));
vi.stubGlobal("fetch", fetchMock);
await run(["agent", "permission:grant", AGENT_ID, "commentary:view"]);
await run(["agent", "permission:revoke", AGENT_ID, "commentary:view"]);
expect(fetchMock.mock.calls.map(call => [call[0], call[1]?.method, JSON.parse(call[1]?.body)])).toEqual([
[`http://localhost:3100/api/agents/${AGENT_ID}/permission-grants`, "PATCH", { permissionKey: "commentary:view", enabled: true }],
[`http://localhost:3100/api/agents/${AGENT_ID}/permission-grants`, "PATCH", { permissionKey: "commentary:view", enabled: false }],
]);
});
it("refuses to delete an instructions file without confirmation", async () => {
const fetchMock = vi.fn().mockImplementation(() => Promise.resolve(jsonResponse()));
vi.stubGlobal("fetch", fetchMock);
cli/src/__tests__/chat-cli-guidance.test.ts
+80
View File
@@ -0,0 +1,80 @@
import { readFileSync } from "node:fs";
import { Command } from "commander";
import { describe, expect, it } from "vitest";
import { registerClientAuthCommands } from "../commands/client/auth.js";
import { registerCompanyCommands } from "../commands/client/company.js";
import { registerContextCommands } from "../commands/client/context.js";
import { registerIssueCommands } from "../commands/client/issue.js";
import { registerRunCommands } from "../commands/client/run.js";
// Chat agents copy these commands into their answers verbatim, so every
// command and flag the chat guidance names must exist in this CLI.
const guidance = readFileSync(new URL("../../../server/src/services/agent-conversations.ts", import.meta.url), "utf8");
function cliProgram() {
const program = new Command();
registerContextCommands(program);
registerIssueCommands(program);
registerCompanyCommands(program);
registerRunCommands(program.command("run"));
registerClientAuthCommands(program.command("auth"));
return program;
}
/** Resolves the command at the start of a guidance sentence; returns its path and the unknown part, if any. */
function resolve(program: Command, text: string): { path: string; error?: string } {
const tokens = (text.match(/<[^<>]+>|\S+/g) ?? []).map(token => token.replace(/[.,;:]$/, ""));
const path: string[] = [];
const result = (error?: string) => ({ path: path.join(" "), error });
let command = program, index = 0;
while (index < tokens.length) {
const next = command.commands.find(child => child.name() === tokens[index]);
if (!next) break;
command = next; path.push(next.name()); index += 1;
}
if (command === program) return result(`unknown command: ${tokens[0]}`);
if (command.commands.length > 0) return result(`unknown subcommand ${tokens[index]} of ${command.name()}`);
let argumentsLeft = command.registeredArguments.length;
const given = new Set<string>();
while (index < tokens.length) {
const token = tokens[index]!;
if (token.startsWith("--")) {
const option = command.options.find(row => row.long === token);
if (!option) return result(`unknown flag ${token} for ${command.name()}`);
given.add(token);
if (option.required || option.optional) {
const value = tokens[index + 1];
if (option.required && (!value || value.startsWith("-"))) return result(`missing value for ${token}`);
index += value && !value.startsWith("-") ? 2 : 1;
} else index += 1;
} else if (token.startsWith("<") && argumentsLeft > 0) {
argumentsLeft -= 1; index += 1;
} else break;
}
if (argumentsLeft > 0) return result(`missing arguments for ${command.name()}`);
const missing = command.options.find(option => option.mandatory && !given.has(option.long!));
return result(missing ? `missing required ${missing.long} for ${command.name()}` : undefined);
}
describe("chat CLI guidance", () => {
const program = cliProgram();
const commands = guidance.split("npx paperclipai ").slice(1).map(rest => resolve(program, rest.slice(0, 400)));
it("names the commands a person runs for tasks, documents, runs and context", () => {
const paths = commands.map(row => row.path);
for (const command of ["issue create", "issue document:get", "run get", "run cancel", "context set", "context show", "company current", "auth whoami"])
expect(paths).toContain(command);
});
it("names only commands, arguments and flags this CLI supports", () => {
for (const command of commands) expect(command.error, command.path).toBeUndefined();
});
it("detects an invented command or flag", () => {
expect(resolve(program, "run cancel <runId> --force now").error).toMatch(/unknown flag --force/);
expect(resolve(program, "context select <name>").error).toMatch(/unknown subcommand select/);
expect(resolve(program, "issue document:get <task> and more").error).toMatch(/missing arguments/);
expect(resolve(program, "context set --profile --use").error).toMatch(/missing value for --profile/);
expect(resolve(program, "issue create --project-id <projectId> creates").error).toMatch(/missing required --company-id/);
});
});
cli/src/commands/client/agent.ts
+56
View File
@@ -6,6 +6,10 @@ import {
updateAgentInstructionsBundleSchema,
updateAgentInstructionsPathSchema,
updateAgentPermissionsSchema,
updateAgentInteractionPermissionsSchema,
AGENT_INTERACTION_PERMISSION_KEYS,
type AgentInteractionPermissionsState,
updateAgentPermissionGrantSchema,
updateAgentSchema,
upsertAgentInstructionsFileSchema,
wakeAgentSchema,
@@ -462,6 +466,58 @@ export function registerAgentCommands(program: Command): void {
}),
);
function printInteractionPermissions(value: AgentInteractionPermissionsState, json: boolean) {
if (json) return printOutput(value, { json: true });
for (const key of AGENT_INTERACTION_PERMISSION_KEYS) {
const source = Object.hasOwn(value.overrides, key) ? "explicit" : "default";
console.log(`${key}: ${value.effective[key]} (${source})`);
}
}
addCommonClientOptions(agent.command("interaction-permissions")
.description("Show individual interaction restrictions and their defaults")
.argument("<agentId>", "Agent UUID")
.action(async (agentId: string, opts: BaseClientOptions) => {
try {
const ctx = resolveCommandContext(opts);
const value = await ctx.api.get<AgentInteractionPermissionsState>(apiPath`/api/agents/${agentId}/interaction-permissions`);
if (!value) throw new Error("Agent interaction permissions not found");
printInteractionPermissions(value, ctx.json);
} catch (err) { handleCommandError(err); }
}));
addCommonClientOptions(agent.command("interaction-permissions:update")
.description("Set individual interaction restrictions; null restores a default (human permission managers only)")
.argument("<agentId>", "Agent UUID")
.requiredOption("--payload-json <json>", "Partial interaction permissions map: true, false, or null")
.action(async (agentId: string, opts: AgentJsonPayloadOptions) => {
try {
const ctx = resolveCommandContext(opts);
const payload = updateAgentInteractionPermissionsSchema.parse(parseJson(opts.payloadJson));
const value = await ctx.api.patch<AgentInteractionPermissionsState>(apiPath`/api/agents/${agentId}/interaction-permissions`, payload);
if (!value) throw new Error("Agent interaction permissions not found");
printInteractionPermissions(value, ctx.json);
} catch (err) { handleCommandError(err); }
}));
for (const [command, enabled] of [["permission:grant", true], ["permission:revoke", false]] as const) {
addCommonClientOptions(
agent.command(command)
.description(`${enabled ? "Grant" : "Revoke"} one agent permission (Board permission manager only)`)
.argument("<agentId>", "Agent UUID")
.argument("<permissionKey>", "Permission key, such as commentary:view")
.action(async (agentId: string, permissionKey: string, opts: BaseClientOptions) => {
try {
const ctx = resolveCommandContext(opts);
const payload = updateAgentPermissionGrantSchema.parse({ permissionKey, enabled });
printOutput(await ctx.api.patch(apiPath`/api/agents/${agentId}/permission-grants`, payload), { json: ctx.json });
} catch (err) {
handleCommandError(err);
}
}),
);
}
addCommonClientOptions(
agent
.command("configuration")
doc/AGENT-ARTIFACTS.md
+18
View File
@@ -27,6 +27,12 @@ remote workspace. Supply a workspace-relative `contentRef`, basename `filename`,
The tool verifies the file, stores an attachment and artifact work product, and
binds it to the response. Generic API tools and a legacy API key are unnecessary.
`register_deliverable` stores the file on its current task. A completion report
can also cite a file on a direct child task that is done, because the child's
own completion verified that file. A parent that delegated a file to a child
links the child's file and does not register a copy. The child's file does not
count as delivery of a file that the parent's own request names.
Named JSON file outputs and explicit publication requests require the same
verified delivery evidence as other file outputs. A saved summary document does
not replace a requested downloadable JSON file. Reading JSON inputs or discussing
@@ -34,6 +40,18 @@ their contents inline does not by itself require a file attachment.
JSON output intent is recognized from a direct filename or an explicit output
destination. Ambiguous descriptions of input files joined by `to` remain text
requests; use `as`, `into`, or an explicit output file when an artifact is required.
A `Deliverable:` or `Deliverables` label also states the output. A file named
on that line, or in the list directly below it (blank lines between items are
allowed), is a requested file. This includes relative paths such as
`dist/index.html` and web page files such as `index.html` and `styles.css`.
Items that describe a repository change, pull request, commit, or branch do
not count, nor do files named as inputs (`a comparison of input.csv and
baseline.csv`), files the item forbids (`do not create report.pdf`), files to
be edited in place (`update styles.css`) unless the item also asks to publish
them, files to be written inline in the reply (`index.html and styles.css inline
in the reply`), or items that say no files are created. An inline or no-upload
instruction about another output in the same item does not cancel a named file
(`index.html and a short summary in the reply`).
The completion gate permits a narrow sentence-bound exception for one requested
file when the following sentence explicitly qualifies it, for example:
doc/AGENT-LIFECYCLE.md
+14 -3
View File
@@ -198,11 +198,22 @@ plugins are selected for all agents, including a sandbox provider that declares
`requiresAgent: true`.
When an agent moves to such an environment, the host adds the plugin at the next
`preparing` or `resuming` step. An agent in `ready` cannot move to it until it is
paused, because the plugin must finish its work before the agent uses that
environment. An agent in `verifying` returns to `preparing`. An agent that moves away keeps the selection until it is terminated,
`preparing` or `resuming` step. Moving a `ready` agent to an environment that
needs a new plugin automatically pauses execution, waits for the host and
existing plugins to stop, resumes with the new plugin, and verifies the saved
configuration before admitting work. The change and pause request commit
together; retries and server restarts retain the pending operation. Manual,
budget, and company holds still prevent automatic resume. An agent in
`verifying` returns to `preparing`. An agent that moves away keeps the selection until it is terminated,
so the plugin keeps its resources until then.
The existing agent configuration permission authorizes this move, including its
temporary system pause. That permission already controls environment selection
and resource allocation. Separate permission to issue a manual pause is not
required for the migration. Configuration and company access checks still run
before the change; an unauthorized request cannot save the environment or pause
the agent.
Saving an environment or the instance default that gives such a plugin to `ready`
or `verifying` agents without that selection returns a conflict. Pause those agents,
save the change, and then resume them.
doc/CLI.md
+12 -1
View File
@@ -9,7 +9,9 @@ Paperclip CLI now supports both:
## Browser-approved Board authentication
`npx paperclipai auth login --api-base <instance-url>` creates a ten-minute
challenge and prints a browser approval link. Sign in to the instance, review
challenge and prints a browser approval link. The link uses the instance's
canonical public origin when configured, so reverse proxies do not send the
browser to an internal upstream hostname. Sign in to the instance, review
the requesting command and full Board access, then approve. The pending token
cannot authenticate before approval. The CLI verifies its resulting identity
before storing its credential in its private auth store. The default Board key
@@ -586,6 +588,8 @@ Agent configuration and runtime endpoints:
```sh
npx paperclipai agent permissions:update <agent-id> --payload-json '{"canCreateAgents":true,"canCreateSkills":true,"canAssignTasks":true}'
npx paperclipai agent permission:grant <agent-id> commentary:view
npx paperclipai agent permission:revoke <agent-id> commentary:view
npx paperclipai agent configuration <agent-id>
npx paperclipai agent config-revisions <agent-id>
npx paperclipai agent config-revision:get <agent-id> <revision-id>
@@ -603,6 +607,13 @@ npx paperclipai agent instructions-file:put <agent-id> --path AGENTS.md --conten
npx paperclipai agent instructions-file:delete <agent-id> --path AGENTS.md --yes
```
`permission:grant` and `permission:revoke` require a Board login with
`users:manage_permissions` in the agent's company. They change one permission
key and preserve all other grants. A grant applies without starting a new run.
Granting a key replaces that key's scope with an unscoped company grant;
revoking removes the key. Use `agent get <agent-id> --json` to inspect
`access.grants`. These commands do not reactivate an inactive membership.
Agent config, instructions, skills, project env, environment, secret, and workspace edits affect the next run. Active runs finish with the config they started with. When a saved session, reused workspace, or sandbox lease no longer matches the effective next-run config, Paperclip may start fresh execution and records non-sensitive freshness categories in run result JSON and workspace operation logs.
`agent local-cli` is the quickest way to run local Claude/Codex manually as a Paperclip agent:
doc/DATABASE.md
+153
View File
@@ -122,10 +122,72 @@ original-volume anchor or filesystem read. These validations add bounded
per-retained-scope database reads during startup. They do not add a worker or a
poll. Missing or invalid byte evidence remains an enrollment blocker.
The original-volume owner can refresh a WEB or full native receipt whose database
facts digest has become stale only when fresh source inspection, current-key checkpoint
validation and ciphertext hash/size readback prove the preserved bindings unchanged.
It reuses the same archive and takes the existing finalizer, task and run fences before
comparing the prior receipt digest and replacing it. The company activity log
retains the prior database and local facts digests, source witness fingerprint,
observation time and receipt digest. Those four changed fields must reconstruct
the exact prior proof from the unchanged current bindings before publication. No source paths
enter the audit delta; normal activity redaction remains enabled. Changed
evidence, concurrent invalid receipts and active sources remain blocked; remote
recovery obligations are unchanged.
Full native reproof still requires canonical destroyed-provider receipts. A
database-only legacy workspace migration marker can stale the facts digest while
the preserved controller, skill assets and local byte proof remain unchanged;
refresh authenticates that same object and does not upload a replacement archive.
Local inode and timestamp observations may change between preservation attempts
without changing byte coverage. Valid receipts remain idempotent after those
changes; stale database facts can refresh with the same archive and fresh local
observations. During one attempt, the local metadata digest must remain stable
from capture through the locked publication check. Changed bytes, paths or modes
still invalidate preservation evidence.
Receipt validation adds bounded per-scope database reads to startup eligibility.
It does not add a recurring scan or network work. Qualification must measure the
selected cohort's startup latency on the serving revision before enrollment.
### Workspace cohort admission proofs
A saved cohort keeps its exact members, operations and snapshots on resume.
Independent companion imports can change the current source-scope proof without
changing those members. When every member has a running journal and its current
fingerprint matches both saved fingerprints, the importer can reprove the current
scope under fresh source, lifecycle and finalizer locks. After current source and
recovery checks, one transaction compares every journal and refreshes only the
cohort proof for all members. Cancellation or a changed journal rolls back the
whole refresh. Partial reservations still require the original aggregate proof.
Archives, recovery obligations and publication checks retain their existing owners.
Git cache expiration treats a saved migration marker or unfinished migration
journal as a durable consumer. It checks this under the lifecycle lock. A failed
migration with `lastFailure` and no `retiredSave` also keeps its source and
snapshot retention hold after process loss.
If an older server already expired members of an unpublished cohort,
`supersedeExpiredRemoteWorkspaceMigrationCohort` accepts the exact saved cohort
and the exact expired member IDs. This is an internal service API for a privileged
operator with a real database handle and reviewed company/cohort inputs. It does
not add a CLI mode or an HTTP route. An operator calls the exported owner directly;
its test and the reviewed one-time administrative qualification use that path.
It requires completed Git expiration and an
unexpired restore-point recipe for each missing member. Other members must retain
their saved source fingerprints. Fresh source, lifecycle, finalizer and row locks
protect one transaction that fails every old journal and removes only its matched
active marker. It does not report migration success or change task state. The
old cohort proof, operations, snapshots and object references remain unchanged.
The failure receipt retains those objects through the existing failed-save hold.
The optional `sourceLockWaitMs` uses the existing bounded source-lock queue.
The default still refuses a busy source immediately. Cancellation targets this
waiter only. All source and recovery checks run again after lock admission.
New canonical imports use the current archived recipe or current online source.
They record the old operation and supersession IDs and reserve a snapshot
generation above the existing ledger maximum. A successful new import does not
retire the old hold. Release of those historical objects requires separate review.
### Original local Git sources for archived workspaces
When an archived runtime branch has an exact successful deletion receipt, its
@@ -333,11 +395,49 @@ by company and normalized session. Migration 0432 builds both indexes
concurrently. A poll therefore reads a few index entries, not a run's full event
history.
The attention feed finds retry-exhausted runs through the partial index
`heartbeat_run_events_retry_exhausted_idx`. The index covers only `lifecycle`
events whose message starts with `Bounded retry exhausted`, keyed by company,
run, and newest event first. The query predicate must stay textually equal to
the index predicate, or the scan reads every company event again. Migration
0454 builds the index concurrently.
A subtree pause release finds the runs that a dispatch gate refused for that pause
through the partial index `heartbeat_runs_tree_hold_suppression_idx`. The index
covers only cancelled runs with a `treeHoldSuppression` marker, keyed by company and
hold. Migration 0436 builds it concurrently.
The budget worker finds committed budget Stops that are not delivered yet
through the partial index `heartbeat_runs_pending_budget_cancellation_idx`. The
index covers only runs whose `result_json.startupCancellation.budgetEnforcement`
is an object without `deliveredAt`, keyed by company. Its predicate must stay
identical to `pendingBudgetRunCancellationPredicate` in
`server/src/services/heartbeat/budget-cancellation.ts`; a regression test checks
that the planner uses it. The worker check, run control, and the idle-sleep
check therefore read a few index entries, not every run's result JSON. The
migration builds the index concurrently. To compare both plans on a disposable
local fixture:
```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-cancellation-scan.ts --runs=20000 --payload-kib=16
```
GitHub review lookups use four indexes from migration 0456, which builds them
concurrently. `chat_message_links_company_comment_idx` finds a source comment's
link for inherited review waits and run-origin checks.
`chat_message_links_delivery_idx` finds a delivery's links.
`chat_deliveries_github_automatic_delivery_idx` finds a delivery by its GitHub
webhook delivery ID on one endpoint. These three are partial.
`heartbeat_runs_company_wake_comment_idx` keys runs by company and
`context_snapshot.wakeCommentId`. It is not partial, because the planner does not
use expression statistics from a partial index; without them it estimates many
matches and walks the run history by creation time instead. The review check
reads the delivery's comment IDs first and then the newest same-agent run woken
by one of them, so it no longer walks the agent's run history. The two link
indexes also serve the comment and delivery `on delete set null` actions. Issue
lists share one external review-wait read per issue between the review and
blocked-inbox attention maps.
The chat wake-notice sweep compares UUID columns directly against guarded,
canonical lowercase UUID payload values. Malformed and noncanonical payloads
still match nothing. The query can use existing primary keys without adding a
@@ -1577,6 +1677,18 @@ Lifecycle polls can page past failed resources using the last returned event id
cursor, never a persisted high-water mark. This retries failures and includes
transactions that commit later with lower ids.
A lifecycle page reads the company's unacknowledged events for the requested
types once. It keeps one head for each resource: the unacknowledged creation,
else the oldest unacknowledged event. Then it applies `afterId`, id order and
the limit. Cost grows with the company's journal size, not with pending events
times acknowledged history. The idle-sleep check only asks whether any
unacknowledged event exists, because each such resource has a head. No index
or migration is needed.
```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=02bfe71a6b
```
## Legacy controller ownership
Legacy run claims atomically record `controller_boot_id`, a database-clock
@@ -1891,6 +2003,22 @@ delivery queries, including complete positional results and database plans.
All data is synthetic. Query timings exclude process liveness probes, HTTP and
provider work.
The reconciler's lost-wake check for user comments reads in three stages.
The first stage keeps recent user comments that no wake request carries. The
second stage removes comments with a live or later run, and it requires an
earlier run. Each stage probes native and legacy run bindings separately with
existing indexes. Only the remaining comments reach the steer check, which
reads run event payloads. An OR across both bindings let PostgreSQL hash the
runs by company only and test each comment against every run of its company.
No new index or migration is needed.
```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-comment-wake-check.ts --baseline-ref=02bfe71a6b
```
This disposable fixture compares the complete check result, the complete
positional SQL rows, plans, and timings with and without nested-loop joins.
### Agent runtime synchronization journal
Migration 0347 adds the opt-in `agent_runtime` resource stream. Database triggers
@@ -2030,6 +2158,31 @@ retains its previous usage in `previous_usage_json`.
Unknown late run receipts remain durable per-receipt debt; resolving one receipt
cannot clear another unresolved receipt.
Live usage receipt writes use READ COMMITTED and the existing run-row lock. All
mutable receipt and accounting reads occur under that lock. A streamed run-log
append between company validation and lock acquisition therefore cannot exhaust
serialization retries. A concurrent finalization is read under the lock and
keeps the normal late-receipt reconciliation path. Other independent accounting
operations retain their repeatable-read snapshot. Receipt identity, replay
conflicts, company scoping and budget-work publication remain transactional.
Run budget admission (`reserveRunBudget`), the final dispatch gate's accounting
phase (resolved-interaction dispatch only), the usage recorder's receipt-source handoff and its capture-failure fence
use the same pattern. The dispatch gate's ownership phase already ran at READ
COMMITTED under the same issue and run locks. Retry promotion and stale-run
cancellation keep their repeatable-read snapshot because they combine many facts
in one decision. Admission reads all
mutable state after the shared admission fences, the agent share lock and the
run-row lock. Its exposure is one statement, and the per-run reservation index
plus the run lock prevent a second hold. Run-log appends and output progress that
update a preparing run therefore cannot exhaust admission retries. Concurrent
admissions of different runs remain best-effort, as before. When any operation
transaction still exhausts its retries, it raises `AccountingContentionError`. The
error carries a bounded message, keeps the SQLSTATE in `code` and the database
error in `cause`, and is logged with both. A run that fails this way keeps the
generic `adapter_failed` code, so existing retry paths apply, and records that
bounded message with the SQLSTATE as its failure cause.
Upgrade all controllers/workers to this accounting contract and drain incompatible
pre-upgrade database transactions before enabling independent writers. Do not run
pre-upgrade synchronous counter/evaluation writers beside unlocked writers.
doc/DEVELOPING.md
+22 -8
View File
@@ -1068,8 +1068,10 @@ npx paperclipai allowed-hostname dotta-macbook-pro
## Agent status recovery
Lifecycle verification stops automatic attempts after missing authentication,
an unavailable model or command, an invalid working directory, or unexpected
probe output. A generic failed test gets up to three failures before automatic
an unavailable model or command, an invalid working directory, unexpected
probe output, or an owned AI account selection refusal. Account selection guidance
names missing defaults, unavailable accounts, incompatible bindings, missing
responsible users, or credential sharing requirements without provider error text. A generic failed test gets up to three failures before automatic
checks stop. The saved `verificationFailures` count only advances when the host
verification test fails; claims deferred before the test do not spend that budget.
An explicit retry or execution configuration change resets that count.
@@ -1138,12 +1140,19 @@ Prepare a new or updated agent worktree with one command:
node scripts/agent-preflight.mjs
```
It does what `pnpm install --frozen-lockfile`, the Runner's `ensure:eval-build-deps`,
`build:rust`, `build` and `build:runner-binaries` did, and skips work whose inputs
are unchanged:
It does what `pnpm install --frozen-lockfile`, the plugin SDK's `ensure-build-deps`,
the Runner's `ensure:eval-build-deps`, `build:rust`, `build` and
`build:runner-binaries` did, and skips work whose inputs are unchanged:
- **install** is skipped when the lockfile, every `package.json`, `patches/`, Node
and pnpm match this worktree's last install.
- **plugin SDK build** runs `scripts/ensure-plugin-build-deps.mjs`, which builds
`@paperclipai/plugin-sdk` and its `@paperclipai/shared` dependency and skips
them when their content fingerprints match the last build. Like the Runner and
the eval kernel, the SDK is imported through its built `dist` (plugin workers
load it under plain Node), so the server and the restart-chaos harness cannot
start without it. Both compile in about two seconds, so they do not use the
shared cache.
- **Runner build** is skipped when the Runner's build stamp is current. When another
worktree built the same inputs (the build-input digest described under
"Reused Runner build in the server build", plus the compiler), the Runner and eval
@@ -2597,7 +2606,7 @@ The default `worktree init` still seeds eagerly. A lean worktree (created withou
- `pnpm paperclipai worktree ensure-seeded` performs the deferred seed **exactly once**. It is lock-guarded and idempotent: only a complete `verified` manifest short-circuits it, so it is safe to call repeatedly and from concurrent processes. Managed workspaces derive the source from the control-plane-provided base project workspace when it carries its own `.paperclip/config.json`, and otherwise from the control plane's own registered instance config; either way the workspace's manifest never selects it. Manual worktrees must pass `--from-config`.
- `paperclipai run` calls `ensureWorktreeSeeded` automatically before doctor/boot. Managed runs transparently seed a lean worktree from their registered base workspace; an unmanaged lean worktree must first run `worktree ensure-seeded --from-config <source-config>`.
- Managed Paperclip git worktrees default to the repository's `scripts/provision-worktree.sh` when the strategy omits `provisionCommand`. When a registered source config exists, setup creates the isolated config and pending manifest. Runtime startup also runs `scripts/provision-worktree-runtime.sh` automatically when no explicit runtime provision command is configured and the manifest is not verified. Explicitly configured provision commands still take precedence.
- Managed git worktrees run a repository's `scripts/provision-worktree.sh` when the strategy omits `provisionCommand` and that file exists. The Paperclip repository deliberately does not ship a file at that path, so agent turns in Paperclip worktrees do not wait on dependency installs or instance setup. Its setup script is `scripts/setup-paperclip-worktree.sh`: run it by hand, or set it as the project's `provisionCommand` to opt in. When a registered source config exists, it creates the isolated config and pending manifest, then installs dependencies. Runtime startup still runs `scripts/provision-worktree-runtime.sh` automatically when no explicit runtime provision command is configured and the manifest is not verified; that script runs `scripts/setup-paperclip-worktree.sh` first when the worktree has no config. Explicitly configured provision commands still take precedence.
- An environment-configured server may have no local seed config. If neither the base checkout nor the default control-plane instance has one, a fresh worktree prepares its dependencies without creating a development instance. This also applies when `PAPERCLIP_CONFIG` names the default `$PAPERCLIP_HOME/instances/$PAPERCLIP_INSTANCE_ID/config.json` path, as the Docker image does. Setup creates no config, environment file, or seed manifest and does not claim runtime or seed readiness. A later request for a seeded development runtime still needs a canonical registered source config. Once that source exists, provisioning the checkout again creates the development instance normally.
- A missing custom `PAPERCLIP_CONFIG`, rejected symlink, or non-regular source file still fails setup. An existing worktree with a config, environment file, or seed state also fails if its source disappears; setup never downgrades that instance to a plain checkout. Repair the source before retrying.
- The built-in deferred seed is recorded as its own terminal `workspace_seed` operation. A zero exit code is not enough for success: the operation succeeds only when `.paperclip/seed-manifest.json` contains complete verified evidence; failed, missing, or malformed manifests produce a failed operation with the seed phase in metadata.
@@ -3189,6 +3198,10 @@ Saved OpenCode agents keep their runner and runtime policies.
Plugin environment drivers can declare `requiresAgent: true` when their leases
need an agent-owned resource. The host selects that plugin's lifecycle work only for
agents on that environment. See [Required plugin work](./AGENT-LIFECYCLE.md#required-plugin-work).
Moving a ready agent to an environment that needs a new plugin automatically
pauses execution, waits for stop completion, starts the required resources, and
verifies the saved configuration before allowing work. Existing pause holds
still apply; the user does not need to pause or resume the agent for this move.
Setup saves the draft and environment, runs its
existing lifecycle preparation, and waits in `verifying` with the `setup` hold.
The worker becomes idle there until the user finishes hiring. Connection tests
@@ -4690,8 +4703,9 @@ whether or not anything is left to compact.
A compacted worktree is in the same state as a freshly created one, so when the
task resumes, whatever produced those files the first time produces them again:
reuse runs the workspace provision command (or `scripts/provision-worktree.sh`),
which reinstalls dependencies when `node_modules` is missing. Repositories without
reuse runs the workspace provision command (or a repository's
`scripts/provision-worktree.sh`), which can reinstall dependencies when
`node_modules` is missing. Repositories without
a provision command rely on the agent's own install step, as they do for a new
worktree.
doc/PRODUCT.md
+2
View File
@@ -499,6 +499,8 @@ The agent performs authorized setup, reuses saved accounts and drafts, and hands
Authorized policy administration includes user-requested changes that loosen the acting agent’s own tool controls. These writes remain bounded by management grants, resource scope and the responsible user’s current authority; they do not need an additional approval solely because the agent is a target. Subsequent tool calls obey the resulting policy. Agents cannot approve their own governed action requests or edit their own principal permission grants.
Ordinary tool approvals remain reviewable without an implicit one-hour deadline. Authenticated OAuth refresh preserves a pending approval for the same saved action and authorization. Sign-in repair retains an accepted decision and resumes through connection-change events after revalidation; changed consent, permissions, credentials or action targets require a new review. Current access, explicit deadlines, task closure and execution receipts remain enforced. See [Delayed tool approvals](execution-semantics.md#delayed-tool-approvals).
The implementation, operation inventory, permission matrix and fixture qualification are recorded in [Agent API parity for connection configuration](plans/2026-10-13-connection-api-parity.md). Provider fixtures establish API behavior; they do not constitute live provider setup or an empirical agent-success claim.
## Budget control behavior
doc/SPEC-implementation.md
+81 -20
View File
@@ -881,6 +881,18 @@ Agent-authenticated changes cannot set or restore host-executed process adapter
configuration, including commands and environment values. Agent-authenticated
rollbacks cannot restore host-executed workspace commands either.
### Individual agent interaction restrictions
The optional `agents.permissions.interactions` object holds independent incoming
and outgoing controls for assignments, mentions, comments, task edits and peer
configuration, plus a separate outgoing assignment-to-humans control. Absence
adds no restriction; explicit false cannot be bypassed by grants or agent roles.
Human permission managers alone write the subtree through the dedicated API/CLI.
No migration, default write, grant backfill or UI is introduced. Mutation and
pending-delivery authorization use current settings while preserving history and
running work. See [the interaction permission contract](agent-interaction-permissions.md)
for the settings, ownership rules, atomic updates and compatibility boundaries.
### 9.3.1 Shared default-open issue writes
For standard-trust agents, issue comments, issue field/status updates, child
@@ -1520,6 +1532,20 @@ timestamps, including on repeated release. For unfinished issues it clears the
agent assignee; only `in_progress` changes to `todo`. Existing company access,
assignee/run ownership checks, and the `issue.released` activity event still apply.
A status change releases the checkout and execution locks but does not stop the
run that holds them. An agent request (`PATCH /issues/:issueId` or a recovery
resolution) that sets a status other than `in_progress` (even its current value) on
a task whose lock is held by a different queued or running run is refused with
`409` and code `issue_status_change_live_run`;
otherwise the run would keep going with no task bound to it, which is the effect
of interrupting another agent's run. The denial is final and names the person's
controls on the linked task: the run's Stop or Interrupt, or More task actions,
then Pause subtree on the parent. The lock-holding run, board users, and paths
that stop the run themselves (cancelling the task, reassigning it, and completing
through `PATCH` a task whose Runner goal is paused first) are unaffected. The write
is conditional on the lock the check saw; a run that acquires the lock in between returns `409`
`issue_execution_lock_changed`.
`POST /issues/:issueId/admin/force-release` is an operator recovery endpoint for stale harness locks. It requires board access to the issue company, clears checkout and execution run lock fields, and may clear the agent assignee when `clearAssignee=true` is passed. The route must write an `issue.admin_force_release` activity log entry containing the previous checkout and execution run IDs.
## 10.5 Projects
@@ -1769,7 +1795,10 @@ hard stop refuses new work even while evaluation lags.
waits on another run's cleanup or accounting completeness. Only these refuse it:
- spend at the limit (ledger plus observed running spend), or unpriced usage
under a blocking policy (the hard stop);
a provider reported, under a blocking policy (the hard stop). A charge that
accounting settled without its receipt (lost, closure evidence that never
arrived, or a failed capture) is recorded and shown in Costs, but it is never
a company gate;
- an explicit pause of the company, project or agent;
- an inactive company.
@@ -1986,12 +2015,12 @@ its display label after fifteen minutes. Elapsed time never discards a late
receipt. It settles the debt only for a native run whose recovery has ended
(terminal failure or an accepted result) and whose closure evidence has not
arrived after ten minutes (the remote runner reconnect grace plus a margin): the known usage is recorded as unpriced with
provenance saying closure evidence never arrived, an activity entry names the
run, and the blocking unpriced-usage policy keeps counting that charge unless
a control-plane restart caused it. A restart-derived receipt (the run replaced
its receipt source, recovered from a restart, or lost a provider turn on
restore) never hard-stops or pauses a company under the blocking policy,
whatever its token count; its activity entry and provenance remain the notice. A lost
provenance saying closure evidence never arrived, and an activity entry names
the run. Like any charge accounting settled without its receipt, it never
hard-stops or pauses a company under the blocking policy, whatever its token
count; its activity entry and provenance remain the notice. A verified local
process stop closes the receipt source for a failed, cancelled, timed-out or
interrupted run alike, so a restart-cancelled run settles at once. A lost
receipt that already carries a provider-reported cost settles at that cost as
an estimate marked as a lower bound, not as unpriced. A failed native run that
never started provider work (no usage receipt, no replaced receipt source, and a
@@ -2001,7 +2030,12 @@ that every usage recorder binds is not evidence that provider work began. A nati
turn that the restored Runner reports as lost on restore can never report its
usage. It does not hold the run's receipt open: the receipt closes when the
run's other turns do, and its provenance says the observed usage is a lower
bound. Raw provider errors and credentials are excluded from
bound. The same applies to a turn that was open when the provider session was
restored after process loss and that the restored Runner then reports
terminal without a complete usage report: the provider process that would have
sent the final report was stopped. Local and remote Runners both apply it; the
remote producer journals the restore as a receipt so a replacement controller
rebuilds the same turn state. Raw provider errors and credentials are excluded from
cost details. ACP usage retains the runtime-selected model, including defaults;
conflicting before/after model IDs do not authorize a single-model estimate.
Remote OpenCode billing uses only credentials forwarded to the execution target.
@@ -2406,6 +2440,13 @@ contract, setup, recovery, boundaries, and qualification status.
### Native task completion
A durable question or approval handoff survives a missing provider shutdown
notification. Paperclip settles its yielded result once the provider terminal
or exact host-owned shutdown proves stopped execution and settled tool outcomes.
Until then it retains the saved handoff and explains the shutdown hold. Recovery
must not repeat completed work or release an overlapping execution. See
`doc/execution-semantics.md` for the bounded settlement and restart contract.
For ordinary low-risk tasks, accept the current agent's structured `done` claim
subject to explicit workflow constraints. Missing independent evidence or a
`needs_review` label alone must not create a human approval. Require a concrete
@@ -2797,18 +2838,36 @@ cherry-pick, revert, merge, and bisect. The source files remain protected. Finis
or abort the Git operation before saving. V1 does not claim a complete saved
copy while these continuation records cannot be restored safely.
Saved-copy browsing restores and verifies the complete archive for each request.
Availability batches share one verified access per execution workspace within
the request and allow at most eight remote workspace accesses. Each path still
checks current authority. V1 has no extraction cache across requests. Large saved workspaces can therefore need
substantial transfer time and temporary disk space for each browse request.
Saved Files access admits at most two open extractions per process. It limits an
encrypted archive to 512 MiB, its manifest to 4 MiB, and extraction to 10,000
entries. It reserves at most 2 GiB of temporary space across those accesses and
checks free disk space with a 256 MiB margin. Capacity failures stop before an
object download. Cleanup releases capacity after it removes the owned files.
These limits apply to saved-copy file access. Agent workspace saves and restores
retain their existing archive limits.
Saved-copy browsing reads the committed archive in place. It range-reads and
decrypts the archive header and manifest, and verifies the manifest against the
snapshot's recorded manifest digest. Listing and file details come from that
verified manifest. Reading or downloading one file range-reads and decrypts only
that file's bytes into a private temporary file, verifies them against the
file's manifest SHA-256, and only then serves them. No browse request downloads or
extracts the whole archive, so archive size does not limit saved Files access.
A preview decrypts and verifies the file in memory, within its preview size limit.
A download reserves the file's size against a 4 GiB per-process budget and checks
free temporary disk with a 256 MiB margin before writing. Cleanup releases the
reservation. A file larger than the budget reports `too_large`. A storage read
that sends no bytes for five minutes is closed. The process keeps parsed
manifests of recent snapshots, charged by manifest size plus their entries, up
to 256 MiB. Concurrent opens of one snapshot share one manifest read, and at most
two manifests are read and parsed at once per process. Manifests are limited to 64 MiB, the archive format's limit, and
500,000 entries; a larger saved copy reports `too_large`.
Saved files carry no modification times, so the recent view of a saved copy is
ordered by path. Availability batches share one verified access per execution
workspace within the request and allow at most eight remote workspace accesses.
Each path still checks current authority.
The changed-files view needs the repository status, so it still restores the
complete archive for that request. That extraction admits at most two open
extractions per process. It limits an encrypted archive to 512 MiB, its manifest
to 4 MiB, and extraction to 10,000 entries. It reserves at most 2 GiB of
temporary space across those accesses and checks free disk space with a 256 MiB
margin. A saved copy beyond those limits reports changed files as unavailable;
its other views stay available. Capacity failures stop before an object download.
Cleanup releases capacity after it removes the owned files. These limits apply to
saved-copy file access. Agent workspace saves and restores retain their existing
archive limits.
Each managed repository preparation also saves a complete workspace archive.
These saves preserve each preparation boundary; V1 does not combine them or use
incremental uploads.
@@ -2963,4 +3022,6 @@ The agent performs authorized setup, reuses saved accounts and drafts, and hands
Authorized policy administration includes user-requested changes that loosen the acting agent’s own tool controls. These writes remain bounded by management grants, resource scope and the responsible user’s current authority; they do not need an additional approval solely because the agent is a target. Subsequent tool calls obey the resulting policy. Agents cannot approve their own governed action requests or edit their own principal permission grants.
Ordinary tool approvals remain reviewable without an implicit one-hour deadline. Authenticated OAuth refresh preserves a pending approval for the same saved action and authorization. Sign-in repair retains an accepted decision and resumes through connection-change events after revalidation; changed consent, permissions, credentials or action targets require a new review. Current access, explicit deadlines, task closure and execution receipts remain enforced. See [Delayed tool approvals](execution-semantics.md#delayed-tool-approvals).
The implementation, operation inventory, permission matrix and fixture qualification are recorded in [Agent API parity for connection configuration](plans/2026-10-13-connection-api-parity.md). Provider fixtures establish API behavior; they do not constitute live provider setup or an empirical agent-success claim.
doc/agent-interaction-permissions.md
+142
View File
@@ -0,0 +1,142 @@
# Individual agent interaction permissions
Interaction permissions are optional restrictions on actions between agents.
They are independent of low-trust presets and principal permission grants. They
have no UI or bundled preset; a human permission manager configures individual
agents through the CLI or API.
Every setting defaults to unrestricted when absent. Existing agents are not
rewritten, new agents do not persist default values, and no migration or grant
backfill is required. An explicit `true` removes this restriction only: company
boundaries, visibility, trust, responsible-user authority, grants, approvals,
checkout and execution rules still apply.
## Settings
| Operation | Outgoing setting | Incoming setting |
| --- | --- | --- |
| Assign/delegate work to an agent | `canAssignTasksToAgents` | `canReceiveAssignmentsFromAgents` |
| Assign work to a human | `canAssignTasksToHumans` | Existing human authorization |
| Mention another agent | `canMentionAgents` | `canReceiveMentionsFromAgents` |
| Comment on another task | `canCommentOnOtherTasks` | `canReceiveCommentsFromAgents` |
| Edit another task | `canEditOtherTasks` | `canReceiveTaskEditsFromAgents` |
| Configure another agent | `canConfigureOtherAgents` | `canReceiveConfigurationChangesFromAgents` |
An operation must pass both applicable sides. An explicit `false` is not
overridden by a CEO role, manager relationship, assignment grant, parent-report
exception, or the human responsible for the acting agent's run. Authenticated
human actions retain their ordinary authorization without these agent-only
restrictions.
An ordinary task is owned by its current agent assignee; an agent's own
conversation belongs to its canonical conversation agent. Human-assigned and
unassigned tasks are other tasks for outgoing comments and edits. Creating a
task, owning its parent, or formerly being its assignee does not establish current
ownership. Self-interactions retain existing authorization.
Assignments to a new agent check the sender and destination. Reassigning,
unassigning, or claiming an existing task also checks edits against its current
owner. Initial content on a newly assigned task is part of assignment; it does
not require permission to edit that agent's existing tasks. Creating or moving
a child under someone else's task checks the affected parent.
Comments and task messages share the comment controls. Mentions are checked
separately, so a permitted comment can still be denied for mentioning a protected
agent. A denial rejects the entire operation. Structured Paperclip agent links
are mentions; an ordinary name in prose is not. Allowed mentions retain their
existing context-link behavior and do not start execution or forward messages.
New or changed mention-bearing text is checked; unrelated edits do not rescan
untouched history. Delegated reports have no automatic exception.
Task edits include instructions, status, assignment, hierarchy, dependencies,
execution policy, documents, attachments, work products, and feedback results.
Compound operations must pass every applicable control. Posting a comment does
not require edit permission solely because the server updates bookkeeping.
Configuration controls cover peer instructions, settings, runtime configuration,
skill/tool assignments, and restores. Shared company resources retain their
existing permissions; changing a shared skill is not treated as editing every
agent that uses it. Initial hiring remains governed by hiring authorization.
## CLI and API
Use a board-authenticated CLI context whose human has `users:manage_permissions`.
Read access follows agent configuration read authorization. Agents, including
CEOs, cannot change these restrictions.
```sh
paperclipai agent interaction-permissions <agent-id>
paperclipai agent interaction-permissions:update <agent-id> \
--payload-json '{"canReceiveCommentsFromAgents":false}'
paperclipai agent interaction-permissions:update <agent-id> \
--payload-json '{"canReceiveCommentsFromAgents":null}'
```
The last command removes that override. Omitted keys are preserved; `null`
restores the default. Human-readable output identifies explicit versus default
values. Both commands support `--json`.
The underlying endpoints are `GET /api/agents/:id/interaction-permissions` and
`PATCH /api/agents/:id/interaction-permissions`. PATCH accepts a nonempty partial map
of the setting names to booleans or `null`. Unknown keys and malformed values
are rejected. Responses contain `agentId`, `companyId`, sparse `overrides`, and
the resolved `effective` map. Effective values describe this restriction layer,
not the outcome of every other authorization rule.
For an agent that should accept only human-directed interactions, explicitly
set all five incoming controls to false:
```sh
paperclipai agent interaction-permissions:update <agent-id> --payload-json '{
"canReceiveAssignmentsFromAgents": false,
"canReceiveMentionsFromAgents": false,
"canReceiveCommentsFromAgents": false,
"canReceiveTaskEditsFromAgents": false,
"canReceiveConfigurationChangesFromAgents": false
}'
```
This is a configuration example, not a stored preset. Outgoing controls remain
independent and can be set individually.
## Persistence and changes in flight
Only the dedicated endpoint changes `agents.permissions.interactions`. It merges
under the agent row lock and records the before/after audit in the same
transaction. Saving does not start work, retry a task, reactivate membership, or
alter unrelated grants. Pending hires cannot be changed before approval.
Other permission/configuration saves preserve the subtree, including when an
older client omits it. Attempts to change it through those APIs are rejected.
Restores cannot revive older interaction permissions. Company package exports
omit this instance's restrictions with a warning; configure the imported agent
explicitly after creation rather than copying another agent's authority.
Current restrictions are checked before pending execution and input delivery,
including queued steering. Coalesced input retains per-item origin, so a human
message does not authorize an otherwise prohibited agent message. Refused
delivery retains an inspectable permanent policy outcome instead of retrying
indefinitely. Removing the restriction later does not automatically replay the
refused delivery.
A refused child-completion report does not undo the child's completed work.
The report keeps a terminal refusal receipt. Dependency completion can still
notify the parent with a system event that contains no agent-authored report.
Permission changes do not delete saved comments, rewrite assignments, terminate
running work, or sanitize historical context. Subsequent actions from running
agents use current permissions. Older queue records use existing source
provenance when available; explicitly restricted work with insufficient
provenance cannot be assumed to be human-authorized.
Deploy compatible servers and workers before configuring restrictions. An older
binary cannot enforce these controls. No existing agent is configured during
deployment. These are addressed-action controls, not filesystem/network
isolation or a new read-access boundary: agents can still read otherwise
accessible history, shared files, skills, and external content.
Direct connection installs, profile bindings, and gateways that target an agent are configuration changes. Shared connection/profile changes and connection reads keep their existing authorization checks without treating every consumer as a configuration recipient. Mutation services recheck direct targets under ordered agent locks, independently of the shared-resource authorization check.
Plugin tool invocations retain the host-issued originating agent and run through host calls and queued delivery. A worker cannot replace that identity with a user or system label. Background plugin work without an originating agent retains its existing authority.
doc/agent-permission-defaults.md
+10
View File
@@ -44,3 +44,13 @@ Connection management always intersects the current human's authority, including
See [the authorization and qualification record](plans/2026-10-13-connection-api-parity.md) and the operation-reviewed [agent access maps](../maps/agent-access-map/connection-setup.md). Human consent, credential entry and identity confirmation remain bounded handoffs to saved setup.
Authorized policy administration includes user-requested changes that loosen the acting agent’s own tool controls. These writes remain bounded by management grants, resource scope and the responsible user’s current authority; they do not need an additional approval solely because the agent is a target. Subsequent tool calls obey the resulting policy. Agents cannot approve their own governed action requests or edit their own principal permission grants.
## Individual interaction restrictions
Optional incoming and outgoing [interaction permissions](agent-interaction-permissions.md)
can narrow assignments, mentions, comments, task edits, and peer configuration.
Missing settings preserve these defaults; no existing agent is backfilled and
new agents do not persist default overrides. A human permission manager sets
individual controls with `agent interaction-permissions:update`; explicit `true`
does not override another authorization denial. These controls are separate from
principal grants, `canAssignTasks`, and low-trust presets.
doc/connections/GITHUB.md
+7 -1
View File
@@ -154,7 +154,13 @@ repository owner first. The authenticated GitHub username and the accounts
where the App is installed are separate. An organization installation does not
grant access to the user's personal repositories. Personal repository creation
needs App access to that personal account; organization creation needs App access
to that organization and the tool's `organization` argument. The agent must
to that organization and the tool's `organization` argument. When the App is
installed on exactly one account, and its access snapshot identifies that
account as an organization, `create_repository` and `fork_repository` calls that omit `organization` use
that organization, because the personal account is unavailable. Paperclip fills
it in before policy, approval, and audit, so approval cards and test results
show the owner. An explicit owner is unchanged, and any other installation set
keeps GitHub's personal-account default. The agent must
keep the owner requested by the user. The user can choose the intended account
from **Add More Repos on GitHub**, then select **Refresh access**. If creation remains
unavailable, the user can create the repository on GitHub and give the App access,
doc/execution-github-identity.md
+17 -1
View File
@@ -14,6 +14,15 @@ Delegated work and interactions persist their originating context. Retries retai
Executions with managed GitHub configured receive token-free `git` and `gh` launchers. Each launcher invocation resolves one eligible credential from the current run's accepted identity at operation start. A `gh` command's child Git processes inherit that command's captured identity; later steering does not change already-started operations.
When the broker returns a managed credential, `gh repo create`, its
`gh repo new` alias, and `gh repo fork` stop before they run. The message
names the connection's `create_repository` or `fork_repository` tool. Those
tools apply the connection's tool access rules, approvals, owner defaults, and
audit, which a raw `gh` command cannot. The agent then uses Git to push to the
new repository or to clone the fork. `--help` still runs. Without a managed
credential, `gh` runs these commands unchanged. This guidance is not a
security boundary: any command that receives the token can call the GitHub API.
A `git` command that never uses the network does not ask the broker. The launcher reads past the global options, such as `-C` and `-c`, and checks the subcommand against a short list (for example `status`, `diff`, `diff-files`, `write-tree`, `log`, `rev-parse`, `add`, `config`, `branch`, `checkout`). This includes the index comparison and tree writing used by workspace snapshots. An unknown subcommand or option still asks. `commit` and `var` still ask, because the managed author comes from the broker answer. The launcher keeps no broker answer between invocations and never writes a credential to a file.
GitHub credential capture reads the current task, project, and retained trust
@@ -55,13 +64,20 @@ the active run through that broker. They do not require a persistent GitHub CLI
login or inherit an ambient host broker capability.
Claude through native ACPX retains the controller-projected broker URL and
session/bridge capabilities alongside its managed launcher paths. Both isolated
session/bridge capabilities alongside its managed launcher paths through both
the Rust daemon-to-sidecar and TypeScript provider launch boundaries. Both isolated
and preconfigured homes use this transport for shell `git` and `gh` operations.
These capabilities stay out of persisted session environment data. Raw GitHub
tokens remain confined to each managed command; ambient host capabilities and
task-supplied broker overrides are excluded. A working connector tool alone does
not prove that the terminal received its broker transport.
The Rust sidecar launch accepts this transport only for Claude with a nonempty
managed launcher directory. Existing saved sessions retain their profile and
checkpoint format. A running provider needs its normal retirement and restart
to receive the repaired launch environment; reconnecting the GitHub account
does not repair a process that already started without the transport.
Without a managed identity, local commits default to
`Paperclip <noreply@paperclip.ing>`. The launcher writes this token-free default
to a private per-operation global Git configuration file, removed on exit.
doc/execution-semantics.md
+451 -16
View File
@@ -244,6 +244,32 @@ writer and finalization holds remain until ordinary snapshot publication and
cleanup complete. This fallback does not remove source files, rerun provider
work, or solve capacity failure while creating the workspace archive itself.
### Delayed tool approvals
Ordinary tool reviews have no implicit one-hour deadline. A person may approve
the saved action after the requesting run ends or the server restarts. Explicit
action deadlines and terminal-task expiry still apply. The shared interaction
payload represents the absence of a deadline with `toolAction.expiresAt = null`.
New connected-tool snapshots bind the reviewed arguments, tool contract,
connection settings, credential owner, consent and permissions. Authenticated
OAuth refresh preserves the vault's credential identity epoch; manual credential
replacement and new consent do not. Token versions and refresh timestamps may
change without invalidating the action. Fixed credential versions and non-OAuth
credentials retain their exact-version checks. Earlier signed snapshots retain
their original strict comparison and saved deadlines; the server does not infer
continuity for historical rotations or revive already-expired requests.
If sign-in is required before dispatch, the accepted decision remains saved.
The card shows that approval is waiting for the app connection and links to its
repair surface. Recovery waits without a recurring timer and wakes on committed
connection changes or startup. It checks the exact saved action and current
authority again. Repair under the same authorization can continue automatically;
new consent, changed permissions or changed credentials require a new review.
The existing execution claim and result receipt prevent repeated approval from
dispatching the action twice. Uncertain provider outcomes retain the existing
no-replay recovery contract.
### Answered Slack conversations
An accepted scheduled-monitor yield remains unfinished task work. Its committed
@@ -514,6 +540,8 @@ Another run's unfinished accounting or cleanup never refuses new work (see `doc/
A user's message refused by a pause (a paused agent, a paused company, or a budget pause that is not a reached hard stop) is saved as a deferred intent that names the refusal and its release trigger; it is never written as a terminal skip. The pause's own stop keeps it. Lifting the pause is the operator's explicit authorization to send what the pause refused, as Send is after a Stop: when the pause stopped the task's run, the lifted pause issues the task's next explicit wake, which adopts the saved message once. A task that never ran keeps the message in the task, and stranded-work recovery starts it.
A reached budget hard stop cancels the scope's running work. When a board member resolves the incident with Raise budget and resume, Paperclip claims every task run that the stop cancelled since the incident opened (runs the budget enforcement fenced, and runs the agent lifecycle pause stopped for this budget: each pause records why it started, and the runs it stops carry that cause), with that member and their session source, in one write before the incident closes, as rows in `budget_resume_claims`, outside the run's result JSON so a Stop settling at the same moment cannot erase them. This includes runs whose stop is still being delivered. Resolving again keeps existing claims. The execution-continuation worker delivers each claim once its run's stop has settled: it files that member's Retry for the task. The Retry clears the task's execution hold and is accepted only while the cancelled run is still the task's latest run and the task is open and assigned, checked under the task lock, so completed or continued work is never repeated. Its idempotency key names the cancelled run, so every pass, restart, or repeated resume returns the same single request. The claim and its activity entry settle in one commit. The Retry starts a new run through normal admission and charges no attempt to the cancelled run. Paperclip records `budget.cancelled_work_resume_requested` on the incident and `budget.cancelled_work_continued` (or `budget.cancelled_work_not_continued`, with the reason) on each task. A run a person stopped is never continued this way, even if budget enforcement also fenced it. A pause's stop is intentional, not a failure: a native run that the agent lifecycle pause or a budget pause cancelled keeps its task's status and never opens a reconciliation incident; resuming continues it. A retryable Retry refusal leaves the claim pending for the next pass.
The same holds for accepted work beyond a first assignment. A temporary refusal never terminally cancels or escalates accepted work: an unstarted queued run (a promoted comment, a Retry, or a recovery) stays queued, and stranded-work recovery leaves the task as it is instead of moving it to `blocked`. These waits are passive, so settling a run's accounting wakes queued-run dispatch, the saved-receipt worker, and stranded-work recovery for the company. The held work then starts through normal admission, once. When a native run reaches a terminal status through recovery, or releases its environment, each message saved behind it is hinted by its own receipt, so delivery does not wait out a company-wide rescan interval.
New comments received during an execution hold retain their individual deferred receipts and ordered comment ids. Release cannot drain those receipts while replay remains blocked; the next eligible wake can adopt them. Authorized external-chat requests also remain deferred with their exact durable receipt. They must use normal promotion and current authorization; a generic wake cannot adopt only their comment ids and discard their actor or session contract. A wait does not authorize replay, reset an incident retry budget, or bypass an interaction's delivery rules.
@@ -569,7 +597,44 @@ writer, measured from the time the writer ended. After the cutoff, preparation
fails with the unfinished-writer error and leaves the held files unchanged.
This wait is also one parked retry, for example after a reassignment stops the
writer. It is re-checked every minute while the ended writer's hold settles, is
admitted once the hold releases, and stops parking at the same cutoff.
admitted once the hold releases, and stops parking at the same cutoff. The
writer's final save commit and an unchanged release (below) make these waiters
due immediately instead of at the next re-check.
A writer that ended (failed, cancelled, interrupted or timed out) without
changing its files releases the hold at once, with no snapshot. Its workspace
keeps the saved version it started from. Preparation records that starting
point only when it began from a clean saved version and added nothing to it:
no recovered attachments, no repository outside the saved version and no
workspace migration. The release requires durable evidence that nothing ran
after that point: no execution or provider event, process, runner checkpoint,
native result or provider turn in a checkpoint for that run; no workspace
operation other than a restore or a finalize or recovery receipt; and no task
repository outside the saved version. The writer's owner fence must also be clear. Any missing or ambiguous evidence,
including a writer prepared by an earlier release without that starting point,
keeps the full save. The next run's reservation, cancelled-workspace
preservation and a later final save each apply this release. A repeated save
returns the same version.
A save that requires a full capture also keeps the current saved version when
the captured manifest, archive hash and size match the saved version. This
comparison applies only to the same company and workspace. The saved encryption
key must be readable. The archive must still exist with its recorded size. The save then completes its
normal ownership checks, run receipt, repository metadata and waiter wake
without uploading another archive. Its unused generation has no object and is
marked deleted. A retry returns the completed receipt. An earlier upload attempt
keeps its recovery path; it cannot be discarded by this comparison. Changed
contents still create a new version and retain the previous version for seven
days after replacement. Active work and recovery references can extend retention.
The reused archive keeps its original source run, generation and encryption
binding. Recovery and native replacement verify the new run's final save through
its succeeded operation and empty retired generation, including its epoch,
workspace, sandbox location, content hash and settled work. They do not borrow
the old run's authority. Read-only historical attachments keep this proof after
a later writer advances the workspace. Existing saves from earlier releases
keep their original receipt checks. Reuse metadata requests, including retries
and failures, count toward the save's storage request statistics.
A writer stopped mid-run, for example by reassignment, keeps its unsaved files
in its stopped sandbox. When the run used a reusable (warm) sandbox, the stop
@@ -593,6 +658,45 @@ failing setup. The save's retry budget bounds that wait. A save that exhausts
its retries is handed to the board as a named recovery action. After that hand-off,
the cutoff no longer advances.
Automatic wakes do not run into a hold that preparation would refuse. A parent or
dependent woken because a child or blocker stalled on the board
(`issue_blocker_stalled`) is not woken again when its assignee's last run on the
task failed setup on its bound workspace's hold and that hold is still past the
cutoff, or held by a writer that cannot be named: another run could only fail
the same way. The wake is delivered once the hold settles or the task changes
assignee, and the sweep checks again at most hourly. Failed runs that carry a
stall, including resource-wait retries of its wake, are retried after ten
minutes, then twenty, and not after the third failed run. The stall stays
listed on the waiting task, and a new stall wakes it again.
An unstarted queued turn names an agent setup or verification wait. A failed
setup that requires explicit Retry shows **Agent setup needs attention**, while
its saved turn remains queued. Temporary errors with automatic retries still show
**Waiting for agent setup**.
Queued setup failures also create the existing inline AI connection card on the
task, addressed to the person whose account host verification checks (the saved
lifecycle responsible user or company default). A queued turn can have a different
responsible user; neither its authority nor its saved identity is rewritten.
Runtime tokens remain unavailable for
queued runs. Completing the card rechecks verification through the lifecycle
worker and retains the original queued turn; it does not enqueue another model
turn. Decline, Stop, reassignment and other lifecycle holds remain effective.
Setup cards stay bound to their queued source turn. A replacement turn supersedes
the prior setup card instead of reusing a stopped turn's delivery.
New cards also name the lifecycle operation, so a stale completion cannot recheck
a newer configuration. Owned AI selection failures park during resume as well as
initial verification. For a routed agent, repair uses the account already pinned
to that lifecycle operation; it never allocates a task route, advances the pool
cursor, or replaces the agent's pool. Legacy direct-account setup cards remain
readable without an operation ID.
AI account selection failures use the selector's owned configuration evidence:
missing defaults, unavailable or incompatible connections, missing responsible
users, and credential sharing refusals park automatic host verification until
configuration repair or explicit Retry. They retain fixed, redacted guidance.
Provider and infrastructure failures keep their existing retry policy. Readiness
must commit before normal queue dispatch; this display grants no execution or
credential authority and does not change the task's lifecycle status.
An instance admission drain keeps accepted new turns queued and displays
**Waiting for maintenance**. Ending the audited drain requests admission for
these existing runs immediately. The shared delivery coordinator owns the drain's
@@ -686,6 +790,28 @@ its task tree created: the task that created the workspace keeps it, and the
relative bound to it (for example a parent that selected its child's workspace)
moves to its own on its next admission.
A legacy child can have no next admission: when another task's writer held the
shared workspace, the child's run failed setup before any provider work, and
stranded-work recovery blocked it with a board-owned escalation. The
active recovery action sweep moves such a child without a person, in one
transaction, when the escalation's latest run failed setup on that workspace
hold, the child's Blocked status came from that escalation (no status decision
at or after the block), the shared dependency-readiness rule allows work, and
the binding release above is allowed. The child's binding is released, the
escalation resolves with the outcome `legacy_shared_workspace_rehomed`
(`issue.legacy_workspace_rehomed` system activity), the child returns to
`todo`, and a pending wake is recorded. The assignee is then woken once with a
fresh session. Only a created run settles the wake: a wake lost to a crash or
refused by the heartbeat stays pending, backs off (5 minutes, doubling, at most
an hour), and is retried at that deadline while the child is still an assigned
`todo`. What the moved child
sees is a fresh workspace from the project base: no parent changes, and none of
its own earlier files from the shared workspace. The child that still holds the
shared workspace with unsaved files is not moved; its save recovery owns the
hold, its files stay in the shared workspace and its saved-work receipts, and it
moves on its first admission after that recovery settles. The parent keeps its
workspace throughout.
### Blockers (`blockedByIssueIds`)
This is dependency semantics.
@@ -1550,6 +1676,24 @@ and any run saved by an older release without these facts, keeps verified
reattachment. A reattachment that loses its runner still requests the user's
Retry as described above.
A local runnerd that dies (for example SIGKILL during a tool call) cannot
suspend, and its provider and tool processes are re-parented, so they cannot be
found from the controller's process records. Durable runnerd therefore journals
each provider's process lineage to `<runner state>/process-lineage/` from the
moment the provider is spawned: the provider, every descendant, and every
process group or session a descendant leads, each named by PID and start time.
When close finds that runnerd exited without suspending and without a signal
from close, it runs `paperclip-runnerd reap-lineage` from the same executable.
The reaper refuses while runnerd is alive or when runnerd never journalled. It
signals only processes whose PID and start time still match the journal, and it
reports `verified` only when none remain. The journal also records how far the
process table was read. A same-user orphan that started after that point cannot
be attributed, so the reaper refuses without signalling it. A failed journal write removes the writer marker, so a
record that may be missing a member never verifies. A verified report ends the session as
a crash stop with its durable record intact, and recovery continues within the
normal attempt budget. Any other report keeps the cleanup quarantine below.
Launcher-owned (remote) and adopted runners are not reaped this way.
When a native attempt fails before launch because its predecessor's session is
quarantined, explicit task Retry verifies the retained journal's actual owner.
That owner must belong to the same company, task, agent, workspace, native session,
@@ -1851,8 +1995,13 @@ A native run can accept its result and then lose its server to a hard kill
before finalization. Its result stays leased to the controller of the earlier
boot, which cannot release the lease the way a graceful drain does. Startup
native recovery releases that lease at once when the lease belongs to a
different boot and its recorded controller process has exited or its PID now
names another process. Only the executor's lease carries that controller
different boot in the same known kernel and PID namespace, and its recorded
controller process has exited or its PID now names another process. The
executor saves that namespace with its claim. Linux uses its boot UUID and PID
namespace; macOS uses its kernel boot-session UUID. An unreadable or unsupported
identity, a different namespace, or an older claim without this proof keeps the
original lease expiry and does not probe the recorded PID. Only the executor's
lease carries that controller
identity; a finalizer or cleanup lease is never released this way and expires
on its own short lease. The stored result is then
finalized without another
@@ -1860,22 +2009,146 @@ provider turn, and saved follow-up messages are reconsidered. A live owner, an
owner without a recorded process, a lease of the current boot, and a remote
workspace keep the lease until it expires. Lease age alone is never proof.
Before claiming an in-flight local native run, restart recovery also requires
this same known process space. Expiry alone does not prove a local controller
stopped. A missing, foreign or unreadable namespace creates one board-owned
recovery action for the exact saved authority, without probing the recorded PID,
spending retry attempts, or arming a polling loop. Renewed authority gets a new
opaque fingerprint and preserves the previous action. Remote and plugin tasks
retain their ordinary Stop/provider-cleanup path and do not offer the local
containment confirmation.
When automatic recovery cannot prove a local owner stopped, an operator can
record containment through the task's recovery action after the original lease
expires. The operator separately confirms the controller and provider/tools
stopped on their original host, and records reviewed action outcomes and evidence.
The server rechecks board `runtime:confirm_owner_stop` permission for a company
owner, company administrator or instance administrator, plus company, assignee, source binding,
opaque authority, exact expiry (including microseconds), pending Stop, accepted
results and cleanup obligations under the normal ownership locks. A renewed or
replaced owner invalidates the confirmation. This is explicit operator evidence,
not a local process-stop observation.
Containment uses normal failed-owner settlement and preserves usage uncertainty,
workspaces and outcome evidence. Containment records the canonical `restored`
outcome while keeping the task status unchanged, including when it is already
`todo`. A separate ordinary Retry then consumes that committed evidence. It does not accept a provider result or queue a
replacement. Only exact unadmitted Retry receipts for the contained source are
closed with an inspectable reason; accepted or newer work stays intact. After
cleanup and outcome review, the user makes a new ordinary Retry request under
current authorization. Unknown outcomes keep the hold; reviewed completed or
mixed effects remain in the continuation context and never authorize automatic
replay. A later successful ordinary recovery claim resolves only its exact prior
namespace attention.
A recorded process is judged only in the PID space that recorded it. Native
controller rows store `controller_host_hash` and run processes store
`process_host_hash` (`"remote"` for a sandbox process). A dead PID on this host
never proves that a process on another host, in another PID namespace, or in a
sandbox is gone; such a record keeps its lease, fence or staleness TTL.
Rows saved before those columns existed have no recorded host. For one release
only, a single-host server keeps the previous local proof for them, and each use
Rows saved before those columns existed have no recorded host. The legacy
compatibility helper and run-process sweep retain one-release single-host local
proof where their existing owners permit it. Each use
writes a `legacy_host_unknown_dead_pid_proof` activity-log entry, so the release that
removes this fallback can confirm that no rows still depend on it. An ephemeral
server (`PAPERCLIP_EPHEMERAL_SERVER=true`) never uses the fallback. Its native
controller rows with no recorded host wait as `controller_host_unrecorded`, with
a board recovery action to confirm that the previous server stopped. Its run
processes with no recorded host are left to the run's Stop control. The next
release removes the fallback and treats every row with no recorded host as
unprovable.
release removes that compatibility fallback. The strict Native in-flight claim,
early accepted-result release, and failed-owner guards above do not use it.
A previous-server-only confirmation is not same-namespace proof and does not
confirm that a provider or its tools stopped. It cannot override those guards.
Its notification requests recovery inspection; unfinished ownership can still
require the separate containment and ordinary Retry described above.
A server boot that holds work keeps a controller instance lease
(`controller_instances`): a row named by its boot id, with an epoch and a lease
on the database clock (90 s TTL).
- **Renewal.** The shared delivery-work coordinator schedules it every 30 s,
only while the boot holds work. A single upsert both creates the row on first
use and extends the lease, but only if the boot stamped ownership in the last
90 s or still has an in-progress record stamped with its boot id: an
environment lease or cleanup claim, a sandbox creation intent, a workspace
finalize attempt, an agent lifecycle step, a native finalizer or cleanup
lease, an open native coordinator claim, or a running legacy run. A stamp
whose ownership write may still be pending also counts until the server's
tracked work settles (at most 10 minutes). Otherwise the lease lapses and
the worker issues no statements until the next stamp, so an idle server's
database can sleep.
- **Renewal failure.** Each renewal has a 10 s bound. A failed or timed-out
renewal quiesces the boot: it reports a task drain and dispatches no new work.
If it renews again before anyone fences it, it resumes. It exits only if
renewal has kept failing for 60 s, counted from the first failed attempt's
start, while it still holds work.
A record owned by the server process itself (a lease, claim or intent with no
recorded process group) that names a boot in another host or PID space is
judged through that boot's row:
- `stopped`: the boot shut down cleanly, so the record's owner is gone.
- live lease: the owner is alive.
- otherwise, the boot's last sign of life is the later of its lease expiry and
the record's last write plus the TTL. A boot that resumes after an idle lapse
renews on its first stamp.
- Within 30 s after that, the owner still counts as alive, so the settle watch
re-checks it.
- After that, the prover fences the boot (only a newer epoch may), and the
owner is gone. A boot that held work has exited by then.
- no row (a boot from before the lease): the record stays unprovable.
A record with a recorded process group stays unprovable across hosts, because
agent processes may outlive their server. The lease is enabled on volume-backed
Railway services too: a process can crash and restart inside the same deployment,
and some saved ownership records do not contain a deployment identity.
`PAPERCLIP_CONTROLLER_INSTANCE_LEASE=off` explicitly disables it.
Native execution and restart claims register held preparation with this lease before
a runner PID or checkpoint exists. Passive identity reads do not register work.
Restart recovery uses the recorded controller
boot and last write to prove its owner stopped or to fence it after the existing
grace period when the PID space is foreign or unknown. Within the same readable
PID space, the existing process and start-time proof still governs takeover.
A live owner or a grace period keeps the preparation held, with a
scheduled recovery check. Repeating the same evidence wait preserves its last-write
timestamp so the recovery check cannot keep extending its own grace period.
After owner proof, the normal launch, checkpoint, provider-evidence and admission
checks still govern resumption; controller fencing alone does not prove an agent
process stopped or authorize repeating its work.
Older native controller records with no verifiable boot lease remain held. Recovery
exposes the existing board stop-confirmation action for an unknown or foreign PID
space. It requires `runtime:confirm_owner_stop` and confirmation of the exact run,
boot, PID, process start and recorded host. A changed owner rejects the confirmation.
The receipt only permits normal recovery to inspect and resume the saved work.
Scoped recovery wakes also select fenced preparations with no runner process or
checkpoint yet. They use the existing finite expired-run inventory and normal
restart classifier; a missing checkpoint cannot silently discard that wake.
A record saved before its owner was fully recorded (`no_owner_stamp`,
`missing_identity`, `process_space_unrecorded`) stays unprovable and is never
released on its own. For a settling kind that opts in, recovery asks the board
on the record's task (`ownership_owner_unprovable`) to confirm that the previous
server stopped. The board action (`POST /api/issues/{id}/recovery-actions/confirm-owner-gone`)
is board-only, needs `runtime:confirm_owner_stop`, writes an
`issue.ownership_owner_gone_confirmed` activity entry and wakes the reconciler.
The confirmation then proves the owner gone (`operator_attested`) and the kind's
own compare-and-swap settle releases the record:
- The environment run lease stores the confirmation on the lease.
- Every other opted-in kind (legacy controller lease, teardown claim, sandbox
creation intent, runtime-control operation, workspace finalize attempt,
native finalizer and cleanup leases) keeps it on the resolved recovery action,
bound to the record's last write. It applies only while the record is
unchanged and its recorded process is still unprovable for that legacy reason
(`missing_identity`, `process_space_unrecorded`). A record that changed since
is judged again, and recovery asks again about its current version. These
settles compare-and-swap on the recorded process, so a record of these kinds
with no owner stamp at all keeps its existing recovery path and is not offered.
- Kinds that are unowned by design, such as queued pending cleanup and
accounting debt, are never offered.
The stranded-work pass closes the gap where issue state survives a crash but the wake/run path does not. The silent-run scan covers the separate case where a live process exists but has stopped producing observable output.
@@ -2153,10 +2426,30 @@ failed or cancelled even when the agent already reported completed work.
The final assistant message is the visible task response. Response selection runs
after preceding event persistence completes; the completion summary cannot replace
an available final answer. Existing fallback and explicit-comment precedence still
apply. Stream closure without a turn terminal is not proof of success. Event
apply for ordinary tasks. In native standing conversations, a completed final
response or accepted non-yielded semantic response takes precedence over earlier
tool comments. Those comments stay in history. Identical text reuses its existing
comment. With no qualified final, the explicit comment remains the response.
Governed waits and interaction cards keep ownership. A server-proven finished
conversation may still have a yielded semantic result. Its completed final reply
remains visible; a completed unknown-channel terminal reply is used only under
that finished-turn authority and after the explicit final and non-yielded summary.
The presentation decision records the conversation-specific choice.
Stream closure without a turn terminal is not proof of success. Event
replay uses the existing source receipts and never repeats provider work merely
to recover recorded output.
In an Agent Chat conversation, a successful turn whose status decision committed
posts its final answer to the conversation even when the task stays held: a
pending or revision-requested approval, a pending execution stage, or a
dependency blocker. A held task does not present the turn. Only a pending
thread interaction (a question or confirmation card) that the turn's own run
created presents the turn in place of its prose. A card from an earlier turn,
or one the board addressed to another person, can hold the task but does not
present the turn, so its answer is still posted. Background presentation recovery applies the same rule, so a
restart between the status commit and the live presentation posts the held
turn's answer once, or records that its card presented the turn.
A delegated-task completion callback can finish native execution before its
accepted final reply is saved. The completion outbox must retain that callback's
queued target and attempt while the existing presentation selector finds pending
@@ -2390,6 +2683,30 @@ recovery worker discovers these holds itself, so holds saved by earlier releases
take the same path after deploy. An exhausted hold whose run's files a sibling
recovery action already saved settles as recovered.
An exhausted save whose last failure was `agent_stop_unconfirmed` is not a
refusal of the saved files: its recovery window ran out before the agent's stop
could be proven. This applies to every workspace recovery the worker owns,
export holds and restore holds alike. Once the source run is provably over (its
run is terminal, no local execution remains, and neither its controller lease
nor its finalizer lease is live), the recovery worker re-arms it with a new
bounded system generation. A live lease defers the re-arm to that lease's
expiry; the worker is not woken earlier. The new generation repeats the strict
stop proof (a provider stop receipt for every allocation of the run) and the
save, then settles the hold exactly like any completed recovery, so other tasks
on the workspace can start. Re-arms are bounded: at most four, the first at once
and the next ones 30 minutes, 2 hours and 8 hours after the previous re-arm,
each recorded in the action's `stopProofRecovery` marker and as
`issue.workspace_recovery_rearmed` system activity. When every re-arm of an
export hold is spent and the stop still cannot be proven, the hold takes the
retained-source fallback above: the unsaved files are attached to the task,
unapplied, and the task continues from the last save. A fallback whose bounded
checks are spent is reconsidered once whenever an allocation of the source run
changes afterwards, for example when a late provider stop receipt arrives.
While a re-arm remains, the hold names the wait and says that Paperclip tries
again automatically; after that it names Retry. Holds exhausted by earlier
releases carry no marker and are re-armed on the worker's first pass after
deploy, with no migration.
An exhausted workspace export records the exact Blocked status version it
creates, bound to its source run, result, contract and identity. Recovery may
restore the accepted passive wait only while that receipt still owns the task
@@ -2480,7 +2797,7 @@ replacement behavior. When a replacement does archive remote continuity state
and the remote move fails, the local state is not moved, and the run error keeps
the bounded, redacted exit status and output of that move.
Bootstrap retries, exact-checkpoint resumes, and fresh replacement sessions share three total provider attempts, including the original attempt. Linked run IDs, controller restarts, and duplicate wakes do not reset this budget. A resume after a graceful shutdown drain suspended the run for reattachment does not consume an attempt, because the run did not cause that restart. A crash restart (no drain record) consumes an attempt only when the attempt it interrupted spent provider work without progress. This is one rule over recorded outcomes, not over intents. The attempt counts as started only when the provider accepted a turn: a `turn.accepted` event or the provider's own `turn.started`. A startup still marked `beforeSpawn`, a runner that spawned but was never admitted, and a `turn.start` the provider did not accept are not charged. Progress means that an accepted turn then completed work: a tool execution or a completed item other than a usage report. An accepted turn with only usage, a failure, or no events is charged, so a resume that fails each time still uses up the budget. The first restart judges the original execution. Each later restart claim records the run's event-log position, and the next claim judges only the attempt recorded after it; a normal retry in between starts a new attempt (at its `native.process_start_requested`), so earlier work does not count for it. A reattachment adopts a provider that already started, so for it only progress counts. A resume that launches a new runner and that the controller died before starting used no provider work, so the next claim replaces it and the attempt ends at the attempt that resume was to run as. Uncharged restarts stop after 12 controller generations, so a crash loop that never starts the provider, or that makes progress and then crashes the controller each time, is still bounded. Legacy runs follow the same rule: a preparation interrupted before provider work started (`providerWorkStarted: false`) is retried in its own `interrupted_preparation` lane, which never spends the failure retry budget and is bounded at 12 retries. Exact-checkpoint resumes retain the 30-second delay. Replacement scheduling and predecessor lineage commit together, with one successor per predecessor and admission through the normal task locks, authorization, pause, approval, and budget gates.
Bootstrap retries, exact-checkpoint resumes, and fresh replacement sessions share three total provider attempts, including the original attempt. Linked run IDs, controller restarts, and duplicate wakes do not reset this budget. A resume after a graceful shutdown drain suspended the run for reattachment does not consume an attempt, because the run did not cause that restart. A crash restart (no drain record) consumes an attempt only when the attempt it interrupted spent provider work without progress. This is one rule over recorded outcomes, not over intents. The attempt counts as started only when the provider accepted a turn: a `turn.accepted` event or the provider's own `turn.started`. A startup still marked `beforeSpawn`, a runner that spawned but was never admitted, and a `turn.start` the provider did not accept are not charged. Progress means that an accepted turn then completed work: a tool execution or a completed item other than a usage report. An accepted turn with only usage, a failure, or no events is charged, so a resume that fails each time still uses up the budget. The first restart judges the original execution. Each later restart claim records the run's event-log position, and the next claim judges only the attempt recorded after it; a normal retry in between starts a new attempt (at its `native.process_start_requested`), so earlier work does not count for it. A reattachment adopts a provider that already started, so for it only progress counts. A resume that launches a new runner and that the controller died before starting used no provider work, so the next claim replaces it and the attempt ends at the attempt that resume was to run as. Uncharged restarts stop after 12 controller generations, so a crash loop that never starts the provider, or that makes progress and then crashes the controller each time, is still bounded. Legacy runs follow the same rule: a preparation interrupted before provider work started (`providerWorkStarted: false`) is retried in its own `interrupted_preparation` lane, which never spends the failure retry budget and is bounded at 12 retries. This holds for a hard kill as well as a graceful drain: once the killed controller's recorded process is proven gone, restart recovery fails the run as `process_lost` with the bootstrap certificate and schedules it in that lane. A run saved by an older release without a controller process stamp cannot be proven and stays in the charged lane. Exact-checkpoint resumes retain the 30-second delay. Replacement scheduling and predecessor lineage commit together, with one successor per predecessor and admission through the normal task locks, authorization, pause, approval, and budget gates.
A due failed remote retry can retain its provider PID after heartbeat completion
releases the task execution lock. Both remote workspace reference formats and
@@ -2662,6 +2979,33 @@ predecessor/successor receipts for every earlier owner. Unrelated, live, or
incompletely linked owners remain ambiguous. The run then fails closed
with `runner_harness_state_mismatch: backup_without_reusable_lease`.
A workspace-restore hold also takes a warm sandbox out of reuse. When a run
that holds a warm lease ends with a workspace-restore failure, the hold stops
the sandbox under a stop-and-retain receipt, changes the lease policy to
`retain_on_failure`, and records a `workspaceRestoreRecovery` marker with the
original policy. Workspace recovery then saves the files. In the transaction
that completes that recovery, Paperclip returns the lease to
`reuse_by_environment` only when all of these are true: the original policy
was `reuse_by_environment` (or, for a marker saved by an older release, the
lease carries its reusable-lease scope), the recovery is complete with a
saved-file receipt from this allocation, the stop receipt is confirmed with
state `stopped`, no unsaved writer or pending native workspace export is bound
to the allocation, the run's recorded disposition is not `destroy`, and the
allocation has one verified owner chain. The update is a compare-and-swap on
the checked row. It moves the marker to `workspaceRestoreRecoverySettled`, so
the next run resumes the warm sandbox. A hold that fails any condition keeps
its protection.
Older releases left settled holds in `retain_on_failure`, and reuse never
selects that policy. For those rows, the hold's lease also counts as a retired
backup source when all of these are true: the lease is `released` with policy
`retain_on_failure`, it has a `workspaceRestoreRecovery` marker for its own
run, it has a confirmed stop-and-retain receipt with state `stopped`, and no
unsaved writer or pending native workspace export is bound to it. The backup
stamp, single-owner, and later-provider-work checks above still apply. A
stopped lease that is still `reuse_by_environment` remains ambiguous and fails
closed.
A Runner close can also fail to prove suspension after the control plane
accepted the run's result (`native.post_result_failure` with category
`runner_settlement_incomplete`). The provider work is complete, so this is not
@@ -3224,7 +3568,8 @@ Recorded system recovery failures retain their original attempt and intent.
When system workspace recovery exhausts its bounded attempts or deadline, the
recovery action stays active and its owner changes from system to board. Its
next action names the attempt count and last failure code and tells the board to
use **Retry recovery**. The task blocker shows that same text. Retry starts one
use **Retry recovery**. An unproven agent stop (`agent_stop_unconfirmed`) instead
names the automatic re-arm described above; Retry remains available. The task blocker shows that same text. Retry starts one
new bounded system generation and returns ownership to the system. A sweep that
finds an action already exhausted and still system-owned applies the same handoff
as bookkeeping only; it grants no attempt. Exhaustion never submits a provider
@@ -3314,6 +3659,19 @@ ownership, active controllers, cleanup leases, pause, approval, budget, and norm
execution gates still apply. Dependency-blocked interaction mode remains limited
to its existing answer/triage contract.
A saved human answer or confirmation can reach the requesting assignee while
dependencies remain unfinished. Admission and dispatch verify the stored response
against its company, task, requesting agent, resolved status, and human resolver.
The response uses dependency-blocked answer/triage mode. It does not check out the
task, set it to `in_progress`, resolve its dependencies, or approve another action.
An absent, changed, pending, or agent-resolved response grants no such exception.
That bounded response remains a continuation after a workspace or AI-account wait,
so final dispatch still verifies it. Finishing a dependency during preparation
does not turn the admitted response into an attempt to check out the task.
Task interruption notices ignore Retry receipts for a superseded run. A later
successful conversation must not inherit the old failure banner. This read-only
projection neither retires Retry authority nor acknowledges independent saved input.
Snapshot preservation and native copyback are distinct receipts. A historical
workspace recovered as an immutable snapshot does not make its old native sync
reference finalized. Once recovery settles, the current inbound owner can
@@ -3530,6 +3888,17 @@ provider event, or process-launch evidence may exist. Environment cleanup still
requires the same receipts. This historical proof permits explicit Retry or a
newer saved user message; it does not replay the cancelled input.
Workspace and AI-capacity waits use the same pre-dispatch classification for
recovery and stop proof. A cancelled legacy run must retain its matching
`workspace_wait` or `ai_connection_wait` receipt with `providerWorkStarted: false`.
The runtime must remain unresolved, and the stage must be `preparing` or
`settled`. Native execution references, workspace recovery obligations, a live
controller lease, retained resources, or dispatch events still reject the proof.
Remote cleanup requires the exact termination receipt. This rule also covers
historical waits without a process ID; neither an old timestamp nor a different
server boot proves that a run stopped. Pause and terminate can then finish,
while a subsequent run still requires ordinary admission and new authorization.
The existing bounded saved-message worker rechecks this proof after restart.
Admission atomically settles an unclaimed coordinator and admits one fresh turn,
preserving history, unknown action outcomes, and attempt counts. Pauses, approvals,
@@ -3545,6 +3914,32 @@ after cancellation. The old task's Stop receipt cannot settle cleanup of a new
replacement task. A lost reply with no known resource identity remains unresolved
until provider inspection or operator reconciliation establishes its outcome.
### Queued messages before a native run's first turn
A user message that arrives while a Paperclip Runner run is still preparing,
or after its coordinator claim but before `turn.start`, is not a reason to stop
that run. Immediately before the first `turn.start`, the runner seals the
turn's input: in one transaction under the task, queue, and run locks it takes
the responsible user's plain queued comments for the task, in their queued
order, adds them to that turn's prompt, and records a
`firstTurnQueuedInput` receipt (`paperclip.first-turn-queued-input.v1`) in the
run context. Interaction answers, interrupt intents, independent
continuations, conversations, and other users' messages keep their own
delivery. Taken messages leave the visible queue.
The receipt does not consume the messages. They stay on their deferred wake
until the provider accepts that run's turn (`turn.accepted` or the provider's
`turn.started`); only then do release promotion, Send, and later first turns
treat them as delivered, whatever the run's later outcome. A run that ends
before acceptance leaves them queued for the next run, and a restart before
acceptance repeats the same sealed input. An edit after the seal is unread.
**Interrupt** on a run that has neither sealed its first turn nor had a turn
accepted records no intent and does not stop the run; the response's
`executionWait.reason` is `first_turn_pending`. Once the input is sealed or a
turn was accepted, Interrupt stops the run as described below. A run saved by
a release without the seal counts as started once its provider accepted a turn.
### Delivering queued messages after a legacy run stops
The legacy queued-message Interrupt action accepts a null `targetRunId` when
@@ -3811,12 +4206,39 @@ through the ordinary control-plane observers without starting another model turn
When a later heartbeat run reuses the provider session, it clears that prior
run's wait marker along with its terminal, result and pending-request authority.
Already-numbered cancellation acknowledgements and cancelled/expired runtime requests
are retained during shutdown so recovery can replay a contiguous journal. They
carry no permission to start work. Stream closure, stream failure or drain timeout
after complete usage leaves the saved wait unfinalized until a provider terminal
is retained; complete token counters alone cannot certify a stopped provider.
A timeout, missing accounting, or an unproven provider terminal remains a failure;
the saved wait does not certify successful provider execution.
are retained during shutdown with usage and the terminal. They carry no permission
to start work. The drain discards the turn's other numbered events, so the retained
journal can have sequence gaps after the trigger. Recovery accepts a gap only when
the trigger is the first replayed event and every event after the gap is one of
those retained facts of the same turn. Any run, session or source mismatch still
fails closed. Stream closure, stream failure or drain timeout
after complete usage leaves the saved wait unfinalized until execution is proven
stopped; complete token counters alone cannot certify that boundary. Codex allows
up to 30 seconds for its interruption terminal, matching the transport command
window; the caller's execution deadline can shorten it. After that bounded drain,
a supporting driver uses its existing owned shutdown.
For Runner, this requires exact run/session/turn identity, provider settlement,
durable event drain, settled semantic tool outcomes and verified Runner
suspension. An interrupt acknowledgement, process signal or ordinary close
promise alone is insufficient. This proof settles the saved yielded result
without asserting provider success or complete accounting. The stopped session
is retired from warm reuse.
The checkpoint may add a shutdown settlement receipt to the existing governed
wait. Previous checkpoints without it follow the same terminal-or-shutdown
settlement path. Recovery validates the original triggering event and receipt
binding, rejects later turns and failed provider terminals, and completes the
same saved handoff without starting a model turn, even if its question has been
answered. A crash after receipt persistence retains that proof. Late settlement
after controller cancellation cannot publish a result.
If shutdown cannot be proven, the run retains its checkpoint and the named
`native_governed_wait_settlement_pending` condition. The existing bounded
same-run recovery schedule owns further settlement; it grants no replacement
or tool replay. The task explains that work is saved and shutdown is pending.
Exhausted or gated recovery holds continuation and names the shutdown problem.
Actual provider failures and integrity failures retain their existing meaning.
Accounting keeps its separate receipt and reconciliation requirements.
An exact governed-cancellation terminal with an explicit turn ID may return its
durable event receipt before postcommit accounting and output observers finish.
@@ -4775,6 +5197,19 @@ Upgrade diagnostics retain their existing `ready` meaning and add bounded retain
header evidence. That inventory is preliminary, reports partial coverage, and
does not certify workspace durability or volume removal.
The upgrade census separates current execution from an old, unowned native
preparation that recovery explicitly retained as `awaiting_evidence`. This requires
an observed preparation with no result, failure, process identity, or checkpoint
header, and an expired, recorded owner from a different controller boot. Such a
run is counted as `retainedBlockedRuns`, rather than `activeRuns` or
`recoverableRuns`. It remains unchanged and held for its normal recovery path;
the observation does not prove that its old controller stopped or authorize
replay, resource release, or volume removal. A run registered in the current
process still blocks unless it has positive recovery evidence. Before/after
execution counters, pending wake operations, drain state, and a truncated census
continue to prevent readiness. Persisted queued wakes are not deleted by an
upgrade; the pending wake counter measures in-process admission operations.
`importNativeControllerRecovery` is an internal one-source operation for the
migration owner while old controller files remain available. It requires the
admitted configuration digest and exact supervisor snapshot, locks the issue,
doc/observability.md
+21 -1
View File
@@ -236,6 +236,12 @@ Early context preparation also records `agent_read`, `chat_completion`,
`conversation`, `runtime_state`, `issue_context`, `run_identity`, `trust_preset`,
and `task_context` beneath `heartbeat.prepare`. These time the existing awaited
operations before environment selection; they do not add reads or record content.
The work between those phases is named too: `chat_retry_admission`,
`workspace_selection`, `dependency_readiness`, `issue_checkout`,
`instance_settings`, `workspace_context`, `project_context`,
`plan_wake_routing`, `routine_environment` and `responsible_user`.
`execution_start` times the run and agent status writes and the "run started"
event before run output starts. A repeated read or write has its own span.
Execution configuration also records separate phases for mentioned skills,
GitHub identity selection, secret resolution, managed AI runtime preparation,
@@ -297,7 +303,9 @@ The `paperclip.native-controller` tracer records
backend construction, connector assignments, assigned MCP discovery, tool
authority, companion selection and Runner binary resolution. It also records
GitHub shutdown, coordinator release, instruction checkpoints and collection,
and warm-session release. These intervals can occur between the existing native
and warm-session release. Before the coordinator claim it also records
attachment staging, warm controller restore, persistence of the preparation
spans to the run log, and Runner state migration. These intervals can occur between the existing native
preparation, execution and settlement spans. They are controller work, not model
turn time, and carry only the existing run hash. Nested phases overlap their
containing backend or lifecycle intervals; do not add both durations.
@@ -772,6 +780,18 @@ queueing, preparation and execution; do not add its duration to those phases.
No text or provider event identifiers are exported. The existing duration attribute
is `paperclip.native.span.duration_ms`.
`task.time_to_first_agent_activity` uses the same start and `anchor` as
`task.time_to_first_token`, and ends when Core commits the first event that
`provider.time_to_first_agent_event` counts: a text delta, a tool-call start, or
a reasoning item, whichever comes first. That is when the user sees the agent
respond, so the native speed budgets judge this span (metric
`time_to_first_activity`, measured from run creation). Its `eventKind` is
`agentMessage`, `reasoning`, `toolCall` or `dynamicToolCall`. Time to first text
stays a reported metric without a budget: tool-first adapters (for example
Claude) often call tools before they write any text, so first text includes
model tool work rather than control-plane latency. Like TTFT, it is a cumulative
child of `task.provider_session`; do not add it to the phases it overlaps.
`tool.request.input_stream` measures the observed request-input window: from
an ACP tool announcement to its last input update before the provider reports
the tool complete. It can include provider buffering and transport. It is not
doc/plans/2026-10-26-chat-catalog-eval-goal.md
+87
View File
@@ -0,0 +1,87 @@
# Chat catalog Product E2E evaluation goal
Date: 2026-10-26
## Refreshed user steering — 2026-10-28 UTC
The user merged credential-custody PR #17949 and asked to resume meaningful human chat feature journeys on the freshly verified merged production baseline. Prioritize sensible evals and a small passing pilot before wider runs, then continue toward broad passing coverage. Exercise real human commands, actual agent execution and independently verified product outcomes; fixture setup must not perform the tested action. No new numerical success threshold or blanket retry budget is introduced by this update.
The proposed pilot uses two previous genuine passes as explicitly requested new-baseline controls, plus three unused useful project/task journeys. Preserve all earlier attempts and correction limits. The prior production50b measurements are historical evidence for this refreshed baseline, not new proof: no old pass silently carries into the new rate. Record the exact new production/eval-source/model identity and actual results before crediting anything. Unmerged core dependencies remain blocked; the summarizer predicate correction remains unaccepted.
GitHub Actions on the existing AWS fleet and the requested OpenCode/OpenRouter model remain the execution lane. Use the existing Evals history and renderer. The original goal and earlier steering below remain preserved where not superseded.
## Authoritative user steering — 2026-10-27
The existing Evals website and family renderer/history are the primary delivery surface: [Evals hub](https://pages.paperclip.ing/evals/) and [Product E2E history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/). Retire the separate custom local HTML as the delivery approach; retain it only as historical evidence. The [published historical partial Codex campaign](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/historical-local-codex-chat-catalog-20261027-158measured-86unrun/) contains158 actually measured IDs,17 original initial passes,141 first-attempt failures and159 retained measurements;86 IDs remain unrun in that historical lane. These are original one-time local outcomes, not current-source/OpenCode qualification. There are238 callback implementations without declared gaps and six incomplete or blocked journeys. All244 are registered, but registration and prerequisite probes do not establish complete intended E2E coverage or verified behavior.
The [published OpenCode setup result](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40530790741-1/) records actual framework attempt 40530790741: tasks.create-backlog initial 1 failed setup before the canonical query. 243 remote initials remain; no canonical chat or feature grading completed. The report was verified rendered in the browser using the existing Product history renderer. Automatic setup hello provider usage/cost is unknown, not zero. The same maximum-one coherent correction budget remains. Two earlier host-only qualification operations consumed no case ordinal.
All new local eval admissions are stopped. Already-active local children may finish their existing work and cleanup; no further local initial, final-B, infrastructure-health or provider qualification launches are authorized. Preserve the complete local measurement/ordinal histories and original evidence as a historical lane.
All future eval execution must run in GitHub Actions on the existing AWS fleet, allowing its normal Spot or On-Demand fallback as directly authorized by the user on2026-10-27 at15:21UTC, using the new Paperclip runner **OpenCode** provider through **OpenRouter**, model **`deepseek/deepseek-v4.1-flash-1017`**. Record and verify the actual AWS host lifecycle; unknown hosts still fail closed. The earlier strict-Spot intents and refused operations remain unchanged and do not become retroactive case admissions. No local eval execution is authorized. The remote campaign requires a separately named execution/product/database/model identity and separately reported rate across all244 scenarios. Do not carry local passes into the remote rate, and do not execute obsolete local source merely to preserve the previous queue.
Carry the audited coherent harness corrections into the remote source. Preserve prior ordinal histories. The manager approved one new remote initial measurement for every scenario and at most one coherent correction measurement per scenario. The representative remote smoke counts as its scenario’s remote initial. A host qualification refusal before evaluation is a retained infrastructure operation, not a case measurement or consumed initial ordinal; the first such operation40519322850 left all244 remote initials unrun. Preserve all prior local ordinals separately; this is not an unlimited repair or unchanged rerun policy. Production/core fixes remain isolated and unmerged unless separately authorized.
Immediate deliverables are an inspectable safe results page, an exact closed local checkpoint, and evidence-based accounting of elapsed wall time, stages reached, harness/product/model/prerequisite outcomes, actual usage and operational duplicate/reconciliation history. The original goal below remains historical context where superseded by this section.
> Execution state: This file preserves the requested goal; it is not completion evidence. Before execution, reconcile the actual state of PR #17794 rather than assuming it merged.
Turn the complete Paperclip chat-scenario catalog into executable Product E2E evals, run every scenario, give each failing scenario one fix-and-rerun cycle, and ship according to the rules below.
Use the current agent/Astra only as the manager for planning, architecture, integration decisions, evidence review, and delegation. Delegate implementation, investigation, testing, and PR work to subagents explicitly using gpt-6.1-sol. Use up to six Sol subagents, within the runtime’s actual concurrency limit; use waves when fewer slots are available. Give workers distinct file ownership and tell them not to revert others’ work.
1. Establish the inventory and baseline.
Read repository instructions and the paperclip-evals, add-product-e2e-eval, and PR workflow skills. Inspect PR #17794 and reconcile its actual state before branching; do not assume it merged or duplicate its work.
Freeze the scenario IDs in tests/runner-e2e/chat-scenarios/ as the denominator. The current inventory is 244 scenarios across 35 areas and 191 capabilities. Verify those numbers and preserve every scenario’s intended request, context, and outcome. Maintain one concise progress record with owners, implementation status, attempts, results, and PR links.
2. Implement real evals for the entire inventory.
Extend the existing Product E2E framework, fixtures, selectors, independent grading, evidence packaging, cleanup, and reports. Reuse the two working chat evals and shared infrastructure.
Each scenario needs executable setup, the real human query through Agent Chat, any explicit user follow-up or approval steps, and assertions appropriate to its expected outcome. Cover the specified conditionals: connected/disconnected, permitted/denied, unique/ambiguous, fresh/existing state, and recovery.
Exercise the real browser, server, database, runner, and model. Use disposable, isolated test data and authorized integration environments. Fixture setup must not perform the action being evaluated or pre-populate its result. Simulated integrations must be labeled and must not count as live proof of the corresponding external integration.
Verify saved state, attribution, permissions, usable results, and persistence where relevant. Calibrate each distinct oracle against a valid result and a plausible wrong result. A plausible reply, generic model judge, registration-only test, skipped test, or missing evidence is not a pass.
3. Run a controlled campaign.
Use one coordinator for live execution and shared resources. Start with a small harness-health check, then run bounded batches; parallel coding does not mean every worker launches databases or full test suites simultaneously.
Record the exact scenario, source revision, definition/grader fingerprint, model/profile, environment, timing, usage, cleanup, and evidence for every attempt. Keep credentials and private data out of tracked files and public reports.
Run every scenario initially with automatic retries disabled. Preserve failures, including setup and infrastructure failures. Missing credentials or unavailable services are explicit blockers, never passing coverage. Continue independent work when one scenario is blocked.
4. Allow one fix cycle per failing scenario.
After its initial failure, diagnose the evidence, make one coherent correction, and rerun that scenario once. A shared root-cause correction may cover several affected scenarios. Preserve both measurements.
Distinguish eval defects, product defects, model behavior, and infrastructure failures. Do not weaken assertions, rewrite the intended outcome, silently extend deadlines, add hidden nudges, or repeatedly rerun unchanged failures to obtain a pass.
After the allowed cycle, retain any remaining failure or blocker and move on. Required CI and review repairs may continue, but do not use them to disguise additional live fix cycles.
5. Separate core fixes from eval work.
If a demonstrated fix changes production behavior outside the eval harness, create a separate focused core PR with regression coverage. Get its required checks and review requirements satisfied, but DO NOT merge it or enable auto-merge.
Keep production fixes out of the eval PR. You may validate them in isolated candidate worktrees, but report those results separately. Passes requiring an unmerged core fix do not count toward the eval PR’s merge threshold.
6. Apply an honest 80% merge gate.
Calculate the final pass rate across the entire frozen inventory, using each scenario’s final allowed result on the eval PR’s code and declared production baseline. Keep failed, blocked, unrun, and unsupported scenarios in the denominator. Report first-attempt and post-fix rates separately.
For 244 scenarios, at least 196 must pass. Do not stop implementing or running the remaining scenarios once 196 passes are reached.
If the full inventory has been implemented and processed and the final rate is at least 80%, create the eval PR, resolve applicable reviews, pass required checks on its latest head, and merge through the normal protected workflow. Preserve and document the remaining failures; do not skip them to manufacture green checks.
If the rate remains below 80% after the allowed cycles, preserve the work and report the results without merging or starting an unlimited repair campaign.
7. Finish with evidence and accurate PR states.
Deliver the complete runnable catalog, per-scenario results, campaign report, initial and final pass rates, remaining failures and blockers, and links to all PRs.
Core-fix PRs must remain unmerged. The qualifying eval PR should actually merge. If a human-only approval or another external gate prevents that, prepare everything possible, enable ordinary auto-merge only for the qualifying eval PR when appropriate, and identify the exact remaining gate. Never bypass protections or report queued auto-merge as completed.
Proceed autonomously without routine approval pauses. Keep the manager focused on coordination and higher-level decisions, and use Sol 6.1 workers for execution.
doc/plans/2026-10-26-chat-catalog-eval-progress.md
+303
View File
@@ -0,0 +1,303 @@
# Chat catalog eval campaign progress
Updated: 2026-10-27. This is the single progress record for [the saved goal](2026-10-26-chat-catalog-eval-goal.md). Exact frozen requests, ownership, readiness, attempt histories and provenance live in [the campaign manifest](../../tests/runner-e2e/chat-campaign-manifest.json). This record distinguishes authored code from live acceptance.
## Refreshed execution checkpoint — 2026-10-28 UTC
- **User steering:** PR #17949 is merged. Resume real human chat feature commands on a freshly verified merged baseline; prioritize a small sensible pilot before wider runs, then continue toward broad passing coverage. No new numerical threshold or reset of failed-attempt budgets is implied.
- **Fresh source preparation:** the composed candidate is based on verified production `39e811c9c5226979fce42e052a27cfb28301018a`, including merged custody commit `38d08847`. Final eval source is still being composed and reviewed; no new-baseline pilot result is credited yet. Proposed controls are `cli-operations.create-from-cli` and `agents.change-manager`; unused initials are `projects.create-project`, `projects.project-work`, and `projects.archive-project`. Exact intents and actual admission remain separate from this proposal.
- **Preserved historical remote results:** [the official final34 report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40584967834-1-final34-report/) records the previous lane: 57 distinct measured scenarios, 6 passed, 51 failed, 187 unrun; 62 attempts (6 passed, 56 failed). That run intentionally stopped for the shared key-custody finding after 34 admitted results. Its 120 unadmitted selections consumed no ordinal. Prior production50b and earlier mixed-baseline results are historical for this refreshed baseline; their passes do not become new-baseline evidence.
- **Evidence custody:** the former ignored private ledger directory is currently unavailable. Selected pilot history is being recovered from the published immutable 244-ID coverage and actual retained result bytes; unavailable private provenance is explicitly unknown. Do not reconstruct attempts from conversation memory. Accepted boundary/overview/preview candidates remain preserved by named Git refs; the comment fix already exists in the prior eval source. Do not integrate the whole semantic branch: the summarizer predicate remains unresolved and unaccepted.
## Earlier status checkpoints (historical)
- **First remote feedback-loop timing:**run40530790741 spanned15:25:15–16:18:09UTC (52m54s). Retained workflow facts: authorization queue≈8m03s, lockfile queue6m12s, catalog queue3m44s, AWS build queue16m40s plus build3m38s, paid case queue10m19s plus worker2m34s. These stage durations are not additive causal attribution or model runtime. Actual host/Docker qualified; the first framework attempt then failed agent-readiness setup before the canonical query.
- **Existing Evals website is primary:**[published historical partial Codex campaign](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/historical-local-codex-chat-catalog-20261027-158measured-86unrun/) is live in the [canonical Product history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/) and was opened/visually verified in the browser. It shows17/158 original initial passes,141 first-attempt failures and159 retained attempts, with86 historical unrun IDs listed in its published summary. It is not new-baseline confidence. The custom localhost page is retired as the delivery approach. Its exact owned preview server PID3634 was rechecked by UID/start/command/loopback listener, terminated and proven absent; historical HTML remains intact (`operations/interim-report-preview-retired.json`). No original result or ledger bytes changed; public projection excludes raw usage/error/transcripts/screenshots. Canonical normalized usage63,126,652tokens includes2,071,424cached tokens omitted from three goal rows in the old61,055,228-token ledger scope; both values are retained, and unpriced costs are not zero.
- **New remote measurement:**[published OpenCode setup result](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40530790741-1/), browser-verified through the existing Product history renderer. Run40530790741 at source0dd8 completed one actual tasks.create-backlog initial as an agent-readiness setup failure before canonical query;243 initials remain,0passes. Host/Docker/source/model/artifact identity is verified; cleanup passed. Heartbeat native runs0, but automatic hello provider calls/tokens/cost remain unknown. Finally grading snapshots do not establish returned feature grading. The maximum-one correction budget is unchanged; this is not reclassified as a host-only operation.
- **New user host authorization15:21UTC:**future GitHub Actions execution may use the normal AWS fleet Spot or On-Demand fallback, with actual lifecycle retained and unknown hosts rejected. This supersedes the strict-Spot interpretation only for an explicitly new source/intent. Old617/941 intents and the two pre-evaluation refusals remain unchanged; those operations consumed no initials. The later actual framework attempt consumed one, leaving243. No local execution is permitted.
- **Second remote qualification operation:**[run40524193545](https://github.com/paperclipai/paperclip/actions/runs/40524193545), target6175603, proved actual Spot/x86_64 then failed host database-image qualification (`owned-linux-amd64-postgres-image-unqualified`) before evaluation. Authenticated artifact12743322992 and exact extracted bytes are verified; the underlying Docker/image cause is not yet established. Operation1 remains separate. Both have0canonical/native/provider work and consume0case ordinals. All244 remote initials were unrun at that operation checkpoint. Receipt:`results/chat-catalog-remote-opencode/operation-40524193545.json`. Separate reporter PR17888 latest manager checkpoint: head ae190635b93af2a2d85c9617de244c7ad1fadd16, required verify/e2e passed, Greptile5/5 and0 unresolved, OPEN/unmerged/auto-merge off; no reporter outcome is inferred from this host receipt.
- **First remote operation:**[Actions run40519322850](https://github.com/paperclipai/paperclip/actions/runs/40519322850), exact target6175603, was refused by the actual-host gate because AWS supplied On-Demand instead of Spot. Authenticated artifact12740329812 and its SHA256-bound host receipt prove evaluation was not admitted:0canonical/model/native work and0case ordinals consumed. All244 remote initials were unrun at that historical operation checkpoint. Separately, trusted reporter dependency installation failed with `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`; this is not a case/model outcome. No redispatch or retry is authorized by this record. Remote operation evidence is under `results/chat-catalog-remote-opencode/operation-40519322850.json`; the historical local ledger is unchanged.
- **User steering supersedes the local queue:**all new local admissions are stopped. All admitted children have finished and exact owned absence is verified; no remaining76/22 local initials, local B or new health launches. Future execution is GitHub Actions on the existing AWS fleet with normal Spot or On-Demand fallback, using the new Paperclip runner OpenCode provider via OpenRouter model `deepseek/deepseek-v4.1-flash-1017`. Local histories remain preserved; their passes cannot qualify the separately named remote lane. The manager approved a new remote initial for every scenario plus at most one coherent correction measurement, with distinct history and no erased local ordinals.
- **Historical detail:**the earlier custom report remains retained evidence, while the existing website above is the user-facing report. Its conservative local stage projection proved105 saved canonical queries,102 exact canonical-native admissions,14 returned feature grades and110 observation-grade snapshots; unproven stages remain unknown. Local reservation span15.539h, union of completed attempt wall intervals4.719h (not model time), summed walls5.879h. Those historical quantities do not describe the new remote lane.
- **Frozen inventory:244 scenarios /35areas /191capabilities;196 eligible passes required.** Inventory digest `9dc9cf6a5426cf1b56bc30a3571851f233f9dd5e93a1292eaee414d33b52df77`. Original queries, contexts and outcomes remain unchanged; blocked, partial and unrun cases stay in the denominator.
- **All244 callbacks registered; six journeys remain incomplete or blocked** in immutable remaining-initial source `68e8534e3127dec27ae979bbf9f33d8a26422e4b`:238 implementations have no declared gaps; six incomplete or blocked journeys retain prerequisite probes (242 lazy definitions +2 legacy). No absent registry entry is implied, but six are not complete intended E2E journeys. Grok, named ChatGPT and private GitHub after-grant callbacks are integrated. This is registered implementation coverage, not complete intended-journey coverage or live acceptance.
- **Closed historical local checkpoint:158 scenarios attempted/processed,159 completed measurements,17 original initial passes and142 failed measurements;86 scenarios unrun,0 running.** Historical initial rate is17/244 (6.97%) across the frozen inventory, or17/158 (10.76%) among attempted IDs. Final current-source qualification was not completed; no local pass carries into the remote lane. Actual retained usage is61,055,228 tokens; subscription dollars remain unpriced. The final12 initials completed with all cleanup passed/DBremoved; exact owner/12launchers,12full container IDs and roots are absent, with zero resources under their exact root labels on the unchanged pinned engine. Original volume names were not reconstructed. The user-stop/state/lock histories and all original result/attempt bytes are retained. Receipts:`operations/user-stop-local-admissions.json`, `operations/qualified-inputs-user-stop-exact-absence-68e.json`, `operations/local-campaign-user-stop-closed.json`, `local-campaign-closed-summary.json`. Two historical cleanup records remain unresolved; this is not a claim of newly observed live resources. No further local execution is authorized.
- **Single coherent final-B retry set remains held.** Goal-tree exhausted two measurements. All original outcomes, evidence and measured digests remain unchanged. Actual current selected identity review is retained in `closed-pass-identity-review-b65.json`; changed selected execution/grader closures require reviewed revalidation within the same max-two budget. Unrelated Git revisions and unchanged failed behavior do not justify reruns.
- **Two-case concurrency qualification completed and manager accepted continued waves of two.** Immutable `c167a54162fadffe665b4d35a7a1308c6233b75d`:companies.archive-company passed and connection-setup.choose-provider failed its actual reply check; each1native/1source human. Worker overlap25.479s; separate roots, ports53206/53205, containers and PostgreSQL system identifiers. Both containers and exact-root-labeled volumes are absent, both roots removed, and coordinator/child PIDs absent. A parent-only duplicate Playwright import prevented the passing completion from initially recording; isolated-process replay confirmed unchanged selected identity and finalized the existing result without rerun/reservation. Original parent error/stop preserved and owner lock retained. Receipt:`tests/runner-e2e/results/chat-catalog-campaign/two-lane-qualification-closed-c167.json`;5 offline replay controls passed. Full repository typecheck/build and final scoped E2E typecheck passed; broad test:run failed/incomplete on host embedded-PostgreSQL bootstrap, with all owned check processes absent. The qualified Docker lane is unchanged.
- **The former local continuation is cancelled:**61 source initials,3 installed initials and22 guarded invitation prerequisites remain unrun (86 total). The prepared source/private-input selections remain provenance only and must not launch locally. Future remote execution uses the audited current coherent corrections and new OpenCode/OpenRouter model/host/database identity; it must not reuse obsolete68e merely to finish the old schedule.
- **Known invite-path dependency quarantine is source-qualified for22 of166 remaining IDs.** Exact selected setup call chains (including observerPerson wrappers) are recorded in `remaining166-invite-setup-dependencies-68e.json`;144 independent IDs remain, including3 distinct installed cases. This is a dependency inventory, not processed live setup proof. No safe equivalent public membership setup is already implemented/proven; no fake identities, database grants, scanner weakening or automatic continuation is introduced. Guarded immutable candidate `e515d60fe396ef1ecd45f3a274a46f310bee4d2b` (96b source plus exact metadata overlay) is prepared with238 definitions without declared gaps and six incomplete or blocked journeys, scoped E2E typecheck passed, and22 selected fingerprints. No invitation-lane admission has occurred. Actual guard/probe or any earlier-publication failure must be recorded, with no token-path accept on the affected baseline and no invented zero-native measurement.
- **New selected external authority:**the user authorized disposable mail only from the sole QA inbox to itself, and read/isolated clone of the existing designated QA private repository with no repository writes. The external owner independently qualified the actual inbox/account and exact repository/HTTPS clone, then removed its owned clone/helper. New private0700/0600 configuration/reference files are staged separately; actual credential values remain private and selected before capture. Five cases have qualified input fields (outbound-email, unknown-email-outcome, add/replace-repository, prepare-repository). Reply-to-email still lacks a distinct authorized reviewer/sender; private-repository-no-access still lacks the human App/browser authorization. The stable coordinator-only HMAC key commits the selected recipient expectation; credentials, private account/repository metadata and key bytes are not public evidence. Existing read-only discovery and its history remain unchanged. Host IPC cleanup is unnecessary and was not performed.
- **Infrastructure is qualified:**owned local Docker PostgreSQL18.6 at the pinned installed image, normal onboarding/migrations/API and final persisted-state scan passed with zero model/native/scenario work; exact resources removed. No further infrastructure health is needed for the unchanged lane. The proposed host SysV cleanup was never performed and is no longer needed. Source/build/native bytes remain bound to production baseline `8373d7867963674ff960d2345d0d28086f799bc7`; no core candidate is used as benchmark proof.
- **Separate core PRs remain OPEN/unmerged, auto-merge off.** Doctor [#17847](https://github.com/paperclipai/paperclip/pull/17847), head `a042fb4a28f2e1574972798ed80de2e616daf9fc`, and invite logging [#17848](https://github.com/paperclipai/paperclip/pull/17848), head `98a2432f88c33afacff2e014bd8c30630b26b339`, each have55 successful checks. Budget [#17857](https://github.com/paperclipai/paperclip/pull/17857), head `b6cd49d4b6926fae61a2c68a31ce7c659b69eaa6`, has55 successful/2skipped/1neutral checks and no outstanding checks (manager exact-head checkpoint). Approval routing [#17858](https://github.com/paperclipai/paperclip/pull/17858), head `b60aaf19343389faf67580fb1ff7a561983bb4f5`, has55 successful/2skipped/1neutral checks and no pending checks, Greptile5/5 with0unresolved threads. Connection choice [#17862](https://github.com/paperclipai/paperclip/pull/17862), latest repair head `a76a6120ec2b0767b32b62be4716f69214692688`, has48/48 required workflow checks green at exact head (workflow40497606929), Greptile5/5 and0unresolved threads;13focused controls/server typecheck passed and prior failed checks remain historical evidence. Performance metric [#17872](https://github.com/paperclipai/paperclip/pull/17872), head `3807ee526d2e761f6aa420452b390287542682bd`, has3 regression controls and UI typecheck/token-gates/build passed, with CI/review pending. None changes measured baseline8373. No eval PR or merge is eligible.
- **Genuine remaining missing inputs:**the sole self-mail inbox/recipient and exact QA repository are now selected, authorized and independently qualified; five unattempted cases carry those new private inputs. Reply-to-email still needs a distinct authorized reviewer/sender, and private-repository-no-access needs the actual human App/browser authorization. Other named provider, SSH/cloud target/image and private operator prerequisites remain unqualified unless actual retained setup proves otherwise. Standard provider-key reference availability alone is not target qualification. No extra send, repository write or account creation is implied.
The efbb coordinator exited0 after30 retained measurements; no old launcher/writer remains. The authoritative ledger is the current manifest; immutable snapshot selection summaries are historical. The remaining initial-only draft originally covered172 IDs after efbb; the qualification pair consumed two, leaving170 unattempted IDs. It includes concrete missing-resource probes; its known two alias fixtures are corrected in151488bf8c before future admission.
## Current measured continuation and held corrections
The current ledger is authoritative; archived selection summaries remain historical. Every completed case in the current two batches has verified cleanup and exact owned Docker removal. Native subscription dollar amounts remain unpriced; synthetic imported fixture costs never enter campaign billing. Counts of human turns refer to the source standing chat, while separate operator actions remain in case evidence.
Coherent final-B corrections are accumulating without reruns. `983f15394e` removes the archive reply-link overconstraint while exercising actual historical project/task UI and API access, accepts truthful common audit actor headers, grades meaningful Orchid plan requirements without nonce repetition, and fixes hire-denied's public permissions payload without widening task authority. `200e7fc3ac` refreshes the API-created company inventory before the canonical turn and binds the retained UI journeys to actual status labels, original interaction IDs and the visible organization toggle. Focused internal controls113, runtime-agent controls26 and relevant identity/runtime controls49 passed; scoped E2E typecheck passed. Raw failed initial evidence remains unchanged.
Necessary-question's retained observation contains no saved task, document or result link. Its original answered question asked for product, audience and CTA; the supplied short product-name answer was followed by another actual pending positioning question. This is retained for exact YAML/fixture diagnosis, not reclassified as a passing saved brief. Goal-tree remains exhausted; all final-B attempts remain held.
## Measured evidence — all failures retained
All results below use local native Codex subscription profile `runner-codex-subscription`, model `gpt-5.6-sol`, one case per launcher and zero automatic retries. The first72 scenarios were sequential; the separately recorded qualification pair used two concurrent isolated launchers. Retained paths are under ignored `tests/runner-e2e/results/`. The launcher numbers each fresh invocation `attempt-1`; the ledger separately preserves governed campaign ordinals.
| Scenario / campaign ordinal | Source | Retained campaign | Outcome and supported cause | Cleanup / accounted tokens |
| --- | --- | --- | --- | --- |
| `goals.create-goal-tree` / 1 | `10d0ce63610` | `local-2026-10-26T22-19-23-608Z` | Eval harness failed before canonical query: Playwright could not resolve the selected `.js` URL to `.ts`; report parsing also rejected the dotted canonical ID. Machine class `candidate_failure` unchanged. | passed / no heartbeat runs |
| `goals.create-goal-tree` / 2, final allowed | `e652b0de6e` | `local-2026-10-26T22-26-37-076Z` | One native run created the requested tree, but final reply had no hierarchy link and left company ownership unset without a question. The unchanged first-phase gate prevented the explicit retry phase. Rendered href arrays and visual screenshot review confirm no alternate result card. Completed-behavior failure; no third attempt. | passed / 44,131 |
| `goals.align-existing-work` / 1 | `e652b0de6e` | `local-2026-10-26T22-31-32-695Z` | Two native runs correctly saved associations and rejected the foreign goal. Manager/domain review found the old three-links oracle exceeded this scenario’s YAML while underchecking its strategic-context explanation. **Eval oracle defect** supersedes the initial model-behavior attribution; raw failure and diagnosis history remain. | passed / 135,401 |
| `goals.explain-purpose` / 1 | `e652b0de6e` | `local-2026-10-26T22-35-40-720Z` | Initial fixture exposed both intended and later-conditional website tasks. Native tool-result event 208 contains both; the reply conflated them. **Eval fixture contamination**; no product/model blame. Missing/inaccessible phases were never sent. | passed / 25,256 |
Each case directory contains `result.json` and independent `snapshots/chat-catalog-{definition,observation,conversation,runs}.json` where reached. Exact definition/grader/global suite hashes, source SHA, evidence paths, timing and usage are retained per attempt. Subscription dollars remain **unpriced**, not zero; production setup hello probes remain outside heartbeat accounting. Accounted tokens across completed work runs total 204,788.
The initial report-parser error was repaired only for presentation at `results/chat-catalog-campaign/initial-postprocessed/index.html`; its sidecar retains original source and report error. This postprocessing made zero provider calls and remains the original failed measurement.
## Frozen snapshots and corrections
- First immutable source snapshot `10d0ce63610dd85ae338b0cc4d9addcba2cd8f80`: all 12,160 tracked files verified against Git blobs. Dependencies/build outputs were copied with APFS clones, 5,191 symlinks audited, one absolute workspace link rebound, and generated shims rebound. No dependency links reached the mutable checkout.
- Eighteen required workspace packages were rebuilt without incremental reuse. Full snapshot E2E typecheck passed. Native runner was rebuilt offline/locked from source; Rust sources match the declared production baseline. Binary SHA-256 `9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4`; build/toolchain/workspace-output receipts are in `results/chat-catalog-campaign/snapshot-10d0ce63610.json`.
- Correction `4c9de65edd` fixed actual source-module dispatch and bounded dotted report IDs; two regressions passed in an actual Playwright worker without fixtures/providers. Capability `e652b0de6e` added genuine pre-query history boundaries, default zero. Corrected immutable snapshot `e652b0de6eff21473a6d9b4915846c9851f292fa` verified 12,182 tracked files, full E2E typecheck, 32 focused controls and both Playwright regressions. Required production/workspace/native sources remained identical, so pinned build bytes were reused with provenance.
- Goals correction `ad031f1c59` is reviewed/frozen: alignment removes invented link obligations and checks phase-specific context explanation; purpose creates missing-association fixtures only after the initial canonical phase, uses semantic rationale rather than nonce repetition, and rejects negated/wrong-phase grounding. Domain reports 66 controls and scoped typecheck passing. **Calibration is not a rerun or passing live result.**
- Attachment capability `1c84384d41` allows exactly one declared production upload receipt for the two relevant cases, with unchanged canonical query prefix, exact formatter suffix, same-source/human/comment identity, independent bytes/SHA checks and allowlisted evidence. The two production composer source hashes are bound as selected fingerprint data only; their UI dependency graphs are not imported. Twenty-five focused shared controls and E2E typecheck passed.
- Finite repository alias binding now validates declared receipts with current company-scoped provider discovery and active connections; the domain adaptation is included in this checkpoint. It binds only catalog aliases to independently discovered numeric repository identities and active same-company GitHub connections. Its focused controls and scoped typecheck pass; final goal retries remain held until the remaining authoring contracts settle. Attachment helpers are frozen.
## Ownership and next work
The finite assignments cover the full denominator; future-wave assignments remain real work, not reduced scope.
| Owner | Scenarios | Areas |
| --- | ---: | --- |
| `eval_domains` | 45 | tasks, projects, goals, documents-artifacts, inbox-search, activity (two legacy task implementations retain integrator ownership) |
| `eval_integrator` | 27 | cases, status-cards, routines, skills |
| `eval_identity` | 44 | access, navigation-preferences, onboarding, questions-and-approvals, companies, budgets-costs |
| `eval_runtime` | 60 | agents, agent-chat, runs-adapters, execution-environments, recovery, steering, task-coordination, workspaces |
| `eval_external` | 52 | connection-setup, app-permissions, chat-channels, assistant-connections, gateways-profiles, plugins, teams, secrets |
| `eval_campaign` | 16 | instance-operations, cli-operations, developer-labs; also sole live coordinator and ledger/report owner |
Next: manager review of a bounded initial-only slice from an immutable snapshot of this checkpoint; remaining area authoring continues. Run each approved initial sequentially with retries zero, fresh owned environment and cleanup. Final retries remain deferred until coherent freeze B after all initial processing. Accumulate one coherent correction set, freeze B, and review the exact eligible final retry set before execution. Alignment and purpose final retries stay deferred into B; goal-tree is exhausted and must not receive a third run. The coordinator retains sole live/ledger ownership. No new reservation or launch is authorized by this metadata checkpoint. No live test suites run concurrently with campaign instances. Preflight rechecks IPC capacity and exact source/binary/dependency identity; no arbitrary IPC cleanup, kernel tuning or live database changes.
Local tools/dependencies/private subscription auth file are present. Earlier absence described exported provider-key variables and `.env.runner-e2e.local`, not absence from credential sources. Exact standard `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `OPENROUTER_API_KEY` and `DAYTONA_API_KEY` names are present in the `~/.secrets` file. Source reference availability does not establish validity or a disposable target; immutable cloud image/resource scope remains unverified. External provider/configuration blockers remain explicit and never passing coverage; simulations cannot establish live provider acceptance. No private artifacts have been published.
Identity and domain probe helpers now retain one attempt-private credential lifetime with fresh reads and explicit revocation. Remaining external review: allow harmless secret-related artifacts only after actual private byte scanning, and compare secret version/identity semantics rather than JSON property order. Named ChatGPT setup is explicitly partial: real generic OAuth/MCP client consent is retained but cannot prove the named ChatGPT human setup path.
Core behavior fixes require separate focused PRs that remain unmerged; their dependent passes cannot count toward the eval threshold. Eval merge requires all 244 authoring-resolved and actually processed (complete implementations or concrete executable blocker probes, with no unfinished branches), at least 196 final eligible passes, latest-head checks/reviews and actual protected merge. Per-case fingerprints, global suite hash and production/source revisions remain distinct; stale passes cannot be carried across changed execution/grader dependencies.
Credential discovery: relevant source metadata is readable through the existing 1Password service account outside the sandbox. Standard source references are available; this does not prove account validity. The user explicitly approved the narrow read-only AgentMail account/inbox and GitHub account/repository metadata discovery, and the external owner completed it. No contents, sends or provider writes were authorized by that discovery. No designated disposable self-mail recipient/repository scope is selected yet. Private item IDs and account metadata remain outside tracked documents.
## Historical authoring handoff at b33f3b8b64
| Owner | Without declared gaps | Partial | Authored |
| --- | ---: | ---: | ---: |
| Domains | 37 | 2 | 39 |
| Identity | 19 | 0 | 19 |
| Runtime | 23 | 1 | 24 |
| External | 8 | 2 | 10 |
| Integrator legacy | 2 | 0 | 2 |
| Coordinator privileged | 14 | 2 | 16 |
All 16 privileged IDs have executable paths. The health case submits the exact canonical body to the same owned backend only after genuine owned PostgreSQL shutdown and actual HTTP/doctor failure; route timing and early evidence survive ordinary admission failure. Cleanup restores the verified owned cluster before generic cleanup. No database fault was performed during authoring.
Backup restoration now includes a declared factual operator receipt and second, phase-attributed native restore-result report. Performance similarly supplies actual documented fixture measurements to a declared follow-up; generic no-measurement interpretation cannot pass. Feedback retains an actual board-cancelled, unfinished native worker with original comment/run/events, actual human CLI export, independent JSON/archive byte and scope checks, unchanged sharing, and a truthful local-location reply. It is labelled interrupted/cancelled work, never model failure or failed-native coverage. CLI returned commands execute their actual supported argv with an explicit source executable mapping and unresolved-identifier receipts; no JSON flags are silently added. Real state readback supplies independent identity checks.
The two privileged partials are `instance-operations.update-instance` and `instance-operations.provision-runtime`: `installed-source-activation-and-artifact-fingerprint-lane`. Ordinary chat currently rejects installed activation; this is a missing harness capability, not credential absence. Their area code requires an actual source installed target and private mode0600 `PAPERCLIP_CHAT_OPERATOR_INSTALL_RECEIPT` inside the owned attempt: exact CLI/server hashes, complete owned npm-prefix dependency-tree hash, source process PID/UID, source HTTP version and disposable ownership. Update additionally requires an exact supported published version/integrity and retains the concrete original-version reinstall option; it applies the public updater, restarts the actual source and independently reads its new version/readiness. Provision requires an explicitly named private OpenRouter environment reference, missing release-matched assets, the actual public provisioner and closure identity, then a separate genuinely running native Pi worker and exact saved file, followed by a phase-attributed result report. Its declared total is three native runs (initial controller, Pi worker, controller interpretation) and two human messages. Download alone never qualifies it.
Integrator owns the narrow allowlisted installed ordinary-chat selector/fingerprint/receipt lane. It must preserve source-mode guards and bind actual installed artifact closure rather than relabel a source pack as a published release. Coordinator resumes the operator cases after that capability; these partials cannot count for acceptance in the meantime. Runtime's fifth pre-chat `agent-chat.chat-disabled` boundary is allowlisted with real authenticated harness context and zero admissions; runtime still owns its case implementation.
Credential-free validation: 32 privileged controls across seven files passed; the later exact-argv change passed five CLI controls and E2E typecheck. Metadata was reconciled only from Git archive b33f3b8b64; dependency symlinks to the earlier immutable candidate were used for metadata imports only, never a live measurement. Newer committed/WIP area changes are pending the next checkpoint. No extra selected-case probe, reservation, live run, credentialed provider discovery or final retry occurred.
## Current finite integration checkpoint at ad0907afb9
| Owner | Without declared gaps | Partial | Authored |
| --- | ---: | ---: | ---: |
| Domains | 39 | 0 | 39 |
| Identity | 34 | 2 | 36 |
| Runtime | 23 | 1 | 24 |
| External | 15 | 2 | 17 |
| Integrator legacy | 2 | 0 | 2 |
| Coordinator privileged | 16 | 0 | 16 |
The archived source includes domain conditional closures, selected case credentials/private input staging (`9be9c65e85` / `5f8a3c8eec`), and the explicit installed operator lane (`8f67a700a4`). Update/provision no longer have that authoring gap: only these two ordinary-chat operator IDs admit an actual public-registry installation in an owned prefix, with publication pins, exact process/tree identity and supported transition checks. No installation or provider call has qualified that lane yet. Newer identity and integrator commits are pending the next immutable checkpoint.
All 244 scenario rows and their original canonical hashes remain unchanged. Three scenarios/four measurements still have no eligible pass. Four historical failed source measurements now append product identity only from their already-retained verified snapshot/native receipts; outcomes, diagnoses, usage, attempt ordinals and evidence are preserved. Any identity without adequate retained proof remains unknown and cannot qualify.
The approved eligibility policy distinguishes three product identities: ordinary source builds must match production source baseline `8373d7867963674ff960d2345d0d28086f799bc7` with exact snapshot/native/workspace artifact receipts; the two declared published operator cases retain independent public source/target release pins plus matching observed artifact identity; `onboarding.new-instance` may use a real published release or a separately labelled source-built installed artifact with its own CLI/server/tree proof. A source pack is never labelled published. Scenario-appropriate published-release passes may count toward the full 244 benchmark under their declared lane, and are never represented as proof of the source-baseline server. Missing identity, wrong baseline/lane, mismatched observed transition and any unmerged core candidate are ineligible. Ledger completion and summary enforce this distinction and report measurement counts by product identity.
Validation used the exact Git archive and its reconciled descriptors: all 132 runtime exports (including four browser boundaries) matched metadata; discovery listed 134 cells; 22 catalog/campaign/boundary/private-input/fingerprint controls passed. Four installed-child controls passed with the narrowly required process-inspection permission. Their first default-sandbox run failed closed because `ps` inspection was denied; that is retained harness-environment evidence, not a product or scenario measurement. No matching owned timer fixture remained afterward. The new ledger identity gates and a real private-envelope deletion/name-only fingerprint replay passed ten focused controls; E2E typecheck passed.
Plugin installation fingerprints two generated child artifacts. CLI and plugin SDK outputs were narrowly rebuilt inside the immutable archive using the existing committed build configurations and dependencies from the earlier immutable candidate. Lock identity matches, and production source/config/lock paths have zero diff from the baseline. Artifact hashes/build provenance are retained privately in `results/chat-catalog-campaign/metadata-ad0907afb9.json`; this is metadata validation only. Final freeze A must rebuild or adopt audited bytes with the actual execution snapshot receipt, never mutable-worktree `dist`.
Selected private credential inputs retain name-only identity for fingerprint replay after raw file cleanup; secret values never become fingerprint data or tracked metadata. Actual value validity and provider/disposable scope are still unverified. The separate narrow AgentMail/GitHub authenticated read-only discovery approval remains pending; no such request ran during this checkpoint. Native model calls in an authored eval are distinct from that discovery gate. Synthetic imported budget cost-events are fixture context, not incurred usage; campaign billing must use actual native run receipts and keep subscription dollars unpriced where appropriate.
No live launch, selected-case probe, reservation, final retry, production edit or core PR occurred. Goal-tree remains exhausted; alignment and purpose final retries remain for coherent freeze B only. Freeze A still processes all remaining initial attempts first, and only then accumulates one correction set and one reviewed final retry set on B. Coordinator pauses for the runtime authoring wave while retaining sole execution/ledger ownership.
## Pending authoring handoffs
These committed area handoffs await the coordinator's next immutable reconciliation. The finite handoff cutoff includes runtime `ac8ac79d68` (`steering.steer-active`), rather than following mutable HEAD. They do not replace the archived `ad0907afb9` checkpoint of 134 executions or establish live acceptance. The fixed denominator remains 244; the production source baseline remains `8373d7867963674ff960d2345d0d28086f799bc7`. Live history is unchanged: three scenarios, four measurements, zero eligible passes.
| Pending owner handoff | Authored | Without declared gaps | Partial | Still absent |
| --- | ---: | ---: | ---: | ---: |
| Domains, including two legacy cases | 41 | 41 | 0 | 4 |
| Identity | 44 | 41 | 3 | 0 |
| Runtime | 37 | 37 | 0 | 23 |
| Integrator | 22 | 19 | 3 | 5 |
| External | 25 | 22 | 3 | 27 |
| Coordinator privileged | 16 | 16 | 0 | 0 |
| Total pending reconciliation | 185 | 176 | 9 | 59 |
Integrator's latest finite area handoff is `37b72e28e3` (genuine SDK-managed routine), following `2d53c3952f` / `6cc0012da0` (actual Morgan primary execution and review handoff) and `fe05c54c11` / `edbf426db3` (partial skill enablement). Its five absent IDs are `cases.update-case-facts`, `cases.case-history`, `cases.close-case`, `routines.webhook-trigger`, and `skills.review-update`. Card explanation still lacks the actual responding-principal inaccessible-source phase; card refresh still lacks an actual persisted failed-generation phase retaining prior output. Skill enablement has concrete intended assignment/future-version callbacks but requires an observed public configuration grant and an applicable effective-denial branch; rejection of the nonexistent `skills.use` action alone is not proof that every denial context is unsupported. Unfinished authoring and source-only/calibration prerequisite receipts never count as passing outcomes or complete inventory processing.
Shared privacy changes `b72f0f7cbb`, `4e73b7bb70`, `69b2e2b1a2`, and `d1fa9efdaf` protect diagnostic streaming, generated secret registration, reporter archives, authenticated API cookies, and sensitive PATCH calls. Worker registrations synchronously update an owned atomic mode0600 registry outside retained evidence; the launcher redacts before console/log emission and deletes that registry unconditionally. Diagnostic sink cleanup runs after independent observations: actual product response/work-product bytes remain available to leak oracles, while leaking opaque HTML/blob reports are omitted with explicit failure metadata and the original failed diagnosis retained. Actual offline Playwright canaries used synthetic values only, with no provider calls or canonical attempts. Closed source/hash/count receipts are retained under ignored `tests/runner-e2e/results/chat-catalog-campaign/privacy/`: `external-original-closed.json`, `shared-diagnostic-boundary.json`, `identity-private-browser-closed.json`, and `authenticated-api-transport-closed.json`. Raw canary reports are not published. Identity's finite child and exact state-file installation proof scanned 17 sinks without findings; the authenticated API canary scanned nine sinks without findings. Shared fingerprints change with these inputs; older passes cannot silently carry forward.
`7154ba9e85` adds only the approved numeric GitHub alias/discovery bindings for the two workspace repository scenarios and removes the erroneous requirement for an already-qualified managed Anthropic profile. Source credential-name availability still does not establish validity or target authorization. Narrow authenticated AgentMail/GitHub resource discovery remains pending explicit approval after automatic review rejected it; no credentialed discovery calls or metadata persistence occurred. No production fix, core PR, live launch, reservation, or final retry was added by these authoring handoffs. The coordinator retains sole ledger/execution ownership and resumes sole progress-document ownership after this bounded update.
## Finite coordinator checkpoint at 994897f542
Archived source contains 200 authored scenarios: domains 39 plus two legacy tasks, identity 44, runtime 43, integrator 26 and external 30, plus 16 privileged scenarios. Frozen catalog remains 244/35/191 with its original digest. Source production paths and lockfile have no diff from baseline `8373d7867963674ff960d2345d0d28086f799bc7`.
Validation: all 198 lazy exports imported through actual selected dispatch and matched their metadata (193 ordinary/companion definitions and five browser boundaries); derived `--list` contains 200 cells. Full archived E2E typecheck passed. Archived input/fingerprint/diagnostic/oracle controls passed 20 tests; coordinator ledger controls passed 12 tests. These checks launch no canonical workflow or provider. TSX CLI IPC was sandbox-denied during discovery; the supported Node import loader completed the same listing without its auxiliary IPC server.
Ledger processing is verified against retained terminal `result.json` execution/source identity, not mere reservations: three scenarios processed, four measurements retained, zero eligible passes. Source-chat human counts exclude separately evidenced auxiliary human task comments/operator actions. Imported synthetic fixture dollars never enter campaign/provider billing; actual heartbeat usage remains authoritative.
Selected semantic private expectations now use an opaque keyed commitment, separate from credential references. Audited Notion fields are `apiVersion`, `olderFactJson`, `olderMarker`, `recentFactJson`, `recentMarker`. The stable random 32-byte mode-0600 campaign key was explicitly initialized once outside tracked artifacts and model inputs. No raw expectations, key bytes or low-entropy hashes are public; missing/replaced key or changed expectations invalidates replay. Case-only missing inputs are sanitized setup blockers inside reserved attempts; profile/environment credentials remain launch prerequisites. No private provider discovery occurred during this checkpoint.
Proposed manager-reviewed initial slice (all ordinal 1, still unreserved): `access.sign-in` (browser boundary, zero native/source-chat human turns); `projects.create-project` (two native/two source-chat human turns); `tasks.comment-with-reference`, `navigation-preferences.broken-link`, `navigation-preferences.edit-profile` (one native/one source-chat human turn each). All use local Codex subscription profile, max parallelism one and zero retries; no private oracle expectations are selected. Exact candidate source/build/fingerprint receipts must be fixed before admission. Published operator source/target release identities remain distinct from ordinary source-baseline evidence.
Read-only IPC preflight found 97 free semaphores and 19 free shared-memory slots; the single-instance reserve guard passed. Recheck immediately before admission. No IPC cleanup, kernel tuning, live database access or canonical health probe occurred.
## Initial slice stopped after access.sign-in
Manager authorized five initials from isolated commit `0fdcb539a832684ab03a55cc3342c0bef4a9a41a`, based on `994897f542`. The candidate verified 12,465 Git blobs and 5,191 dependency links, with no mutable escapes, unchanged production sources/lock, pinned source-built native binary, full E2E typecheck and 32 focused controls. Native/source/build receipts are retained under `results/chat-catalog-campaign/`.
Only `access.sign-in` ordinal 1 was reserved and actually launched, campaign `local-2026-10-27T04-01-47-918Z`, launcher PID 86819 (now dead). It failed at embedded PostgreSQL `initdb` bootstrap, before the webserver/browser journey or native/model admission. The original actual fallback result is retained verbatim under `results/local-2026-10-27T04-01-47-918Z/abnormal-finalization/access.sign-in/result.json`; its sidecar explicitly labels postprocessing, the original result/log hashes and the secondary finalization defect. This is one failed measurement, not an uncounted probe. Zero native runs/tokens were admitted; provider dollars remain unpriced.
Underlying PostgreSQL stderr was not captured: the pinned `embedded-postgres` initialise implementation wires only stdout despite its stderr comment. The retained log proves bootstrap exit 1, not its precise resource cause. Pre- and post-failure headroom remained 97 semaphores/19 shared-memory slots; no IPC exhaustion claim or limit change is justified. No arbitrary cleanup or PostgreSQL retry was performed.
Secondary harness failure: launcher credentials were replaced by expanded diagnostic variants; the reporter expanded those variants again, incorrectly indexing JSON structural punctuation. Reporter `.last-run.json` became invalid JSON and evidence packaging aborted. The original case-private fallback result survived; original campaign reports zero packaged results. A narrow offline correction retains raw registered secrets, makes repeated expansion stable, serializes reporter JSON structurally, and preserves primary failure plus abnormal-finalization error in a failed result. Synthetic controls cover real failing child stderr, short multiline secrets, repeated synthetic auth/reporter processing, valid JSON and preserved failure attribution. No corrective scenario run occurred.
Initial cleanup remains **unconfirmed/failed** in the ledger. A later read-only exact-owned-root process scan found zero matches and the launcher dead; this is appended verification, not proof of complete resource cleanup. The original private recovery directory remains retained. The other four cases (`projects.create-project`, `tasks.comment-with-reference`, `navigation-preferences.broken-link`, `navigation-preferences.edit-profile`) remain unreserved and unrun. No final retry was consumed; sign-in's one permitted correction belongs to B.
Next safe step is a source-qualified test-scoped PostgreSQL diagnostic path that captures the real owned initdb stderr, plus independently verified cleanup and a meaningful environment/correction change. No further canonical admission is authorized until manager review. A production/dependency stderr behavior fix, if required, stays in the separate core lane; no production code was changed here.
## One infrastructure-only bootstrap diagnostic
Manager authorized one separate owned `initdb` diagnostic with private stderr capture, after five offline wrong-binary/root/owner/privacy/process-guard controls and fresh read-only IPC/ownership checks passed. No scenario, browser, provider or native runner was admitted; campaign attempt counts remain four scenarios/five measurements/zero eligible passes. Original sign-in and goal budgets are unchanged; the other four initial cases remain unreserved.
The exact immutable embedded binary was `@embedded-postgres/darwin-arm64` 18.1.0-beta.16, SHA-256 `5a6ad92c2ce915b1a13aa2f8f2b2199996daf0cbe6b9f1d0044a387303b368ec`. Its bytes matched the prior immutable dependency copy. Command matched the failed server's password auth, paperclip user, C locale and UTF-8 flags, against a new mode-0700 owned directory with mode-0600 password/stderr files. No production code, arguments, requests or kernel settings were rewritten.
Captured stderr proves **semaphore resource exhaustion**: PostgreSQL reported `FATAL: could not create semaphores: No space left on device`, with a failed `semget` allocation of 17 members and an explicit SEMMNI/SEMMNS (not disk-space) explanation. Before and after, host capacity was 97 free semaphores/19 shared-memory slots. The existing 34-semaphore preflight reserve therefore failed to establish bootstrap capacity; no guessed replacement threshold was adopted. The original scenario's missing stderr was not retrospectively fabricated: this is separate diagnostic evidence of the current environment.
The diagnostic's strict process-group ownership guard rejected a member after bootstrap admission. It captured stderr but raised before waiting, so the bootstrap exit code is **unknown**; controller exit code 1 is recorded separately. No second bootstrap was launched and the guard was not relaxed to conceal that failure. A subsequent read-only inspection proved process group 16180 empty and the diagnostic cluster directory already absent. Its exact owned regular mode-0600 password file was removed after those checks; raw logs and the closed receipt remain private. The original failed scenario recovery root was untouched and its cleanup remains unconfirmed.
Receipt: `tests/runner-e2e/results/chat-catalog-campaign/initdb-diagnostic-receipt.json`; pre/post-capacity receipts are alongside it. These record exact command, binary/version/helper hashes, start/verification times, known PIDs, status limits, original stderr/stdout hashes and cleanup. A future diagnostic would first need a reviewed ownership-observation correction; none is authorized by this record. Another canonical initial needs meaningful environment correction or independently qualified capacity plus manager review. No arbitrary IPC deletion, kernel tuning, unrelated process stop or database access occurred.
Read-only resource census distinguishes the binding limit: `SEMMNI=87381` with 5,135 sets leaves 82,246 set slots, while `SEMMNS=87381` with 87,284 members leaves only 97 semaphore members. A bounded sample of 32 current-user 17-member sets all had last-member value 537 and zero waiters; 20 had every recorded member PID dead, but no campaign creator/run receipt association was established. No set was released. The other sampled sets did not satisfy all-PIDs-dead proof and are not cleanup candidates.
The helper's offline guard correction now retains observations and waits for its direct owned child when strict executable validation fails; it never signals an unverified group. Six offline controls pass. No second bootstrap was run. The exact executed earlier helper was reconstructed and verified against its original SHA before being retained privately; its abnormal status remains separate from the corrected helper source.
Source qualification found no existing external-Postgres substitute in this Product E2E lane: `server-config.ts` creates embedded-Postgres configuration; `harness-env.ts` strips and rejects ambient database URLs; `instance-isolation.ts` rejects a nonembedded mode or connection string. The product itself supports external Postgres, but using it here would require an explicit isolated-target capability and provenance review. No credential access, network request or database creation was performed for that investigation. Current capacity is still the known failed 97-member headroom; another canonical launch requires a reviewed meaningful resource/environment correction. Original sign-in cleanup and all attempt budgets remain unchanged.
A concrete dry-run cleanup manifest now lists exactly 20 currently user-owned PostgreSQL-marker sets with all recorded member PIDs dead and no waiters. It records IDs, owner/creator UID/GID, key/sequence, mode, change/operation timestamps and every member PID/value/waiter hash, checked twice without mutation. Campaign association is not established, so explicit user authorization is pending and **zero resources were released**. Recipe aborts on any identity/creation/state/PID reuse/permission/waiter change and rechecks each target immediately before a separately authorized removal. A bounded first eight sets would free 136 members, raising headroom from 97 to 233; this is a proposal, not a proven minimum or permission to expand. All 20 would free 340, but no removal is authorized by this document. Manifest: `results/chat-catalog-campaign/ipc-cleanup-dry-run-20.json`.
Privacy review supersedes the first root-inference correction in `025bf9a420`: two independently registered raw credentials can overlap as encodings, so deriving roots from their bytes was unsafe. Current correction preserves every explicitly registered raw root and carries a process-local provenance-bound expansion for replay. An overlapping-root control requires `base64(base64(A))` to remain protected when raw B equals `base64(A)`; unbound replay is rejected. The original actual report corruption/failure remains retained. No scenario or bootstrap was rerun while correcting this offline defect.
## Integrated finite metadata checkpoint e4b62f48aa
Exact archive imports verified 205 lazy definitions plus two legacy tasks: domains 43+2 legacy (41 no-gap/two partial), identity 44 (41/three), runtime 43 (43/zero), integrator 27 (26/one), external 32 (29/three), privileged 16 (16/zero). Total 207 authored/198 no-gap/nine partial/37 absent. No newer WIP is included. The canonical 244/35/191 inventory and original four-scenario/five-measurement history are preserved. Five concrete blockers and four unfinished branches remain distinct. Registered native Slack metadata is supported, but future Slack callback authoring remains outside this source checkpoint.
Diagnostics/reporting integration preserves isolated source refs `0fdcb539a8` and `025bf9a420` and adopts their coordinator-owned changes without modifying the executed snapshot. Current raw-root expansion uses explicit process-local provenance and preserves independently registered overlapping credentials; seven narrow controls and 400 related evidence/recovery controls passed. Six Python diagnostic guards passed. No production code was edited. No second bootstrap, canonical admission, final retry or IPC release occurred while authorization for the bounded dry-run recovery proposal remained pending.
## Integrated finite archive transport checkpoint 4043cf8984
Exact committed-source archive reconciliation found 208 lazy definitions plus two legacy cases: domains 43+2 legacy (42 no-gap/one concrete blocker), identity 44 (41/three blockers), runtime 44 (44/zero), integrator 27 (26/one blocker), external 34 (31/three unfinished) and privileged 16 (16/zero). This supersedes e4b62f48aa metadata while preserving its history. Total **210 authored/202 no-gap/eight partial/34 absent**, with 207 authoring-resolved. Newly reconciled actual paths are Slack reply-in-thread, wrong-person connection resolution and recovery post-stop-message; their declared transport, source-chat human and native-run bounds match their actual exports.
The archive partial-failure case now has its required pre-server lifecycle and authenticated cross-process control handoff. Only its exact single local execution can start the owned fixed-loopback proxy. The product inherits the ordinary API-origin override only; control receipt, URL and bearer remain in the test host. Actual public agent/run/responsible-person reads restrict the three-task fault. Two real backend completions precede a third pre-admission drop; repeated failed-item calls remain dropped until the explicit human retry. Independent saved-state/audit oracles and first-phase checkpoint remain required. Teardown closes owned connections/control and removes its private receipt after backend shutdown; cleanup errors remain failures. This closes unfinished authoring, **not live coverage**.
Fifty offline controls in six files passed from the pinned archive, including a real separate Node control client, HTTP/upgrade byte preservation, malformed/oversized passthrough, exact actor/person/run restrictions, duplicate fault/retry behavior, private product-environment exclusion, cleanup and fingerprint/accounting checks. Archived E2E typecheck passed, all 208 runtime descriptors matched, and actual `--list` returned 210 cells. Generated CLI/SDK artifacts were adopted only from the previously audited immutable candidate after exact owning-source/lock equality and byte hashes; no mutable worktree dist was imported. Production source diff from 8373 remains empty. Receipt: `results/chat-catalog-campaign/metadata-4043cf8984.json`.
All original four-scenario/five-measurement records remain byte-for-byte unchanged. Zero live/DB/provider/browser/native runs, reservations, second bootstraps or IPC removals occurred during this closure. The eight-set recovery proposal still awaits user approval. Original sign-in stderr was not captured: the later same-binary one-shot diagnostic directly proves current semaphore exhaustion and strongly explains the earlier startup failure, but does not replace missing original stderr or unknown original details. Original cleanup uncertainty remains preserved.
### Archive reporting oracle follow-up
Manager review identified a count-binding defect in the first-phase prose check: any occurrence of two plus a failure word could admit reversed counts. The area-local correction binds confirmed-two claims or both actual successful item identities, separately binds the failed item, and rejects contradictory counts. Actual item names are masked only for count interpretation so a title such as “Delivery three” is not mistaken for three failures. Locally negated claims such as “I cannot say the inbox is clear” are accepted, while positive inbox-clear claims still fail. Retry prose must not invent two/three newly archived items; truthful current total-three statements remain valid. Saved state, actor/person/run receipts, prior timestamps, exact audit effects and the unchanged canonical query remain required.
Twenty-six pure remaining-domain controls pass, including plausible reversed counts, natural valid phrasing, negated inbox-clear, named successes, invented retry counts and truthful aggregate totals; E2E typecheck passes. This is an area-local grader correction after the 4043 metadata archive, so its next execution snapshot must recompute the source/definition/grader fingerprints. Authored counts stay 210/202/eight/34 and eligible passes stay zero. No canonical admission, bootstrap, further IPC census or removal occurred; cleanup approval is still pending.
## Optional Docker database recovery checkpoint
The coordinator paused five-computer authoring for a finite test-harness-only Docker PostgreSQL lane. Default embedded behavior remains unchanged; ambient URLs never select an external target. The optional lane pins the already installed official PostgreSQL18.6 Linux/arm64 image, local Unix context/engine, one per-attempt exact labeled container and named data volume, loopback publication, private mode0600 configuration and password-file handoff. No Docker pull, engine start, container start or database bootstrap has occurred during authoring. A separate infrastructure-only health entry point awaits manager review before its single actual onboarding/migration/API/isolation/cleanup check; it is not a canonical attempt.
Lost create acknowledgements are recorded before resource creation and recovered only through exact names/labels; unknown inspection/engine state preserves cleanup uncertainty. Docker selection never triggers Darwin shared-memory reaping, even if startup throws before returning a handle. Failed startup retains an explicit Docker identity or unconfirmed identity and private recovery receipt. Stable database lane/image/version enter case fingerprints; retained Docker proof must match the selected lane, while old embedded failures retain their original bytes/provenance. No passwords, URLs or per-attempt resource IDs enter the stable fingerprint. New source closures must be recomputed before future canonical admission.
## Docker infrastructure health and appended cleanup (2026-10-27)
The single infrastructure-only health command from immutable `772c29dad96b161e8f2a893df9d4ac1d3779d102` exited 1 during owned PostgreSQL startup. Actual official image/context/engine identities matched; the container remained `created`, never running. Every ownership predicate except literal password bind source matched. Docker Desktop exposes both actual bind records as `/host_mnt` plus the exact owned canonical host path. Original health and initial `cleanup-failed` receipts remain unchanged.
After explicit manager authorization, immediate full engine/context/ownership/private-file revalidation permitted removal of only full container ID `ec05ccb6b21e1e8724de792d314597ab10d27c22cfb6819d364eed67e75b976c` and its exact labeled local volume. Independent inspect calls proved both absent before the exact password file was removed. The append-only verification is `results/chat-catalog-campaign/docker-health-cleanup-772c29dad9.json`; diagnosis and original health receipt are linked there separately. This adds no canonical attempt or provider usage. The narrow correction admits only the exact Docker Desktop translation with an owned canonical regular0600 password, preserving all other guards; positive/negative offline controls cover wrong root, other engine OS, writable bind and symlink. A second infrastructure health requires separate authorization.
The separately authorized corrected infrastructure-only health from immutable `325af67e3dcf2e13c5380d01047ec4be91fc95a0` passed in 21.749 seconds: actual PostgreSQL `18.6 (Debian 18.6-1.pgdg13+2)`, normal Paperclip onboarding/migrations/API, zero companies/agents/native runs/models/scenarios. Exact labeled container `9ad4d09649748f41bb0f2009968cf1b540b01fb6f2d296b76138855b94d5b81f` and its owned volume were removed with independently verified absence; cleanup failures are empty and the private root was removed. Retained receipt: `results/chat-catalog-campaign/docker-infrastructure-health-2026-10-27T05-48-52-510Z.json`. This qualifies only the explicit owned-Docker PG18.6 lane, not embedded PG18.1. Production baseline remains8373; original failed infrastructure/canonical evidence is unchanged. Host semaphore cleanup is no longer needed. No canonical reservation was added.
## Finite computer authoring handoff (pending batched registration)
Five execution-environment callbacks are authored in `runtime-computers-ssh.ts` and `runtime-computers-cloud.ts`: add-computer, choose-capable-target, test-computer, assign-computer and prepare-image. They remain **unmeasured**. SSH creation uses a standalone human browser for the selected pinned-host/private-key target; transport and actual remote Claude adapter validation are separate, and a failed login leaves runtime testing explicitly unrun. Source inventory/provision/probe authority is independently denied to the ordinary agent. The cloud paths reuse the new-attempt-owned Daytona target helper; assignment checks exact saved Morgan configuration and reload, while image preparation verifies actual Node/npm/Git through the supported private operator terminal, captures a real provider template, reads persisted session/template attribution and performs a subsequent actual-target/lease probe before the declared factual human result phase. No live SSH/cloud call occurred during authoring.
Fifteen bounded oracle/closed-receipt controls and the E2E typecheck pass. Six actual standalone synthetic browser/terminal privacy controls cover success, private child failure and expected parent failure; scans of 40 report files and 37 ZIP members found 0 synthetic credential findings. Closed source-hashed evidence is `results/chat-catalog-campaign/privacy/computer-private-ui-closed.json`. Missing allowlisted targets/provider references remain genuine setup blockers; code/calibration is not successful remote qualification. Authoritative244inventory counts await one exact-archive batched metadata reconciliation.
Manager review correction to the five-computer handoff: `test-computer` now reads the actual saved Morgan harness/config/account binding in the private child and submits that exact public saved-agent validation on the intended SSH environment. It no longer substitutes a Claude test for a Codex Morgan. The explicit choose-capable-target query retains its actual Claude check. Cloud assignment now permits only `defaultEnvironmentId` to change: the already-native Morgan needs no adapter normalization, so model/account/config substitutions fail even within the same harness. Seventeen oracle controls and eight actual synthetic private process/UI/terminal controls cover these cases; current source-hashed privacy evidence supersedes the earlier six-case calibration without adding live proof.
## Immutable metadata reconciliation — 32b472dfa5
All 227 concrete lazy exports and 2 legacy definitions were reconciled from the exact Git archive 32b472dfa5, with all 244 original requests/capabilities/inventory hashes preserved. Every historical attempt object was compared against its original ledger bytes: 4 scenarios / 5 measurements remain unchanged, with 0 eligible passes. Source-only/unfinished probes remain distinct from concrete supported-surface blockers and live processing. The accepted preinitial fixture correction 2e6a5dbe27 removes outcome instructions from broken-link descriptions without changing its grader; the separate note correction requires an affirmative instruction with a parsed same-origin actual design href and rejects negated/wrong-target/foreign links. No attempt budget was consumed. The execution candidate incorporates these exact source corrections separately from the metadata cutoff; newer area handoffs remain queued.
## Latest measured slice: project creation initial (2026-10-27)
`projects.create-project` ordinal 1 ran from reviewed immutable `6951f67ecd1aaedec789703dc9c7227bddd2b59a`, production baseline `8373d7867963674ff960d2345d0d28086f799bc7`, profile `runner-codex-subscription`, model `gpt-5.6-sol`, automatic retries zero. Campaign `catalog-initial-2026-10-27T07-03-53-222Z-projects-create-project` retains the original failed result and persisted-state scan. Two real native runs and two source-chat human turns satisfied all ten independent behavior checks, including exact retry reusing the first project. The supervisor then reported `secret_leak` in private disposable `config.json`; the scan sidecar preserves original result status `passed`, but the actual measured outcome remains **failed**.
Source diagnosis: the owned Docker helper registers its generated password and connection string as protected roots; normal server configuration intentionally stores that connection string in the private disposable config; the generic persisted-home scan has no corresponding narrowly validated custody handling. This identifies a shared harness custody conflict, pending manager review; it does not establish a product or model leak. Further admissions stopped and the other three selected cases remain unreserved. Exact result: `tests/runner-e2e/results/catalog-initial-2026-10-27T07-03-53-222Z-projects-create-project/chat-features/runner-codex-subscription/local/projects.create-project/attempt-1/result.json`.
Cleanup passed; actual database identity reports the exact pinned PostgreSQL 18.6 artifact and container status `removed`. Actual heartbeat usage is **704,239 total tokens including cache** across two runs; subscription dollars remain unpriced. No imported fixture cost events were included. Original earlier failures and cleanup overlays remain unchanged.
### Docker custody correction and remaining-three proposal
The authorized harness-only correction `96058c7e2f5364342917ebeb598089b5195330d9` uses the existing product-supported server `DATABASE_URL` configuration. Seeded Docker config contains no DSN; only the exact receipt-validated server child environment receives it. Ambient URLs remain scrubbed, actual native Codex environment projection excludes both database variables, and the persisted-state scanner is unchanged. Controls reject an arbitrary server URL, undeclared override, persisted DSN, a DB password elsewhere in config, and unrelated provider credentials. The project initial remains failed; no postprocessing changes its outcome or consumes its held final-B retry.
New immutable remaining-three candidate: `f73390f925edcddd3af977e337adf16d5818fb41`, parent `6951f67ecd1aaedec789703dc9c7227bddd2b59a`, with only the six custody paths overlaid. Actual archived E2E typecheck exited 0; 109 targeted controls across four files passed. Production sources/lock still exactly match baseline `8373d7867963674ff960d2345d0d28086f799bc7`; 46 dependency links and copied build/native bytes match prior audited immutable receipts. Read-only Docker preflight confirms the same qualified engine and installed PostgreSQL 18.6 digest. Three fresh case fingerprints and name-only private-input replay receipts are in `results/chat-catalog-campaign/initial-selection-f73390f925.json`; scoped control/provenance summary is `custody-correction-f73390f925.json`. Comment-with-reference, broken-link and edit-profile remain unreserved pending manager admission. No new bootstrap, model/provider calls, canonical launches or reservations occurred during correction.
### Comment initial startup failure (2026-10-27)
Campaign `catalog-initial-2026-10-27T07-15-25-823Z-tasks-comment-with-reference` consumed only comment-with-reference ordinal 1 from reviewed `f73390f925edcddd3af977e337adf16d5818fb41`. Actual CLI doctor reports “PostgreSQL mode selected but no connection string configured” and refuses server start. Source `cli/src/checks/database-check.ts` checks only `config.database.connectionString`; it does not use the server’s `DATABASE_URL` override. The prior custody correction missed this normal-startup contract, so this is an eval-harness startup failure, not feature/model evidence. There were zero browser/scenario/native/model admissions or tokens; the owned PostgreSQL infrastructure did start. No retry was performed.
Original fallback result and logs are retained at `tests/runner-e2e/results/catalog-initial-2026-10-27T07-15-25-823Z-tasks-comment-with-reference/chat-features/runner-codex-subscription/local/tasks.comment-with-reference/attempt-1/`. Result database identity reports pinned PostgreSQL 18.6 and container status `removed`. Overall cleanup nevertheless remains failed/unconfirmed; private root `paperclip-runner-e2e-hZhllt` is retained. This distinction is preserved without inferring all cleanup from Docker removal. Broken-link and profile remain unreserved pending the shared startup/custody correction and explicit admission.
### Exact persisted-DSN custody correction; infrastructure qualification proposed
Commit `1fc30a0ed1d3c2d3770b0e70d94710f0b3310589` restores the product-required file DSN and original server environment scrubbing. New `docker-config-custody.ts` validates the independently observed live owned Docker receipt/password/container and exact current-UID canonical private attempt/home/instance directories (0700), regular nonsymlink config (0600), and its single exact `database.connectionString`. The directory scanner removes only that one validated raw JSON string literal from its scan input; it scans all remaining original bytes and all secret roots. Wrong field/path/owner/modes, duplicate DSN, extra password, unrelated provider credentials, symlinks or non-ready lifecycle fail. Native Codex DB environment remains absent; no public/raw report sanitization was relaxed.
Immutable infrastructure qualification proposal `6beacfe0a94e87b9b834f0f1d46d42e8ec7aec42` overlays only ten owned correction paths on failed snapshot f733. Archived E2E typecheck exited 0 and 120 focused controls across five files passed. Production baseline8373/lock, copied workspace artifacts, native binary and 46 audited dependency links remain unchanged. The health helper now executes the same final persisted-home secret scanner and exact custody policy, recording `sameFinalPersistedStateSecretScan` explicitly. One infrastructure-only health qualification has been proposed to the manager; no new health/canonical launch or reservation occurred while authoring.
Append-only read-only cleanup evidence `results/chat-catalog-campaign/comment-hZhllt-readonly-cleanup.json` verifies the retained comment root is owned0700, exact recorded Docker container and volume are absent, four known observed PIDs are absent, and no current command references that exact root. This limited census does not establish a complete descendant history; original cleanup failed/unconfirmed remains unchanged and no resource was removed. The CLI doctor’s file-only DSN check versus the server’s supported `DATABASE_URL` override is recorded as a candidate core consistency issue for a later separately qualified PR, with no production edit here.
### Qualified custody health and broken-link initial
The one authorized infrastructure health from6be passed in19.656s. Retained `docker-infrastructure-health-2026-10-27T07-32-21-104Z.json` confirms PostgreSQL18.6, API/normal migrations, `sameFinalPersistedStateSecretScan:true`, zero companies/agents/native/model/scenarios, cleanupFailures[] and exact owned container status removed. No canonical attempt was charged for infrastructure qualification.
Broken-link ordinal1 then ran as `catalog-initial-2026-10-27T07-35-44-803Z-navigation-preferences-broken-link`, one actual native run and323,491 tokens, subscription dollars unpriced. It failed after the bounded poll; original result and usage remain unchanged. Retained setup proves `stale.id === similar.id`: production’s recent-open-title dedup returned the first task for the second same-title create. Deleting the old task therefore also deleted the fixture alias intended as the separate distractor. The expected404 status helper worked; the later unchecked `similarAfter` read caused the timeout. Browser receipts still show the genuine missing URL and private-resource nondisclosure. This is a fixture defect, not model navigation failure.
Separately, diagnostic privacy evidence shows the registered setup human invitation token in the ordinary product HTTP URL `POST /invites/[REDACTED]/accept202`. The stream/report safeguards redacted both text sinks and omitted the affected blob; no raw credential value was inspected for diagnosis. All admissions stopped on this shared privacy finding despite cleanup passing and exact Docker status removed. Profile plus the approved12 initial list remain unreserved, with no rerun or final-B measurement. Bounded diagnosis: `broken-link-6be-diagnosis.json`.
The separately authorized registered-artifact transport fix `cbb6593441` uses the existing authenticated `getAttachmentContent` path, retaining attachment origin/size/hash gates.33 focused controls and scoped tsc passed; no executed archive was modified. Selection-summary provenance sidecar explicitly retains historical snapshot4/5 counts versus current ledger7/8 and the previously acknowledged external-fixture source overlap.
### Unaffected initial batch resumed
After actual selected-setup review, only profile and quick-search invoke `enrollAuthenticatedPerson` among the pending13. They remain unreserved. The manager authorized continuing the other11 from unchanged6be; `initial-invite-setup-disposition-6beacfe0a9.json` records exact disposition. Create-backlog ordinal1 **passed** with1native/346,471tokens and confirmed cleanup/Docker removal. Original passed measurement is preserved. `initial-pass-identity-review-tasks.create-backlog.json` records actual changed selected source hashes, not a new Git SHA alone; affected final eligibility waits for the single allowed coherentB measurement. Delegation began next with its declared3-native bound.
The area-only broken-link fixture correction `61c863364a` uses a distinct similarly named replacement and independent saved ID/project readback guard before deletion;10 controls plus scoped tsc pass. It preserves the original request/oracle and failed initial; its correction measurement stays held forB. Shared artifact transport correction remains separate `cbb6593441`. Source metadata for runtime067f unknown-email and later external callbacks is queued until this batch naturally ends.
doc/plans/2026-11-07-durable-ownership-and-reconciliation.md
+7 -1
View File
@@ -37,7 +37,7 @@ These become reconciler checks and crash-suite assertions. They should also be a
5. **Work-based budgets:** a restart consumes run budget only when the resumed attempt started provider work and made no progress.
6. **Generation fencing:** every runner and provider event carries its generation and turn. Events from an earlier generation are recorded as history and never fail the current turn.
7. **Visible next action:** every non-terminal task has a next action, either automatic or a working control. "Waiting" with no owner is a violation.
8. **Fast turns:** TTFT and agent boot stay within their measured budgets. Restart safety work never adds waiting or file transfer before `turn.start` (see "Performance constraint" below).
8. **Fast turns:** time to first visible agent activity and agent boot stay within their measured budgets (TTFT, time to first text, is reported; since 2026-11-08 the budget judges first activity, because tool-first adapters call tools before writing text). Restart safety work never adds waiting or file transfer before `turn.start` (see "Performance constraint" below).
## Performance constraint: a turn must start as fast as possible
@@ -99,6 +99,12 @@ Restart recovery latency, from `ready` to the recovered run's next provider even
| codex | 6.5 / 8.0 |
| opencode | 25.5 / 38.5 |
Since 2026-11-09 this full latency is reported, not judged: the provider's saved-session load (`provider.session.load`) grows with session size and with how many sessions resume at once (Codex at concurrency 6: 3.0–6.3 s of a 5.5–8.5 s recovery). The judged row is Paperclip's own share, `restart_recovery_control_plane`: from `ready` to the start of the provider's session load, or to the next provider event when nothing reloads. It covers claim, prepare, runtime stage and runner launch, which measured 1.2–2.0 s per run with six Codex runs resuming at once (wave `w-20261109T045320-s9807`).
| Adapter | Paperclip share budget (s, p50 / p95) |
|---|---|
| claude, codex, opencode | 2.5 / 4.0 |
These budgets measure the product default `lifecycleMode: per_turn`. Phase 1's TTFT work (runner and provider-session reuse, baseline finding 1) should lower them. Lower budgets go in the same table.
### Phase 1 report-only data: local chaos waves (2026-11-07)
doc/plans/2026-11-07-online-git-source-fence.md
+68
View File
@@ -38,12 +38,28 @@ their existing reconstruction path.
Preparation, lifecycle, runtime-service, expiration, and reopen owners retain
the same physical keys before filesystem mutation or lifecycle exclusion. The
importer holds all cohort task/project and lifecycle keys through publication.
Each cohort reserves its members in fully joined batches of at most eight
independent transactions. It joins every reservation before starting capture or
releasing the source fence, including on failure. Partial journals keep the full exact member
and operation set for resume; publication stays serial in one transaction.
It also holds the existing per-run physical finalizer keys for each exact native
recovery proof. It re-inspects the complete source proof before capture and
publication. Changed descriptors, allocations, owners, aliases, or Git metadata
reject publication. Prepared sources and unresolved recovery warnings retain
their original state and evidence. They are not marked finalized or cleared.
A terminal failed run can hold recovery for a separate durable remote workspace
on the same task. A strict remote-only proof lets the old local source import
without resolving that recovery. The proof binds the exact run, finalizer,
action, accepted result, task, remote authority, allocation and any committed
baseline snapshot. It uses the canonical remote source inspector, takes the
run's finalizer fence and is rechecked before publication. Mixed legacy/remote
references, active owners and changed bindings remain blocked. The remote
source never enters the legacy export list; its dirty state, files, result and
recovery obligation remain intact. Previous-release prepared references keep
their original epoch and prior snapshot identity; the baseline need not have
been created by the held run.
Present nested workspaces must import before their parent. The existing owning
nested-import path downloads and restores each immutable child snapshot and
compares it with a fresh source capture before excluding that child. Missing or
@@ -88,3 +104,55 @@ serialized remote function, compare archive bytes and digests, remove every
original source, restore uncommitted/staged work, and run fsck. No hosted import,
hold, cancellation, provider call, or shared-cache Git command is part of these
local tests.
A cohort shares its initial nested-source verification only within its held
source and finalizer fences, keyed by physical root, Git roots and the exact
recorded child set (or discovery). After every capture joins, a separate fresh
verification checks each physical root and child set before publication. A
changed child still rejects the whole cohort. Singleton imports keep their
per-import checks. No local-byte proof survives the invocation.
The online migration owner can checkpoint completed immutable nested-snapshot
verification in the child snapshot ledger. This optional versioned proof binds
the complete encrypted reference, child lineage, source operation and a known
S3 endpoint/region/bucket/prefix/style namespace. A new process can reuse it only
after fresh strong-ETag/length, current ledger/source-fence checks and canonical
metadata-only unwrapping of each child key under the current master key. This
key check also runs for in-memory hits; it returns no secret or payload. The proof
is published only after canonical decryption, manifest/file restore and a stable
post-verification HEAD. Missing or unknown proof versions, implicit provider
endpoints and weak or changed validators require full verification. Existing
records need no migration. No plaintext scratch, parent journal CAS, authority,
retention deadline or local/Git source proof is reused. Every resumed invocation
reacquires the source locks and captures current local bytes again. Read-only
verifier callers do not opt into checkpoint publication.
A change that strengthens immutable archive verification must advance the proof
version so earlier verification cannot bypass the new checks.
A cohort with nine to 32 members is admitted only when every canonical resolved
physical root is exactly the same. Cohorts of at most eight members retain the
existing mixed-leaf rules. Source resolution, admission, reservation, recovery
checks, final readers, replay and cleanup remain bounded to eight peers per
stage. Capture concurrency remains at most eight, and all running peers join
before an error or cancellation can release the source fence.
With shared initial and final child verification, 30 fresh aliases need one
initial and one final child pass instead of four pairs. For N distinct covered
children this changes 8N child checks to 2N. Thirty parent captures, uploads and
restores remain required. This is a work-count reduction; hosted wall-clock
improvement is not yet measured. Saved six- and eight-member cohorts retain their
exact membership. The canonical maintenance request accepts the same bounded
32-member list; single-workspace and capture-worker limits stay unchanged.
The journal shape is unchanged; older tools that enforce the
eight-member bound refuse larger cohorts and must not regroup them.
After a successful online cohort capture, members without archived repository
intents release their parent archive, authenticated readback, fresh comparison
archive and restored tree before the worker starts another member. The scratch
root and nested-child verification areas remain until atomic publication, and
the final fresh child checks still run. Archived repository intents retain their
restored payload for the final publication proof. Select a larger cohort
only after checking the actual parent capture footprint, its verification and
restore scratch, and current free capacity on the target filesystem. The worker
limit does not bound retained nested-child scratch. Keep the existing resource guards.
doc/plans/2026-11-07-online-workspace-migration-entry.md
+9
View File
@@ -96,6 +96,15 @@ The entry does not assert stopped writers, terminate an agent, clear a hold or
change remote authority. An unrelated active run is not a global-idle blocker.
An actual scoped writer, foreign alias or unsupported source still refuses.
Programmatic administrative calls to `migrateRemoteWorkspace` and
`migrateRemoteWorkspaceCohort` can pass `sourceLockWaitMs` with their existing
`signal` to join the source-lock queue. The budget covers acquisition of the
complete sorted source scope, rather than restarting for each key. Timeout or
abort rolls back that admission before any journal reservation. This opt-in uses
an independent source transaction and preserves the lifecycle and publication
checks; activity is published after commit. Omitted options retain try-only
admission. This is a canonical API option, not an additional JSON entry field.
Before each apply, use the existing per-operation memory, CPU and temporary-disk
resource preflight. A concurrency bound is not resource admission. Budget every
logical capture and up to two nested-child verifications. Qualify one single
doc/plans/2026-11-07-ownership-inventory.md
+2 -2
View File
@@ -346,12 +346,12 @@ Scope: workspace operations (#19807 intents), runtime leases, runtime services,
| Managed runtime-control operation (start/stop/restart/repair/run) | `workspace_operations.status` (+ `metadata.runtimeControlOwner`) | workspace-operations.ts:559-578 (stamp :539-550, 10 s heartbeat :586-612) | `running` | `{ownerId: random UUID per process, pid, heartbeatAt}`. No host, boot id, or process start time (:25-29, :69-74) | Owner settle :671-690, owner failure :701-728, own deadline 30 min / 4 h (:46, :53, :656-668). Sweep `sweepRuntimeControlOperations` :297-415, run at startup (index.ts:1087) and before each control (:448-454, routes/execution-workspaces.ts:484, :570) | Same ownerId and not in the in-process set → abandoned (:334-335). Otherwise `process.kill(pid,0)` (:126-135) or **TTL/age**: 60 s without a heartbeat (:38, :346). CAS on `updated_at` (:142-147, :379-383) | ownership | 1 (no host/boot), 2 | A, C |
| Native workspace finalize attempt (copyback export) | `workspace_operations.status`, phase `workspace_finalize`, `metadata.owningService='native_workspace_finalizer'` | native-runtime/native-workspace-finalizer.ts:170 → workspace-operations.ts:559 | `running` (`pending` is also read as unresolved) | none on the row. Run id only. Physical owner is the `nativeWorkspaceFinalizationOwner` receipt (row below) | (a) Normal settle workspace-operations.ts:671-728. (b) #19807 supersession native-workspace-finalization-supersession.ts:21-91, called from native-workspace-finalizer.ts:94. (c) Offline maintenance offline-workspace-operation-reconciliation.ts:30-118, called from maintenance/ephemeral-migration-data.ts:52. (d) Online variant :152-157 (**no production caller**) | (b) Caller holds finalization ownership, plus the in-process registry `isNativeWorkspaceFinalizationOperationActive` (workspace-operations.ts:114-119) and a later terminal attempt. (c) Advisory lock :41, committed coordinator with no lease :52-57, finalized sync hash :58-60, accepted result :65-70, later succeeded receipt :83-92. Never age | workspace-preservation | 1, 2 | B, C |
| Remote provision command intent | `workspace_operations.status` + `metadata.uncertain`, phase `remote_workspace_provision` | remote-workspaces.ts:973-975 (`running`, `uncertain:true`) | `running` / `uncertain=true` | `heartbeatRunId` + `metadata.epoch` (fenced by authority epoch/writerRunId :964) | Success only: remote-workspaces.ts:989-990. On a non-zero exit, a timeout (:980-982), or a crash, **no write settles the row** (grep finds no other writer) | none. Reservation for any other run is refused with `remote_workspace_provision_unverified` (:220-225). Offline migration counts it as a blocker (remote-workspace-migration-tree.ts:160, remote-workspace-migration.ts:237) | workspace-preservation | 1, 2, 3, 7 | B, C |
| Remote snapshot save (initial/checkpoint/final) | `workspace_operations.status`, phase `remote_workspace_snapshot` | remote-workspaces.ts:729 | `running`, `pending` | `heartbeatRunId` + `metadata.epoch`. Fenced by `assertOwned` and the authority writerRunId/epoch (:776-779) | Owner settle in `saveRemoteWorkspace`. Earlier saves are retired only after a final save commits: `retireQuiescedSaves` :592-630 | Save-work quiescence (`readSaveWork` activeIds/quiescedAt :616-618) plus a committed final snapshot (:597-608). No owner-death proof | workspace-preservation | 1, 2 | B, C |
| Remote snapshot save (initial/checkpoint/final) | `workspace_operations.status`, phase `remote_workspace_snapshot` | remote-workspaces.ts:729 | `running`, `pending` | `heartbeatRunId` + `metadata.epoch`. Fenced by `assertOwned` and the authority writerRunId/epoch (:776-779) | Owner settle in `saveRemoteWorkspace`. Earlier saves are retired only after a final save commits: `retireQuiescedSaves` :592-630 | Save-work quiescence (`readSaveWork` activeIds/quiescedAt :616-618) plus a committed final snapshot (:597-608). Each save invocation records its controller process in `saveWork.owners`. The ownership reconciler (`settleAbandonedRemoteSnapshotSave`) settles a save `failed` when its active invocations' process is proven gone, or when it is quiesced (or saved before owner stamps) and its run is terminal | workspace-preservation | 1, 2 | B, C |
| Remote workspace migration op | `workspace_operations.status`, phase `remote_workspace_migration` | remote-workspace-migration.ts:375 | `running` | none (workspace id) | Explicit publication or repair by migration (unverified line). The runtime-control sweep skips it on purpose (workspace-operations.ts:319-323) | none | workspace-preservation | 1, 2 | B, C |
| Execution workspace runtime lease | `execution_workspace_runtime_leases` (row existence) | workspace-runtime-leases.ts:194-209 (create), :235-247 (renew), :255-273 (reclaim). Called only from routes/execution-workspaces.ts:567 | row present | `owner_key` `issue:`/`run:`/`agent:` (+ FK columns). No process identity | `release` :291-313. In production it is called only with `force` on archive (routes/execution-workspaces.ts:1270). Otherwise the lease is reclaimed lazily at the next claim (:250-283) | `evaluateStaleOwner` :85-119: issue hidden or terminal, run terminal, or **TTL/age** 30 min (:39, :90). An agent-scoped owner has TTL only (:116-118) | ownership | 1, 2 | A, C |
| Workspace runtime service (local process) | `workspace_runtime_services.status` | workspace-runtime.ts (insert line unverified) | `provisioning`, `starting`, `running` | `startedByRunId`, `ownerAgentId`. The pid lives in a local service registry file, not on the row | Startup `reconcilePersistedRuntimeServicesOnStartup` workspace-runtime.ts:8636, which adopts or stops each service (index.ts:1101, not awaited) | Registry record match (workspace-runtime.ts:8781-8797) plus `isPidAlive` local-service-supervisor.ts:218, :245-250. Same host only | ownership | 1, 2 | A, C |
| Remote workspace writer lane | `execution_workspaces.metadata.remoteWorkspace.writerRunId` (+ `epoch`) | remote-workspaces.ts:230 (epoch+1), :243 (new) | non-null writerRunId | writer run id + epoch | Final save sets it to null: remote-workspaces.ts:780-781. Migration: remote-workspace-migration.ts:516 | `readRemoteWorkspaceWriterHold` remote-workspace-writer-hold.ts:25-79: run status (:42), lease termination/stop receipts (:53-56), recovery-save progress (:63-76). The waiter gives up after **TTL/age** 60 min of writer inactivity (heartbeat/workspace-preparation.ts:1050-1051, run-dispatch/domain/wake-context.ts:36) | ownership | 1, 2, 3, 7 | A, E |
| Remote workspace unsaved-files flag | `execution_workspaces.metadata.remoteWorkspace.dirty` | remote-workspaces.ts:780-781 (`dirty: boundary !== "final"`) | `dirty=true` | writerRunId (same object) | Final save :781. Failed-run recovery native-runtime/native-failed-workspace-recovery.ts:52-60 (pins/saves) | Writer-hold rule (above). Lease termination receipt | workspace-preservation | 1, 2 | C |
| Remote workspace unsaved-files flag | `execution_workspaces.metadata.remoteWorkspace.dirty` | remote-workspaces.ts:780-781 (`dirty: boundary !== "final"`) | `dirty=true` | writerRunId (same object) | Final save :781. Failed-run recovery native-runtime/native-failed-workspace-recovery.ts:52-60 (pins/saves). A writer that died during preparation (terminal, no resolved runtime, no native finalization, no save in flight) is ended at the last committed save by the ownership reconciler (`settlePreparationOnlyRemoteWorkspaceEpoch`) | Writer-hold rule (above). Lease termination receipt | workspace-preservation | 1, 2 | C |
| Remote snapshot ledger | `execution_workspace_snapshots.status` | remote-workspace-snapshot-storage.ts:73-83 (`pending`, deleteAfter = now + 7 d) | `pending`, `uploaded` | `source_operation_id`, `source_run_id` | Commit remote-workspaces.ts:782-784. Orphans become `delete_requested` by **TTL/age** 7 d from createdAt (remote-workspace-snapshot-retention.ts:9, :113-122), deferred while ops are running (:36) | none (age) | workspace-preservation | 2 | C |
| Git restore point | `execution_workspace_restore_points.status` | git-restore-point-storage.ts:146-155 | `pending` (and `expiring`) | none (workspace + generation) | → `active` :187-188. → `superseded` :141-145, :182-185, :208-216 | none found | workspace-preservation | 1, 2 | C |
| Git cache expiration intent | `execution_workspaces.metadata.gitCacheExpiration.completed` | git-workspace-expiration.ts:384-390, :401-406 | `completed=false` | `lockToken` (random, :384, :405). File locks git-workspace-expiration-locks.ts:72-76 | Deleting the key on completion git-workspace-expiration.ts:454. Restore/rehome deletes it (execution-workspaces.ts:4516, isolated-repository-rehome.ts:485) | Transaction advisory lock `execution_workspace_lifecycle:` (:302-303). Resumable by any worker using the same token (recover: git-workspace-expiration-locks.ts:57-63) | workspace-preservation | 1 | B, C |
doc/plans/2026-11-07-phase2-reconciler-data.md
+4 -1
View File
@@ -235,7 +235,10 @@ Phase 3 is not started. It needs a separate go-ahead.
- **Daytona** is covered by one wave only (see "Daytona wave"). The first apparent content-ID mismatch was a misreading: CI run 42422571919 built target commit `a37a2e4`, not master. Local content IDs reproduce CI's for the same commit.
- The remote, workspace-operation, sandbox and accounting kinds (see "Per-kind data").
- **Follow-up: `environment.stop_retain.after_claim_before_stop` has no live coverage.** No chaos workload creates a stop-and-retain intent (a native workspace export resume on a sandbox), so the crash point is never reached. A workload that forces an export resume on Daytona would cover it. `environment.lease_release.after_provider_before_lease` is also unreached by native runs, which release through teardown.
- `environment.stop_retain.after_claim_before_stop` is covered since #20049 (`--daytona-reuse-lease`; passed in `cp-20261108T133351`).
- **Known gap: `environment.lease_release.after_provider_before_lease` is unreached on Daytona.** Native runs release their sandbox through teardown, not this path. Recorded, not pursued.
- **Known gap: orphans confirmed stopped while `canary/v2026.1108.0-canary.6` or `canary.7` ran.** #19964 shipped without #19976 for about three hours, and those orphans kept no durable marker, so they cannot be told apart from an intentional stop-and-retain. The provider's auto-delete bounds them. A stop still pending at upgrade is covered by #20043. Recorded, not pursued.
- A replaced container's stamp (Docker, Kubernetes, Fly, a rebooted host) is now provable through the controller instance lease (see `doc/execution-semantics.md`). Boots from before the lease remain unprovable.
- Crash points that sit between two durable writes. Random waves hit 20 ms–1 s windows only by luck. The crash-point suite (Phase 5) is the tool for those.
## Reproduce
doc/plans/2026-11-07-ttft-baseline.md
+1
View File
@@ -45,6 +45,7 @@ Plan: [`2026-11-07-durable-ownership-and-reconciliation.md`](2026-11-07-durable-
| Metric | Definition (run-log spans) |
|---|---|
| **TTFT** | `task.time_to_first_token`. It runs from the accepted input to the first deliverable text committed by Core. The accepted input is the wake comment, the answered question, or the durable wake request when that is earlier (#19838). Otherwise it is run creation. `anchor` names which one. |
| **First visible activity** | `task.time_to_first_agent_activity`: the same start as TTFT, to the first text delta, tool-call start or reasoning item committed by Core. Added 2026-11-08: the speed budgets judge this metric (from run creation) with the TTFT numbers below, and TTFT is reported without a budget, because tool-first adapters such as Claude call tools before they write text. |
| **Agent boot** | Run claimed (end of `heartbeat.queue`) → provider session accepted `turn.start` (end of `runner.turn.submit`, or of `runner.session.resume` for a resumed session) |
| **Pre-turn steps** | Every span that starts before the turn boundary above, without cumulative wrappers (`task.prepare`, `runner.session.startup`, `environment.startup`) |
| **Restart recovery latency** | Fault `readyAt` (the harness saw `/api/ready` = `ready` and a healthy `/api/liveness`) → the recovered run's next provider event, grouped by `native.recovery.transition.runnerDisposition` |
doc/plans/2026-11-07-workspace-cohort-capture.md
+34 -12
View File
@@ -35,22 +35,25 @@ The batch acquires an independent complete source fence, all component lifecycle
locks, and the union of required physical copyback locks. It refuses to borrow
an enclosing transaction because a savepoint cannot confirm the outer commit.
Reservation runs serially on the original pool. Capture, encryption, upload,
authenticated readback, restore, and recapture run concurrently on that pool.
Reservation runs serially on the original pool. Each member makes one archive,
computes its checksums while copying files into it, encrypts it, and
uploads it. Migration trusts a successful upload. It does not pre-hash source
files, download the upload, extract a test copy, or capture the source again.
Encryption computes the ciphertext checksum as it writes, without a separate
read of the encrypted file. These member operations run concurrently.
Under the complete fence, preflight compares every selected workspace fingerprint
with the aggregate source proof's full preimage before the first reservation.
Each reservation repeats its member check before writing a journal. Earlier
source changes refuse the attempt; a fresh attempt can replan when no journal
was reserved. Source changes after a partial reservation remain blocked by the
saved aggregate proof.
One nested-child verifier belongs to the complete cohort fence. Its prepare
calls are queued; each call retains the existing two-child bound and completed
authenticated immutable proof. Every member still rechecks current local bytes,
Git metadata, database references and recovery bindings. The verifier closes
only after all member work joins, before the fence can leave.
One nested-child selector belongs to the complete cohort fence. Its prepare
calls are queued. It uses committed snapshot and operation records to retain
already imported children as dependencies. It does not download, restore, or
recapture those children. Publication still checks their database bindings.
The selector closes after all member work joins, before the fence can leave.
The caller must budget for the number of logical captures, even though there is
only one Node process. Include up to two nested-child verification workers in
addition to the configured member concurrency. The first hosted qualification
only one Node process. The first hosted qualification
uses two same-component leaves without imported children. Eight captures can use much more scratch space, CPU,
storage bandwidth, and database capacity than one capture. Start with a
two-member canary and measure the actual host before increasing concurrency.
@@ -71,9 +74,10 @@ eligibility or snapshot work. Publication repeats that check. The owner does
not reset or reopen those journals. A single-workspace import refuses a cohort
journal.
After capture, the owner transaction locks and freshly validates every member,
source, journal, recovery binding, and uploaded snapshot before it writes any
workspace authority. It then publishes members serially. The outer commit must
After upload, the owner transaction checks every member's ownership, journal,
recovery binding, and snapshot record before it writes workspace authority.
It does not reconstruct the data to prove the copy. It publishes members
serially. The outer commit must
succeed before the batch returns success or publishes live activity events.
The originals remain intact.
@@ -84,6 +88,24 @@ reservation. A changed or incomplete membership is rejected. Fresh physical
source checks still run after a process or volume handover; the durable logical
fingerprint does not treat an inode number as lasting authority.
An acknowledged upload is reused directly. If the previous process lost its
upload reply, a journaled upload found by HEAD is reused. Neither case rebuilds
or downloads the archive. An explicit upload failure remains a failure.
## Archive compatibility
Migration keeps the PCWS1 archive format. The file checksums required by existing
readers are computed during the single source copy, without a preliminary file
read. Their fixed-length fields are filled in the archive header after copying.
The complete archive is checksummed for storage. Existing readers, saved
snapshots, and interrupted upload journals remain compatible. Ordinary run
captures keep their existing mode. Object loading still authenticates and
decrypts the complete encrypted archive when a real restore is needed.
This deliberately trades migration readback and repeat source comparisons for
speed. It relies on the existing source/writer fences and the storage upload
acknowledgement. The original source is retained for recovery.
An uncertain commit must first be observed through the canonical inventory and
journals. An entirely remote set is an idempotent replay only when its selected
snapshots and succeeded journals prove the same complete cohort membership.
doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md
+139
View File
@@ -0,0 +1,139 @@
# Per-turn follow-ups without re-uploading the repository
Status: measured. Snapshot seed chosen; not built. It is a scoped multi-day feature awaiting a decision (see "Experiment results").
## Problem
Per-turn remote sandboxes are the default (warm reuse, #20048, is on hold).
Each follow-up turn gets a fresh Daytona sandbox, and the controller uploads the
workspace into it again.
Measured on the paperclip repository clone (Daytona, Codex, per-turn), from the
run log's `workspace_sync.*` spans (baseline `ws-daytona-phases2`):
| Span | Cold turn | Follow-up turn |
|---|---|---|
| `workspace_sync.pack` (changed files) | 4.1 s | **0.54 s** |
| `workspace_sync.transfer` | 13.6 s | **13.3 s** |
| `workspace_sync.stage_workspace` | 17.8 s | 13.8 s |
| `runner.session.startup` | 5.5 s | 9.1 s |
| time to first token | 33.0 s | 31.2 s |
The follow-up's changes pack in half a second, but the transfer barely shrinks.
Each fresh sandbox receives the whole seed: the repository archive is 220 MB
(70 MB with gzip, 14,769 files), and 70 MB in about 13 s is about 5 MB/s from
the controller host to Daytona. Earlier per-turn runs measured 13–22 s per turn
for this transfer.
The host side already avoids rebuilding the seed: the workspace seed cache
(`workspace.seed_lookup`, `seed_replay`, overlay archive, deleted-file
manifest) packs only the overlay against a cached seed generation. The cost is
re-sending that seed over the controller's link on every turn.
## Goal
A follow-up turn on a fresh per-turn sandbox sends only the overlay (the
changed files and a deleted-file manifest against a content-hashed seed
manifest), and gets the seed from somewhere already inside the provider's
network. When anything is uncertain, it falls back to today's full upload.
Expected saving: about 12–13 s of the 13.3 s transfer per follow-up turn
(overlay upload under 1 s for typical turns), taking per-turn follow-up time to
first token from about 31 s to about 18 s on this repository. The remaining
large item is `runner.session.startup` (about 9 s: runner boot plus provider
session resume), which this design does not change.
## Design
**Seed identity.** The workspace seed cache already names a seed generation.
Key the provider-side seed by `(repository, base commit, seed generation
digest)`, with a content-hashed manifest (path → sha256, mode, size) stored next
to the seed.
**Where the seed lives, one of:**
1. **A Daytona snapshot per seed (preferred if it is fast to create).** The
Daytona plugin already creates sandboxes from a named snapshot
(`config.snapshot`). Build a snapshot whose image layers the seed over the
runner image, once per seed key, and create follow-up sandboxes from it.
- **Open question:** time to build a snapshot from a 70 MB context, and
whether snapshot creation can run in the background after the first turn.
2. **A Daytona volume per seed.** Upload the seed archive to a volume once, then
mount it in each per-turn sandbox and extract inside the sandbox.
- **Open question:** volume read and extract time for 220 MB, which must stay
well under 13 s to be worth it.
**Turn flow.**
1. Look up the seed key on the provider (snapshot or volume present and
verified against its manifest digest).
2. If present: create the sandbox from the snapshot (or mount the volume and
extract), then upload only the overlay through the existing `syncIn` path
(files plus the deleted-file manifest), and verify the result with the
existing workspace sentinel and `baselineSha256`.
3. If absent, unverifiable, or the overlay exceeds a threshold (for example
20% of seed bytes, because the base moved far): do today's full upload, and
build or refresh the provider seed in the background for the next turn.
**Safety.**
- The seed is read-only, keyed by content. A mismatch in the manifest digest
falls back to the full upload, never to a partial workspace.
- The existing post-upload verification (sentinel, baseline hash) stays the
final check before `turn.start`.
- Provider seeds are cleaned up with the environment and expire by age.
- Credentials never enter the seed: it is built from the same filtered
workspace as today's upload.
## Next step: one bounded experiment
Before building anything, measure both seed locations with the existing harness
(`tests/restart-chaos/ttft-baseline.ts --env daytona --workspace-repo <paperclip clone>`):
1. Time to create a Daytona snapshot from the paperclip seed, and time to create
a sandbox from it.
2. Time to mount a volume holding the seed archive and extract it inside a
sandbox.
3. Overlay upload time for a typical follow-up (pack output 0.5 s).
Choose the location whose first-token time is lowest, then build it red/green
with a before-and-after measurement on the clone. That needs one Phase 3 harness
slot.
## Experiment results (2026-11-08)
Setup:
- Daytona `us`, 2 vCPU sandboxes, runner image `paperclip-daytona-runner@sha256:4b13c70a…`.
- The seed is a git archive of the paperclip clone: 58.7 MB gzip, 14,766 files.
- Timed with the Daytona SDK from the controller host, using a restricted key with snapshot and volume permissions.
- Every resource was deleted, and an account-wide listing confirmed none remained.
| Path | Per turn | One-time cost per seed generation |
|---|---|---|
| Direct upload (today) | create 0.97 s + upload 3.97 s + extract 1.43 s = **6.4 s** | none |
| Volume | create 0.86 s + extract from the mounted volume 5.95 s = **6.8 s** | create 6.8 s + writer sandbox 6.2 s + upload 5.3 s, about 18 s |
| Snapshot | create from snapshot 1.83 s; the seed is already in the image (copy 0.26 s; extracting it adds about 1.4 s unless it is extracted at build time) = **about 2–3.5 s** | **build 177 s** |
Findings:
- **The volume is ruled out.** Reading the archive from a mounted volume is slower than uploading it from the controller.
- **The snapshot wins.** It saves about 3–4 s per remote turn against this run's direct upload. It saves more when the controller link is slow: production follow-up transfers measured 6–16 s with zstd. That is below the 12–13 s estimated in "Goal", because direct upload was fast in this run.
- **Fork is unsupported** for these sandboxes (measured earlier the same day).
- **Transfer compression is exhausted.** The runner image already ships `zstd`, so production transfers are already compressed. #20104 adds a gzip fallback for images without zstd.
- **Snapshot builder facts:**
- It runs as uid 1001 (`daytona`).
- `Image.addLocalFile` copies an archive without extracting it.
- Build steps must write somewhere that user owns. `/opt` is not writable by it, and it cannot delete a root-owned file in `/tmp`.
## Work to build the snapshot seed
This is a multi-day feature, not a single PR:
1. **Seed snapshot lifecycle.** Build a snapshot in the background after a full upload, keyed by repository, base commit, seed digest and the resolved runner image digest. A build takes about 177 s. Never block a turn on it.
2. **Sandbox creation.** The Daytona plugin creates per-turn sandboxes from the seed snapshot, which is layered on the environment's configured runner image. A runner image update changes the key, so it needs a new build; until that build finishes, turns use the full upload on the new image. A sandbox never starts from a snapshot of an older image.
3. **Verification and fallback.** Check the manifest digest before using the seed. On any mismatch or missing snapshot, do today's full upload. Keep the workspace sentinel and `baselineSha256` checks as the final gate.
4. **Expiry and cleanup.** Expire seeds by age, delete them when their environment is deleted, and handle provider snapshot quotas.
5. **Rebuild policy.** A base-commit move needs a new 177 s build; until then the overlay grows. Rebuild when the overlay passes a threshold.
## Out of scope
- Runner and provider session startup (`runner.session.startup`, about 9 s).
Keeping the runner alive is the warm path, which stays on hold.
- Local environments; they have no upload.
doc/plans/2026-11-08-restore-source-salvage-and-warm-retirement.md
+195
View File
@@ -0,0 +1,195 @@
# Retiring kept sandboxes: salvage, then destroy
Status: proposed design, for its own reviewed task. Not implemented.
## Problem
Paperclip keeps a remote sandbox after its run in two cases:
- **Warm reuse** (`reuse_by_environment`): the next turn on the same workspace
resumes the sandbox.
- **Workspace-restore source** (`retain_on_failure`, lease and run carry
`workspaceRestoreRecovery`): after a failed run, the stopped sandbox may hold
the only copy of files that were never saved back.
Some of these sandboxes are never retired:
| Case | Why it stays | What ends it today |
|---|---|---|
| Restore source whose recovery actions were closed by a user Retry (`explicit_user_continuation`), where a later run on the task succeeded | Restore recovery only destroys after a verified repair with disposition `destroy` (`scheduleRecoveredWorkspaceCleanup`), and only for open or exhausted actions (`workspaceRecoverySourceEligible`). The retained-orphan kind treats `workspaceRestoreRecovery` as permanently held. | Environment deletion (#20085) |
| Warm sandbox, suspended, whose task is done or cancelled but still bound to the workspace | The idle timer lives in the server process. After a restart, Daytona's own auto-stop suspends it. Nothing destroys it while the task stays bound. | Environment deletion |
Both are suspended (stopped) and owned, so they leak storage cost, not compute.
In the warm Daytona chaos wave w-20261108T170423-s9501, all four real
"orphans" were closed restore sources.
Decision so far (2026-11-08): grade a stopped, owned restore source as
retained, not as a leak (#20086). Environment deletion destroys both kinds with
consent (#20085). This note covers retiring them earlier without losing work
(R2: no unsaved work is lost).
## Rule
One owned kind on the shared ownership reconciler,
`environment_kept_sandbox_retirement`, lists a kept sandbox when:
- **Restore source:**
- every recovery action for the source run is closed (resolved or cancelled,
including by explicit user continuation);
- a later run on the same task has succeeded;
- no native workspace export or preservation obligation is pending.
- **Warm sandbox:**
- the task is `done` or `cancelled`;
- the sandbox has been idle (`releasedAt`, or the last turn's end) longer
than the retention window: default 24 h, configurable per instance;
- its newest lease row is `released` or `retained`.
A new turn on the task inside the window adopts the sandbox as it does today.
Its owner is the lease's run (`runOwner`), which is terminal. Most holds from
`environment-retained-orphan-settlement.ts` still apply: shared provider
resource, stop or restore intents in flight, pending cleanup claim, open runs
on the task.
Two existing guards are deliberately replaced for this kind, and only for it:
- The unconditional hold on `workspaceRestoreRecovery` lease metadata
(`alwaysHeldSql`, `heldBy`). For a restore source it is replaced by the
closed-actions and later-success conditions above, plus the salvage receipt.
- The `resourceDisposition === "destroy"` requirement in
`scheduleRecoveredWorkspaceCleanup`. Restore sources (`retain_on_failure`)
and warm leases (`keep_running`) never record `destroy`, so retirement uses
the salvage receipt instead.
**Adoption versus retirement.** Reuse eligibility today is status-only
(`reusableLeaseCanBeResumed` accepts any `released` or `retained` lease), so a
metadata stamp cannot stop a new run from resuming a sandbox mid-salvage. At
claim time, the settle moves the lease to a non-resumable state with a
compare-and-swap on its status (the stop-only `pending_cleanup` claim shape).
A run admitted after that provisions a fresh sandbox. A run admitted before the
claim adopts the sandbox, and the claim's compare-and-swap then fails.
## Settle: salvage, then retire
Reconciler settles never call providers; they hand provider work to a sweep
with a durable claim. Retirement does the same, as a three-stage state machine
recorded on the lease (`keptSandboxRetirement`), each stage idempotent and
retried with backoff.
1. **Compare.** Claim the lease (`pendingCleanupInFlight`-style attempt id with
a controller stamp), then resume the stopped sandbox read-only: the same
path `recoverRetainedWorkspace` uses (`resumeWorkspaceRecoveryLease`), under
`withNativeWorkspaceFinalizationOwnership`.
- **Remote workspace reference:** capture the read-only manifest digest and
compare it with the committed `executionWorkspaceSnapshots.manifestDigest`
of the reference's snapshot. (`sha256` hashes the encrypted archive bytes and
can never match a manifest digest.)
- **Legacy native descriptor:** compare `captureDirectorySnapshotOffThread` +
`directorySnapshotSha256` with the descriptor's `finalHostSha256`, else its
`baselineSha256`.
- **Warm sandbox after a successful turn:** the run already recorded
`finalHostSha256` when it synced back, so a match needs no capture of new
content.
- **Equal:** record `compare: clean` and go to stage 3. **Unequal or
unknown:** go to stage 2.
- **Every exit after a resume attempt re-stops the sandbox**, clean or not,
in a `finally`. `recoverRetainedWorkspace` does the same with
`retryPendingSandboxTeardown` and `settleStopOnlyCleanup`. If the stop
cannot be confirmed, a durable stop obligation stays on the lease.
Preparing the resume clears `remoteExecutionTermination`, so stage 3 uses
the new stop receipt from this re-stop, not the old one.
2. **Export.** Capture the filtered workspace: extract `quarantineRetainedRoot`'s
capture, `omitDeniedWorkspaceFiles` and `putFile` steps from
`workspace-recovery-fallback.ts` into an exported helper.
- Store the archive under a deterministic object key derived from the lease
and the attempt.
- **Write the receipt with the attachment, in one transaction:** the asset,
the issue attachment and `artifact` work product, the comment, and a
`workspaceOperations` receipt (`kept_sandbox_salvage`, archive sha256),
as `quarantineRetainedRoot` already does. `createAttachment` always
inserts and assets are unique on `(company_id, object_key)`, so a retry
first looks up an existing receipt or asset by object key and reuses it.
It never creates a second artifact or comment.
- The re-stop happens in the single resume window's `finally` (see Fencing),
after the export, not between compare and export.
- **Exposure.** The filter (`omitDeniedWorkspaceFiles`) only denies known
secret names and directories; a credential in an ordinary file would be
included. The export is therefore restricted to the issue's existing
readers, stored with a bounded retention, and run through the run-log
secret redaction on file contents. If the design review prefers, the
first version can instead record "unsaved files exist" on the issue and
wait for a user's decision before attaching the archive.
- **First version:** export the whole filtered workspace. A file-level diff
needs new code on top of the manifest and `selectDirectorySnapshot`, and
can follow.
3. **Retire.** With `compare: clean` or a verified export receipt, hand the
lease to the pending-cleanup sweep for destruction. Use the same lease update
as `scheduleRecoveredWorkspaceCleanup` (`pending_cleanup`, policy
`ephemeral`, `workspaceRestoreRecovery` removed, stop receipt kept as
`workspaceRecoveryCleanup.stopReceipt`, so successors are never blocked),
with the salvage receipt in place of `hasVerifiedWorkspaceRepair`.
**Failure keeps the sandbox.** If the resume, capture, upload or attachment
fails, the lease stays suspended, and the stage retries with backoff up to a
bounded retention window. Past the window it is handed to the board as an
issue recovery action ("salvage failed; the sandbox is kept"). It is never
destroyed without a clean compare or a confirmed export.
## Fencing (required before implementation)
- **One claim status that the sweep skips.** The retirement claim sets the lease
to `pending_cleanup` and records `keptSandboxRetirement` with `stage` set to
`compare` or `export` and an attempt id. The pending-cleanup sweep must treat
a lease with an in-flight `keptSandboxRetirement` stage as held, exactly as
it treats `workspaceRecoveryCleanup` or an in-flight cleanup claim. Only
stage 3 clears the marker and queues the destroy. Otherwise the sweep could
destroy the sandbox while its files are still being saved.
- **Admission claims before it resumes.** Run admission must win a
compare-and-swap on the reusable lease's status before it calls the
provider's resume. A resume that loses the claim provisions a fresh sandbox
and never touches this one. This closes two overlaps: a turn that already
selected the sandbox, and a provider resume in flight when retirement claims.
Today `environmentResumeLease` runs before `acquireLease` updates the old row,
so this ordering change is part of the work. Test: a turn racing a retirement
claim leaves exactly one owner of the sandbox, and the loser never calls the
provider for it.
- **One resume lifetime.** Compare and export run inside a single resumed
window. The re-stop `finally` wraps both stages together, so the sandbox is
running for the capture and stopped on every exit (clean, dirty or failed).
## Provider I/O steps and their receipts
| Step | Provider call | Durable receipt before the next step |
|---|---|---|
| Claim | none | attempt id plus controller stamp on the lease |
| Resume read-only | `environmentResumeLease` | none needed; the step is re-done on retry |
| Capture | sandbox exec/read | manifest digest on `keptSandboxRetirement` |
| Upload | storage `putFile` | object key plus sha256 |
| Attach and comment | none (database) | `issueWorkProducts` row, comment id, `workspaceOperations` receipt |
| Re-stop | `environmentStopLease` | stop receipt (`remoteExecutionTermination.state = stopped`) |
| Destroy | `environmentDestroyLease` (pending-cleanup sweep) | `expired` with a destroyed receipt |
## Tests
- Old-saved/new-resumed: restore sources and warm leases written by the
current release, in their exact row shape.
- Clean compare → destroyed without export. Dirty compare → one artifact work
product and one comment, then destroyed. Export failure → kept and retried,
never destroyed.
- Holds: open run, open recovery action, pending export, another lease on the
sandbox, an operator owner-gone attestation.
- Concurrency: two reconciler processes settle once (one work product).
- Crash between the attachment write and the receipt: the retry reuses the
artifact by object key and records the receipt once.
- Every exit after resume re-stops the sandbox (clean compare, capture
failure, upload failure); an unconfirmed stop leaves a durable obligation.
- A turn admitted at the claim boundary: before the claim it adopts the
sandbox and the claim fails its compare-and-swap; after the claim it gets a
fresh sandbox.
- Warm: a done task within the window is adopted by a new turn; past the window
it is retired; a still-running task's warm sandbox is never listed.
## Out of scope
- Hard-deleted issues: the lease's `issue_id` is set null on delete, so no rule
can prove the task moved. The issue-delete path should hand its kept leases
to this teardown. That is a separate follow-up.
doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md
+86
View File
@@ -0,0 +1,86 @@
# Restart chaos: config and aged-state faults (CF1–CF4)
Status: CF1–CF3 are implemented in `tests/restart-chaos/config-faults.ts` and `config-wave.ts`. CF4 is designed here but not built.
## Why
Every chaos wave starts from a new instance that has default, valid settings. On
2026-11-09, production failures came from state that a new instance never has:
- **#20269.** A project's explicit setup command named `scripts/provision-worktree.sh`,
a script that had been renamed. Workspace preparation failed before the provider ran.
- **#20361.** An OpenRouter binding selected the responsible user's personal account
default, but that default did not exist. Verification retried forever, and the task
stayed Queued.
## Oracle (CF1–CF4)
"The task completes" is not the oracle. A task passes when all of these hold:
- **Bounded rest.** Within a bound, the task either completes or rests where the task
page names the broken setting.
- **No silent waiting.** No run retries unchanged configuration forever. Nothing stays
Queued without a live owner or a named wait.
- **Restart safety.** The rest survives a K1 or K2 restart.
- **Repair.** After the harness repairs the setting and uses the action the task offers
(card, Retry, or agent setup Retry), the task completes.
See the README section "Config faults" for the exact checks.
## CF4: aged instance (design)
**Goal.** Start the wave on an instance that an older release created. That release saves
tasks and settings, including CF1–CF3 breaks and parked tasks. Then switch the same
database and instance directory to the current build. Grade with the CF oracle.
**Reuse.** K14 already has the switching machinery:
- `ChaosSupervisor.create({ upgradeFrom })` starts the server from a prepared older
checkout.
- `supervisor.fault("K14")` stops that server gracefully and starts the current checkout
against the same database, port, secrets and instance home.
- The prepared checkout is made with `git worktree add <dir> <commit>` and
`node scripts/agent-preflight.mjs`, as in the K14 section of the README.
Build CF4 on these. Do not add a second upgrade path.
**Changes to `config-wave.ts`** (about half a day):
1. **New flag.** Add `--upgrade-from <dir>`. Check that the directory is prepared, the
same way `main.ts preflight` does.
2. **Seed and break on the older build.** The older build runs `seedChaosLab`, creates
each task's resources, applies the break, and launches the task. Use only routes that
both builds have. Where a route differs, use the older build's route.
3. **Barrier.** Wait until every task has finished its broken phase, so it has rested or
reached its bound. The parked state is then saved by the older build.
4. **One switch.** Fire `K14` once for the wave. This replaces each task's own K1/K2.
Re-check the rest on the current build, using the same check as after a restart.
5. **Repair on the current build.** Run repair and the offered action on the current
build, then grade.
6. **Record both commits.** Write both commits into the summary. The K14 receipt already
includes them.
**What it covers that CF1–CF3 do not:**
- Older-release run state is resumed by the current build:
- setup cards created by the older build
- lifecycle operations whose old retry policy was "retry every 60 s"
- interruptions projected by older code
- Settings written by older validators that newer validators would reject. A
`refused_on_write` setting can still exist in an aged database. CF4 is the only way to
reach that case without writing directly to the database.
**Commit choice.** Use the commit one or two releases back. Also use the commit just
before a known fix. For example, the merge base of #20361 makes the old build save a
CF2 queued-without-card state, which the fixed build must then recover.
**Cost and limits.**
- A prepared older checkout takes about 3 minutes with the shared preflight cache, plus
about 2 GB of disk.
- Waves stay at 4 tasks or fewer and a concurrency of 2 or less.
- It runs locally only, as K14 does.
**Not designed here.** Months-old data shapes, such as rows from migrations long since
squashed. A seeded fixture database taken from an older release would cover them.
Capture one only when an upgrade incident shows the need.
doc/plans/2026-11-09-durable-tool-approvals.md
+40
View File
@@ -0,0 +1,40 @@
# Durable tool approvals
Outcome: a person can approve the saved action later, after ordinary OAuth refresh
and a server restart. Paperclip executes its reviewed arguments once and returns
the result to the task. Delivery ends at a reviewed PR with passing checks.
The approval binds the action, task, company, selected authorization, tool contract,
destination, and audience. OAuth token lifetime is separate from that decision.
Authenticated refresh may preserve credential identity; manual replacement and
new consent may not. Current access and task state still gate provider dispatch.
| Acceptance | Evidence |
| --- | --- |
| Original failure reproduced | Real local OAuth HTTP exchange and two refreshes; original approval fails with `approved_tool_target_changed` before the fix |
| Approve later | Same HTTP fixture, ended requesting run, rebuilt gateway, approval after 25 hours |
| Execute once | Repeated approval returns the receipt without another provider call |
| Preserve boundaries | Changed consent, scope, identity, tool, destination, arguments, closed task, manual rotation and explicit deadlines refuse execution |
| Connection repair | Keep an accepted decision for a provably unchanged authorization during a pre-dispatch authentication wait; show repair in context and resume on connection changes |
| Upgrade compatibility | Old signatures retain strict version checks; no inferred continuity for already-rotated historical credentials |
Owners: OAuth credential storage owns authenticated refresh continuity. The vault
owns credential identity epochs. Tool approval target validation owns the saved
contract. Existing approval execution, recovery and delivery own claims, results
and continuation. Shared interaction cards own user-facing repair and deadlines.
Progress: reproduced the original 409 for both OAuth client modes before the
repair. The local OAuth HTTP journey now passes for personal and shared grants,
public and confidential clients, delayed review, ended runs, reconstructed gateway,
duplicate approval, changed scopes/consent/tool/destination, manual token replacement,
and event-driven authorization repair. The journey injects the production OAuth
refresher and proves that an invalid refresh preserves the decision before action
dispatch. It also covers equivalent scope-reporting changes, withdrawn formal
approvals during sign-in waits, and explicit deadlines on saved decisions.
Gateway service regressions,
connected-tool acceptance and vault suites passed after making their intentional
expiry fixtures explicit. UI tests, server/UI typechecks, token gates and catalog
checks passed. The production task-thread repair state was inspected in local
Storybook. Providers are simulated by the local HTTP fixture; no staging rollout
or real Notebook publication is claimed. Verification planning returned exit 2
and requires broad CI coverage. PR and review checks are next.
doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md
+86
View File
@@ -0,0 +1,86 @@
# Explicit claim tokens for native execution leases
Status: proposed follow-up (step 2). Step 1 shipped with the host-sleep lease fix.
## Problem
`native_run_finalizations.lease_expires_at` means two things today. It means
"nobody renewed this lease", and many actors also read it as "the owner is gone".
A host sleep makes the first true without the second: the wall clock and the
database clock jump past the 20-minute executor lease, while the server's
monotonic timers barely move. When the next renewal tick ran, it found its own
lease expired and interrupted a healthy turn (`native_session_lease_lost`, and the
user had to click Retry). The restart-chaos fault TM2 reproduces this.
## Step 1 (shipped)
The owner can take back its own lapsed lease in one conditional UPDATE. The UPDATE
runs under the coordinator row lock, followed by a share lock on the run. It
succeeds only when all of these hold:
- the owner token, attempt, controller generation, and controller boot, pid and
process start are unchanged;
- the phase is `observed` and no result was accepted;
- `recovery_state`, `recovery_request_id`, `failure_code` and `next_attempt_at`
are null;
- `controller_host_hash` is not the board's attested-stopped stamp;
- the run is still `running`, without the ownership-unverified error code and
without a `nativeStartupCleanup` marker;
- the run is local.
After a result is accepted, the owner may only relinquish a lapsed lease, and
never renews it. The rule lives in `nativeExecutionLeaseHeldCondition` and
`lockNativeExecutionLeaseForReacquire` (`native-execution-lease.ts`). Renewal,
failure settlement and the final release use it.
This works because every consumer that acts on a lapsed executor lease of a
running, pre-result run already writes one of those fields under the coordinator
lock first. Step 1 depends on that convention, so step 2 replaces it with an
explicit token.
## Gaps step 1 leaves open
1. **Workspace finalization** (`native-workspace-finalization-work.ts` →
`attemptNativeWorkspaceFinalization`). It copies back on a lapsed post-result
lease. It is fenced only by the copyback advisory lock and the
`nativeWorkspaceFinalizationOwner` CAS, never by a coordinator write. For this
reason a post-result lease is never renewed.
2. **Another server's orphan sweep** (`heartbeat/recovery.ts`
`markNativeOwnershipUnverified`). It writes the run's error code without the
coordinator lock. If the marker lands first, the owner is fenced. If the sweep
reads the row before the owner takes the lease back and writes afterwards,
a live run carries an ownership-unverified error. This needs two server
processes on one database, and the timing that strands a run is as rare as before.
Closed without the epoch (2026-11-10): the sweep now writes its marker under the
coordinator row lock and only while no executor lease on the run is live
(`setRunStatusFromLive` option `onlyWhileNativeLeaseLapsed`). Test:
`server/src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts`.
3. **Shared-workspace holder detection** (`remote-workspace-writer-hold.ts`
`runOwnerFenceHeldSql`). This is a read. A run silent past the stale threshold
with both leases lapsed stops counting as the holder, and nothing is claimed.
4. **Board copyback-owner confirmation on another host**
(`native-workspace-owner-recovery.ts`). It is accepted on the board's word.
5. **Remote leases.** Unchanged. The remote runner is fenced by its own
controller lease, and step 1 never takes a remote lease back.
## Step 2
Add `lease_claim_token text` (nullable) and `lease_epoch integer not null default 0`
to `native_run_finalizations` (AGENTS.md §6 migration).
- An owner claim sets `lease_epoch = lease_epoch + 1` and clears `lease_claim_token`.
- A consumer that acts on a lapsed lease first runs
`update … set lease_claim_token = $consumer, lease_epoch = lease_epoch + 1 where run_id = $1 and lease_epoch = $seen and lease_expires_at <= clock_timestamp() and lease_claim_token is null`.
Only the winner acts.
- The owner takes a lease back with
`… where lease_epoch = $mine and lease_claim_token is null`. Expiry is no
longer part of that decision.
- Move gaps 1, 2 and 4 to that claim. Workspace finalization claims before its
copyback, and the orphan sweep claims before it writes its marker. Then
post-result renewal can take the lease back as well.
- Upgrade: rows saved by step 1 have epoch 0 and a null token. A step-1 server
that is still draining never reads the new columns. Ship the consumers' claims
before the owner path relies only on the token, so that both orders stay fenced
during a rolling upgrade.
- Tests: one race test per consumer that runs its real claim against the owner
taking the lease back, and an old-saved/new-resumed row.
doc/plans/2026-11-09-task-loading-performance.md
+269
View File
@@ -0,0 +1,269 @@
# Task loading performance: latest chat in ~0 ms
Date: 2026-11-09 · Branch: `claude/task-loading-perf-154fea` (base `33a23e0246`)
## Goal
Opening a task from the "Mine" issues list should show the composer and the latest
messages almost immediately. If an agent is live, the user should catch up on its
progress. If the run is done, the user should see the last messages. Older history
loads only when the user asks for it. The page must not move content around:
skeletons must have the same size as the content that replaces them, and the latest
message must not move once it is visible.
### Finish conditions
The user-facing endpoint is reviewable PRs with these results remeasured on staging.
| # | Condition | Baseline (below) | Target |
|---|---|---|---|
| F1 | Revisit a task whose data is cached → composer and latest message visible | 144–170 ms (14.6 s when the run log gates) | ≤ 100 ms, for every task |
| F2 | Hover, then click a row → composer and latest message visible | not prefetched today | ≤ 150 ms |
| F3 | Click a row with no prefetch (data not cached) → composer and latest message visible | 1.1–2.0 s (13.9 s when the run log gates) | ≤ 600 ms p50 (one round trip) |
| F4 | Hard load of a task URL → content painted | 5.2 s | ≤ 1.5 s |
| F5 | Layout stability | CLS 0.035 on every click | 0 inside the chat; the latest message never moves after it first appears |
| F6 | API requests started before the page is ready | ~30 on a click, ~70 on a hard load | ≤ 3 on the critical path |
| F7 | Live run | — | Shows the current run status within F1–F3 and streams new output |
The evidence for each condition is a timing sample from the staging harness (Phase 0):
5 tasks × 3 starting states, run serially, reporting the median and range. The same
journey is repeated after each PR is deployed.
## Baseline on staging, measured 2026-11-09
- **Environment:** the shared staging instance, one company with production-like data, desktop Chrome, HTTP/2, no throttling.
- **JS cache:** the JS bundle was already in the browser cache.
- **Harness:** an in-page harness that checks every animation frame for the thread mount, a visible composer, the latest `[data-conversation-message]` being visible in the viewport, layout shifts, and resource timing.
### Click from the list (data not cached in React Query)
| Task | Composer + latest message | Thread mounted | API requests (total / before ready) | What it waited on |
|---|---|---|---|---|
| Task A | 1,107 ms | 38 ms | 35 / 33 | `GET /issues/:id` 1,016 ms; activity, runs, recent-comments ~710 ms each |
| Task B | 1,190 ms | 56 ms | 49 / 31 | `GET /issues/:id` 1,289 ms; ~10 parallel queries of 800–900 ms |
| Task C | 1,977 ms | 935 ms | 41 / 28 | a serial chain: work-products, activity and runs started at ~900 ms |
| Task D | 2,014 ms | 540 ms | — | — |
| **Task E** | **13,862 ms** | 40 ms | 78 / 76 | **`GET /heartbeat-runs/:id/log` 11,150 ms** |
### Revisit (React Query cache warm)
| Task | Composer + latest message | Note |
|---|---|---|
| Task D | 144 ms | |
| Task B | 170 ms | |
| **Task E** | **14,651 ms** | The run log is fetched again and gates the page again (13.2 s) |
### Hard load (direct URL, JS from cache)
| Task | Document TTFB | First contentful paint | Header paint | **Content paint** | API requests before content |
|---|---|---|---|---|---|
| Task F | 478 ms | 676 ms | 2,416 ms | **5,211 ms** | 70 |
| Task G | 4,558 ms (outlier; a later fetch of the same HTML took 207–389 ms) | 4,980 ms | 6,587 ms | 7,841 ms | 72 |
The page parses about 7.7 MB of JS, even with all of it served from the cache.
### Server latency floor
Each endpoint was requested 3 times in a row, without competing requests.
- `/api/cli-auth/me` 272–365 ms. This trivial authenticated request sets the floor for each round trip from this machine.
- `/api/issues/:id/recent-comments` 300 ms.
- `/api/issues/:id?view=core` 464–595 ms, with one sample at 3.2 s.
- `/api/issues/:id` full 863–996 ms.
- `/api/issues/:id/activity?view=recent` 520–1,061 ms.
- `/api/issues/:id/runs?view=recent` 374–1,125 ms.
During a click, ~30 requests run at the same time, and each slows to 700–1,500 ms.
**Server-Timing** samples:
- `issue_lookup` took 893–908 ms in one cold sample and ~100–160 ms in others.
- `paperclip_issue` (full view) took 736–1,813 ms.
### Run log read
Task E has a failed run with no saved reply. Its log comes from the object-store journal (`s3_segments`).
| Read | Time |
|---|---|
| Tail, 32 KB at offset 414,581, with `alignStartToRecord=true` (what the UI sends) | 11.9 s, 11.2 s |
| Same tail without alignment | 6.1 s |
| Same 32 KB at offset 0 | 2.0 s |
### Repeated and polling requests
These requests were made during one 17 s visit (Task E):
| Request | Count |
|---|---|
| `issues/summaries?ids` | 22 |
| `/issues/:id/live-runs` (two id forms) | 11 |
| `/issues/:id/browsers` | 6 |
| `companies/:id/agents` (60 KB each) | 6 |
| `companies/:id/live-runs` | 3 |
| descendant, created-from and parent lists | 2–3 each |
## Where the time goes
1. **The run transcript blocks the whole chat.**
- The composer and messages stay hidden until history is ready: `historyPending = initialHistoryPending || planLoading || transcriptHistoryPending || chatMetadataPending` (`ui/src/components/TaskChatThread.tsx:3216`).
- The composer dock is hidden until then too (`concealed={!historyRevealed}`, `TaskChatThread.tsx:3524`).
- `transcriptHistoryPending` waits for every failed run, live run, or run without a reply to finish hydrating.
- The only cap is the 15 s reveal timeout. One slow log read therefore hides even saved messages the page has already fetched.
2. **Tail reads from the object-store log are serial.**
- `readRange` in `server/src/services/object-byte-journal.ts:525` visits index nodes and payload segments one by one, with one S3 GET for each.
- A tail read crosses many small segments.
- `alignStartToRecord` repeats the whole traversal for the 32 KB before it (`object-run-log-store.ts:388`), which doubles the cost.
- Nothing caches these objects, although segments cannot change after they are written.
3. **The page waits on many queries.**
- `initialHistoryPending` waits on about ten queries: recent comments, interactions, attachments, work products, activity, runs, comment runs, comment input facts, live runs and active run.
- The thread also waits on the plan, transcripts and conversation metadata.
- Even when every query is fast, the reveal waits for the slowest of ~30 concurrent requests that compete on the server.
4. **Prefetch covers only part of what the page waits on.**
- `prefetchIssueDetailForNavigation` (`ui/src/lib/issueDetailCache.ts:227`) warms only the core view and recent comments.
- Everything else that gates the reveal still starts on click.
- Nothing is kept across reloads (no React Query persister). A hard load is therefore always cold.
5. **The full issue request waits for the core request.**
- The full `GET /issues/:id` starts only after `view=core` resolves (`IssueDetail.tsx:3361`).
6. **Non-critical work competes with the reveal.**
- chat-binding, email tasks, browsers, external-objects, feedback votes, tree holds, projects, related-issue lists and polling all start on the click and share server capacity with the requests that matter.
7. **The hard-load path is heavy.**
- It parses about 7.7 MB of JS.
- It sends 70 API requests before content appears.
- The header paints 1.5 s after navigation starts, and the content 4.3 s after.
## Feedback loop
| Question | Loop and environment | Cost per iteration |
|---|---|---|
| Is the user-visible timing better? | Staging harness (Phase 0) run through Chrome against real data, after each deploy | ~5 min per sample set, plus the deploy |
| Does a UI gate or prefetch change help? | Local `tests/perf/issue-detail` Playwright harness with production assets (`PAPERCLIP_UI_DEV_MIDDLEWARE=false`), plus a CDP network RTT of ~150 ms so local results match staging round trips | ~2–3 min |
| Is the object-store log read faster? | Server unit or bench test against a fake object provider with ~60 ms latency per GET and a fixture log with many small segments. Confirm on staging with Honeycomb traces (GET count per request) | seconds |
| Did we break layout? | Existing `layout-stability`, `loading-layout`, `compact-startup` and `saved-reply-startup` specs, plus the frame recorder in the harness | ~3 min |
Two things local runs cannot show: real multi-tenant contention and real S3 latency. Every
result therefore gets a staging check before we call it done.
## Plan: PR slices in priority order
### P0. Measurement, before any changes (small PR)
- Commit the staging click harness as `tests/perf/issue-detail/staging-click-harness.js`. It runs in the page and records, per sample:
- T_composer and T_latest
- T_thread
- layout shifts and moves of the latest message
- requests before ready, and the request that released the reveal
- Use the same harness against local runs, so every comparison uses the same measurement.
- Add `Server-Timing` to the endpoints the reveal waits on:
- `recent-comments`
- `activity`
- `runs`
- `live-runs`
- `heartbeat-runs/:id/log`, including its object GET count and total GET time
- `heartbeat-runs/:id/events`
- Add a local fixture with a failed run that has no reply and a multi-segment object log. Give the fake provider realistic per-GET latency so it reproduces Task E.
### P1. Show the composer and saved messages without waiting for transcripts (UI, biggest win)
- Remove `transcriptHistoryPending` from the reveal gate.
- Runs that are still hydrating render in a **reserved, fixed-height run slot**: a collapsed "agent activity" row with the same geometry as the loaded collapsed state. That row expands only when the user acts, so its content never pushes the latest message.
- Live runs show their status line right away and stream output as it arrives.
- Never hide the composer dock behind history.
- It is pinned to the bottom of the viewport and its height comes from the composer itself, so it can render on the first frame with the thread skeleton above it.
- Keep the draft and focus behaviour.
- Narrow `initialHistoryPending` to what defines the structure of the latest view: recent comments and interactions.
- Activity, runs, attachments and work products go into reserved slots or load after the reveal.
- Expected result on staging:
- Task E drops from 13.9 s to the time of the other tasks (~1.1–2 s).
- The tail of every other cold click gets shorter, because the slowest of ~10 queries no longer decides the reveal.
- Regression journeys:
- deep link to an older comment
- Back restore
- a draft during refetch
- failed or stalled requests
- a run that finishes while the page is open
### P2. Fix object-store log tail reads (server)
- In `readRange`, fetch the payload segments for a range in parallel, with limited concurrency (~8).
- Align to a record in **one pass**: read `[offset − 32 KB, end)` once and cut at the last newline. Do not re-traverse.
- Cache immutable segments and index nodes per process, in a bounded LRU keyed by content key and company.
- Optionally:
- Store a tail pointer or the last index node in the head, so a tail read skips the walk from the root.
- Compact small segments when the log closes.
- Target: a 32 KB tail read under 500 ms on staging, with no stalls of several seconds. The staging check is Honeycomb GET counts plus the harness.
- Also benefits: every live run viewer and the polling fallback.
### P3. Load the data before the click (target ~0 ms)
- **Prefetch everything the reveal waits on.**
- Add the P1-narrowed set to `prefetchIssueDetailForNavigation` (`issueDetailCache.ts:227`).
- Trigger it on `pointerover`, `pointerdown`, `focusin` and keyboard selection, which already exist for code and core data.
- Also trigger it on idle for the first few visible rows of "Mine".
- Respect `staleTime` and company scope.
- **Add a chat bootstrap endpoint**, `GET /issues/:id/chat-bootstrap`. It returns one company-scoped response with:
- the core issue
- the latest N timeline items, already merged: comments, interactions, run summaries and the latest status of a live run
- the composer facts: assignee, model and permissions
- a `cursor` used to catch up after the response
- This replaces ~10 gating round trips with one, so the floor becomes ~300 ms on a cold click.
- The existing granular queries are seeded from the bootstrap response, so invalidation keeps working unchanged.
- Contracts change in shared, server and UI.
- **Keep the last-seen chat window per task in IndexedDB.**
- Bounded size, scoped by company and user, cleared on logout or company change.
- On revisit or hard load, render from it on the first frame, then revalidate with `?since=cursor`.
- New items append under the bottom anchor, so the reading position does not move.
- A small "syncing" indicator sits in a reserved slot.
- Freshness is guaranteed by the catch-up read plus the websocket subscription, not by a time window.
- **Run full detail in parallel with core.** Start full detail at the same time as `view=core` instead of after it (`IssueDetail.tsx:3361`). Drop that ordering if the bootstrap endpoint makes the core view unnecessary.
### P4. Remove fan-out and polling noise
- Defer the requests the reveal does not need (chat-binding, email tasks, browsers, external-objects, feedback votes, tree holds, projects, descendant, created-from and parent lists). Start them in `requestIdleCallback` after the reveal, or when their UI is opened.
- Find out why `issues/summaries?ids` runs about every 0.5–1 s, and why `agents` (60 KB), `live-runs` and `browsers` refetch repeatedly while a task is open.
- Batch those requests.
- Remove duplicates.
- Rely on websocket invalidation instead.
- Goal: ≤ 3 requests on the critical path, and a quiet page after load.
- These changes also lower load on the staging server.
### P5. Hard-load path
After P3, a hard load can render from the persisted chat window. The remaining work:
- Profile the 1.5 s between navigation and header paint: JS parse and evaluate of 7.7 MB, and module initialisation.
- Measure the 70 requests made before content and remove the ones not needed.
- Consider inlining the chat bootstrap for the requested URL into the HTML response, so the first paint needs no extra round trip.
### P6. Layout contract
- Find the `DIV` that shifts (CLS 0.035) ~25–40 ms after every click. It is most likely the swap from the list to the detail skeleton.
- Make the skeleton's geometry match the header, thread and composer exactly.
- Add the harness checks "latest message never moves after first visible" and "composer never moves" to the existing frame-recording specs.
## Acceptance matrix (applies to every slice)
| Case | Check |
|---|---|
| Done task with a saved reply, short and long thread | Latest message and composer within target; no movement |
| Failed run with no reply (Task E shape) | Not gated by the transcript; run slot is reserved |
| Live run | Status visible right away; new output streams; no duplicate after catch-up |
| Deep link to an older comment, Back restore | Anchor kept |
| Draft typed during background refresh, navigating away and back | Draft kept; no duplicate send |
| Company switch, logout | Persisted cache isolated and cleared |
| Websocket down | Polling fallback still catches up |
## Deferred / follow-ups
- Server-side `issue_lookup` by identifier sometimes takes ~900 ms. Hand it to `fix-slow-queries` with Neon evidence if it persists after P4 reduces contention.
- The 4.5 s document TTFB outlier on one hard load. Watch for it in the harness; it may be an instance wake-up.
## State
- [x] Staging baseline captured (this doc)
- [ ] P0 harness + Server-Timing
- [ ] P1 ungate transcripts / composer
- [ ] P2 object log reads
- [ ] P3 prefetch + bootstrap + persisted window
- [ ] P4 fan-out/polling
- [ ] P5 hard load
- [ ] P6 layout contract
doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md
+28
View File
@@ -435,3 +435,31 @@ while being read. Return canonical base64, byte size, and a SHA-256 digest in a
before publishing it. Deliverables additionally verify the caller's expected
size and digest. Providers without this operation fail closed; Core never falls
back to reading remote paths on the server. Existing artifact uploads are unchanged.
## Login allocations on an existing computer
A login-capable environment driver can host authorization on a saved agent's
computer. Core checks the company, environment, agent lifecycle, manifest, and
live worker hooks before acquisition. The chosen agent is a transport target;
it does not change the signed-in account's owner or agent grants.
Core acquires this resource with `purpose: "login"`, the fixed command key, and
an absolute expiry. The returned lease must attest an expiry no later than that
deadline. Drivers must declare `creationOperationVersion: 1` and expose the live
`environmentAcquireLease`, `environmentInspectLease`, and `environmentReleaseLease`
hooks so Core can acquire, recover, and clean up the allocation.
Core does not submit a Runner task or mount project volumes.
The plugin keeps the computer binding in durable lease metadata. PTY close stops
and detaches the terminal while preserving its result. Lease release/destroy
removes only the login resources after Core commits the credential or ends the
attempt. It must preserve the computer and unrelated tasks.
The provider may assign a lease ID different from Core's creation operation ID.
Inspection by `creationOperation` must return that exact allocation as
`creationLease`, with its provider ID matching the observation envelope, so a
lost acquisition reply still reaches the existing cleanup path.
Claude uses the existing setup-token output parser. Codex uses the optional
`loginPtyReadCredential` hook to read its fixed auth artifact. Both retain the
existing owner-bound credential completion and durable cleanup. A restart
interrupts an unfinished login; it does not resume the terminal.
doc/plugins/PLUGIN_RUNNER_EXECUTION.md
+31
View File
@@ -218,6 +218,26 @@ Legacy intents without a recovery hint retain their deterministic ID. A hinted
intent without a recorded ID is ambiguous and fails closed. None of these
receipts replaces the provider's checks.
A reservation is a workspace whose `prepare` never returned a binding. When its
source task is done or cancelled, no task points at it, and no run for the task
is queued or running, Core archives it and sends `remove` with `binding: null`
and `expectedHeadSha: null` instead of reconciling it with an agent-free
`prepare`. The provider must resolve its own record for that execution
workspace, discard anything the unfinished preparation created, and return
`{ kind: "removed" }`, including when nothing was created. The removal is
persisted and replayed with the same operation ID like any other removal. A
provider that refuses it fails the removal, which backs off like any other
cleanup failure. A reservation that a task still points at, or whose task is
open, keeps the `prepare` reconciliation.
When terminal cleanup inspection or removal fails, Core records
`metadata.terminalCleanupRetry` on the workspace with the failure count, the
next attempt time, and the failure `kind`, `reason`, and Core's message. Retries
back off from 5 minutes, doubling up to 4 hours. A successful inspection or
removal clears the record, and a reopen, which raises the lifecycle generation,
discards it. Other wakes, including provider upgrades, wait for the next attempt
time. Core keeps the record through workspace metadata updates.
The optional `prepared.destinationRef` records the resolved delivery branch as
`refs/heads/<branch>`, up to 266 characters including the prefix. Providers should
return null when preparation cannot establish an unambiguous branch destination
@@ -324,6 +344,17 @@ unsettled Core tool result, other close error, or continuation of the same
session still blocks replacement.
A controller restart is not required for this stopped-owner reset path.
Core ends a dispatched task lease with `stop`, `status`, and `complete`. When
that sequence confirms the stop and completes the same task, Core does not also
call `onEnvironmentReleaseLease`. Core still calls it when the release did not
complete a dispatched task, and the cleanup retry calls it after any failure.
When `onEnvironmentResumeLease` returns no provider lease, Core sends `status`
first. It sends `stop` only when that `status` does not report
`executionStopped: true` and any required `preparationStopped: true` proof.
An old warm-start preparation marker does not force a repeated stop when both
proofs are present. A driver must release task-scoped resources in its
`status` or `complete` handling, not only in `onEnvironmentReleaseLease`.
Termination receipts cover all run leases that owned the retired physical task.
A stopped task gets a new provider task ID for checkpoint resumption. Inspection
precedes in-memory session selection so a replacement cannot inherit the stopped
doc/run-log-events.md
+141 -5
View File
@@ -5,6 +5,21 @@ Run-log events write to the `heartbeat_run_events` table
Paperclip Telemetry events, and they are not OpenTelemetry exports. A run-log
event needs no operator endpoint.
## ACPX sidecar shutdown
The ACPX sidecar emits `runtime.diagnostic` with code
`sidecar_shutdown_requested` once at the first local shutdown request, before
waiters are removed. Its JSON message records the local reason, pending tool,
input and permission counts, completion-normalization count, and at most 16
oldest pending tools. Each tool sample contains elapsed milliseconds and a
SHA-256 call ID. Hash the corresponding `runtime.tool_called` call ID to match
it. Arguments, results, operation names and raw call IDs are excluded.
The existing diagnostic path also writes this message to sidecar stderr.
A controller that has already stopped may not ingest the frame. Stdin EOF
identifies a closed controller pipe, not the upstream reason for that closure.
This is local run evidence; no first-party Telemetry event or OpenTelemetry
span is added. Retry, cancellation and authorization behavior stay unchanged.
## Startup reuse evidence
`run.startup.reuse` uses schema `paperclip.startup-reuse.v1`. Each receipt names
@@ -64,6 +79,15 @@ instance run log and add no first-party Telemetry or OpenTelemetry export.
## Provider-pack preparation timing
Every `run.performance.span` payload also carries `wallStartedAtMs`, the span's own
start time in epoch milliseconds (since 2026-11-09). Span rows can be written late
through deferred diagnostics, so readers take a span's start from this field (or
from the trace start plus `startOffsetMs`), never from the row's write time.
The preparation spans of a native run (`heartbeat.queue`, `heartbeat.prepare_*`,
`environment.*`) are always written this way: in the order they ended, while the
provider starts, and before the run's trace finishes. They can appear after the
first Runner events of the run.
`run.performance.span` also records `task.time_to_first_token` once per native
dispatch at the first newly committed deliverable text: assistant or reasoning
`item.delta`/`item.completed`, a redaction tail released at completion, or an
@@ -93,18 +117,27 @@ earlier output timing. The event contains no text or provider identifiers.
level `warn`) records one native run that exceeded its p95 speed budget. The
budgets come from the Phase 0 local baseline and are listed in
`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` ("Speed budgets").
`metric` is `ttft` (`task.time_to_first_token`), `agent_boot` (end of
`metric` is `time_to_first_activity` (`task.time_to_first_agent_activity`:
the first text delta, tool-call start or reasoning item; named `ttft` and
judged on `task.time_to_first_token` before 2026-11-08), `agent_boot` (end of
`heartbeat.queue` to the end of `runner.turn.submit` or `runner.session.resume`)
or `restart_recovery` (startup recovery `ready` to the recovered run's first
provider event). The payload names `adapter` (`codex`, `claude` or `opencode`),
or `restart_recovery_control_plane` (startup recovery `ready` to the start of the
provider's saved-session load, `provider.session.load`, or to the recovered run's
first provider event when nothing reloads). That is Paperclip's own share of a
recovery: claim, prepare, runtime stage and runner launch. The full recovery
(`ready` to the first provider event) and the provider's session load grow with
session size and with how many sessions resume at once, so they are reported in
the payload as `restartTotalMs` and `providerLoadMs` and are not judged; events
written before 2026-11-09 judged the full latency as `restart_recovery`.
The payload names `adapter` (`codex`, `claude` or `opencode`),
`environment` (`local`), `startKind` (`cold`, `warm` or `unknown`; an unknown
start uses the looser budget), `measuredMs`, `budgetP50Ms`, `budgetP95Ms`,
`percentile` (`p95`), `baseline`, and, for restart recovery, `disposition`.
For `ttft`, `measuredMs` counts from run creation: when the input waited for its
For `time_to_first_activity`, `measuredMs` counts from run creation: when the input waited for its
run to exist (for example a comment posted while the previous turn was still
running), that wait is excluded from the judgment and reported as `inputWaitMs`.
Each metric is checked at most once per dispatch. A restart-recovery dispatch is
judged only on `restart_recovery`, because its TTFT and boot span the outage. Remote runs and other adapters have
judged only on `restart_recovery_control_plane`, because its first activity and boot span the outage. Time to first text (`task.time_to_first_token`) is reported without a budget. Remote runs and other adapters have
no budget yet. The event is written in detached diagnostic work after the
measurement, so it adds no wait before `turn.start`. It is a run-log row only
and never goes to Telemetry.
@@ -477,6 +510,15 @@ server-authored event among these three types; provider
source events cannot supply stop authority. These records stay in the local run
log and do not add Telemetry or OpenTelemetry data.
The server writes `native.prior_owner_fence_refused` when its runner-state
check refuses a run before any launch, because the conversation's saved runner
state belongs to an earlier run that cannot be proven finished. The payload
holds only the closed refusal `reason` (for example
`prior_owner_terminal_state_indeterminate`). Continuation admission reads it,
with no launch, identity, `adapter.invoke` or runner-sourced event, as proof
that the failed run never had a runner, so Retry and new input need no stop
record for it.
Explicit continuation writes `native.stopped_conversation_verified` after it
verifies the retained worker and provider processes stopped. Its payload binds
the requesting `runId`, retained `ownerRunId`, native session, runner instance,
@@ -600,6 +642,64 @@ settlement record. The record does not imply exactly-once emission across
controller recovery, and failure to write it is logged without changing the
already-settled outcome.
## Storage outage waits
A run waits out a file-storage outage on its critical path: when it starts its
run log, restores its saved workspace, or saves it. While it waits, the run's live
status says so, with phase `waiting_for_storage`, for example "Waiting for file
storage before starting the run log (45s so far)". That status lives in server
memory and on the live-event stream only. It writes nothing to the run row or
the run log during the wait.
When the wait ends, the controller writes one record with `eventType:
"lifecycle"`, `stream: "system"`, and `level: "warn"`. Its message is "Waited Ns
for file storage before <step>." or, when storage stayed unavailable, "Waited Ns
for file storage before <step>; it was still unavailable." Its payload holds
`reason: "storage_outage_wait"`, `purpose` (the step), `waitedMs`, and
`recovered`. The record stays in the instance run log.
## Durable storage timing
A saved remote workspace is one encrypted archive in durable object storage.
Restoring it before a turn, and saving it after one, runs these steps as named
run phases:
| Phase | Step |
| --- | --- |
| `storage.snapshot.download` | Get the encrypted archive from storage and hash it while it streams. |
| `storage.snapshot.decrypt` | Decrypt the archive to private scratch. |
| `storage.snapshot.verify_plaintext` | Hash the decrypted archive against its sealed digest. |
| `storage.snapshot.publish` | Copy the verified archive to its destination. |
| `workspace.snapshot.sandbox_restore` | Upload and unpack the archive in the sandbox. |
| `workspace.snapshot.sandbox_upload` | Inside `sandbox_restore`: copy the archive into the sandbox. |
| `workspace.snapshot.sandbox_extract` | Inside `sandbox_restore`: verify and unpack the archive, then publish the restored folder. |
| `storage.snapshot.prior_attempts` | Check whether an earlier upload attempt already landed. |
| `storage.snapshot.encrypt` | Hash and encrypt the archive in one pass. |
| `storage.snapshot.digest` | Hash the encrypted file that is uploaded. |
| `storage.snapshot.put` | Upload the encrypted archive (multipart above 16 MiB). |
| `storage.snapshot.verify` | Confirm the stored object (its size when the backend checked every part, otherwise a full read-back). |
A restore runs inside run preparation, so each of its steps of 50 ms or more is
a `run.performance.span` row of the run. A save runs after the turn. Its step
durations are recorded on its `remote_workspace_snapshot` workspace operation,
in `metadata.saveTiming.storage`:
- `priorAttemptCheckMs`, `encryptMs`, `digestMs`, `putMs` and `verifyMs`;
- `bytes`;
- `requests`, the storage requests the save sent. These are `requests`,
`attempts` (SDK attempts across them), `failedRequests`, `retryDelayMs` (SDK
waits between its own retries), `parts`, `throttled` (part sends the backend
rate-limited with 503, 429 or SlowDown) and `backoffMs` (the uploader's waits
before resending a part).
A restore also records its step durations, `bytes` and `requests` on its
`remote_workspace_restore` workspace operation, in `metadata.restoreTiming`
(schema `paperclip.remote-workspace-restore-timing.v1`). A save that reuses an
earlier upload still reads it back; that read-back is counted in `verifyMs`.
A throttled part send counts in `throttled` even when it ends the retries,
and a multipart abort sent during cleanup counts in `requests`.
These are counts and durations only: no object keys, paths or provider text.
## System notices and final replies
Automatic comments with a stored `system_notice` presentation, including
@@ -927,3 +1027,39 @@ references in Postgres. Local-disk instances remain local. Raw provider
traces have their own restricted, encrypted object path. See
[run evidence storage](run-evidence-storage.md) for batching, durability,
configuration, migration, retention, and qualification boundaries.
### Event commit stall diagnostics
A Runner event commit stall records the first safe error code in the error and
up to three nested `cause` objects. This retains database and transport codes
when an adapter wraps the original error. Each code has the existing 80-character
identifier limit. If no safe code exists, the diagnostic uses the outer error's
safe name. It never copies exception messages, SQL, or query arguments. Cyclic or
long cause chains stop at the same four-object limit.
This only improves the recorded cause. Event acknowledgement, replay integrity,
the bounded stall policy, and operator recovery requirements stay unchanged.
A later successful continuation does not establish the cause of an earlier
failure that recorded only `Error`.
### Failure cause of generic run failures
A run that fails with a generic code (`adapter_failed`, `setup_failed`) keeps
that code, because callers compare against it, and records
`resultJson.failureCause`: one identifier of at most 80 characters
(`[A-Za-z0-9_.:-]`). For a thrown error it is the provider's HTTP status
(`provider_http_429`) or category, else the first safe code in the error and up
to three nested `cause` objects (`40001`, `ECONNRESET`, the same extraction as
the Runner's event commit stall diagnostics), else the error's class name. For
an adapter result without a code it is the error family, `signal_<name>`,
`process_cancellation_failed`, `exit_code_<n>` or `adapter_reported_error`. It
never copies message text. The server's failure log line carries the same
cause, and the board's run list shows it, for example "adapter failed (40001)".
Older rows have no cause and read as before.
Run error text keeps SQL statements off the board: a Drizzle `Failed query:`
statement and its parameters, or unwrapped statement text with `$n`
placeholders, is replaced with "Database query failed (<cause>)". The wrapper
text before the statement is kept.
docs/adapters/claude-local.md
+1 -1
View File
@@ -97,7 +97,7 @@ What the adapter does automatically:
On-call checklist if you see this in production:
- Confirm `errorCode` is `claude_poisoned_previous_message_id` in the run row — that means the guards fired correctly and the issue auto-recovers on the next heartbeat.
- If the same issue still loops after one heartbeat, check that `agentTaskSessions` for that `(agentId, taskKey)` was cleared. If not, the adapter return value was lost (e.g. a malformed run finalization) — escalate; do **not** manually edit the row, file a child issue with the run id.
- If the same issue still loops after one heartbeat, check that `agentTaskSessions` for that `(agentId, taskKey)` was cleared. If not, the adapter return value was lost (e.g. a malformed run finalization) — escalate; do **not** manually edit the row, include the run id in the escalation.
- For remote execution targets (sandbox/SSH), the poisoned JSONL is on the remote and the adapter only logs the cleanup intent. The fresh-session retry still succeeds because it uses a new session id, and the server-side `clearSession: true` is authoritative regardless of remote disk state.
## Skills Injection
docs/agents-runtime.md
+1 -1
View File
@@ -132,7 +132,7 @@ If the connection drops, the UI reconnects automatically.
1. Disable timer or set a long interval
2. Keep wake-on-assignment enabled
3. Use child issues, comments, and on-demand wakeups for handoffs instead of loops that poll agents, sessions, or processes
3. Use comments and on-demand wakeups for existing handoffs instead of loops that poll agents, sessions, or processes
## 7.3 Safety-first loop
docs/guides/agent-developer/handling-approvals.md
+1 -1
View File
@@ -55,7 +55,7 @@ For normal issue implementation plans, use the issue-thread confirmation surface
2. Create `request_confirmation` bound to the latest `plan` revision.
3. Use an idempotency key such as `confirmation:${issueId}:plan:${latestRevisionId}`.
4. Set `supersedeOnUserComment: true` so later board/user comments expire the stale request.
5. Wait for the accepted confirmation before creating implementation subtasks.
5. Wait for the accepted confirmation before implementation.
## Responding to Approval Resolutions
docs/guides/agent-developer/heartbeat-protocol.md
+5 -5
View File
@@ -68,9 +68,9 @@ Read ancestors to understand why this task exists. If woken by a specific commen
Use your tools and capabilities to complete the task. If the issue is actionable, take a concrete action in the same heartbeat. Do not stop at a plan unless the issue asked for planning.
Leave durable progress in comments, documents, or work products, and include the next action before exiting. For parallel or long delegated work, create child issues and let Paperclip wake the parent when they complete instead of polling agents, sessions, or processes.
Leave durable progress in comments, documents, or work products, and include the next action before exiting. Continue work on the current issue. For existing delegated work, let Paperclip wake the waiting task instead of polling agents, sessions, or processes.
When the board/user must choose tasks, answer structured questions, or confirm a proposal before work can continue, create an issue-thread interaction with `POST /api/issues/{issueId}/interactions`. Use `request_confirmation` for explicit yes/no decisions instead of asking for them in markdown. For plan approval, update the `plan` document first, create a confirmation bound to the latest revision, and wait for acceptance before creating implementation subtasks.
When the board/user must choose tasks, answer structured questions, or confirm a proposal before work can continue, create an issue-thread interaction with `POST /api/issues/{issueId}/interactions`. Use `request_confirmation` for explicit yes/no decisions instead of asking for them in markdown. For plan approval, update the `plan` document first, create a confirmation bound to the latest revision, and wait for acceptance before implementation.
### Step 8: Update Status
@@ -90,9 +90,9 @@ Headers: X-Paperclip-Run-Id: {runId}
{ "status": "blocked", "comment": "What is blocked, why, and who needs to unblock it." }
```
### Step 9: Delegate if Needed
### Optional Task Creation
Create subtasks for your reports:
Task creation supports an assignee and parent relationship:
```
POST /api/companies/{companyId}/issues
@@ -108,7 +108,7 @@ Always set `parentId` and `goalId` on subtasks.
- **Always comment** on in-progress work before exiting a heartbeat
- **Start actionable work** in the same heartbeat; planning-only exits are for planning tasks
- **Leave a clear next action** in durable issue context
- **Use child issues instead of polling** for long or parallel delegated work
- **Do not busy-poll** while waiting for existing delegated work
- **Use `request_confirmation`** for issue-scoped yes/no decisions and plan approval cards
- **Always set parentId** on subtasks
- **Never cancel cross-team tasks** — reassign to your manager
docs/guides/agent-developer/task-workflow.md
+2 -2
View File
@@ -52,7 +52,7 @@ Never sit silently on blocked work. Comment the blocker, update the status, and
## Delegation Pattern
Managers break down work into subtasks:
If work is delegated, task creation supports a parent relationship:
```
POST /api/companies/{companyId}/issues
@@ -99,7 +99,7 @@ When a plan needs approval before implementation:
2. Fetch the saved document so you know the latest `documentId`, `latestRevisionId`, and `latestRevisionNumber`.
3. Create a `request_confirmation` targeting that exact `plan` revision.
4. Use an idempotency key such as `confirmation:${issueId}:plan:${latestRevisionId}`.
5. Wait for acceptance before creating implementation subtasks.
5. Wait for acceptance before implementation.
6. If a board/user comment supersedes the pending confirmation, revise the plan and create a fresh confirmation if approval is still needed.
Plan approval targets look like this:
docs/guides/board-operator/creating-a-company.md
+1 -1
View File
@@ -31,7 +31,7 @@ The CEO is the first agent you create. Choose an adapter type (Claude Code is a
- **Prompt template** — instructions for what the CEO does on each heartbeat
- **Budget** — monthly spend limit in cents
The CEO's prompt should instruct it to review company health, set strategy, and delegate work to reports.
The CEO's prompt should instruct it to review company health, set strategy, and carry out assigned work.
## Step 4: Build the Org Chart
docs/guides/board-operator/delegation.md
+15 -15
View File
@@ -1,13 +1,13 @@
---
title: How Delegation Works
summary: How the CEO breaks down goals into tasks and assigns them to agents
summary: How the CEO assigns work when delegation is part of the requested scope
---
Delegation is one of Paperclip's most powerful features. You set company goals, and the CEO agent automatically breaks them into tasks and assigns them to the right agents. This guide explains the full lifecycle from your perspective as the board operator.
Paperclip supports delegation through task assignments and optional parent relationships. Work can stay on the current task. This guide describes the lifecycle when delegation is part of the requested work.
## The Delegation Lifecycle
When you create a company goal, the CEO doesn't just acknowledge it — it builds a plan and mobilizes the team:
For a goal whose approved scope includes delegation:
```
You set a company goal
@@ -21,7 +21,7 @@ You set a company goal
→ You see results in the dashboard and activity log
```
Each step is traceable. Every task links back to the goal through a parent hierarchy, so you can always see why work is happening.
Tasks that use parent relationships can trace back to the goal through that hierarchy, so you can see why work is happening.
## What You Need to Do
@@ -40,16 +40,16 @@ Your role is strategic oversight, not task management. Here's what the delegatio
- Is an agent paused or in an error state?
- Is the CEO's budget exhausted (above 80%, it focuses on critical tasks only)?
## What the CEO Does Automatically
## What an Approved Delegation Can Include
You do **not** need to tell the CEO to engage specific agents. After you approve its strategy, the CEO:
When your request or approved plan assigns work to a team, the CEO can:
- **Breaks goals into concrete tasks** with clear descriptions, priorities, and acceptance criteria
- **Assigns tasks to the right agent** based on role and capabilities (e.g., engineering tasks go to the CTO or engineers, marketing tasks go to the CMO)
- **Creates subtasks** when work needs to be decomposed further
- **Hires new agents** when the team lacks capacity for a goal, with hire approvals available when enabled in company settings
- **Monitors progress** on each heartbeat, checking task status and unblocking reports
- **Escalates to you** when it encounters something it can't resolve — budget issues, blocked approvals, or strategic ambiguity
- **Break goals into concrete tasks** with clear descriptions, priorities, and acceptance criteria
- **Assign tasks to the right agent** based on role and capabilities (e.g., engineering tasks go to the CTO or engineers, marketing tasks go to the CMO)
- **Use parent relationships** when included in the requested task structure
- **Hire new agents** when the team lacks capacity for a goal, with hire approvals available when enabled in company settings
- **Monitor progress** on each heartbeat, checking task status and unblocking reports
- **Escalate to you** when it encounters something it can't resolve — budget issues, blocked approvals, or strategic ambiguity
## Common Delegation Patterns
@@ -79,7 +79,7 @@ CEO
└── Content Writer
```
The CEO assigns high-level tasks to the CTO and CMO. They break those into subtasks and assign them to their own reports. You only interact with the CEO — the rest happens automatically.
The CEO assigns high-level tasks to the CTO and CMO. They break those into subtasks and assign them to their own reports. This is an optional structure, not a requirement for each request.
### Hire-on-Demand
@@ -106,11 +106,11 @@ If you've set a goal but nothing is happening, check these common causes:
| **Budget** | If the CEO is above 80% of its monthly budget, it focuses only on critical tasks and may skip lower-priority delegation. |
| **Goals** | If no company goals are set, the CEO has nothing to work from. Create a goal first. |
| **Heartbeat** | Is the CEO's heartbeat enabled and running? Check the agent detail page for recent heartbeat history. |
| **Agent instructions** | The CEO's delegation behavior is driven by its `AGENTS.md` instructions file. Open the CEO agent's detail page and verify that its instructions path is set and that the file includes delegation directives (subtask creation, hiring, assignment). If AGENTS.md is missing or doesn't mention delegation, the CEO won't know to break down goals and assign work. |
| **Agent instructions** | Check that its `AGENTS.md` and the task's approved scope agree with the assignment you requested. A goal or plan approval alone does not require subtasks. |
### "Do I have to tell the CEO to engage engineering and marketing?"
**No.** The CEO will delegate automatically after you approve its strategy. It knows the org chart and assigns tasks based on each agent's role and capabilities. You set the goal and approve the plan — the CEO handles task breakdown and assignment.
Specify the assignments you want in your request or approved plan. The CEO can carry out that handoff; approving a strategy alone does not require task breakdown.
### "A task seems stuck"
docs/guides/board-operator/org-structure.md
+2 -2
View File
@@ -10,7 +10,7 @@ Paperclip enforces a strict organizational hierarchy. Every agent reports to exa
- The **CEO** has no manager (reports to the board/human operator)
- Every other agent has a `reportsTo` field pointing to their manager
- You can change an agent’s manager after creation from **Agent → Configuration → Reports to** (or via `PATCH /api/agents/{id}` with `reportsTo`)
- Managers can create subtasks and delegate to their reports
- Managers can assign work to their reports
- Agents escalate blockers up the chain of command
## Viewing the Org Chart
@@ -28,7 +28,7 @@ GET /api/companies/{companyId}/org
Every agent has access to their `chainOfCommand` — the list of managers from their direct report up to the CEO. This is used for:
- **Escalation** — when an agent is blocked, they can reassign to their manager
- **Delegation** — managers create subtasks for their reports
- **Delegation** — managers can assign work to their reports
- **Visibility** — managers can see what their reports are working on
## Rules
docs/start/core-concepts.md
+6 -6
View File
@@ -52,14 +52,14 @@ The transition to `in_progress` requires an **atomic checkout** — only one age
## Delegation
The CEO is the primary delegator. When you set company goals, the CEO:
When your request includes delegation, the CEO can:
1. Creates a strategy and submits it for your approval
2. Breaks approved goals into tasks
3. Assigns tasks to agents based on their role and capabilities
4. Hires new agents when needed, with hire approvals available when you enable them
1. Create a strategy and submit it for your approval
2. Break approved goals into tasks
3. Assign tasks to agents based on their role and capabilities
4. Hire new agents when needed, with hire approvals available when you enable them
You don't need to manually assign every task — set the goals and let the CEO organize the work. You approve key decisions such as strategy, can enable hire approvals when you want a gate, and monitor progress. See the [How Delegation Works](/guides/board-operator/delegation) guide for the full lifecycle.
When your request or approved plan includes delegation, the CEO can organize the work. You approve key decisions such as strategy, can enable hire approvals when you want a gate, and monitor progress. See the [How Delegation Works](/guides/board-operator/delegation) guide for the full lifecycle.
## Heartbeats
evals/promptfoo/prompts/heartbeat-system.txt
+2 -2
View File
@@ -24,12 +24,12 @@ The Heartbeat Procedure:
6. Understand context: GET /api/issues/{issueId}/heartbeat-context
7. Do the work
8. Update status: PATCH /api/issues/{issueId} with status and comment
9. Delegate if needed: POST /api/companies/{companyId}/issues
9. Optional task creation: POST /api/companies/{companyId}/issues
Execution Contract:
- If the issue is actionable, start concrete work in this heartbeat. Do not stop at a plan unless the issue asks for planning.
- Leave durable progress in comments, documents, or work products, with a clear next action.
- Use child issues for parallel or long delegated work instead of polling agents, sessions, or processes.
- Continue work on the current issue. Do not busy-poll agents, sessions, or processes.
- If blocked, PATCH the issue to blocked and name the unblock owner and action.
- Respect budget, pause/cancel, approval gates, and company boundaries.
maps/agent-access-map/agents.md
+9 -1
View File
@@ -63,6 +63,12 @@ bypass hiring authorization. Source: [runner resolver](../../server/src/services
Policy: Agent creation/configuration uses action grants, resource policies and change
consent; some controls require a board actor.
`GET /api/agents/{id}/interaction-permissions` follows configuration-read access.
`PATCH /api/agents/{id}/interaction-permissions` is restricted to board-authenticated
human permission managers (`users:manage_permissions`); agents cannot change it.
Optional [interaction restrictions](../../doc/agent-interaction-permissions.md)
can independently deny peer configuration despite a configure/suggest grant.
`agents:read` permits redacted configuration, configured skills, instruction content,
and configuration/instruction history reads. It does not permit changes, restore,
or environment probes. `agents:configure` still includes read/write authority;
@@ -83,7 +89,7 @@ An unfinished record keeps compatibility status `paused` with a
`setup` hold and cannot receive API keys or authenticate with keys or JWTs.
Its standard grants apply at hire. Authorized completion retries recover the
same record and approval without changing the submitted configuration.
Normal agent-authored hiring keeps its existing approval policy.
Normal agent-authored hiring keeps its existing approval policy. A pending hire may propose a budget, but only board approval activates that policy. Direct creation or an immediate hire with a positive budget requires a board actor.
An agent-owned plugin may prepare the draft and leave its lifecycle in
`verifying` while that setup hold remains. Only bounded Core-issued setup probes
@@ -212,6 +218,8 @@ CLI.
Policy: Agent creation/configuration uses action grants, resource policies and change
consent; some controls require a board actor.
Changes to the legacy `budgetMonthlyCents` field and built-in provision budget overrides require a board actor, including budget-changing rollbacks.
The roster intent preserves unrelated settings, rejects stale model/connection
previews and harness changes, and checks the full execution configuration under
the lifecycle write lock. It uses the same configuration, company, credential,
maps/agent-access-map/documents-artifacts.md
+6
View File
@@ -5,6 +5,12 @@ products](../feature-map/documents-artifacts.md).
Feature ID: `documents-artifacts`.
[Agent interaction restrictions](../../doc/agent-interaction-permissions.md)
apply to task document, attachment and work-product mutations, and to annotation
comments. Newly submitted or changed structured mentions are checked separately.
Initial content on a new task uses assignment authority. Existing read access
and historical content remain unchanged.
Review date: 2026-10-12. Source commit: `f981fa3cc10cb8663a87dec60345fbff46d54d0a`.
This is a source-reviewed baseline of representative operations and known gaps.
maps/agent-access-map/recovery.md
+11
View File
@@ -100,6 +100,17 @@ Gaps: Component host test doesn't prove actual filesystem/provider repair.
## `bounded-continuation`
Original-host Native controller containment requires board-only
`runtime:confirm_owner_stop` authority on the existing recovery resolution route:
a company owner, company administrator, or instance administrator. Agents cannot submit it.
It requires separate controller/provider stop confirmations, reviewed outcomes,
a server-bound exact authority digest, elapsed lease and unchanged locked
source/assignee/cleanup fences. It records containment without queuing work; a
separate ordinary task Retry is required. Unknown process space never authorizes
a local PID probe or automatic physical takeover, even after expiry.
Evidence: `source-only` — candidate authority, route, UI and real-owner controls;
no deployed/original-host stop verification is claimed.
Capability ID: `recovery/bounded-continuation`.
Ordinary task Retry uses `POST /api/issues/{id}/retry`. This is authenticated
maps/agent-access-map/steering.md
+6
View File
@@ -4,6 +4,12 @@ Product map: [Steering and queued messages](../feature-map/steering.md).
Feature ID: `steering`.
Pending agent-authored input is rechecked against current
[interaction restrictions](../../doc/agent-interaction-permissions.md) per item,
including when a human steers it. A denied item receives a permanent refusal;
other allowed items retain their own authority. History and already running
execution remain intact. Restoring a setting does not replay refused input.
Review date: 2026-10-12. Source commit: `58e16dbddfc9c8d0cf60867f9d9847f0feda055b`.
Native REST transport reconciled in this change; see the
[shared transport contract](README.md#native-rest-transport).
maps/agent-access-map/tasks.md
+6
View File
@@ -4,6 +4,12 @@ Product map: [Task creation and lifecycle](../feature-map/tasks.md).
Feature ID: `tasks`.
Optional [agent interaction restrictions](../../doc/agent-interaction-permissions.md)
add sender and recipient checks to assignments, comments, task mutations and
structured mentions. Current assignment establishes ownership. Self-claim and
reassignment also check the previous owner; child placement checks the parent.
These restrictions add no grants and leave absent settings unrestricted.
Source revision: `3bbc2104b65caf9f7763938a01260a5673e12f7a`.
Review depth and executed proof are separate. The operation records enumerate the named behavior and access boundaries. Their evidence states describe the reviewed sources; run results and environment limits belong in the PR or verification artifacts. A listed API or source test does not establish a live authorized journey. Follow the [map contract](../README.md).
maps/feature-map/recovery.md
+8
View File
@@ -64,6 +64,14 @@ and the [disconnect recovery journey](../../tests/e2e/disconnect-write-recovery.
### `source-task`
A Native run held for unknown controller process space remains blocked even
after its lease expires. A company owner, company administrator, or instance
administrator can record separate original-host controller and provider/tool stop confirmations with reviewed
action outcomes through the existing task recovery card. Recording containment
does not replay work. After normal settlement/cleanup, use a separate ordinary
Retry; stale authority or unresolved cleanup must keep it blocked. This candidate
path is source-only pending its focused real-owner qualification.
Open the affected task from Tasks, Inbox/Blocked, search, or a direct task link.
Observe one working, backoff, failed or waiting-for-user state. Use Retry after
an infrastructure failure. No process or file certification is required.
package.json
+1 -1
View File
@@ -61,7 +61,7 @@
"smoke:notion-generic-live": "node scripts/smoke/notion-generic-live.mjs",
"smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
"smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
"test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/ci/retry-reporter.test.mjs scripts/__tests__/runner-vitest-package-lanes.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/public-install-registry.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs scripts/__tests__/agent-build-cache.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs scripts/typecheck-test-programs.test.mjs scripts/__tests__/local-verify.test.mjs scripts/test-sandbox-providers.test.mjs",
"test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/ci/retry-reporter.test.mjs scripts/__tests__/runner-vitest-package-lanes.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/public-install-registry.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs scripts/__tests__/agent-build-cache.test.mjs scripts/__tests__/agent-preflight.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs scripts/typecheck-test-programs.test.mjs scripts/__tests__/local-verify.test.mjs scripts/test-sandbox-providers.test.mjs",
"storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
"test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
"test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",
packages/adapter-utils/src/acpx-engine/execute.test.ts
+1 -1
View File
@@ -971,7 +971,7 @@ describe("shared ACPX engine runtime behavior", () => {
expect(String(reset.meta[0]?.prompt)).toContain(custom ? "Custom agent instructions." : "You are agent agent-1");
const ordinary = await runExecutor({ ...config, promptTemplate: "" }, { context: { ...context, conversationMode: false } });
expect(String(ordinary.meta[0]?.prompt)).not.toContain("Execution contract:");
expect(String(ordinary.meta[0]?.prompt)).toContain("Create child issues from the approved plan");
expect(String(ordinary.meta[0]?.prompt)).toContain("Implement the accepted plan on this issue.");
});
it("uses only the guarded external-chat contract for a default ACPX prompt", async () => {
packages/adapter-utils/src/github-launcher.test.ts
+36
View File
@@ -189,6 +189,42 @@ process.exit(result.status ?? 98);
expect(result.stderr).toContain("More than one managed GitHub identity matches this run");
expect(result.stderr).not.toMatch(/host-token|must-not-be-used|run-capability/);
});
it("sends repository creation and forks to the connection's tools when a connection is in use", async () => {
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-github-repository-tools-"));
cleanups.push(() => rm(root, { recursive: true, force: true }));
const bin = path.join(root, "managed"), realBin = path.join(root, "real");
await mkdir(bin); await mkdir(realBin);
await writeFile(path.join(bin, "gh"), githubLauncherSource(), { mode: 0o700 });
await writeFile(path.join(realBin, "gh"), '#!/usr/bin/env node\nprocess.stdout.write(JSON.stringify({args:process.argv.slice(2),token:process.env.GH_TOKEN ?? null}));', { mode: 0o700 });
let status = "available";
const server = createServer((_req, res) => {
res.setHeader("content-type", "application/json");
res.end(JSON.stringify({ status, env: { GH_TOKEN: "managed-token" } }));
});
await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
cleanups.push(() => new Promise<void>(resolve => server.close(() => resolve())));
const { port } = server.address() as { port: number };
const env = { ...process.env, ...githubBrokerEnvironment({}, { url: `http://127.0.0.1:${port}`, token: "run-capability" }),
PATH: `${bin}:${realBin}:${process.env.PATH}` };
const gh = (...args: string[]) => exec(path.join(bin, "gh"), args, { env });
for (const [args, tool] of [
[["repo", "create", "demo", "--private"], "create_repository"],
[["repo", "new", "demo"], "create_repository"],
[["repo", "fork", "octo-org/tools", "--org", "acme"], "fork_repository"],
] as const) {
const failure = await gh(...args).catch((error: { code?: number; stdout?: string; stderr?: string }) => error);
expect(failure).toMatchObject({ code: 1, stdout: "" });
expect(failure.stderr).toContain(`GitHub connection's ${tool} tool`);
expect(failure.stderr).not.toContain("managed-token");
}
expect(JSON.parse((await gh("repo", "create", "--help")).stdout))
.toEqual({ args: ["repo", "create", "--help"], token: "managed-token" });
expect(JSON.parse((await gh("repo", "view", "octo-org/tools")).stdout))
.toEqual({ args: ["repo", "view", "octo-org/tools"], token: "managed-token" });
// Without a connection there are no connection tools; gh runs unchanged.
status = "absent";
expect(JSON.parse((await gh("repo", "create", "demo")).stdout)).toEqual({ args: ["repo", "create", "demo"], token: null });
});
it.each(["before-write", "partial-write"])("keeps Git and managed credentials available after a %s failure", async (failure) => {
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-github-default-write-failure-"));
cleanups.push(() => rm(root, { recursive: true, force: true }));
packages/adapter-utils/src/github-launcher.ts
+17
View File
@@ -43,6 +43,13 @@ function isLocalGit(args) {
}
return false;
}
// With a Paperclip GitHub connection, repositories are created and forked
// through its tools so access rules, approvals, owner defaults and audit apply.
const connectorRepositoryTools = new Map([['create', 'create_repository'], ['new', 'create_repository'], ['fork', 'fork_repository']]);
function connectorRepositoryTool(args) {
if (program !== 'gh' || args[0] !== 'repo' || args.includes('--help') || args.includes('-h')) return null;
return connectorRepositoryTools.get(args[1]) || null;
}
async function main() {
let env = { ...process.env };
${options.brokerFile ? String.raw`
@@ -60,6 +67,7 @@ ${options.brokerFile ? String.raw`
} catch {}
` : ''}
const diagnostic = (code) => process.stderr.write('Paperclip: GitHub ' + code + '; continuing without managed credentials.\n');
let connected = false;
const configRoot = env.GH_CONFIG_DIR || os.tmpdir();
// A missing/unwritable scratch directory must not break local Git. The
// fallback deliberately cannot load the host's gh authentication files.
@@ -131,6 +139,7 @@ ${options.brokerFile ? String.raw`
: 'Check the GitHub connection in Paperclip';
process.stderr.write('Paperclip: GitHub access unavailable: ' + reason + '. Continuing without GitHub credentials.\n');
}
connected = result.status === 'available';
if (result.status === 'available' && configReady) {
for (const [key, value] of Object.entries(result.env || {})) {
if (/^(GH_TOKEN|GITHUB_TOKEN|PAPERCLIP_GIT_TOKEN|GIT_TERMINAL_PROMPT|GIT_AUTHOR_(NAME|EMAIL)|GIT_COMMITTER_(NAME|EMAIL)|GIT_CONFIG_COUNT|GIT_CONFIG_(KEY|VALUE)_\d+)$/.test(key) && typeof value === 'string') env[key] = value;
@@ -140,6 +149,14 @@ ${options.brokerFile ? String.raw`
} else if (askBroker) { diagnostic('capability_missing'); }
} catch { diagnostic('broker_transport_unavailable'); }
}
const connectorTool = connected ? connectorRepositoryTool(process.argv.slice(2)) : null;
if (connectorTool) {
process.stderr.write("Paperclip: use the GitHub connection's " + connectorTool + " tool instead of gh repo " + process.argv[3] +
", so the connection's access rules apply. " + (connectorTool === 'fork_repository'
? 'Then clone the fork or add it as a git remote.' : 'Then add the repository as a git remote and push.') + '\n');
process.exitCode = 1;
return;
}
// Only this invocation and its children inherit the captured credential.
env.PAPERCLIP_GITHUB_OPERATION_ACTIVE = '1';
// Its Git children use the real binary, so steering cannot split a gh operation.
packages/adapter-utils/src/server-utils.test.ts
+35 -7
View File
@@ -1109,7 +1109,7 @@ describe("renderPaperclipWakePrompt", () => {
};
const ordinary = renderPaperclipWakePrompt(payload, { resumedSession: true });
expect(ordinary).not.toContain("Execution contract:");
expect(ordinary).toContain("Create child issues from the approved plan");
expect(ordinary).toContain("Implement the accepted plan on this issue.");
for (const resumedSession of [false, true]) {
const chat = renderPaperclipWakePrompt(payload, {
resumedSession, conversationMode: true, includeExecutionContract: true,
@@ -1117,7 +1117,7 @@ describe("renderPaperclipWakePrompt", () => {
expect(chat).not.toContain("Execution contract:");
expect(chat).not.toContain("clear final disposition");
expect(chat).not.toContain("Create child issues");
expect(chat).not.toContain("you may create child implementation issues");
expect(chat).not.toContain("Implement the accepted plan on this issue.");
}
});
@@ -2552,7 +2552,7 @@ describe("renderPaperclipWakePrompt", () => {
);
expect(prompt).not.toContain("accepted-plan continuation");
expect(prompt).not.toContain(
"Create child issues from the approved plan only",
"Implement the accepted plan on this issue.",
);
});
@@ -2574,10 +2574,10 @@ describe("renderPaperclipWakePrompt", () => {
});
expect(prompt).toContain("accepted-plan continuation");
expect(prompt).toContain("Create child issues from the approved plan only");
expect(prompt).toContain("may create child implementation issues");
expect(prompt).toContain("Implement the accepted plan on this issue.");
expect(prompt).toContain("Plan acceptance alone does not require subtasks.");
expect(prompt).toContain(
"must not start implementation work on the planning issue itself",
"continue the approved work on this issue unless the approved plan explicitly assigns it elsewhere",
);
});
@@ -2920,7 +2920,7 @@ describe("renderPaperclipWakePrompt", () => {
expect(prompt).toContain("selected text: Passed in Chrome");
expect(prompt).toContain("Attach the run id.");
expect(prompt).toContain(
"propose a child issue before making code changes",
"obtain authorization before making code changes",
);
expect(prompt).toContain(
"prefer replying and resolving the thread over rewriting the snapshot",
@@ -3008,6 +3008,34 @@ describe("renderPaperclipWakePrompt", () => {
expect(JSON.parse(stringifyPaperclipWakePayload(payload)!)).toMatchObject({ dispositionRepair: payload.dispositionRepair });
});
it("tells the agent what it can do when a child or blocker stalls on the board", () => {
const payload = {
reason: "issue_blocker_stalled",
issue: { id: "parent-1", identifier: "CHA-5", title: "Coordinate the rollout", status: "blocked", priority: "medium" },
blockerStall: {
stalledIssueId: "child-1",
stalledIssueIdentifier: "CHA-8",
stalledIssueTitle: "Ship the migration",
stalledIssueStatus: "blocked",
relation: "child",
recoveryAction: { id: "action-1", kind: "run_recovery", cause: "execution_recovery_budget_exhausted",
status: "escalated", ownerType: "board", nextAction: "Retry the run or reassign the task" },
},
};
expect(JSON.parse(stringifyPaperclipWakePayload(payload) ?? "{}")).toMatchObject({
blockerStall: { stalledIssueIdentifier: "CHA-8", relation: "child",
recoveryAction: { cause: "execution_recovery_budget_exhausted", nextAction: "Retry the run or reassign the task" } },
});
const prompt = renderPaperclipWakePrompt(payload);
expect(prompt).toContain("## Stalled dependency");
expect(prompt).toContain("CHA-8 Ship the migration (blocked)");
expect(prompt).toContain("execution_recovery_budget_exhausted");
expect(prompt).toContain("Retry the run or reassign the task");
expect(prompt).toContain("reassign its existing task");
expect(prompt).toMatch(/escalate/i);
expect(prompt).toMatch(/block this issue/i);
});
it("includes continuation and child issue summaries in structured wake context", () => {
const payload = {
reason: "issue_children_completed",
packages/adapter-utils/src/server-utils.ts
+68 -4
View File
@@ -229,7 +229,7 @@ export const WATCHDOG_DEFAULT_MANDATE = [
"- Treat every terminal, cancelled, blocked, in-review, or otherwise stopped leaf in the watched subtree as a claim that must be verified against comments, documents, work products, screenshots, tests, blockers, and review state.",
'- Do not accept "I could not" or "waiting for approval" as automatically valid. Read the evidence before deciding.',
"- If a stopped leaf is genuinely complete, leave it alone and record why you believe so.",
"- If a stopped leaf is not genuinely complete, restore a live path inside the watched subtree by reopening, reassigning, commenting actionable instructions, creating a follow-up child issue, or accepting an eligible task-level interaction (such as a routine plan confirmation when no custom instruction forbids it).",
"- If a stopped leaf is not genuinely complete, restore a live path inside the watched subtree by reopening, reassigning, commenting actionable instructions, or accepting an eligible task-level interaction (such as a routine plan confirmation when no custom instruction forbids it).",
"- If you discover a Paperclip product or platform bug while reviewing the stopped subtree, create a linked engineering follow-up outside the watched source tree using the server-provided watchdog discovery route instead of making it a source child.",
"- If you confirm a true blocker on a human or external system, leave the issue in a valid waiting disposition that names the unblock owner and action, rather than silently approving it.",
"",
@@ -704,6 +704,22 @@ type PaperclipWakeChildIssueSummary = {
summary: string | null;
};
/** `issue_blocker_stalled`: the child or blocker that stalled on a board-owned recovery action. */
type PaperclipWakeBlockerStall = {
stalledIssueId: string | null;
stalledIssueIdentifier: string | null;
stalledIssueTitle: string | null;
stalledIssueStatus: string | null;
relation: "child" | "blocker";
recoveryAction: {
kind: string | null;
cause: string | null;
status: string | null;
ownerType: string | null;
nextAction: string | null;
};
};
type PaperclipWakeBlockerSummary = {
id: string | null;
identifier: string | null;
@@ -817,6 +833,7 @@ type PaperclipWakePayload = {
annotationDeltas: PaperclipWakeAnnotationDelta[];
childIssueSummaries: PaperclipWakeChildIssueSummary[];
childIssueSummaryTruncated: boolean;
blockerStall: PaperclipWakeBlockerStall | null;
commentIds: string[];
latestCommentId: string | null;
comments: PaperclipWakeComment[];
@@ -1322,6 +1339,29 @@ function normalizePaperclipWakeChildIssueSummary(
return { id, identifier, title, status, priority, summary };
}
function normalizePaperclipWakeBlockerStall(value: unknown): PaperclipWakeBlockerStall | null {
const stall = parseObject(value);
const text = (field: unknown) => asString(field, "").trim() || null;
const stalledIssueId = text(stall.stalledIssueId);
const stalledIssueIdentifier = text(stall.stalledIssueIdentifier);
if (!stalledIssueId && !stalledIssueIdentifier) return null;
const action = parseObject(stall.recoveryAction);
return {
stalledIssueId,
stalledIssueIdentifier,
stalledIssueTitle: text(stall.stalledIssueTitle),
stalledIssueStatus: text(stall.stalledIssueStatus),
relation: stall.relation === "blocker" ? "blocker" : "child",
recoveryAction: {
kind: text(action.kind),
cause: text(action.cause),
status: text(action.status),
ownerType: text(action.ownerType),
nextAction: text(action.nextAction),
},
};
}
function normalizePaperclipWakeBlockerSummary(
value: unknown,
): PaperclipWakeBlockerSummary | null {
@@ -1895,6 +1935,7 @@ export function normalizePaperclipWakePayload(
payload.childIssueSummaryTruncated,
false,
),
blockerStall: normalizePaperclipWakeBlockerStall(payload.blockerStall),
commentIds,
latestCommentId: asString(payload.latestCommentId, "").trim() || null,
comments,
@@ -2639,14 +2680,14 @@ function renderPaperclipWakePromptBody(
if (acceptedPlanContinuation) {
directive = acceptedPlanWithMissingWakeComment
? "Continue the accepted-plan review only. Do not write code or perform implementation work on the planning issue."
: "Create child issues from the approved plan only. Do not write code or perform implementation work on the planning issue.";
: "Implement the accepted plan on this issue. Follow any task assignments explicitly included in the approved plan. Plan acceptance alone does not require subtasks.";
}
lines.push(`- planning directive: ${directive}`);
if (acceptedPlanContinuation) {
lines.push(
acceptedPlanWithMissingWakeComment
? "- accepted-plan continuation: fetch and reconcile the missing wake comment; do not create a child merely because a plan was accepted"
: "- accepted-plan continuation: you may create child implementation issues from the approved plan, but must not start implementation work on the planning issue itself",
: "- accepted-plan continuation: continue the approved work on this issue unless the approved plan explicitly assigns it elsewhere",
);
}
}
@@ -2857,7 +2898,7 @@ function renderPaperclipWakePromptBody(
"## Open document annotations",
"",
"These open annotations are grouped by issue document. Resolved annotations were intentionally omitted.",
"Scope: a document annotation authorizes document edits and thread replies only; propose a child issue before making code changes.",
"Scope: a document annotation authorizes document edits and thread replies only; obtain authorization before making code changes.",
"For snapshot documents such as QA evidence and run summaries, prefer replying and resolving the thread over rewriting the snapshot.",
`- open annotation threads included: ${context.totals.includedThreadCount}/${context.totals.openThreadCount}`,
`- annotation comments included: ${context.totals.includedCommentCount}/${context.totals.commentCount}`,
@@ -2942,6 +2983,9 @@ function renderPaperclipWakePromptBody(
`You are waking as the active ${executionStage.wakeRole} for this issue.`,
"Do not execute the task itself or continue executor work.",
"Review the issue and choose one of the allowed actions above.",
"Record your decision on this issue with PATCH /api/issues/{issueId} and a required decision comment: approve with {\"status\":\"done\",\"comment\":\"...\"}, or request changes with {\"status\":\"in_progress\",\"comment\":\"...\"}.",
"A Paperclip Runner run cannot hand the task back itself; to request changes from one, report the required changes as a blocker with paperclip_block.",
"A run completion report does not record the decision; record it before you report.",
"If you request changes, the workflow routes back to the stored return assignee.",
"",
);
@@ -3067,6 +3111,26 @@ function renderPaperclipWakePromptBody(
}
}
if (normalized.blockerStall) {
const stall = normalized.blockerStall;
const label = stall.stalledIssueIdentifier ?? stall.stalledIssueId ?? "unknown";
const action = stall.recoveryAction;
lines.push(
"",
"## Stalled dependency",
"",
`Your ${stall.relation === "child" ? "child issue" : "blocker"} ${label}${stall.stalledIssueTitle ? ` ${stall.stalledIssueTitle}` : ""}${stall.stalledIssueStatus ? ` (${stall.stalledIssueStatus})` : ""} cannot continue on its own: it waits on a recovery action that needs the board.`,
`- recovery: ${action.kind ?? "unknown"}${action.cause ? ` — ${action.cause}` : ""}${action.status ? ` (${action.status})` : ""}`,
...(action.nextAction ? [`- next action for the board: ${action.nextAction}`] : []),
"",
"Choose one, then stop:",
`- Reassign: if the work can be done another way, reassign its existing task and stop depending on ${label}.`,
`- Escalate: comment on this issue with what is stuck and what the board must do on ${label} (its Retry or reassignment), so the board sees it here.`,
`- Block this issue with a reason that names ${label} and its recovery action, if nothing else can move until the board acts.`,
`Do not redo ${label}'s work yourself while it is assigned, and do not wait silently.`,
);
}
if (normalized.childIssueSummaries.length > 0) {
lines.push("", "Direct child issue summaries:");
for (const child of normalized.childIssueSummaries) {
packages/adapters/hermes/src/server/prompt-rendering.test.ts
+4 -4
View File
@@ -190,7 +190,7 @@ test("renders comment wake batch guidance without defaulting to a full-thread re
expect(prompt).toContain("- fallback fetch needed: no");
});
test("renders accepted-plan continuation without authorizing implementation on the planning issue", () => {
test("renders accepted-plan continuation on the current issue without requiring subtasks", () => {
const prompt = buildPrompt(baseContext({
paperclipWake: {
reason: "issue_commented",
@@ -210,9 +210,9 @@ test("renders accepted-plan continuation without authorizing implementation on t
},
}), {});
expect(prompt).toContain("- planning directive: Create child issues from the approved plan only. Do not write code or perform implementation work on the planning issue.");
expect(prompt).toContain("- accepted-plan continuation: you may create child implementation issues from the approved plan");
expect(prompt).toContain("must not start implementation work on the planning issue itself");
expect(prompt).toContain("- planning directive: Implement the accepted plan on this issue.");
expect(prompt).toContain("- accepted-plan continuation: continue the approved work on this issue unless the approved plan explicitly assigns it elsewhere");
expect(prompt).toContain("Plan acceptance alone does not require subtasks.");
expect(prompt).not.toContain("- planning directive: Make the plan only.");
expect(prompt).not.toContain("Update the plan only");
});
packages/adapters/openclaw-gateway/src/server/execute.ts
+3 -3
View File
@@ -449,9 +449,9 @@ function buildWakeText(
" - GET /api/issues/{issueId}",
" - GET /api/issues/{issueId}/comments",
" - Execute the issue instructions exactly. If the issue is actionable, take concrete action in this run; do not stop at a plan unless planning was requested.",
" - Leave durable progress with a clear next action. Use child issues for long or parallel delegated work instead of polling agents, sessions, or processes.",
" - Create child issues directly when you know what needs to be done; use POST /api/issues/{issueId}/interactions with kind suggest_tasks, ask_user_questions, or request_confirmation when the board/user must choose, answer, or confirm before you can continue.",
" - For plan approval, update the plan document first, then create request_confirmation targeting the latest plan revision with idempotencyKey confirmation:{issueId}:plan:{revisionId}; wait for acceptance before creating implementation subtasks.",
" - Leave durable progress with a clear next action. Do not busy-poll agents, sessions, or processes waiting for completion.",
" - Continue work on the current issue; use POST /api/issues/{issueId}/interactions with kind suggest_tasks, ask_user_questions, or request_confirmation when the board/user must choose, answer, or confirm before you can continue.",
" - For plan approval, update the plan document first, then create request_confirmation targeting the latest plan revision with idempotencyKey confirmation:{issueId}:plan:{revisionId}; wait for acceptance before implementation.",
" - If blocked, PATCH /api/issues/{issueId} with {\"status\":\"blocked\",\"comment\":\"what is blocked, who owns the unblock, and the next action\"}.",
" - If instructions require a comment, POST /api/issues/{issueId}/comments with {\"body\":\"...\"}.",
" - PATCH /api/issues/{issueId} with {\"status\":\"done\",\"comment\":\"what changed and why\"}.",
packages/db/src/client.ts
+5
View File
@@ -614,6 +614,11 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
"0435_premium_puff_adder.sql",
"0436_legal_reavers.sql",
"0447_legacy_controller_process.sql",
"0450_petite_frightful_four.sql",
"0453_curious_ben_parker.sql",
"0454_lumpy_ronan.sql",
"0455_pending_budget_cancellation_index.sql",
"0456_chat_review_lookup_indexes.sql",
"0384_funny_adam_warlock.sql",
packages/db/src/heartbeat-workspace-companion-index.test.ts
+63
View File
@@ -0,0 +1,63 @@
import { createHash, randomUUID } from "node:crypto";
import { readFile } from "node:fs/promises";
import postgres from "postgres";
import { describe, expect, it } from "vitest";
import { applyPendingMigrations } from "./client.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js";
const support = await getEmbeddedPostgresTestSupport();
const indexName = "heartbeat_runs_company_native_workspace_idx";
const migrationFile = "0453_curious_ben_parker.sql";
(support.supported ? describe : describe.skip)("workspace companion expression index", () => {
it("builds outside a transaction and preserves ordered prepared/finalized references", async () => {
const database = await startEmbeddedPostgresTestDatabase("workspace-companion-index-");
const sql = postgres(database.connectionString, { max: 1, onnotice: () => {} });
try {
const migration = await readFile(new URL(`./migrations/${migrationFile}`, import.meta.url), "utf8");
const hash = createHash("sha256").update(migration).digest("hex");
await sql`DELETE FROM drizzle.__drizzle_migrations WHERE hash=${hash}`;
await sql.unsafe(`DROP INDEX ${indexName}`);
const companies = await sql`INSERT INTO companies(name,issue_prefix) VALUES ('Index fixture','CIX1'),('Foreign fixture','CIX2') RETURNING id`;
const company = companies[0]!.id, foreignCompany = companies[1]!.id;
const [agent] = await sql`INSERT INTO agents(company_id,name) VALUES (${company},'Local fixture') RETURNING id`;
const [foreignAgent] = await sql`INSERT INTO agents(company_id,name) VALUES (${foreignCompany},'Foreign fixture') RETURNING id`;
const issues = await sql`INSERT INTO issues(company_id,title) VALUES (${company},'Source issue'),(${company},'Other source issue') RETURNING id`;
const workspace = randomUUID(), expected: string[] = [];
for (const [state, issue] of [["prepared", issues[0]!.id], ["finalized", issues[1]!.id], ["prepared", null], ["finalized", null]] as const) {
const id = randomUUID(); expected.push(id);
await sql`INSERT INTO heartbeat_runs(id,company_id,agent_id,native_issue_id,runner_profile_json)
VALUES (${id},${company},${agent!.id},${issue},${sql.json({ nativeWorkspaceSync: { workspaceId: workspace, state } })})`;
}
await sql`INSERT INTO heartbeat_runs(company_id,agent_id,runner_profile_json)
VALUES (${foreignCompany},${foreignAgent!.id},${sql.json({ nativeWorkspaceSync: { workspaceId: workspace, state: "prepared" } })})`;
for (const sync of [{ workspaceId: { malformed: true }, state: "prepared" }, { workspaceId: workspace.toUpperCase(), state: "prepared" }, { workspaceId: workspace, state: "other" }, { workspaceId: workspace }, "malformed", null]) {
await sql`INSERT INTO heartbeat_runs(company_id,agent_id,runner_profile_json)
VALUES (${company},${agent!.id},${sql.json({ nativeWorkspaceSync: sync })})`;
}
const before = await sql`SELECT id,native_issue_id,runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}' workspace_id,
runner_profile_json#>>'{nativeWorkspaceSync,state}' state FROM heartbeat_runs
WHERE company_id=${company} AND runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}'=${workspace}
AND runner_profile_json#>>'{nativeWorkspaceSync,state}' IN ('prepared','finalized') ORDER BY id`;
expect(before.map(row => row.id)).toEqual(expected.sort());
await applyPendingMigrations(database.connectionString);
const after = await sql`SELECT id,native_issue_id,runner_profile_json->'nativeWorkspaceSync'->>'workspaceId' workspace_id,
runner_profile_json->'nativeWorkspaceSync'->>'state' state FROM heartbeat_runs
WHERE company_id=${company} AND runner_profile_json->'nativeWorkspaceSync'->>'workspaceId'=${workspace}
AND runner_profile_json->'nativeWorkspaceSync'->>'state' IN ('prepared','finalized') ORDER BY id`;
expect(after).toEqual(before);
const [index] = await sql`SELECT indisvalid,indisready,pg_get_expr(indpred,indrelid) predicate,pg_get_indexdef(indexrelid) definition
FROM pg_index WHERE indexrelid=${indexName}::regclass`;
expect(index).toMatchObject({ indisvalid: true, indisready: true, predicate: null });
expect(index!.definition).toContain("company_id");
expect(index!.definition).toContain("nativeWorkspaceSync");
expect(index!.definition).toContain("workspaceId");
await applyPendingMigrations(database.connectionString);
const [history] = await sql`SELECT count(*)::int count FROM drizzle.__drizzle_migrations WHERE hash=${hash}`;
expect(history!.count).toBe(1);
} finally {
await sql.end();
await database.cleanup();
}
}, 90_000);
});
packages/db/src/migrations/0448_agent_harness_verified_digest.sql
+2
View File
@@ -0,0 +1,2 @@
ALTER TABLE "agents" ADD COLUMN "harness_verified_digest" text;--> statement-breakpoint
ALTER TABLE "agents" ADD COLUMN "harness_verified_at" timestamp with time zone;
packages/db/src/migrations/0449_controller_instances.sql
+15
View File
@@ -0,0 +1,15 @@
CREATE TABLE "controller_instances" (
"boot_id" text PRIMARY KEY NOT NULL,
"epoch" bigserial NOT NULL,
"process_space" text NOT NULL,
"pid" integer NOT NULL,
"started_at" timestamp with time zone DEFAULT now() NOT NULL,
"renewed_at" timestamp with time zone DEFAULT now() NOT NULL,
"lease_expires_at" timestamp with time zone NOT NULL,
"fenced_at" timestamp with time zone,
"fenced_by_epoch" bigint,
"stopped_at" timestamp with time zone
);
--> statement-breakpoint
CREATE UNIQUE INDEX "controller_instances_epoch_idx" ON "controller_instances" USING btree ("epoch");--> statement-breakpoint
CREATE INDEX "controller_instances_lease_idx" ON "controller_instances" USING btree ("lease_expires_at");
packages/db/src/migrations/0450_petite_frightful_four.sql
+1
View File
@@ -0,0 +1 @@
CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS "agent_wakeup_requests_issue_blocker_stalled_uq" ON "agent_wakeup_requests" USING btree ("company_id","idempotency_key") WHERE "agent_wakeup_requests"."idempotency_key" LIKE 'issue_blocker_stalled:%' AND "agent_wakeup_requests"."status" IN ('queued', 'deferred_issue_execution', 'claimed', 'coalesced', 'completed');
packages/db/src/migrations/0451_budget_resume_claims.sql
+18
View File
@@ -0,0 +1,18 @@
CREATE TABLE "budget_resume_claims" (
"run_id" uuid PRIMARY KEY NOT NULL,
"company_id" uuid NOT NULL,
"incident_id" uuid NOT NULL,
"actor_user_id" text NOT NULL,
"actor_source" text,
"state" text DEFAULT 'pending' NOT NULL,
"claimed_at" timestamp with time zone,
"settled_at" timestamp with time zone,
"reason" text,
"request_id" uuid,
"created_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
ALTER TABLE "budget_resume_claims" ADD CONSTRAINT "budget_resume_claims_run_id_heartbeat_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."heartbeat_runs"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "budget_resume_claims" ADD CONSTRAINT "budget_resume_claims_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "budget_resume_claims" ADD CONSTRAINT "budget_resume_claims_incident_id_budget_incidents_id_fk" FOREIGN KEY ("incident_id") REFERENCES "public"."budget_incidents"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
CREATE INDEX "budget_resume_claims_pending_idx" ON "budget_resume_claims" USING btree ("company_id","run_id") WHERE "budget_resume_claims"."state" = 'pending';
packages/db/src/migrations/0452_secret_proposal_secret_cascade.sql
+3
View File
@@ -0,0 +1,3 @@
ALTER TABLE "company_secret_proposals" DROP CONSTRAINT IF EXISTS "company_secret_proposals_secret_id_company_secrets_id_fk";
--> statement-breakpoint
ALTER TABLE "company_secret_proposals" ADD CONSTRAINT "company_secret_proposals_secret_id_company_secrets_id_fk" FOREIGN KEY ("secret_id") REFERENCES "public"."company_secrets"("id") ON DELETE cascade ON UPDATE no action;
packages/db/src/migrations/0453_curious_ben_parker.sql
+1
View File
@@ -0,0 +1 @@
CREATE INDEX CONCURRENTLY IF NOT EXISTS "heartbeat_runs_company_native_workspace_idx" ON "heartbeat_runs" USING btree ("company_id",("runner_profile_json"->'nativeWorkspaceSync'->>'workspaceId'));
packages/db/src/migrations/0454_lumpy_ronan.sql
+1
View File
@@ -0,0 +1 @@
CREATE INDEX CONCURRENTLY IF NOT EXISTS "heartbeat_run_events_retry_exhausted_idx" ON "heartbeat_run_events" USING btree ("company_id","run_id","id" DESC NULLS LAST) WHERE "heartbeat_run_events"."event_type" = 'lifecycle' and "heartbeat_run_events"."message" like 'Bounded retry exhausted%';
packages/db/src/migrations/0455_pending_budget_cancellation_index.sql
+2
View File
@@ -0,0 +1,2 @@
CREATE INDEX CONCURRENTLY IF NOT EXISTS "heartbeat_runs_pending_budget_cancellation_idx" ON "heartbeat_runs" USING btree ("company_id") WHERE jsonb_typeof("heartbeat_runs"."result_json"->'startupCancellation'->'budgetEnforcement') = 'object'
and "heartbeat_runs"."result_json"->'startupCancellation'->'budgetEnforcement'->>'deliveredAt' is null;
packages/db/src/migrations/0456_chat_review_lookup_indexes.sql
+4
View File
@@ -0,0 +1,4 @@
CREATE INDEX CONCURRENTLY IF NOT EXISTS "chat_deliveries_github_automatic_delivery_idx" ON "chat_deliveries" USING btree ("endpoint_id",("normalized_event"->'githubAutomatic'->'context'->>'deliveryId')) WHERE ("chat_deliveries"."normalized_event"->'githubAutomatic'->'context'->>'deliveryId') is not null;--> statement-breakpoint
CREATE INDEX CONCURRENTLY IF NOT EXISTS "chat_message_links_company_comment_idx" ON "chat_message_links" USING btree ("company_id","comment_id") WHERE "chat_message_links"."comment_id" is not null;--> statement-breakpoint
CREATE INDEX CONCURRENTLY IF NOT EXISTS "chat_message_links_delivery_idx" ON "chat_message_links" USING btree ("delivery_id") WHERE "chat_message_links"."delivery_id" is not null;--> statement-breakpoint
CREATE INDEX CONCURRENTLY IF NOT EXISTS "heartbeat_runs_company_wake_comment_idx" ON "heartbeat_runs" USING btree ("company_id",("context_snapshot"->>'wakeCommentId'));
packages/db/src/migrations/meta/0443_snapshot.json
Generated
-67227
View File
File diff suppressed because it is too large. Load diff
packages/db/src/migrations/meta/0447_snapshot.json → packages/db/src/migrations/meta/0452_snapshot.json
Generated
+287 -3
View File
@@ -1,6 +1,6 @@
{
"id": "76f06e29-53a8-4ecc-809a-d52b6bbb6520",
"prevId": "782688f1-7843-4a93-8ea3-2cec362d269e",
"id": "64e8be1e-e0cb-4cd8-85e0-0fdcce5eafde",
"prevId": "b5699366-77c2-48c8-b7dc-658a75de7658",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -4102,6 +4102,28 @@
"method": "btree",
"with": {}
},
"agent_wakeup_requests_issue_blocker_stalled_uq": {
"name": "agent_wakeup_requests_issue_blocker_stalled_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "idempotency_key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"agent_wakeup_requests\".\"idempotency_key\" LIKE 'issue_blocker_stalled:%' AND \"agent_wakeup_requests\".\"status\" IN ('queued', 'deferred_issue_execution', 'claimed', 'coalesced', 'completed')",
"concurrently": true,
"method": "btree",
"with": {}
},
"agent_wakeup_requests_company_payload_issue_idx": {
"name": "agent_wakeup_requests_company_payload_issue_idx",
"columns": [
@@ -4252,6 +4274,18 @@
"primaryKey": false,
"notNull": false
},
"harness_verified_digest": {
"name": "harness_verified_digest",
"type": "text",
"primaryKey": false,
"notNull": false
},
"harness_verified_at": {
"name": "harness_verified_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reports_to": {
"name": "reports_to",
"type": "uuid",
@@ -7848,6 +7882,150 @@
},
"isRLSEnabled": false
},
"public.budget_resume_claims": {
"name": "budget_resume_claims",
"schema": "",
"columns": {
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"incident_id": {
"name": "incident_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"actor_user_id": {
"name": "actor_user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"actor_source": {
"name": "actor_source",
"type": "text",
"primaryKey": false,
"notNull": false
},
"state": {
"name": "state",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"claimed_at": {
"name": "claimed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"settled_at": {
"name": "settled_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reason": {
"name": "reason",
"type": "text",
"primaryKey": false,
"notNull": false
},
"request_id": {
"name": "request_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"budget_resume_claims_pending_idx": {
"name": "budget_resume_claims_pending_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"budget_resume_claims\".\"state\" = 'pending'",
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"budget_resume_claims_run_id_heartbeat_runs_id_fk": {
"name": "budget_resume_claims_run_id_heartbeat_runs_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "heartbeat_runs",
"columnsFrom": [
"run_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"budget_resume_claims_company_id_companies_id_fk": {
"name": "budget_resume_claims_company_id_companies_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"budget_resume_claims_incident_id_budget_incidents_id_fk": {
"name": "budget_resume_claims_incident_id_budget_incidents_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "budget_incidents",
"columnsFrom": [
"incident_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.built_in_managed_resources": {
"name": "built_in_managed_resources",
"schema": "",
@@ -17221,7 +17399,7 @@
"columnsTo": [
"id"
],
"onDelete": "set null",
"onDelete": "cascade",
"onUpdate": "no action"
},
"company_secret_proposals_secret_proposal_id_company_secret_proposals_id_fk": {
@@ -21965,6 +22143,112 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.controller_instances": {
"name": "controller_instances",
"schema": "",
"columns": {
"boot_id": {
"name": "boot_id",
"type": "text",
"primaryKey": true,
"notNull": true
},
"epoch": {
"name": "epoch",
"type": "bigserial",
"primaryKey": false,
"notNull": true
},
"process_space": {
"name": "process_space",
"type": "text",
"primaryKey": false,
"notNull": true
},
"pid": {
"name": "pid",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"started_at": {
"name": "started_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"renewed_at": {
"name": "renewed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"lease_expires_at": {
"name": "lease_expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"fenced_at": {
"name": "fenced_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"fenced_by_epoch": {
"name": "fenced_by_epoch",
"type": "bigint",
"primaryKey": false,
"notNull": false
},
"stopped_at": {
"name": "stopped_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
}
},
"indexes": {
"controller_instances_epoch_idx": {
"name": "controller_instances_epoch_idx",
"columns": [
{
"expression": "epoch",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"controller_instances_lease_idx": {
"name": "controller_instances_lease_idx",
"columns": [
{
"expression": "lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.cost_events": {
"name": "cost_events",
"schema": "",
packages/db/src/migrations/meta/0446_snapshot.json → packages/db/src/migrations/meta/0453_snapshot.json
Generated
+330 -3
View File
@@ -1,6 +1,6 @@
{
"id": "782688f1-7843-4a93-8ea3-2cec362d269e",
"prevId": "f0d8430b-7abf-4232-86a9-11a268a0d20e",
"id": "d54bfeb4-dedb-4f8b-9195-a4d4aa67d71f",
"prevId": "64e8be1e-e0cb-4cd8-85e0-0fdcce5eafde",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -4102,6 +4102,28 @@
"method": "btree",
"with": {}
},
"agent_wakeup_requests_issue_blocker_stalled_uq": {
"name": "agent_wakeup_requests_issue_blocker_stalled_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "idempotency_key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"agent_wakeup_requests\".\"idempotency_key\" LIKE 'issue_blocker_stalled:%' AND \"agent_wakeup_requests\".\"status\" IN ('queued', 'deferred_issue_execution', 'claimed', 'coalesced', 'completed')",
"concurrently": true,
"method": "btree",
"with": {}
},
"agent_wakeup_requests_company_payload_issue_idx": {
"name": "agent_wakeup_requests_company_payload_issue_idx",
"columns": [
@@ -4252,6 +4274,18 @@
"primaryKey": false,
"notNull": false
},
"harness_verified_digest": {
"name": "harness_verified_digest",
"type": "text",
"primaryKey": false,
"notNull": false
},
"harness_verified_at": {
"name": "harness_verified_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reports_to": {
"name": "reports_to",
"type": "uuid",
@@ -7848,6 +7882,150 @@
},
"isRLSEnabled": false
},
"public.budget_resume_claims": {
"name": "budget_resume_claims",
"schema": "",
"columns": {
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"incident_id": {
"name": "incident_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"actor_user_id": {
"name": "actor_user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"actor_source": {
"name": "actor_source",
"type": "text",
"primaryKey": false,
"notNull": false
},
"state": {
"name": "state",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"claimed_at": {
"name": "claimed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"settled_at": {
"name": "settled_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reason": {
"name": "reason",
"type": "text",
"primaryKey": false,
"notNull": false
},
"request_id": {
"name": "request_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"budget_resume_claims_pending_idx": {
"name": "budget_resume_claims_pending_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"budget_resume_claims\".\"state\" = 'pending'",
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"budget_resume_claims_run_id_heartbeat_runs_id_fk": {
"name": "budget_resume_claims_run_id_heartbeat_runs_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "heartbeat_runs",
"columnsFrom": [
"run_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"budget_resume_claims_company_id_companies_id_fk": {
"name": "budget_resume_claims_company_id_companies_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"budget_resume_claims_incident_id_budget_incidents_id_fk": {
"name": "budget_resume_claims_incident_id_budget_incidents_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "budget_incidents",
"columnsFrom": [
"incident_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.built_in_managed_resources": {
"name": "built_in_managed_resources",
"schema": "",
@@ -17221,7 +17399,7 @@
"columnsTo": [
"id"
],
"onDelete": "set null",
"onDelete": "cascade",
"onUpdate": "no action"
},
"company_secret_proposals_secret_proposal_id_company_secret_proposals_id_fk": {
@@ -21965,6 +22143,112 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.controller_instances": {
"name": "controller_instances",
"schema": "",
"columns": {
"boot_id": {
"name": "boot_id",
"type": "text",
"primaryKey": true,
"notNull": true
},
"epoch": {
"name": "epoch",
"type": "bigserial",
"primaryKey": false,
"notNull": true
},
"process_space": {
"name": "process_space",
"type": "text",
"primaryKey": false,
"notNull": true
},
"pid": {
"name": "pid",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"started_at": {
"name": "started_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"renewed_at": {
"name": "renewed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"lease_expires_at": {
"name": "lease_expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"fenced_at": {
"name": "fenced_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"fenced_by_epoch": {
"name": "fenced_by_epoch",
"type": "bigint",
"primaryKey": false,
"notNull": false
},
"stopped_at": {
"name": "stopped_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
}
},
"indexes": {
"controller_instances_epoch_idx": {
"name": "controller_instances_epoch_idx",
"columns": [
{
"expression": "epoch",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"controller_instances_lease_idx": {
"name": "controller_instances_lease_idx",
"columns": [
{
"expression": "lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.cost_events": {
"name": "cost_events",
"schema": "",
@@ -34856,6 +35140,12 @@
"primaryKey": false,
"notNull": false
},
"controller_process": {
"name": "controller_process",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"execution_stage": {
"name": "execution_stage",
"type": "text",
@@ -35125,6 +35415,22 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_legacy_controller_lease_idx": {
"name": "heartbeat_runs_legacy_controller_lease_idx",
"columns": [
{
"expression": "controller_lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'legacy' AND \"heartbeat_runs\".\"status\" = 'running' AND \"heartbeat_runs\".\"controller_boot_id\" is not null",
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_runs_cost_accounting_pending_idx": {
"name": "heartbeat_runs_cost_accounting_pending_idx",
"columns": [
@@ -35256,6 +35562,27 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_company_native_workspace_idx": {
"name": "heartbeat_runs_company_native_workspace_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "(\"runner_profile_json\"->'nativeWorkspaceSync'->>'workspaceId')",
"asc": true,
"isExpression": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": true,
"method": "btree",
"with": {}
},
"heartbeat_runs_company_issue_created_idx": {
"name": "heartbeat_runs_company_issue_created_idx",
"columns": [
packages/db/src/migrations/meta/0445_snapshot.json → packages/db/src/migrations/meta/0454_snapshot.json
Generated
+396 -3
View File
@@ -1,6 +1,6 @@
{
"id": "f0d8430b-7abf-4232-86a9-11a268a0d20e",
"prevId": "93cbf124-1264-4bc3-a27d-b651ce9640e7",
"id": "f98af245-4a2e-4c3d-b89e-6ed3e81a074c",
"prevId": "d54bfeb4-dedb-4f8b-9195-a4d4aa67d71f",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -4102,6 +4102,28 @@
"method": "btree",
"with": {}
},
"agent_wakeup_requests_issue_blocker_stalled_uq": {
"name": "agent_wakeup_requests_issue_blocker_stalled_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "idempotency_key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"agent_wakeup_requests\".\"idempotency_key\" LIKE 'issue_blocker_stalled:%' AND \"agent_wakeup_requests\".\"status\" IN ('queued', 'deferred_issue_execution', 'claimed', 'coalesced', 'completed')",
"concurrently": true,
"method": "btree",
"with": {}
},
"agent_wakeup_requests_company_payload_issue_idx": {
"name": "agent_wakeup_requests_company_payload_issue_idx",
"columns": [
@@ -4252,6 +4274,18 @@
"primaryKey": false,
"notNull": false
},
"harness_verified_digest": {
"name": "harness_verified_digest",
"type": "text",
"primaryKey": false,
"notNull": false
},
"harness_verified_at": {
"name": "harness_verified_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reports_to": {
"name": "reports_to",
"type": "uuid",
@@ -7848,6 +7882,150 @@
},
"isRLSEnabled": false
},
"public.budget_resume_claims": {
"name": "budget_resume_claims",
"schema": "",
"columns": {
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"incident_id": {
"name": "incident_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"actor_user_id": {
"name": "actor_user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"actor_source": {
"name": "actor_source",
"type": "text",
"primaryKey": false,
"notNull": false
},
"state": {
"name": "state",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"claimed_at": {
"name": "claimed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"settled_at": {
"name": "settled_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reason": {
"name": "reason",
"type": "text",
"primaryKey": false,
"notNull": false
},
"request_id": {
"name": "request_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"budget_resume_claims_pending_idx": {
"name": "budget_resume_claims_pending_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"budget_resume_claims\".\"state\" = 'pending'",
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"budget_resume_claims_run_id_heartbeat_runs_id_fk": {
"name": "budget_resume_claims_run_id_heartbeat_runs_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "heartbeat_runs",
"columnsFrom": [
"run_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"budget_resume_claims_company_id_companies_id_fk": {
"name": "budget_resume_claims_company_id_companies_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"budget_resume_claims_incident_id_budget_incidents_id_fk": {
"name": "budget_resume_claims_incident_id_budget_incidents_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "budget_incidents",
"columnsFrom": [
"incident_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.built_in_managed_resources": {
"name": "built_in_managed_resources",
"schema": "",
@@ -17221,7 +17399,7 @@
"columnsTo": [
"id"
],
"onDelete": "set null",
"onDelete": "cascade",
"onUpdate": "no action"
},
"company_secret_proposals_secret_proposal_id_company_secret_proposals_id_fk": {
@@ -21965,6 +22143,112 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.controller_instances": {
"name": "controller_instances",
"schema": "",
"columns": {
"boot_id": {
"name": "boot_id",
"type": "text",
"primaryKey": true,
"notNull": true
},
"epoch": {
"name": "epoch",
"type": "bigserial",
"primaryKey": false,
"notNull": true
},
"process_space": {
"name": "process_space",
"type": "text",
"primaryKey": false,
"notNull": true
},
"pid": {
"name": "pid",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"started_at": {
"name": "started_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"renewed_at": {
"name": "renewed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"lease_expires_at": {
"name": "lease_expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"fenced_at": {
"name": "fenced_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"fenced_by_epoch": {
"name": "fenced_by_epoch",
"type": "bigint",
"primaryKey": false,
"notNull": false
},
"stopped_at": {
"name": "stopped_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
}
},
"indexes": {
"controller_instances_epoch_idx": {
"name": "controller_instances_epoch_idx",
"columns": [
{
"expression": "epoch",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"controller_instances_lease_idx": {
"name": "controller_instances_lease_idx",
"columns": [
{
"expression": "lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.cost_events": {
"name": "cost_events",
"schema": "",
@@ -34268,6 +34552,34 @@
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_run_events_retry_exhausted_idx": {
"name": "heartbeat_run_events_retry_exhausted_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": false,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_run_events\".\"event_type\" = 'lifecycle' and \"heartbeat_run_events\".\"message\" like 'Bounded retry exhausted%'",
"concurrently": true,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
@@ -34856,6 +35168,12 @@
"primaryKey": false,
"notNull": false
},
"controller_process": {
"name": "controller_process",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"execution_stage": {
"name": "execution_stage",
"type": "text",
@@ -35125,6 +35443,22 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_legacy_controller_lease_idx": {
"name": "heartbeat_runs_legacy_controller_lease_idx",
"columns": [
{
"expression": "controller_lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'legacy' AND \"heartbeat_runs\".\"status\" = 'running' AND \"heartbeat_runs\".\"controller_boot_id\" is not null",
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_runs_cost_accounting_pending_idx": {
"name": "heartbeat_runs_cost_accounting_pending_idx",
"columns": [
@@ -35256,6 +35590,27 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_company_native_workspace_idx": {
"name": "heartbeat_runs_company_native_workspace_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "(\"runner_profile_json\"->'nativeWorkspaceSync'->>'workspaceId')",
"asc": true,
"isExpression": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": true,
"method": "btree",
"with": {}
},
"heartbeat_runs_company_issue_created_idx": {
"name": "heartbeat_runs_company_issue_created_idx",
"columns": [
@@ -44766,6 +45121,12 @@
"primaryKey": false,
"notNull": false
},
"lease_holder": {
"name": "lease_holder",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"controller_boot_id": {
"name": "controller_boot_id",
"type": "text",
@@ -44930,6 +45291,38 @@
"method": "btree",
"with": {}
},
"native_run_finalizations_finalizer_lease_idx": {
"name": "native_run_finalizations_finalizer_lease_idx",
"columns": [
{
"expression": "lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"native_run_finalizations\".\"lease_owner\" like 'native-finalizer:%'",
"concurrently": false,
"method": "btree",
"with": {}
},
"native_run_finalizations_cleanup_lease_idx": {
"name": "native_run_finalizations_cleanup_lease_idx",
"columns": [
{
"expression": "lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"native_run_finalizations\".\"lease_owner\" like 'native-cleanup:%'",
"concurrently": false,
"method": "btree",
"with": {}
},
"native_run_finalizations_accepted_result_lease_idx": {
"name": "native_run_finalizations_accepted_result_lease_idx",
"columns": [
packages/db/src/migrations/meta/0442_snapshot.json → packages/db/src/migrations/meta/0455_snapshot.json
Generated
+428 -3
View File
@@ -1,6 +1,6 @@
{
"id": "b3f3d712-65da-4b5a-bbf5-3c22fe8aae40",
"prevId": "a2ec96c6-3cf1-4c51-8ee8-b35f090bfd42",
"id": "2be53987-902a-41b6-8cb4-6e89a768f8cf",
"prevId": "f98af245-4a2e-4c3d-b89e-6ed3e81a074c",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -4102,6 +4102,28 @@
"method": "btree",
"with": {}
},
"agent_wakeup_requests_issue_blocker_stalled_uq": {
"name": "agent_wakeup_requests_issue_blocker_stalled_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "idempotency_key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"agent_wakeup_requests\".\"idempotency_key\" LIKE 'issue_blocker_stalled:%' AND \"agent_wakeup_requests\".\"status\" IN ('queued', 'deferred_issue_execution', 'claimed', 'coalesced', 'completed')",
"concurrently": true,
"method": "btree",
"with": {}
},
"agent_wakeup_requests_company_payload_issue_idx": {
"name": "agent_wakeup_requests_company_payload_issue_idx",
"columns": [
@@ -4252,6 +4274,18 @@
"primaryKey": false,
"notNull": false
},
"harness_verified_digest": {
"name": "harness_verified_digest",
"type": "text",
"primaryKey": false,
"notNull": false
},
"harness_verified_at": {
"name": "harness_verified_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reports_to": {
"name": "reports_to",
"type": "uuid",
@@ -7848,6 +7882,150 @@
},
"isRLSEnabled": false
},
"public.budget_resume_claims": {
"name": "budget_resume_claims",
"schema": "",
"columns": {
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"incident_id": {
"name": "incident_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"actor_user_id": {
"name": "actor_user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"actor_source": {
"name": "actor_source",
"type": "text",
"primaryKey": false,
"notNull": false
},
"state": {
"name": "state",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"claimed_at": {
"name": "claimed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"settled_at": {
"name": "settled_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reason": {
"name": "reason",
"type": "text",
"primaryKey": false,
"notNull": false
},
"request_id": {
"name": "request_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"budget_resume_claims_pending_idx": {
"name": "budget_resume_claims_pending_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"budget_resume_claims\".\"state\" = 'pending'",
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"budget_resume_claims_run_id_heartbeat_runs_id_fk": {
"name": "budget_resume_claims_run_id_heartbeat_runs_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "heartbeat_runs",
"columnsFrom": [
"run_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"budget_resume_claims_company_id_companies_id_fk": {
"name": "budget_resume_claims_company_id_companies_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"budget_resume_claims_incident_id_budget_incidents_id_fk": {
"name": "budget_resume_claims_incident_id_budget_incidents_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "budget_incidents",
"columnsFrom": [
"incident_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.built_in_managed_resources": {
"name": "built_in_managed_resources",
"schema": "",
@@ -17221,7 +17399,7 @@
"columnsTo": [
"id"
],
"onDelete": "set null",
"onDelete": "cascade",
"onUpdate": "no action"
},
"company_secret_proposals_secret_proposal_id_company_secret_proposals_id_fk": {
@@ -21965,6 +22143,112 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.controller_instances": {
"name": "controller_instances",
"schema": "",
"columns": {
"boot_id": {
"name": "boot_id",
"type": "text",
"primaryKey": true,
"notNull": true
},
"epoch": {
"name": "epoch",
"type": "bigserial",
"primaryKey": false,
"notNull": true
},
"process_space": {
"name": "process_space",
"type": "text",
"primaryKey": false,
"notNull": true
},
"pid": {
"name": "pid",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"started_at": {
"name": "started_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"renewed_at": {
"name": "renewed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"lease_expires_at": {
"name": "lease_expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"fenced_at": {
"name": "fenced_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"fenced_by_epoch": {
"name": "fenced_by_epoch",
"type": "bigint",
"primaryKey": false,
"notNull": false
},
"stopped_at": {
"name": "stopped_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
}
},
"indexes": {
"controller_instances_epoch_idx": {
"name": "controller_instances_epoch_idx",
"columns": [
{
"expression": "epoch",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"controller_instances_lease_idx": {
"name": "controller_instances_lease_idx",
"columns": [
{
"expression": "lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.cost_events": {
"name": "cost_events",
"schema": "",
@@ -34268,6 +34552,34 @@
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_run_events_retry_exhausted_idx": {
"name": "heartbeat_run_events_retry_exhausted_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": false,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_run_events\".\"event_type\" = 'lifecycle' and \"heartbeat_run_events\".\"message\" like 'Bounded retry exhausted%'",
"concurrently": true,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
@@ -34856,6 +35168,12 @@
"primaryKey": false,
"notNull": false
},
"controller_process": {
"name": "controller_process",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"execution_stage": {
"name": "execution_stage",
"type": "text",
@@ -35103,6 +35421,22 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_pending_budget_cancellation_idx": {
"name": "heartbeat_runs_pending_budget_cancellation_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "jsonb_typeof(\"heartbeat_runs\".\"result_json\"->'startupCancellation'->'budgetEnforcement') = 'object'\n and \"heartbeat_runs\".\"result_json\"->'startupCancellation'->'budgetEnforcement'->>'deliveredAt' is null",
"concurrently": true,
"method": "btree",
"with": {}
},
"heartbeat_runs_tree_hold_suppression_idx": {
"name": "heartbeat_runs_tree_hold_suppression_idx",
"columns": [
@@ -35125,6 +35459,22 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_legacy_controller_lease_idx": {
"name": "heartbeat_runs_legacy_controller_lease_idx",
"columns": [
{
"expression": "controller_lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'legacy' AND \"heartbeat_runs\".\"status\" = 'running' AND \"heartbeat_runs\".\"controller_boot_id\" is not null",
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_runs_cost_accounting_pending_idx": {
"name": "heartbeat_runs_cost_accounting_pending_idx",
"columns": [
@@ -35256,6 +35606,27 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_company_native_workspace_idx": {
"name": "heartbeat_runs_company_native_workspace_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "(\"runner_profile_json\"->'nativeWorkspaceSync'->>'workspaceId')",
"asc": true,
"isExpression": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": true,
"method": "btree",
"with": {}
},
"heartbeat_runs_company_issue_created_idx": {
"name": "heartbeat_runs_company_issue_created_idx",
"columns": [
@@ -44766,6 +45137,12 @@
"primaryKey": false,
"notNull": false
},
"lease_holder": {
"name": "lease_holder",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"controller_boot_id": {
"name": "controller_boot_id",
"type": "text",
@@ -44930,6 +45307,54 @@
"method": "btree",
"with": {}
},
"native_run_finalizations_finalizer_lease_idx": {
"name": "native_run_finalizations_finalizer_lease_idx",
"columns": [
{
"expression": "lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"native_run_finalizations\".\"lease_owner\" like 'native-finalizer:%'",
"concurrently": false,
"method": "btree",
"with": {}
},
"native_run_finalizations_cleanup_lease_idx": {
"name": "native_run_finalizations_cleanup_lease_idx",
"columns": [
{
"expression": "lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"native_run_finalizations\".\"lease_owner\" like 'native-cleanup:%'",
"concurrently": false,
"method": "btree",
"with": {}
},
"native_run_finalizations_accepted_result_lease_idx": {
"name": "native_run_finalizations_accepted_result_lease_idx",
"columns": [
{
"expression": "lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"native_run_finalizations\".\"result_id\" is not null AND \"native_run_finalizations\".\"lease_owner\" is not null AND \"native_run_finalizations\".\"phase\" not in ('committed', 'terminal_failure', 'applied')",
"concurrently": false,
"method": "btree",
"with": {}
},
"native_run_finalizations_control_deadline_idx": {
"name": "native_run_finalizations_control_deadline_idx",
"columns": [
packages/db/src/migrations/meta/0456_snapshot.json
Generated
+67733
View File
File diff suppressed because it is too large. Load diff
packages/db/src/migrations/meta/_journal.json
Generated
+63
View File
@@ -3123,6 +3123,69 @@
"when": 1794105389611,
"tag": "0447_legacy_controller_process",
"breakpoints": true
},
{
"idx": 448,
"version": "7",
"when": 1794135798799,
"tag": "0448_agent_harness_verified_digest",
"breakpoints": true
},
{
"idx": 449,
"version": "7",
"when": 1794147993995,
"tag": "0449_controller_instances",
"breakpoints": true
},
{
"idx": 450,
"version": "7",
"when": 1794170960477,
"tag": "0450_petite_frightful_four",
"breakpoints": true
},
{
"idx": 451,
"version": "7",
"when": 1794185816322,
"tag": "0451_budget_resume_claims",
"breakpoints": true
},
{
"idx": 452,
"version": "7",
"when": 1794205579544,
"tag": "0452_secret_proposal_secret_cascade",
"breakpoints": true
},
{
"idx": 453,
"version": "7",
"when": 1794260223713,
"tag": "0453_curious_ben_parker",
"breakpoints": true
},
{
"idx": 454,
"version": "7",
"when": 1794266236755,
"tag": "0454_lumpy_ronan",
"breakpoints": true
},
{
"idx": 455,
"version": "7",
"when": 1794268118742,
"tag": "0455_pending_budget_cancellation_index",
"breakpoints": true
},
{
"idx": 456,
"version": "7",
"when": 1794270045155,
"tag": "0456_chat_review_lookup_indexes",
"breakpoints": true
}
]
}
packages/db/src/schema/agent_wakeup_requests.ts
+7
View File
@@ -99,6 +99,13 @@ export const agentWakeupRequests = pgTable(
chatCompletionIdempotencyUq: uniqueIndex("agent_wakeup_requests_chat_completion_uq")
.on(table.companyId, table.idempotencyKey)
.where(sql`${table.idempotencyKey} LIKE 'chat-completion:%'`),
// One delivered or pending wake per stall (issue_blocker_stalled). Refused
// wakes (skipped, failed, cancelled) are not covered, so a stall retries
// once its agent takes wakes again; concurrent sweeps cannot both deliver.
issueBlockerStalledIdempotencyUq: uniqueIndex("agent_wakeup_requests_issue_blocker_stalled_uq")
.on(table.companyId, table.idempotencyKey)
.concurrently()
.where(sql`${table.idempotencyKey} LIKE 'issue_blocker_stalled:%' AND ${table.status} IN ('queued', 'deferred_issue_execution', 'claimed', 'coalesced', 'completed')`),
companyPayloadIssueIdx: index("agent_wakeup_requests_company_payload_issue_idx").on(
table.companyId,
sql`(${table.payload} ->> 'issueId')`,
packages/db/src/schema/agents.ts
+3
View File
@@ -31,6 +31,9 @@ export const agents = pgTable(
lifecycleVersion: integer("lifecycle_version").notNull().default(0),
lifecycleError: text("lifecycle_error"),
lifecycleOperation: jsonb("lifecycle_operation").$type<AgentLifecycleOperation>(),
/** Digest of the effective configuration and runner image whose harness verification passed. */
harnessVerifiedDigest: text("harness_verified_digest"),
harnessVerifiedAt: timestamp("harness_verified_at", { withTimezone: true }),
reportsTo: uuid("reports_to").references((): AnyPgColumn => agents.id),
capabilities: text("capabilities"),
adapterType: text("adapter_type").notNull().default("process"),
packages/db/src/schema/budget_resume_claims.ts
+32
View File
@@ -0,0 +1,32 @@
import { sql } from "drizzle-orm";
import { index, pgTable, text, timestamp, uuid } from "drizzle-orm/pg-core";
import { budgetIncidents } from "./budget_incidents.js";
import { companies } from "./companies.js";
import { heartbeatRuns } from "./heartbeat_runs.js";
/**
* A run a budget stop cancelled, owed one continuation because a board member
* resolved the incident with Raise budget and resume. Kept outside the run's
* result JSON, which terminal writers replace, so a Stop that settles at the
* same moment cannot erase it.
*/
export const budgetResumeClaims = pgTable(
"budget_resume_claims",
{
runId: uuid("run_id").primaryKey().references(() => heartbeatRuns.id, { onDelete: "cascade" }),
companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }),
incidentId: uuid("incident_id").notNull().references(() => budgetIncidents.id, { onDelete: "cascade" }),
actorUserId: text("actor_user_id").notNull(),
actorSource: text("actor_source"),
state: text("state").notNull().default("pending"),
claimedAt: timestamp("claimed_at", { withTimezone: true }),
settledAt: timestamp("settled_at", { withTimezone: true }),
reason: text("reason"),
requestId: uuid("request_id"),
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
},
(table) => ({
pendingIdx: index("budget_resume_claims_pending_idx").on(table.companyId, table.runId)
.where(sql`${table.state} = 'pending'`),
}),
);
packages/db/src/schema/chat_channels.ts
+15
View File
@@ -487,6 +487,11 @@ export const chatDeliveries = pgTable(
table.endpointId,
table.deduplicationKey,
),
// Review checks find their delivery by the GitHub webhook delivery id.
index("chat_deliveries_github_automatic_delivery_idx")
.on(table.endpointId, sql`(${table.normalizedEvent}->'githubAutomatic'->'context'->>'deliveryId')`)
.where(sql`(${table.normalizedEvent}->'githubAutomatic'->'context'->>'deliveryId') is not null`)
.concurrently(),
foreignKey({
columns: [table.companyId, table.endpointId],
foreignColumns: [chatEndpoints.companyId, chatEndpoints.id],
@@ -638,6 +643,16 @@ export const chatMessageLinks = pgTable(
table.conversationId,
table.providerMessageId,
),
// Source-comment and delivery lookups (GitHub review waits and checks).
// These also serve the comment and delivery `on delete set null` actions.
index("chat_message_links_company_comment_idx")
.on(table.companyId, table.commentId)
.where(sql`${table.commentId} is not null`)
.concurrently(),
index("chat_message_links_delivery_idx")
.on(table.deliveryId)
.where(sql`${table.deliveryId} is not null`)
.concurrently(),
foreignKey({
columns: [table.companyId, table.conversationId],
foreignColumns: [chatConversations.companyId, chatConversations.id],
packages/db/src/schema/company_secret_proposals.ts
+3 -1
View File
@@ -21,7 +21,9 @@ export const companySecretProposals = pgTable(
valueCiphertext: jsonb("value_ciphertext").$type<Record<string, unknown> | null>(),
valueFingerprintSha256: text("value_fingerprint_sha256"),
valueLength: integer("value_length"),
secretId: uuid("secret_id").references(() => companySecrets.id, { onDelete: "set null" }),
// Binding proposals require exactly one source reference (shape check), so
// they go with the secret they reference, like secretProposalId below.
secretId: uuid("secret_id").references(() => companySecrets.id, { onDelete: "cascade" }),
secretProposalId: uuid("secret_proposal_id").references((): AnyPgColumn => companySecretProposals.id, { onDelete: "cascade" }),
targetType: text("target_type"),
targetId: uuid("target_id").references(() => agents.id, { onDelete: "cascade" }),
packages/db/src/schema/controller_instances.ts
+28
View File
@@ -0,0 +1,28 @@
import { bigint, bigserial, index, integer, pgTable, text, timestamp, uniqueIndex } from "drizzle-orm/pg-core";
/**
* One row per server process boot. A live process renews its lease on the
* database clock; a newer boot fences an older one only after its lease has
* expired, and a fenced or lease-less process stops. Stamps name the boot id,
* so a record owned by a replaced container is proven gone without sharing a
* host or PID space with it.
*/
export const controllerInstances = pgTable(
"controller_instances",
{
bootId: text("boot_id").primaryKey(),
epoch: bigserial("epoch", { mode: "number" }).notNull(),
processSpace: text("process_space").notNull(),
pid: integer("pid").notNull(),
startedAt: timestamp("started_at", { withTimezone: true }).notNull().defaultNow(),
renewedAt: timestamp("renewed_at", { withTimezone: true }).notNull().defaultNow(),
leaseExpiresAt: timestamp("lease_expires_at", { withTimezone: true }).notNull(),
fencedAt: timestamp("fenced_at", { withTimezone: true }),
fencedByEpoch: bigint("fenced_by_epoch", { mode: "number" }),
stoppedAt: timestamp("stopped_at", { withTimezone: true }),
},
(table) => ({
epochIdx: uniqueIndex("controller_instances_epoch_idx").on(table.epoch),
liveIdx: index("controller_instances_lease_idx").on(table.leaseExpiresAt),
}),
);
packages/db/src/schema/heartbeat_run_events.ts
+7
View File
@@ -54,5 +54,12 @@ export const heartbeatRunEvents = pgTable(
compactionSessionIdx: index("heartbeat_run_events_compaction_session_idx")
.on(table.companyId, sql`(${table.payload} -> 'prpEvent' ->> 'normalizedSessionId')`)
.where(sql`${table.eventType} = 'context.compacted'`),
// Retry-exhaustion receipts for the attention feed: a few rows per company.
// The predicate must stay textually equal to the query in
// server/src/services/attention-exhausted-runs.ts so the planner can use it.
retryExhaustedIdx: index("heartbeat_run_events_retry_exhausted_idx")
.on(table.companyId, table.runId, table.id.desc())
.where(sql`${table.eventType} = 'lifecycle' and ${table.message} like 'Bounded retry exhausted%'`)
.concurrently(),
}),
);
packages/db/src/schema/heartbeat_runs.ts
+18
View File
@@ -152,6 +152,15 @@ export const heartbeatRuns = pgTable(
),
scheduledRetryCancellationIdx: index("heartbeat_runs_scheduled_retry_cancellation_idx").on(table.id)
.where(sql`${table.status} = 'scheduled_retry' and jsonb_typeof(${table.resultJson}->'startupCancellation') = 'object'`),
// Committed budget Stops that are not delivered yet. The predicate must stay
// identical to `pendingBudgetRunCancellationPredicate` in the server, so the
// budget worker, run control, and the idle check read only these few rows
// instead of every run's result JSON.
pendingBudgetCancellationIdx: index("heartbeat_runs_pending_budget_cancellation_idx")
.on(table.companyId)
.where(sql`jsonb_typeof(${table.resultJson}->'startupCancellation'->'budgetEnforcement') = 'object'
and ${table.resultJson}->'startupCancellation'->'budgetEnforcement'->>'deliveredAt' is null`)
.concurrently(),
// Runs a dispatch gate refused for a pause; that hold's release resumes them.
treeHoldSuppressionIdx: index("heartbeat_runs_tree_hold_suppression_idx")
.on(table.companyId, sql`(${table.resultJson}->'treeHoldSuppression'->>'holdId')`)
@@ -161,6 +170,12 @@ export const heartbeatRuns = pgTable(
legacyControllerLeaseIdx: index("heartbeat_runs_legacy_controller_lease_idx")
.on(table.controllerLeaseExpiresAt)
.where(sql`${table.runtimeMode} = 'legacy' AND ${table.status} = 'running' AND ${table.controllerBootId} is not null`),
// Runs woken by one chat comment. Not partial: the planner ignores the
// expression statistics of a partial index and would then walk the run
// history by creation time to satisfy `order by created_at limit 1`.
companyWakeCommentIdx: index("heartbeat_runs_company_wake_comment_idx")
.on(table.companyId, sql`(${table.contextSnapshot}->>'wakeCommentId')`)
.concurrently(),
costAccountingPendingIdx: index("heartbeat_runs_cost_accounting_pending_idx").on(table.updatedAt, table.id).where(sql`${table.costAccountingPending} = true`),
costDataQualityIdx: index("heartbeat_runs_cost_data_quality_idx")
.on(table.companyId, sql`coalesce(${table.finishedAt}, ${table.createdAt}) desc`, table.id)
@@ -190,6 +205,9 @@ export const heartbeatRuns = pgTable(
table.agentId,
table.startedAt,
),
companyNativeWorkspaceIdx: index("heartbeat_runs_company_native_workspace_idx")
.on(table.companyId, sql`(${table.runnerProfileJson}->'nativeWorkspaceSync'->>'workspaceId')`)
.concurrently(),
companyIssueCreatedIdx: index("heartbeat_runs_company_issue_created_idx").on(
table.companyId,
table.issueId,
packages/db/src/schema/index.ts
+2
View File
@@ -5,6 +5,7 @@ export { companyTransferRuns } from "./company_transfer_runs.js";
export { companyOnboardingSeeds } from "./company_onboarding_seeds.js";
export { authUsers, authSessions, authAccounts, authVerifications } from "./auth.js";
export { instanceSettings } from "./instance_settings.js";
export { controllerInstances } from "./controller_instances.js";
export { instanceUserRoles } from "./instance_user_roles.js";
export { chatVoiceCallbacks } from "./chat_voice_callbacks.js";
export { userSidebarPreferences } from "./user_sidebar_preferences.js";
@@ -21,6 +22,7 @@ export { invites } from "./invites.js";
export { joinRequests } from "./join_requests.js";
export { budgetPolicies } from "./budget_policies.js";
export { budgetIncidents } from "./budget_incidents.js";
export { budgetResumeClaims } from "./budget_resume_claims.js";
export { agentConfigRevisions } from "./agent_config_revisions.js";
export { agentApiKeys } from "./agent_api_keys.js";
export { agentRuntimeState } from "./agent_runtime_state.js";
Loaded 100 of 1994 files, more files were not shown because too many files have changed in this diff. Show more