From 69f4bfc3c543ae8d60746b0c0f61f90ca90aed7a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 13 Oct 2026 09:45:19 -0500 Subject: [PATCH] docs: add external bot integration skill (#16181) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External agents need a connection, a work lifecycle, and clear authority. > - The Dot integration exposed gaps in setup, idle work, recovery, and Cloud delivery. > - Future integrations need a practical procedure that preserves those lessons. > - This pull request adds a core skill and six focused reference documents. > - The skill gives contributors concrete prompts, test cases, and failure diagnostics. ## Linked Issues or Issue Description **Issue type** Missing documentation. **Where is the issue?** Repository contributor skills for external agent integration. **What's wrong?** The integration process has no single reusable guide for provider discovery, both work directions, setup, managed Runner execution, and live proof. Lessons from Dot are spread across implementation changes. **Suggested fix** Add `create-external-bot-integration` under `.agents/skills`. Keep the core workflow short. Put detailed contracts, examples, tests, diagnostics, and dated evidence in references. Related public implementation: Refs #15402, #15414, #15581, #15691, #15722, #15734. A search found no duplicate skill PR. ## What Changed - Add a core skill for external agents and its Codex interface metadata. - Add references for the release contract, Runner architecture, onboarding, qualification tests, deployment, and the Dot case study. - Include idle intake, native tool use, scoped authority, receipt recovery, and resource cleanup. - Mark provider facts and code paths as dated evidence. Use public PRs for the source trail. - Change only eight files under `.agents/skills/create-external-bot-integration`. ## Verification - Skill validator: passed. - Interface metadata, Markdown fences, and 49 local links and anchors: passed. - `git diff --check`: passed. - PR file scope: verified that all eight paths are inside the skill folder. - `pnpm build` and `pnpm -r typecheck`: passed in the isolated worktree. - `pnpm exec vitest run --project @paperclipai/skills-catalog packages/skills-catalog/src/shipped-catalog.test.ts`: 10 tests passed after the description-budget fix. - The broad local `pnpm test:run` attempt reported 12 native-session lease test failures. It is not a passing full-suite result. One isolated lease case passed. The PR changes no runtime files. - Final commit `bd7aed82f623ee279c800af2b8b87f7bedd6be5c`: all GitHub checks completed, with 55 success results and 2 skipped results. - Greptile reviewed that exact commit at 5/5. There are no open review threads. ## Risks - Provider behavior and code paths can change. The skill requires current code and official documentation checks before use. - The case study describes one integration campaign. It does not claim that every provider supports the same transport or capabilities. - This change adds contributor instructions and metadata. It changes no application code or database schema. ## Model Used OpenAI Codex, GPT-6, with reasoning, filesystem access, shell tools, and code review support. The exact serving endpoint ID and context window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../create-external-bot-integration/SKILL.md | 152 ++++++++++++ .../agents/openai.yaml | 4 + .../references/conformance-tests.md | 211 ++++++++++++++++ .../references/dot-case-study.md | 188 +++++++++++++++ .../references/onboarding-and-prompts.md | 200 ++++++++++++++++ .../references/release-contract.md | 141 +++++++++++ .../references/runner-architecture.md | 226 ++++++++++++++++++ .../troubleshooting-and-deployment.md | 215 +++++++++++++++++ 8 files changed, 1337 insertions(+) create mode 100644 .agents/skills/create-external-bot-integration/SKILL.md create mode 100644 .agents/skills/create-external-bot-integration/agents/openai.yaml create mode 100644 .agents/skills/create-external-bot-integration/references/conformance-tests.md create mode 100644 .agents/skills/create-external-bot-integration/references/dot-case-study.md create mode 100644 .agents/skills/create-external-bot-integration/references/onboarding-and-prompts.md create mode 100644 .agents/skills/create-external-bot-integration/references/release-contract.md create mode 100644 .agents/skills/create-external-bot-integration/references/runner-architecture.md create mode 100644 .agents/skills/create-external-bot-integration/references/troubleshooting-and-deployment.md diff --git a/.agents/skills/create-external-bot-integration/SKILL.md b/.agents/skills/create-external-bot-integration/SKILL.md new file mode 100644 index 00000000000..6694d9643cb --- /dev/null +++ b/.agents/skills/create-external-bot-integration/SKILL.md @@ -0,0 +1,152 @@ +--- +name: create-external-bot-integration +description: >- + Build and qualify external bots such as Dot in Paperclip. Use for two-way work, + invitation, identity, wakeup, Runner lifecycle, native tools, recovery, and + Cloud delivery. CLI adapters and app-only connectors use their existing skills. +--- + +# Create an External Bot Integration + +Make an existing external agent a useful Paperclip participant: Paperclip can +assign work to it, and it can initiate governed work from its own conversation. +It uses its relevant native capabilities, returns inspectable results to +Paperclip, and participates in normal admission, authority, and finalization. + +This workflow incorporates the September 30–October 9, 2026 Dot integration. +Provider facts and code paths in the references are dated evidence, not a claim +that every future bot supports the same APIs. Verify the current implementation +and official provider documentation before choosing a transport or changing code. + +## 1. Establish the release contract before narrowing the implementation + +Read [release-contract.md](references/release-contract.md). Infer already-stated +requirements and authorization from the conversation. Ask only about consequential +unknowns; do not ask the user to repeat “two way,” Cloud, or use of Runner. + +Record the target environment, acting identity, capabilities, allowed human setup +steps, delivery/stop expectations, and checkable finish line. Keep one compact +matrix in the existing plan or task. Follow repository conventions if creating a +plan file. “Works” must name its level: prototype, real provider, real Runner, +managed sandbox, deployed tenant, or unattended first-time onboarding. + +Reuse the current implementation and qualification evidence. Inspect the relevant +checkout, instructions, app/Cloud PRs, and Runner artifact. Do not recreate work +already merged or apply an old plan to a different deployed revision. + +## 2. Prove the uncertain provider boundary first + +Identify supported ways to authenticate, wake the agent, load tools, correlate +work, return results, and stop or revoke authority. Check events/subscriptions as +well as direct APIs and chat channels. Do not assume that a bot mention wakes +another bot, that MCP notifications wake an offline agent, or that a plugin listed +in settings is callable in the agent's current conversation. + +Use a synthetic company and stable public HTTPS endpoint. Start with a scripted +client for deterministic protocol tests, then the actual provider account: + +1. Fresh connection → callable read tool → verified event round trip. +2. Paperclip assignment → autonomous acceptance → persisted result → finalization. +3. Idle bot request → one governed intake → useful action → finalization. + +If idle autonomy is in scope, the third journey is a first milestone, not an +optional expansion. Do not make the user create a placeholder task for the bot. +Label every nudge, operator repair, simulated dependency, and manual tool call. +A successful assisted run does not prove prompt-only first-time setup. + +## 3. Reuse Runner while keeping placement and capability separate + +Read [runner-architecture.md](references/runner-architecture.md) before changing +execution or authority. For Paperclip, prefer the existing Runner infrastructure: +shared lifecycle, receipts, recovery, and a future sandbox boundary for workspace +tools. The bot's model and native tools can remain entirely at its provider. + +Use the ordinary configured managed execution environment on Cloud. One admitted +assignment reuses its Runner across messages and tool calls; idle connection +traffic does not allocate compute. A sandbox hosting Runner does not grant the +bot filesystem or shell access. Never add a fallback to the control-plane host. + +Implement dedicated agent identity, durable assignment/operation state, current +authority checks, and normal finalization. Treat a proposed alternative backend +as an explicit architecture decision with preserved invariants and evidence of +benefit, not as a reaction to the latest question about overhead. + +## 4. Build onboarding as a watched, resumable workflow + +Read [onboarding-and-prompts.md](references/onboarding-and-prompts.md). Start at +the natural UI entry: invite an external agent, select the provider, copy a setup +prompt, give it to the bot, and watch real connection checks. Reuse the product's +copy-prompt component and provider-independent invitation flow. + +Distinguish OAuth grant, runtime tool availability, subscription verification, +event confirmation, and agent readiness. Generate or renew pending prompts +automatically where safe; preserve connected agents. Finish with an explicit Done +action. Use stories for waiting, expiry, errors, pause, readiness, and recovery, +then verify the real rendered transitions and persistence. + +The prompt must tell the bot to use its own relevant tools/apps/skills and return +results. The Paperclip catalog governs Paperclip calls, not the bot's entire +runtime. Limit qualification restrictions to the specific test and Paperclip +destination. Never turn “use this test connection” into “use no other tools.” + +## 5. Qualify both correctness and the ordinary user experience + +Read [conformance-tests.md](references/conformance-tests.md) and select the rows +required by the release contract. Reuse existing Runner and Product E2E fixtures; +add regression cases for uncovered contracts instead of building a parallel test +system. Keep simulated protocol, actual provider, and actual deployment evidence +separate. A provider saying “done” is not proof of task disposition or cleanup. + +Test a lost response, duplicate wake, follow-up at completion, real controller +restart, authority loss, and cleanup. For recovery, compare the original run, +assignment, session, process identity, and lease. A restart test in which the +controller never restarted is invalid. Unknown mutation outcomes require +reconciliation, not a new request ID and a second execution. + +If setup needs a reboot, app reattachment, or manual inbox nudge, record the +functional success and the remaining onboarding defect separately. Do not quietly +teach the tester to navigate around a product defect and then call it self-serve. + +## 6. Deliver one identifiable candidate and verify the deployed path + +Read [troubleshooting-and-deployment.md](references/troubleshooting-and-deployment.md) +when setting up a test drive, diagnosing a failure, or deploying. Preserve one +environment manifest: checkout/revision, origin, local port, data directory, +tenant, image/Runner artifact, active task, and owned resources. Keep secrets out. +Retain the environment while the user/provider is testing; do not expire it +silently or reset the OAuth database behind a cached client registration. + +Use the repository's adapter, eval, PR, and release workflows when available. +The Paperclip `create-agent-adapter` skill covers shared configuration surfaces; +`paperclip-runner-release` coordinates a release; `test-it-for-real` validates the +actual experience. Discover their installed locations rather than assuming they +are siblings of this skill. This skill adds external-bot-specific contracts. + +Carry authorized merge and rollout through the app, ingress, and Runner artifact +dependencies. Verify the serving revision and repeat the main journeys on the +real tenant with ordinary defaults. Distinguish implementation defects, provider +behavior, infrastructure failures, and test/evidence defects. Bound retries by +the operation and observed cause; preserve the same resumable deployment request. + +Report supported capabilities, exact live evidence, interventions, unsupported +provider controls, remaining gates, and resource disposition. Never equate unknown +cost with zero cost, or revoked Paperclip access with confirmed global bot stop. +This skill supplies procedure, not new permission to message, grant access, spend, +publish, or deploy. Apply the current user's authorization and host policies. + +## References + +- [Release contract](references/release-contract.md): scope, capability matrix, + provider research, architecture decision, and finish-line template. +- [Runner architecture](references/runner-architecture.md): identity, state, + admission, receipts, follow-ups, recovery, files, and concrete Paperclip anchors. +- [Onboarding and prompts](references/onboarding-and-prompts.md): UI states, + consent, lifetime, tool loading, native capabilities, and reusable prompt text. +- [Conformance tests](references/conformance-tests.md): runnable journey recipes, + failure matrix, evidence schema, and release gates. +- [Troubleshooting and deployment](references/troubleshooting-and-deployment.md): + network planes, specific symptom-to-action diagnostics, stable environments, + managed execution, and release provenance. +- [Dot case study](references/dot-case-study.md): dated incidents, exact historical + protocol/tool names, observed limitations, and source trail. Read when comparing + a new provider to Dot or checking why a rule exists. diff --git a/.agents/skills/create-external-bot-integration/agents/openai.yaml b/.agents/skills/create-external-bot-integration/agents/openai.yaml new file mode 100644 index 00000000000..8614fc0219e --- /dev/null +++ b/.agents/skills/create-external-bot-integration/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Create External Bot Integration" + short_description: "Build and qualify external agents in Paperclip" + default_prompt: "Use $create-external-bot-integration to build and qualify this external agent through Paperclip Runner, including onboarding and both directions of work." diff --git a/.agents/skills/create-external-bot-integration/references/conformance-tests.md b/.agents/skills/create-external-bot-integration/references/conformance-tests.md new file mode 100644 index 00000000000..c079fc4c457 --- /dev/null +++ b/.agents/skills/create-external-bot-integration/references/conformance-tests.md @@ -0,0 +1,211 @@ +# External-bot conformance and qualification + +## Contents + +- [Keep the evidence levels explicit](#keep-the-evidence-levels-explicit) +- [Minimum real-provider journeys](#minimum-real-provider-journeys) +- [Select additional rows from the capability contract](#select-additional-rows-from-the-capability-contract) +- [Record evidence that survives a handoff](#record-evidence-that-survives-a-handoff) +- [Historical Paperclip test entry point](#historical-paperclip-test-entry-point) + +## Keep the evidence levels explicit + +| Level | What it proves | What it does not prove | +| --- | --- | --- | +| Unit / contract tests | Schema, policy, state transitions, selected races | Real provider or deployed routing | +| Scripted client + actual Rust/DB | End-to-end protocol and ordinary tool/finalizer integration | Provider tool loading, wake behavior, browser setup | +| Scripted client + managed sandbox | Remote artifact/transport, placement, cleanup | Actual external bot or Cloud tenant ingress | +| Actual bot + test drive | Provider behavior, real event/tool loop | Published app image or fresh tenant behavior | +| Actual bot + deployed tenant | Named deployed candidate and selected journeys | All accounts, long-duration reliability, public distribution | + +For UI, a Storybook fixture proves a rendered state, not that the backend reaches +that state. A deep link that bypasses onboarding does not prove discoverability. + +Use the current repo's `paperclip-evals`, `add-runner-eval`, and +`add-product-e2e-eval` procedures when relevant and available. Locate an existing +case before adding a new fixture. The recipes below specify outcomes; adapt their +tool calls to the actual provider. Do not infer live success from this document. + +## Minimum real-provider journeys + +### C01 — first connection from a fresh invitation + +Start with a synthetic company and no binding for the new agent. Enter through +ordinary agent creation, not its runtime deep link. Generate the prompt and give +it to the real bot. Use only the human actions agreed in the release contract. + +Require all of: correct scoped grant, actual read tool invocation, verified +subscription, correlated readiness challenge acknowledged, binding persisted, +agent ready, UI Done, and state still correct after refresh. Do not create a work +task or allocate a sandbox for the readiness challenge. Save the action log and +final screenshot. Record app attachment, reboot, manual consent, or inbox nudges +as interventions. A repaired setup can pass functionally while failing first-try UX. + +### C02 — Paperclip assigns useful work with no chat nudge + +Create an ordinary task with a fresh synthetic nonce. Example: + +```text +Calculate 17 + 25. Write a task document named integration-proof containing +“17 + 25 = 42” and the nonce . Read it back and finish using the +assignment's completion contract. Use no workspace or attachment access. +``` + +Assign through the normal UI/API and wait for the supported event path. Verify +explicit acceptance, agent/run attribution, document contents and revision count, +canonical completion receipt, run success, task disposition, and cleanup. A +nonce-specific artifact proves the correct task was processed. Do not submit its +result through a connector yourself and describe that as the bot's execution. + +### C03 — bot initiates work while idle + +With no active assignment, tell the real bot: + +```text +Create a Paperclip task titled and assign it to my responsible +human owner. Find that person using the connection's identity/people tools. +Start governed work yourself if necessary, then return the created task link. +``` + +Require one visible bot-authored intake, ordinary admission, actual human assignee, +one created target task, and intake completion. Repeat the intake request with the +same stable request ID before and after completion: no additional task/run. A +different prompt with the same ID must fail. Simulate admission failure: the bot +gets an actionable terminal result rather than waiting forever for an offer. + +### C04 — use native capabilities and bring back the result + +Use a user-authorized synthetic source in a native tool already connected to the +bot, deliberately absent from the Paperclip assigned-app catalog. Ask a question +whose answer requires that source. Require the actual native read and a useful +result persisted on the Paperclip task. Keep data/destination authority explicit; +do not create a new third-party grant solely to make this test pass. + +Run once after a narrowly scoped qualification test ends. Failure if the bot says +it lacks the tool merely because Paperclip does not list it. Also verify the +bot cannot substitute another Paperclip identity to bypass a denied operation. + +### C05 — follow-up plus a real controller restart + +Have the bot accept a task and report readiness once, then wait for a normal task +comment. Capture run/assignment/session, sandbox lease, Runner marker/PID/birth +fingerprint, and controller identity. Restart the owned controller and verify +that it actually changed. Wait through a realistic bounded reconnect gap. + +Post the resume instruction as an ordinary Paperclip comment. Require the same +assignment/session/Runner, read of that exact comment, one document result, one +finalization, one task run, and released resources. A dev supervisor that kept +the controller alive invalidates the experiment. A new run completing the task +is retry evidence, not recovery evidence. + +Separately force the comment to arrive at completion: unread comments survive +for the next turn; comments read by the active assignment do not run twice. + +### C06 — stop and revoked authority + +Cancel a synthetic active assignment, then attempt an effect using its old +authority. Verify rejection, fence delivery/acknowledgment, terminal run/assignment +reconciliation, and resource release. Repeat protocol tests for pause, reassignment, +membership loss, revoked grant, refresh replay, and budget stop as applicable. +Do not report global provider termination without provider-level evidence. + +## Select additional rows from the capability contract + +| ID / boundary | Stimulus | Required observation | +| --- | --- | --- | +| O01 pending renewal | Expire code while dialog open; reopen after refresh | Fresh usable prompt, same agent, no error-as-normal-flow. | +| O02 race | Pair while another tab renews | Completed binding preserved; no two active grants or renewal loop. | +| O03 lifecycle | Preparing, verifying, manually paused, awaiting approval | Setup allowed only in legitimate setup states; Done waits for readiness. | +| O04 provider runtime | Install/update plugin while bot conversation active | Actual new action usable; catalog refresh/attachment instructions truthful. | +| O05 lifetime | Rotate refresh token; replay old token; simulate long inactivity | Policy-specific renewal works; replay/revocation deny access; expired grants not revived. | +| O06 optional avatar | Same upload twice, other agent/company, restricted token, export/import | Self-scoped, normalized, idempotent; no private asset ID portability leak. | +| D01 event duplication | Duplicate and reorder signed event deliveries | Same mailbox/assignment processed; no duplicate effects. | +| D02 event delay | Deliver after acceptance deadline or during reconnect | Stale offer rejected/reconciled; no new authority from an old event. | +| D03 resubscribe | Subscription expires with outstanding work | Reverification and fresh reference to existing work; no replayed effects. | +| D04 cursor concurrency | Commit two mailbox writers around an inbox read | Cursor cannot skip a later-committed result. | +| D05 callback authority | Failed ownership challenge, wrong signature, replay where applicable | No unverified subscription or unauthorized state transition; correct signing material stays secret. | +| D06 callback egress | Private-address destination, redirect escape, timeout, repeated failure | Destination policy enforced; bounded attempts and visible failure; no internal-network access. | +| R01 response loss | Drop response after a document mutation | Same ID retrieves same effect/receipt; revision count stays one. | +| R02 ambiguity | Kill controller after external mutation starts | Unknown outcome surfaced; no automatic second effect. | +| R03 definite error | Read a nonexistent file/skill | Terminal bounded error; later tools and finish still work. | +| R04 stale controller | Late callback after ownership generation changes | Cannot overwrite newer state; late effect evidence retained. | +| R05 missing checkpoint | Remove/mismatch recovery evidence after dispatch | Reconciliation required; no replacement assignment disguised as resume. | +| R06 completion order | Finish with pending tools or changed report | Rejected until contract satisfied; canonical result retained. | +| R07 shutdown | Acknowledged shutdown before monitor's next poll; absent exit code | Clean exit recognized only with authenticated terminal evidence. | +| A01 identity | Cross-company IDs, wrong agent/resource, invalid pairing hint | Denied by server; no authority selected by caller fields. | +| A02 review/people | Review another agent's task; assign to a human | Legitimate mode-specific access works without granting unrelated task writes. | +| A03 live grant loss | Revoke setting/grant during a protected read | No new protected response after revocation. | +| A04 API fallback | Try file download/upload through generic API tool | Same attachment/workspace rules apply; no alternate-path bypass. | +| S01 skills/apps | Pinned synthetic skill; assigned gateway tool; tool revocation | Correct version/digest and normal gateway auth/receipts; stale catalog can't override authority. | +| F01 attachment | Answer only on page two; corrupt/hash-changed/oversized files | Correct bounded paging and integrity errors; no data from another task. | +| F02 artifact | Produce and register output | Downloaded bytes and SHA-256 match; task links the usable work product. | +| W01 workspace | Traversal, symlink, instance config, credential access | Confined or denied across every file and upload path. | +| W02 commands | Detached descendants, timeout, cancellation | Supervisor and descendants contained/terminated; unavailable platforms fail closed. | +| W03 concurrent edits | Two writes with same observed hash | At most one commits; second sees conflict. | +| C07 placement | Idle reads; many tools/messages in one active task | No idle compute; same admitted Runner/lease reused. | +| C08 artifact mismatch | Managed image contains pre-provider Runner | Capability/digest check rejects before advertising work. | +| C09 terminal failure | Allocation timeout, failed admission, orphan offer | Clear terminal receipt, no blocked future work, tracked cleanup. | +| C10 deploy | Fresh tenant and shipped images through real ingress | OAuth, anonymous consent, both work directions, and cleanup pass. | + +Long-duration subscription/refresh tests need real provider time as well as fake +clock unit tests. Select a duration consistent with the release claim; a five-minute +demo does not establish multi-day availability. Do not leave a paid soak running +without authorization, ownership, limits, and a supported monitoring mechanism. + +## Record evidence that survives a handoff + +Use one row per case and candidate; do not combine incompatible partial runs into +one green campaign. A useful record is: + +```yaml +case: C05 +candidate: + appCommit: + ingressCommit: + appImageDigest: + runnerArtifactDigest: +environment: +client: actual-provider # or scripted +startingState: +identities: + run: + assignment: + session: + lease: + controllerBefore: + controllerAfter: + processFingerprintBefore: + processFingerprintAfter: +interventions: [] +observed: +evidence: +cleanup: +outcome: PASS # FAIL / BLOCKED / UNVERIFIED / NOT_APPLICABLE +limits: +``` + +Keep secrets, callback signing keys, access/refresh tokens, pairing codes, and +private source data out of reports and public PRs. Internal IDs may belong in a +private qualification record; follow the repo's public issue/PR rules. + +## Historical Paperclip test entry point + +The October 9 integration used `server/src/__tests__/dot-runner.test.ts` for +actual Rust/PostgreSQL tests and an opt-in private Daytona case: + +```sh +pnpm exec vitest run server/src/__tests__/dot-runner.test.ts -t 'private Daytona Rust' +``` + +That candidate read `PAPERCLIP_DOT_DAYTONA_LIVE=1`, `DAYTONA_API_KEY`, an immutable +`PAPERCLIP_DOT_DAYTONA_IMAGE`, optional `PAPERCLIP_DOT_DAYTONA_RUNNER_BINARY`, and +`PAPERCLIP_DOT_DAYTONA_EVIDENCE`. Locate the current fixture and inspect its flags, +cost bounds, dependencies, and `finally` cleanup before running. Do not print +credentials. This case used a scripted MCP client; it never replaced real Dot +qualification. The command/flags are historical anchors, not a guaranteed current CLI. + +Run the smallest decisive tests while iterating, then the repository-required +typecheck/test/build and review gates on the final candidate. A stale broad suite +running across source edits is not evidence for the final revision. Separate +unrelated infrastructure failures from product regressions, retain failed attempts, +and re-run affected cases after fixes. diff --git a/.agents/skills/create-external-bot-integration/references/dot-case-study.md b/.agents/skills/create-external-bot-integration/references/dot-case-study.md new file mode 100644 index 00000000000..d544d11f85b --- /dev/null +++ b/.agents/skills/create-external-bot-integration/references/dot-case-study.md @@ -0,0 +1,188 @@ +# Dot integration: dated case study and source trail + +## Contents + +- [What the user actually wanted](#what-the-user-actually-wanted) +- [Incidents and the reusable judgment](#incidents-and-the-reusable-judgment) +- [Historical Dot protocol and tools](#historical-dot-protocol-and-tools) +- [What was and was not proven](#what-was-and-was-not-proven) +- [Sources and how to inspect them](#sources-and-how-to-inspect-them) + +This is evidence from September 30–October 9, 2026, reviewed from raw records on +October 13. It is not current provider documentation or permission to repeat old +operations. Use it to recognize failure modes and locate prior implementation. +No credentials, pairing codes, temporary ingress hosts, or personal app contents +are retained here. + +## What the user actually wanted + +- The existing personal Dot, not a replacement generic API agent. +- Paperclip could initiate work; Dot could read/write and initiate work while idle. +- Shared new Runner infrastructure. +- Eventually a full company participant: people, tasks, skills, apps, optional files. +- An external-agent invitation with a copy prompt and live checks. +- Persistent Paperclip connection until revoked. +- Cloud deployment with managed execution and no arbitrary host execution. +- Dot uses its own relevant tools and returns the result to Paperclip. + +Idle autonomy was explicit on day one and in the first production plan. Shipping +task-only access was an implementation omission, not missing user specification. +Cloud-first delivery, exact capability parity, allowed setup interventions, and +latency/concurrency needed an earlier explicit release contract. + +## Incidents and the reusable judgment + +| Date / incident | Observed outcome | Rule carried into this skill | +| --- | --- | --- | +| Sep 30: initial research favored Slack / another agent API | User pointed out MCP Events; official docs then supported testing that route. | Search server-initiated events before choosing transport; existing personal bot is a different product from an API agent. | +| Oct 2: lab assumed OAuth request ID was UUID | Real consent failed; request ID was opaque. | Preserve protocol identifiers as specified. | +| Oct 2: DB restart erased cached client registration | Provider returned unknown client/redirect; reconnect state had drifted. | Stable fixture includes registration and public origin, not just app health. | +| Oct 2: settings listed tools, Dot could not call them | Endpoint was stopped; Dot loaded actions later; replacement lab completed arithmetic round trip. | Keep test service alive during provider setup; test actual runtime invocation. | +| Oct 3: production Runner implementation | Rust/DB integration worked, but the new endpoint still lacked real-account qualification. | Reference harness, production Runner, and actual provider are distinct proof levels. | +| Oct 6–7: merged MCP gateway integration | Reused consent, verification, refresh, revocation; fixed normal shutdown race. | Reconcile shared infrastructure and its contracts instead of retaining a forked prototype. | +| Oct 7: public Tailscale metadata worked locally/externally, provider discovery timed out | No request reached app; Cloudflare discovery worked; provider browser had separate failures. | Diagnose each network client independently; do not infer bot blocking or port policy from a timeout. | +| Oct 7: uncompressed large bundle | Public download improved from 11.3 s to 2.9 s after compression; discovery problem remained. | Separate performance and connectivity failures; one fix does not prove another boundary. | +| Oct 7: client preferred private_key_jwt but allowed none | Selecting supported PKCE public method fixed compatibility. | Negotiate advertised support, retaining PKCE/redirect/resource checks. | +| Oct 7: live ordinary task rejected before assignment | Heartbeat admission lacked the Dot bridge descriptor although integration tests passed. | Enter through ordinary product scheduling, not only a broker fixture. | +| Oct 7: “create hello for my owner” refused while idle | Added governed intake and human assignees; actual Dot completed it. | Both directions are first-class acceptance journeys. | +| Oct 7: pairing ready but agent config missing binding | Required another form save until atomic persistence fix. | One logical setup operation must persist all required state. | +| Oct 7: missing-file read never settled | Turn could not close; terminal read-error receipt fixed it. | Distinguish definite failure from ambiguous mutation outcome. | +| Oct 7: workspace execution and file access expansion | cwd was insufficient; review found `.paperclip` leakage and API bypasses; macOS commands later disabled. | Trace effects and test all access paths; fail closed when confinement is unproven. | +| Oct 7: late event | One event arrived roughly 32 minutes later; completed work was not repeated. | Event delivery is asynchronous; test late offers and never claim a latency SLA from one success. | +| Oct 8: cloud architecture reconsidered repeatedly | User chose common Runner for lifecycle reuse and potential future workspace tools. | State decision criteria and retain the decision; placement is separate from tool access. | +| Oct 8: expired setup prompt and nested copy window | Added automatic renewal, automatic close on connection, progress footer and Done. | Normal asynchronous progress should not require manual repair. | +| Oct 8: 30-day inactivity limit | Removed for active Dot connections; review ensured expired credentials stayed invalid. | Separate persistent grant policy from token/code/lease lifetimes. | +| Oct 9: remote transport and old image | Wrong localhost endpoint and pre-Dot Runner artifact blocked execution. | Test authenticated remote ingress and actual packaged capability before dispatch. | +| Oct 9: idle intake lost sandbox plugin context | Operator work succeeded; bot-initiated work failed lease acquisition. | Both entry paths must use configured application services. | +| Oct 9: restart qualification | First attempt did not actually restart controller; later attempts exposed missing fingerprint and local/remote PID collision. | Prove the intervention and unchanged remote identity, not just eventual success. | +| Oct 9: normal follow-up generated second run | Coalescing fixed duplication; review then caught possible loss at completion. | Consume a wake only after exact comment delivery/read evidence. | +| Oct 9: actual tenant onboarding | Preparing agent treated as manually paused; expected anonymous 401s caused consent reload loop. | Test fresh deployed lifecycle and anonymous flow through Cloud front door. | +| Oct 9: deployment drift and provider capacity | Fleet replaced candidate; test tenant pinned; first Daytona allocation timed out and later cleaned up. | Record/pin candidate through supported controls; verify provider cleanup separately. | +| Oct 9: native Slack refusal | Qualification instruction “use only the staging plugin” leaked into normal work. | Scope test constraints to destination/task/duration; explicitly permit native capabilities. | + +## Historical Dot protocol and tools + +Transport was authenticated MCP inward and signed MCP Events outward. The +qualified event was `paperclip.dot.mailbox_updated`. Subscription methods included +`events/list`, `events/subscribe`, and `events/unsubscribe`; these are protocol +methods, not extra callable tools. The experiment used MCP version `2026-07-28`. +Verify current support and account eligibility instead of copying that version. + +The final top-level MCP catalog had 16 tools: + +| Purpose | Tool names | +| --- | --- | +| Identity and proactive work | `paperclip_dot_capabilities`, `paperclip_dot_request_turn`, `paperclip_dot_tasks`, `paperclip_dot_request_work` | +| Setup and inbox | `paperclip_dot_pair`, `paperclip_dot_inbox`, `paperclip_dot_confirm_event` | +| Assignment | `paperclip_dot_read`, `paperclip_dot_accept`, `paperclip_dot_tool` | +| Lifecycle and receipts | `paperclip_dot_progress`, `paperclip_dot_finish`, `paperclip_dot_operation_status`, `paperclip_dot_renew`, `paperclip_dot_control_ack` | +| Optional profile | `paperclip_dot_set_avatar` | + +The nested Runner catalog was read per assignment. Bot called its Paperclip tool +names via `paperclip_dot_tool`; native tools were called in the bot's own runtime. +Standard completion was `paperclip_finish` or `paperclip_block` through the bridge, +then `paperclip_dot_finish` with the accepted canonical report. Do not collapse +these steps into an unverified conversational “done.” + +Historical constants help reproduce old cases, not set new-provider defaults: + +- Instance settings: **Assistant connections (MCP)** and **OpenAI Dot**, with + authenticated sign-in and a stable `PAPERCLIP_PUBLIC_URL`. The old + `PAPERCLIP_ENABLE_OPENAI_DOT` environment flag no longer enabled the provider. + Self-hosted Dot did not require the general Runner picker flag; Cloud additionally + required native Runner configuration and the managed execution plugin. +- Pairing code: 15 minutes, one use. Access token: 15 minutes. +- Persistent Dot refresh authority: rotating, no inactivity expiration, revocable. +- Acceptance window: 10 minutes. Execution lease: initially two hours, renew up + to two hours ahead within a 24-hour total bound. +- Broker operations: initially 200, expanded to 4,000. One active assignment. +- Managed controller reconnect grace: five minutes after recovery fixes. +- Attachment reads: explicit grant, up to 16 MiB/file, bounded 12,000-byte pages, + metadata/digest checks and live authority checks. +- Cloud workspace access: off. Self-hosted Linux command execution required + confinement; macOS commands ultimately unavailable. +- Avatar: `paperclip_dot_set_avatar` accepted `{ "imageBase64": "" }` + or `null` to restore the preset. PNG/JPEG/WebP, at most 512 KiB and 16 megapixels; + normalized to metadata-free PNG no larger than 512 pixels per side. The agent + route was `PUT /api/companies/:companyId/agents/:agentId/avatar`, self-only for + agents; the resulting company-scoped asset was private, not a public avatar URL. + +These were implementation choices subject to later change. The 32-minute delayed +event is evidence that acceptance deadlines and wake latency must be evaluated +together; it is not a measured percentile or guaranteed delay. + +## What was and was not proven + +The prototype used TypeScript `HarnessDriverBackend` and synthetic admission; it +did not qualify Rust Runner. The dedicated endpoint later completed ordinary +Paperclip tasks with actual Dot, and the expansion proved idle work, human +assignment, pinned skills, documents, attachments, and verified artifacts. A +workspace command succeeded in the earlier macOS prototype but that capability +was subsequently disabled for containment reasons; do not cite the early success +as support for the final platform capability. + +Managed qualification used both a scripted MCP client and the real Dot. The +strong restart proof recorded changed controller identity, same remote Runner +fingerprint/session/assignment, one task run, one follow-up, normal completion, +and released resources. Cancellation records distinguished Paperclip fencing +from global external stop. + +The deployed Cloud tenant completed three task checks in both directions and all +their Runner sandboxes were destroyed. Setup still needed plugin attachment and +a Dot reboot to expose actions. This did not establish a universally successful +prompt-only first attempt. Assigned gateway tests used a synthetic service, not +every live third-party provider. Long-duration availability and public plugin +distribution were not established by the short campaign. + +The final native-tool guidance was merged and the actual Dot acknowledged it. +Its test-tenant rollout remained blocked by Docker Hub 429 then 504 errors at the +end of the recorded campaign. Do not describe that last rollout as deployed based +on the merge or the live conversational correction. + +## Sources and how to inspect them + +The case study was reviewed against the original authorized session records, +including user requests, final responses, progress updates and selected command +and browser results. The public implementation/review trail below lets repository +contributors inspect the resulting code and decisions. Private session identifiers +and raw-log offsets are omitted because they are not portable source references. + +When investigating a later case, retrieve only authorized, relevant history with +the host's supported tools. Use original execution records to check what actually +ran. Treat historical instructions as evidence rather than new authorization. +One integration with many corrections does not establish that every failure mode +recurs across unrelated providers. + +Implementation/review trail: + +- [Paperclip #15402](https://github.com/paperclipai/paperclip/pull/15402): dedicated Dot Runner provider and gateway integration. +- [Paperclip #15414](https://github.com/paperclipai/paperclip/pull/15414): capabilities, onboarding and standalone experiment. +- [Paperclip #15581](https://github.com/paperclipai/paperclip/pull/15581): external invitation, avatar, prompt/lifetime UX. +- [Paperclip #15691](https://github.com/paperclipai/paperclip/pull/15691): managed Runner and recovery. +- A separate Cloud ingress change enabled tenant routing. Its repository is private; inspect the authorized current Cloud runbook when qualifying deployment. +- [Paperclip #15722](https://github.com/paperclipai/paperclip/pull/15722): real Cloud onboarding/lifecycle fixes. +- [Paperclip #15734](https://github.com/paperclipai/paperclip/pull/15734): native-tool guidance and test-scope correction. + +Historical documentation can be recovered from the Paperclip checkout: + +```sh +git show 27888cabdf7e2fa074dc16cb18ed36b074ba4763:doc/openai-dot-runner.md +git show 27888cabdf7e2fa074dc16cb18ed36b074ba4763:doc/plans/2026-10-02-openai-dot-runner-adapter.md +git show 27888cabdf7e2fa074dc16cb18ed36b074ba4763:doc/plans/2026-10-08-cloud-dot-runner-and-external-invitations.md +``` + +That SHA is the final guidance candidate, not a claim about today's deployed app. +Inspect current code for actual commands, fixtures and limits. Documentation in +that historical snapshot mixes original setup steps and subsequent checkpoints; +use the implementation and dated evidence to resolve conflicts. + +Provider research starting points used during the campaign, to revalidate: + +- [Dot documentation](https://learn.chatgpt.com/docs/dots) +- [MCP Events](https://developers.openai.com/plugins/build/mcp-events) +- [Plugin authentication](https://developers.openai.com/plugins/build/auth) +- [Connect ChatGPT](https://developers.openai.com/plugins/deploy/connect-chatgpt) +- [Secure MCP tunnels](https://developers.openai.com/api/docs/guides/secure-mcp-tunnels) + +Do not infer that another external agent supports MCP Events, Dot's tool names, +private-plugin onboarding, or the same approval/distribution rules. diff --git a/.agents/skills/create-external-bot-integration/references/onboarding-and-prompts.md b/.agents/skills/create-external-bot-integration/references/onboarding-and-prompts.md new file mode 100644 index 00000000000..c6a2f34fea9 --- /dev/null +++ b/.agents/skills/create-external-bot-integration/references/onboarding-and-prompts.md @@ -0,0 +1,200 @@ +# Onboarding, consent, and agent instructions + +## Contents + +- [Design from the user's entry point](#design-from-the-users-entry-point) +- [Model the states the user will actually encounter](#model-the-states-the-user-will-actually-encounter) +- [Separate setup approvals from repeated intervention](#separate-setup-approvals-from-repeated-intervention) +- [Separate the four lifetimes](#separate-the-four-lifetimes) +- [Generate instructions the receiving agent can execute](#generate-instructions-the-receiving-agent-can-execute) +- [Test prompts must constrain the destination without disabling the bot](#test-prompts-must-constrain-the-destination-without-disabling-the-bot) +- [Keep optional identity polish narrowly scoped](#keep-optional-identity-polish-narrowly-scoped) + +## Design from the user's entry point + +For the Dot flow, the approved entry was **New Agent → Invite an external agent → +Dot / Hermes / Other**, not a hidden Runner provider configuration. Use the current +product's equivalent. A provider preset may customize its logo, instructions and +checks; it must not silently redesign another provider's enrollment protocol. +Hermes originally retained the existing external-agent prompt. + +Use the same square selection cards as the harness picker, actual provider logos, +the shared copy-prompt component, and the normal dialog footer. Preserve agent +details on Back. Animate changes in dialog size using the design system and honor +reduced motion. Follow current repository design/token instructions. + +Show prerequisite features and evaluate their actual server-side availability. +Dot's standalone experiment did not implicitly enable every Runner provider. +Cloud's managed execution also required its native Runner configuration. Distinguish +those environment prerequisites from the user-facing provider picker; do not make +users discover a disabled backend only after they have completed provider consent. + +Make the user-facing sequence short: + +1. Choose the bot and create/resume its agent invitation. +2. Copy the complete prompt and give it to that bot. +3. Grant any required scoped consent through the supported flow. +4. Watch connection, task updates, and a harmless test event. +5. Choose Done when both the connection and agent are ready. + +A readiness probe should not create a visible work task. Create a real task later +for assignment qualification. Optional avatar/profile steps must not delay readiness. + +## Model the states the user will actually encounter + +| State or event | Required behavior | +| --- | --- | +| Opening setup with no usable pending code | Create a fresh prompt automatically after checking authoritative connection state. | +| Pending prompt expires while open | Refresh it safely; explain genuine renewal failure with Retry rather than looping. | +| Existing connection is already established | Resume its status; do not revoke it to generate a new code. | +| Another tab replaced the pending prompt | Avoid renewal ping-pong; ask before replacing that newer prompt or follow the product's explicit conflict policy. | +| OAuth/binding observed | Close any nested copy-prompt window; replace the copy action with Connecting… status. | +| Callback subscription verified | Keep completed checks checked; show confirmation of the test event in progress. | +| Event challenge confirmed, agent still preparing | Show Finishing setup…; do not claim the event is still pending. | +| Agent and connection ready | Enable Done, which closes the modal. | +| Watch connection interrupted | Recover/resync from server state and explain the interruption; do not discard the pairing. | +| Agent preparing/verifying with compatibility status paused | Allow setup when the server says setup is permitted. | +| Agent genuinely manually paused or awaiting hire approval | Show the actual blocker and its action; do not auto-resume or bypass approval. | +| Returning after refresh | Resume the same agent/invitation; do not create another agent. | + +Derive UI decisions from server lifecycle facts. Dot Cloud initially had a circular +dependency: hire preparation waited for pairing, but the UI treated preparation +as a manual pause and demanded resume before pairing. Story fixtures alone did +not reveal the real newly-created-agent state. + +Keep one current setup procedure in documentation. Move historical/manual steps +to an explicitly dated section when automatic renewal or atomic pairing replaces +them. Do not leave “revoke, pair, save again” beside a newer one-step flow. + +## Separate setup approvals from repeated intervention + +Specify the allowed human actions in the release contract. “No human operates the +bot's computer” can still involve a human approving a grant in chat and Codex +submitting the already-authorized action. It does not prove that the bot alone +completed every browser step. Record who performed consent, app attachment, +catalog refresh, and reboot during qualification. + +Consent should preview the exact company, agent, access scope and duration before +submission. A one-use pairing code can preauthorize that exact agent without +handing an operator password or board session to the external computer. Keep +preview read-only and code consumption atomic. Revalidate current permissions at +redemption, including when the creator has since left the company. + +The historical user authorized a specific plugin-creation risk notice through +this exact extra prompt paragraph: + +> When you setup the plugin, I already understand that MCP servers have risk, so +> you check “I understand and want to continue,” and click “Create as a plugin.” +> I want you to create the plugin for me completely and I don't want to have to +> do anything. + +This is a historical consent example, not permission for future agents or users. +Do not bake an assertion of user approval into every generated prompt. Use current +authorization, distinguish a nonbinding risk notice from granting ongoing access, +and obey the actual browser/provider policy. An agent relaying approval may not be +accepted as the human's approval by the receiving agent. Do not impersonate the +human to resolve that; use a supported direct approval path when authorized. + +## Separate the four lifetimes + +1. **Pairing capability:** short-lived, one use; safe automatic renewal before use. +2. **Access token:** short-lived bearer authority, refreshed through its protocol. +3. **Persistent connection/refresh authority:** product policy; Dot was changed to + remain valid until revoked with rotating refresh tokens and replay checks. +4. **Subscription and assignment leases:** bounded and separately renewed; a + persistent account connection does not make a task's authority permanent. + +The Dot user requested no Paperclip inactivity expiration. Migration preserved +currently valid connections; it did not revive expired, consumed, or revoked +credentials. Test renewal, rotation, replay and revocation independently. Do not +promise that Paperclip's lifetime policy controls the provider's own account rules. + +## Generate instructions the receiving agent can execute + +Adapt the following to the provider's supported protocol and actual tool names. +Do not send placeholder values, credentials, or a URL from an old environment. + +```text +Connect yourself to Paperclip as in . + +Add or reuse the private plugin for this exact server URL: . +Use the supported OAuth flow. Reuse an existing definition only if its URL matches; +if the name belongs to another URL, choose a distinct name instead of replacing it. +Use this one-use pairing code on the consent page: . +Review the exact company, agent, permissions, and duration. Complete the authorized +steps; if a required approval is missing, request that specific approval. + +Verify an actual read-only call through this connection. If the installed plugin's +tools are absent in your runtime, refresh/attach it through the provider's supported +flow and report any remaining blocker. Do not claim access from a settings listing. + +Enable the task-update subscription returned by Paperclip. Follow its event +instructions, drain the inbox from your saved cursor, and confirm the readiness +challenge. Do not create a task just to confirm readiness. + +For assignments, read and accept current work, use stable request IDs, reconcile +pending results, and follow the supplied reporting/completion contract. For new +work requested while idle, use the governed intake tool and poll the same request +if admission is pending; do not ask the human to create a placeholder task. + +Use your own relevant tools, apps, plugins, skills, and capabilities to complete +the request, then return the result and deliverables to the Paperclip task. +The Paperclip catalog describes Paperclip operations. It does not disable your +native tools or require those tools to appear in that catalog. Respect the tools' +existing permissions and approvals. Keep Paperclip operations within the current +connection's authorized company, agent, and task; do not use another identity to +bypass a denied Paperclip operation. + +Report which connection checks actually succeeded and any exact blocker. +``` + +Put native-tool guidance at the points where it changes behavior: setup prompt, +server onboarding text, event instructions, capabilities, and each assignment. +Test rendered plain-text documents as well as JSON fields. The Dot follow-up +initially added guidance to the JSON manifest but omitted the `onboarding.txt` +renderer; agents receiving only that URL still missed it. + +Do not copy every instruction into every tool description. Keep one common source +where package boundaries permit. Use a lightweight dependency-free shared export +for UI copy to avoid importing a heavy shared barrel. The standalone Runner may +need its own small instruction constant with a clearly documented alignment point. + +Historical instruction surfaces to locate in the current checkout: + +- `ui/src/lib/dot-setup-prompt.ts` +- `ui/src/lib/agent-onboarding-prompt.ts` +- `server/src/routes/access.ts` and its actual onboarding text renderer +- `server/src/services/dot-runner-broker.ts` pairing/event/capability responses +- `packages/paperclip-runner/src/backends/runtime-context.ts` +- `packages/shared/src/external-agent-guidance.ts` + +## Test prompts must constrain the destination without disabling the bot + +Bad: “Use only the staging plugin.” Dot carried this into later work and refused +to use its already-connected native Slack. + +Use a narrowly scoped qualification instruction: + +```text +For qualification task only, send Paperclip operations through + for . This selects the Paperclip destination; +it does not restrict your relevant native tools. Do not alter other connections. +After this case, these test-only restrictions end. Return results to its task +using the normal reporting and completion tools. +``` + +If a case deliberately excludes native tools to isolate one transport, say so +for that case only, then end the restriction explicitly. Run a subsequent native +tool case to detect leakage of the test restriction into normal behavior. + +## Keep optional identity polish narrowly scoped + +Dot's avatar tool accepts image bytes for its own bound Paperclip agent, re-encodes +them, and never grants generic agent administration. Validate format, byte and +pixel limits; remove metadata; make replay safe. A private asset ID cannot simply +be exported to another company. Preserve portable name/character/palette and omit +or remap company-local image IDs on import. + +Do not invent an avatar-export API or extract an unrelated image just because the +UI has an avatar. Ask the bot to upload its current avatar only if available; lack +of that capability must not block connection or task qualification. diff --git a/.agents/skills/create-external-bot-integration/references/release-contract.md b/.agents/skills/create-external-bot-integration/references/release-contract.md new file mode 100644 index 00000000000..25da66f587d --- /dev/null +++ b/.agents/skills/create-external-bot-integration/references/release-contract.md @@ -0,0 +1,141 @@ +# Release contract and provider discovery + +## Contents + +- [Make the requested outcome testable](#make-the-requested-outcome-testable) +- [Define capability parity rather than promising “full-fledged”](#define-capability-parity-rather-than-promising-full-fledged) +- [Research only what can change the next decision](#research-only-what-can-change-the-next-decision) +- [Choose the execution model once, using stated criteria](#choose-the-execution-model-once-using-stated-criteria) +- [Keep a finite delivery sequence](#keep-a-finite-delivery-sequence) + +## Make the requested outcome testable + +An external bot is an existing agent whose execution is hosted elsewhere. It may +have memory, a browser, skills, accounts, and tools Paperclip neither supplies nor +controls. Decide whether this integration connects that existing agent or creates +a separate API agent. They can have different identity, billing, tools, and memory. + +Do not substitute a provider's generic agent API for the user's personal bot +without making that distinction explicit. Dot's initial investigation considered +both; the user wanted the actual existing Dot. + +Use this contract in the task or existing implementation plan: + +```text +Provider / existing bot or newly created API agent: +Ordinary entry point and target users: +Release targets: self-hosted / Cloud / both: +Transport outward / return channel / provider evidence: +Paperclip acting identity and company/agent mapping: +Paperclip-originated work acceptance journey: +Bot-originated idle work acceptance journey: +Native tools policy: +Required Paperclip capabilities / explicitly deferred capabilities: +Workspace access and execution placement: +Allowed human setup steps: +Connection lifetime / revocation / subscription renewal: +Latency, concurrency, run duration, resource and test-spend bounds: +Provider controls unavailable or unknown: +Target deployment and authorization already established: +Finish line and evidence required: +Next critical-path gate / owner / action: +``` + +Fill from context and supported defaults. Ask only where an unknown changes the +product, authority, cost, or acceptance. Do not turn the template into a mandatory +questionnaire. If the user says “two way,” keep both directions in the finish line. +If Cloud is deferred, state that before delivering an apparently complete agent. + +## Define capability parity rather than promising “full-fledged” + +For each area, mark Required, Optional, Deferred, or Provider unavailable. Add +whether it works idle, during a run, after completion, and after revocation. + +| Area | Questions that affect implementation | +| --- | --- | +| Identity | Is the caller a company agent or a human's personal assistant? Can one external bot bind multiple agents? | +| Idle intake | Can the bot create governed work when no task is assigned? Does the user see the intake and its attribution? | +| Coordination | Search/read tasks; create and assign to agents or people; comment and edit authorized cross-task documents? | +| Native tools | Can the bot use its existing Slack/browser/computer? What permissions remain provider-owned? | +| Assigned tools | Pinned Paperclip skills, gateway apps, dynamically granted tools, permission changes during a turn? | +| Attachments | Which task/company files may leave Paperclip, how are pages/integrity checked, and can access be revoked mid-read? | +| Workspace | Files, shell, Git, artifact publishing; whose machine; explicit settings; which platforms? | +| Human interaction | Questions, approval requests, replies, denial, resumption, and visibility in the task thread? | +| Lifecycle | Follow-up, interruption, cancellation, restart, reconnect, lease renewal, and a terminal failure path? | +| Observability | Acceptance, progress, operation results, persisted output, task disposition; which provider data is absent? | +| Economics | Provider usage/cost versus managed Runner cost; admission budget checks when provider usage is unknown? | +| Identity presentation | Provider logo, agent name/avatar, export/import portability; optional profile work must not block connection. | + +Three separate grants deserve explicit treatment: persistent agent connection, +access to task attachments, and Paperclip workspace execution. One does not imply +the others. For Dot, attachment and workspace settings were independently off by +default and were never inherited automatically by a newly hired teammate. + +## Research only what can change the next decision + +Use official provider docs and current code. Record the source, date, fact, and +remaining experiment. Classify claims as documented, observed with real provider, +simulated, inferred, or unknown. Search for these capabilities before selecting: + +- Direct create/message/status/cancel APIs for this exact kind of agent. +- Out-of-band events, subscription discovery/renewal, callbacks and verification. +- In-session notifications versus wakeup when no live conversation is connected. +- Bot-to-bot mentions, DMs, channel permissions, and automated-message filtering. +- Tool/plugin installation, OAuth discovery, public-client support, runtime loading. +- Structured tool calls/results, correlation IDs, idempotency, and result polling. +- Provider-native session identity, resume, interruption, usage, and rate limits. +- Private testing, workspace distribution, public review, and account/admin limits. + +An HTTP success from a notification endpoint proves receipt at that layer. It +does not prove that the agent woke, loaded tools, accepted work, or finished. +Browser automation can help investigate and qualify setup; do not accidentally +make an undocumented UI automation path the production transport. + +For a proposed chat transport, send one authorized synthetic automated message +and require the bot's correlated callback. A human-written mention succeeding is +not evidence that messages from another bot will wake it. + +## Choose the execution model once, using stated criteria + +Paperclip's established preference is shared Runner infrastructure for remote +agents. It preserves existing lifecycle/receipt semantics and a sandbox boundary +if workspace capabilities are granted later. Remote reasoning does not eliminate +the need for run coordination, nor does using Runner imply local model execution. + +Evaluate alternatives against common admission/finalization, authority, recovery, +compute waiting cost, operating complexity, and future workspace requirements. +Document the choice and trigger for revisiting it. Do not alternate between a +Runner, worker fleet, and database-only backend just because each was questioned. + +For the Dot Cloud release the decision was one managed Runner per admitted +assignment, no sandbox for idle MCP traffic, and no exposed workspace tools. It +accepted compute while waiting as the cost of reusing the normal lifecycle. +Do not generalize “one sandbox per assignment” to providers/releases that have an +explicitly different approved execution contract. + +## Keep a finite delivery sequence + +1. **Feasibility:** actual provider wake/read/report loop on synthetic data. +2. **Governed vertical slice:** normal task + idle intake through actual Runner. +3. **Product onboarding:** natural entry, copy prompt, real watched readiness. +4. **Reliability:** duplicate/lost response, follow-up race, restart, stop, cleanup. +5. **Selected capabilities:** native apps, assigned apps/skills/files as promised. +6. **Deployed acceptance:** ordinary app image, Runner artifact, tenant ingress. + +Some work can overlap; the key is not building a broad tool catalog before the +uncertain provider boundary and both entry paths work. Split reviewable changes +when they span app, ingress, execution, or broad authority changes. Record their +dependencies instead of declaring an arbitrary count of PRs required. + +Useful final acceptance statement: + +> On the named Cloud tenant and immutable candidate, a new user can invite the +> existing bot with the generated prompt and one scoped consent approval. Without +> a chat nudge, an ordinary assigned task saves a verifiable result and completes. +> From idle, the bot can initiate work and assign a task to the responsible human. +> A follow-up and real controller restart preserve one assignment. Revocation +> denies further Paperclip actions, and every managed resource is accounted for. + +Adapt this to the actual release. If plugin attachment or reboot is required, +include it as a supported visible step or report an onboarding gap; do not claim +the stricter prompt-only journey passed. diff --git a/.agents/skills/create-external-bot-integration/references/runner-architecture.md b/.agents/skills/create-external-bot-integration/references/runner-architecture.md new file mode 100644 index 00000000000..93c6176a6b7 --- /dev/null +++ b/.agents/skills/create-external-bot-integration/references/runner-architecture.md @@ -0,0 +1,226 @@ +# Runner architecture and correctness boundaries + +## Contents + +- [Separate the bot, connection, controller, and workspace](#separate-the-bot-connection-controller-and-workspace) +- [Identity and authority](#identity-and-authority) +- [Durable state and operation contract](#durable-state-and-operation-contract) +- [Follow-up delivery must be coordinated with ordinary wake scheduling](#follow-up-delivery-must-be-coordinated-with-ordinary-wake-scheduling) +- [Recovery needs identity evidence, not merely a reachable process](#recovery-needs-identity-evidence-not-merely-a-reachable-process) +- [Revocation and native powers are different boundaries](#revocation-and-native-powers-are-different-boundaries) +- [Files and commands are a separate capability decision](#files-and-commands-are-a-separate-capability-decision) +- [Paperclip implementation anchors](#paperclip-implementation-anchors) + +## Separate the bot, connection, controller, and workspace + +```text +External provider + Bot: reasoning, native browser/computer/apps, private memory + | authenticated tools / signed wake events +Paperclip control plane + MCP + OAuth + pairing + durable mailbox + operation broker + | authenticated Runner protocol +Managed execution environment (Cloud) + Rust Runner: assignment lifecycle, catalog validation, durable receipts + | semantic tool requests / correlated results +Paperclip authority services + Task/company permissions, mutations, app gateway, result finalization +``` + +Application tools execute through the existing authorized services. If filesystem +or shell tools are enabled, their execution target must be the assigned confined +workspace. Moving Rust into a sandbox does not magically move a server-side file +handler there. Trace the actual effect destination for every workspace operation. + +Dot's model runs at OpenAI. Runner does not create its personal memory, run its +model, control its computer, or guarantee interruption of other native tools. +Do not invent model/session/usage fields to satisfy a local-provider schema. + +## Identity and authority + +Use a dedicated agent-purpose grant/resource where the provider is acting as an +employee. Personal assistant grants must not silently supply board/operator +permissions to a run. Bind authorization to company, agent, active grant, binding +generation, assignment, and run/turn. A supplied companyId or agentId is not proof +of authority. Recheck live permissions at execution and before returning protected +data, not only at initial pairing. + +Pairing uses an operator-issued, expiring, one-use capability for a specific +agent. Redemption checks current membership/role, feature prerequisites, agent +lifecycle, company/resource hints, expiry and revocation. Preview does not consume +the code. Grant creation, code consumption, and saving the binding reference must +be atomic. Otherwise setup can display ready while the scheduler still sees an +unpaired agent. Saving the form a second time is not an acceptable hidden step. + +Persistent connection identity and per-turn mutation authority are different: + +| Request | Expected behavior | +| --- | --- | +| Pair, inspect own capability/inbox status | Connection-scoped operation; no managed assignment needed. | +| Start work while idle | Durable, agent-attributed intake; ordinary admission before task mutation tools. | +| Read/accept offered assignment | Current binding and offer, explicit acceptance, no arbitrary run selection. | +| Invoke a task/app tool | Same admitted assignment and live permission checks; normal receipts/audit. | +| Update own optional avatar | Narrow self-only operation; do not grant generic agent-config authority. | +| Finish | Accepted canonical result, settled required operations, normal task finalizer. | +| Acknowledge a fence | Narrow control operation may remain available while task tools are denied. | + +Idle intake must retain plugin/environment context from the configured application +services. In Dot, constructing a separate heartbeat service without the sandbox +plugin registry made idle requests fail even though operator-assigned work ran. +Test both entry points against the same managed environment. + +## Durable state and operation contract + +Persist distinct facts for grant/binding, subscription, mailbox item, assignment, +operation receipt, run/controller ownership, and final result. Avoid a single +boolean “connected” or “done.” Enforce active-assignment concurrency in storage +as well as the UI. Dot's initial contract allowed one active assignment per binding. + +An illustrative lifecycle is: + +```text +intake/admission pending -> offered -> accepted -> working -> result accepted + -> turn finished -> task disposition committed -> resource cleanup +``` + +Any active phase can instead become fenced, expired, failed, or reconciliation +required. These are conceptual states; use the repository's actual enums. + +Every side-effecting external request needs a stable request identity plus an +argument digest and authority generation. Implement these semantics: + +- Same identity + same arguments: return the same pending/settled receipt. +- Same identity + different arguments: reject a conflict. +- Completed intake retried: return its original receipt, never another task. +- Old authority generation: deny new execution, even if a historical receipt exists. +- Definite read failure: terminal bounded error; later tools and finish can proceed. +- Effect possibly happened but result was lost: explicit unknown/pending outcome + and reconciliation. Do not reissue under a fresh ID to make progress. + +Persist reservation before effects where appropriate. Use existing transactional +mutation receipts and external-effect reconciliation. Do not promise universal +exactly-once execution across arbitrary remote services. Distinguish command +acknowledgment from completion of the tool's effect. + +Do not block the Runner command lane waiting for a semantic tool result delivered +on that same lane. Reserve/admit promptly, deliver a pending receipt, then settle +with a correlated event. Bound pending operations, results, bytes, and authority +duration. Pagination and control acknowledgments must still work at limits. + +Serialize mailbox cursor visibility with writers so a cursor cannot skip a result +whose transaction commits later. Wake events should contain references, not task +contents, credentials, or copied app data. An event receiver drains the durable +mailbox and validates current authority rather than trusting a stale event body. + +Treat event callbacks as an authenticated protocol, not arbitrary URLs to POST to. +Use the provider's current ownership challenge and signing specification. Keep +callback signing material out of prompts, descriptors, logs and task data. Enforce +the callback destination policy, including redirect and DNS/IP handling, so a +subscription cannot turn the control plane into an internal-network request proxy. +Apply bounded delivery timeouts/retries and record failures without logging secrets. +Where receiving signed callbacks, validate the expected signature, timestamp and +replay policy before changing state. Test the actual directions this provider uses; +do not invent a symmetric webhook API merely because the integration is two way. + +## Follow-up delivery must be coordinated with ordinary wake scheduling + +A normal comment can enter two paths: active-assignment mailbox delivery and the +ordinary scheduler. If both remain independent, a finished bot can receive a +second assignment for a comment it already handled. If the normal wake is simply +discarded when the mailbox entry is made, the comment can be lost at completion. + +The Dot fix retained the normal wake as deferred until the bot successfully read +that exact comment ID through `get_task_history`. Then the existing run consumed +it. If the assignment ended before the read, normal admission kept the unread +comment for another turn. An inbox reference or webhook acknowledgment did not +prove the comment was read. + +Test both sides of this race, including controller recovery. Keep explicit new +session requests, interaction replies, and self-authored comments under their +normal policy; don't blanket-suppress every wake for a connected bot. + +## Recovery needs identity evidence, not merely a reachable process + +Record and compare the run, session, assignment, binding generation, controller +generation, managed lease, process marker, and trustworthy process birth identity. +On Linux the process fingerprint can include boot identity and start information. +Read remote identity from the sandbox, not a same-numbered PID on the controller. + +Before advertising external work, require the evidence needed to recover it. +Recovery reads the checkpoint from the owning execution target and validates it +against saved authority. Missing/invalid state after dispatch fails closed for +reconciliation; a new process or conversation must not masquerade as a resume. + +Incoming operations need a durable draining/replay mechanism owned by the current +controller. An in-memory forwarding optimization alone can leave valid requests +stranded after restart. Fence old controllers and callbacks so a late result +cannot overwrite the new generation's journal. Preserve late effect evidence. + +A verified terminal/shutdown receipt can establish a clean exit when a remote +monitor lacks an exit code. An unexplained process exit still requires recovery. +Configure a bounded reconnect grace consistent with the deployment lifecycle; +Dot used five minutes after shorter restart gaps exposed failures. Do not copy +that number as a universal provider requirement. + +Terminal run reconciliation must clear/fence a still-active assignment, notify the +bot, and release resources. Otherwise a failed run can block every future task +even though its old tools are already denied. + +## Revocation and native powers are different boundaries + +Cancellation, pause, reassignment, membership loss, expired run authority, and +grant revocation must prevent subsequent Paperclip effects. Cached tool results +or UI binding references cannot restore revoked access. Old-grant revocation must +not revoke a replacement binding accidentally. + +Report “Paperclip authority revoked” separately from “external bot stopped.” A +fence acknowledgment proves receipt, not termination of all provider activity. +Likewise, sandboxing the Paperclip Runner does not confine the bot's native Slack +or provider computer. Those tools retain their own permissions. If strict total +capability isolation is required, use a suitably isolated bot/account and verify +the provider's actual controls; Paperclip cannot enforce it with prompt text. + +## Files and commands are a separate capability decision + +Do not add workspace tools merely to make an external agent look like a coding +CLI. A bot can research or write code on its own computer without Paperclip +workspace access. The first Dot Cloud release intentionally offered none. + +When the release includes workspace tools: + +- Resolve paths within the admitted workspace; reject traversal, absolute paths, + symlinks/escapes, and Paperclip instance state such as `.paperclip`. +- Check reads, writes, API downloads, uploads, and artifact publication. A generic + `call_api` path must not bypass attachment/workspace grants. +- Serialize optimistic hash checks with writes. Two concurrent writes must not + both commit against the same observed hash. +- A process working directory is not a sandbox. Use a proven OS/container boundary; + test outside-workspace access, environment secrets, detached descendants, timeout, + cancellation and cleanup. No unrestricted fallback when confinement is unavailable. +- In the historical Dot implementation, Linux used bubblewrap/private PID + namespaces. macOS command execution was disabled after detached descendants + escaped reliable supervision; file tools and artifacts remained available. +- For attachments, enforce current task/company, bounded pages and total bytes, + integrity checks, and live revocation before returning each page. Treat filenames + and contents as untrusted data. Test the generic API escape path too. + +## Paperclip implementation anchors + +Paths below existed in the qualified October 9 candidate. Locate their current +equivalents before editing; do not paste line numbers from historical notes. + +| Concern | Historical anchor | +| --- | --- | +| Bot broker and tool definitions | `server/src/services/dot-runner-broker.ts` | +| Prototype public MCP bridge | `server/src/services/public-mcp/dot-runner.ts` | +| Rust assignment backend | `packages/paperclip-runner/runner/crates/runner-core/src/dot_provider_backend.rs` | +| TypeScript Runner driver | `packages/paperclip-runner/src/drivers/dot/runnerd-dot-driver.ts` | +| Driver/backend selection | `packages/paperclip-runner/src/backends/native-backend-factory.ts` | +| Per-turn instructions | `packages/paperclip-runner/src/backends/runtime-context.ts` | +| Governed tool authority | `server/src/services/native-runtime/paperclip-runner-tool-authority.ts` | +| Admission, wake, terminal reconciliation | `server/src/services/heartbeat.ts`, `server/src/services/heartbeat/queue.ts`, `server/src/services/heartbeat/run-lifecycle.ts` | +| Real Rust/DB and managed qualification | `server/src/__tests__/dot-runner.test.ts` | + +Keep the standalone Runner package independent of server/DB implementation +imports. Extend the actual production backend and negotiated capability contract, +not just a convenient reference harness. diff --git a/.agents/skills/create-external-bot-integration/references/troubleshooting-and-deployment.md b/.agents/skills/create-external-bot-integration/references/troubleshooting-and-deployment.md new file mode 100644 index 00000000000..df851f386af --- /dev/null +++ b/.agents/skills/create-external-bot-integration/references/troubleshooting-and-deployment.md @@ -0,0 +1,215 @@ +# Troubleshooting, test environments, and Cloud delivery + +## Contents + +- [Start with the failing boundary](#start-with-the-failing-boundary) +- [Stable public ingress is part of the test fixture](#stable-public-ingress-is-part-of-the-test-fixture) +- [Concrete OAuth and routing checks for the Dot implementation](#concrete-oauth-and-routing-checks-for-the-dot-implementation) +- [Symptom → discriminator → next action](#symptom--discriminator--next-action) +- [Keep one reproducible environment manifest](#keep-one-reproducible-environment-manifest) +- [Coordinate all Cloud pieces](#coordinate-all-cloud-pieces) +- [Classify a failure before retrying](#classify-a-failure-before-retrying) +- [Close the environment deliberately](#close-the-environment-deliberately) + +## Start with the failing boundary + +Do not use “OAuth is broken” for every setup failure. Trace these independent paths: + +```text +A. Provider backend -> public origin -> discovery, MCP, token exchange +B. Provider browser -> authorization page -> preview/consent -> provider callback +C. Agent runtime -> loaded plugin actions -> authenticated read/tool calls +D. Paperclip event worker -> verified callback -> bot wake -> mailbox/challenge +E. Paperclip scheduler -> managed lease -> Runner -> tools -> finalizer -> cleanup +``` + +For each, collect a timestamp, exact failing operation, status/visible error, +correlated access log, and which party observed it. The developer's browser, the +provider's browser, and the provider's backend can have different reachability. +An absent request in complete ingress logs means that attempt did not reach the +app; it does not identify the upstream blocking rule. + +## Stable public ingress is part of the test fixture + +Prefer one stable HTTPS origin on port 443 reachable by both provider clients. +Keep issuer/resource, token exchange, authorization redirect, and callback +configuration consistent. Do not rotate tunnel hostnames as the first response to +every failure: each new URL can require a different plugin, fresh registration, +new pairing prompt, and subscription cleanup. + +Self-hosted testing may need a tunnel or authorized public reverse proxy. Dot +eventually used a stable public proxy forwarding to the test machine. That is an +observed useful arrangement, not a requirement to provision new infrastructure +for every integration. Use the user's supported deployment/network options. + +Tailscale-specific diagnostic: local MagicDNS may resolve a hostname to a private +tailnet address while outside clients use Funnel. Test public ingress independently; +local curl success does not establish public access. Tailscale Serve and Funnel +have different reachability. Never infer a provider port ban from one timeout. + +`ERR_BLOCKED_BY_CLIENT` alone does not identify Cloudflare or a server bot filter. +Use server logs and the actual consent page, not only `/api/health`: Dot once +blocked the health navigation while the required sign-in page loaded. Do not +bypass browser security warnings or a provider's access controls. Use supported +configuration or an authorized reachable endpoint and preserve the exact uncertainty. + +## Concrete OAuth and routing checks for the Dot implementation + +These routes describe the October 9 candidate; discover current equivalents. + +| Surface | Historical expectation | +| --- | --- | +| `/mcp/runner` unauthenticated | `401` and `WWW-Authenticate` with resource metadata; not an HTML sign-in page. | +| `/.well-known/oauth-protected-resource/mcp/runner` | Public JSON for the exact resource and authorization server. | +| `/.well-known/oauth-authorization-server/mcp/runner/oauth` | Public JSON for issuer and endpoints. | +| `/mcp/runner/oauth` | Dedicated agent-resource issuer; personal MCP issuer remained separate. | +| `/dot-connect/:requestId` | Public pairing page, scoped by request and one-use pairing capability. | +| `/api/dot-mcp/requests/:requestId` and exact pairing operations | Publicly routable aliases with request/code checks; reject personal assistant request IDs. | +| Normal board, connection management, ordinary personal consent | Retain normal session/role requirements. | + +For read-only health checks, substitute the configured origin; do not send tokens: + +```sh +curl --max-time 15 -sS -D - -o /tmp/dot-resource-metadata.json \ + 'https://YOUR_CONFIGURED_ORIGIN/.well-known/oauth-protected-resource/mcp/runner' +curl --max-time 15 -sS -D - -o /tmp/dot-auth-metadata.json \ + 'https://YOUR_CONFIGURED_ORIGIN/.well-known/oauth-authorization-server/mcp/runner/oauth' +``` + +Inspect JSON and redirect behavior, then exercise the actual provider discovery. +Do not interpret these curl checks as provider qualification. Keep browser cookies +and tokens out of pasted logs; use the existing authenticated test client when a +protected operation must be inspected. + +ChatGPT's historical client metadata preferred `private_key_jwt` while also +advertising `none`. Paperclip originally rejected it instead of negotiating its +supported public-client method with PKCE. Inspect the complete advertised methods; +do not disable authentication or exact redirect/resource validation to work around +a preference mismatch. Keep S256 PKCE and current registration checks. + +An optional `PAPERCLIP_MCP_AUTHORIZATION_ORIGIN` supported a different authorized +browser ingress to the same instance, while `PAPERCLIP_PUBLIC_URL` retained the +MCP resource/issuer/token origin. This was a workaround for different client +reachability, not an OAuth requirement. Use only explicitly configured trusted +origins and verify callback/CSRF/issuer behavior. Prefer a single working origin. + +## Symptom → discriminator → next action + +| Symptom | Evidence to inspect | Action and proof | +| --- | --- | --- | +| OAuth discovery times out | Public metadata vs actual provider access logs | Fix the failing ingress/client path; do not keep editing valid metadata when no request arrives. | +| Unknown client or redirect URI after restart | Persisted registration vs provider's cached client | Restore the intended persistent fixture or re-register through supported setup. Do not reset DB repeatedly. | +| Invalid request ID | Actual ID representation | Treat protocol IDs as opaque unless contract says otherwise; the lab incorrectly required a UUID. | +| Grant exists but provider is not connected | Complete same browser flow through callback/token exchange | A server grant row alone is not completed OAuth. Avoid mixing separate callback sessions. | +| Tools listed in settings, absent to bot | Actual tool invocation in active conversation | Supported refresh/attach/restart, bounded once per diagnosis; preserve endpoint and mark intervention. | +| Subscription verified but no bot action | Delivery record, mailbox, challenge/acceptance receipt | Separate callback receipt from model execution. Report latency; inspect actionable work before resending. | +| Ready connection, task admission says unpaired | Saved agent binding/config revision | Fix atomic pairing persistence; do not require another manual form save. | +| Agent created paused before pairing | Lifecycle preparing/verifying vs manual pause | Use lifecycle-specific setup eligibility; preserve actual pause/approval gates. | +| Anonymous consent spins/reloads | Background session probes returning expected 401 | Public consent must tolerate them without global reload; protected board routes retain session recovery. | +| Ordinary assignment works, idle work cannot acquire sandbox | Service construction / plugin context | Use app-configured admission service and test both entry points. | +| Remote Runner connects to localhost | Launched transport endpoint vs private managed ingress | Use the authenticated remote transport supplied by the execution target. | +| Image lacks provider support | Actual packaged Runner digest/capabilities | Fail before dispatch; publish compatible artifact and test installed path. | +| Recovery finds sandbox but no process identity | Marker, fingerprint and original checkpoint | Capture identity before dispatch; reconcile missing evidence rather than launch replacement work. | +| Remote PID has implausible start time | Whether controller inspected its own PID table | Obtain birth identity from the remote sandbox, not a colliding local PID. | +| Finished task starts again after comment | Mailbox + scheduler wake receipt | Consume only when exact comment read succeeds; test unread-at-finish and read-before-finish. | +| Missing file leaves run waiting forever | Pending tool receipt and error dispatch | Settle definite read failures; preserve unknown outcome only for genuinely ambiguous effects. | +| Terminal run blocks all new tasks | Assignment still active despite terminal run | Reconcile/fence it and release resources without claiming external stop. | +| Cancelled job has creating/pending-cleanup sandbox | Provider allocation and deletion receipts | Track until destroyed or explicitly pending with owner; retry only per provider semantics. | +| New deployment vanishes mid-test | Fleet auto-rollout vs direct candidate | Use supported test-tenant pinning with authorization; record current serving revision. | + +## Keep one reproducible environment manifest + +Before a test drive or deployment, resolve current worktree/artifacts and reuse a +suitable isolated checkout. Do not reset the user's primary checkout. Create a new +managed worktree when appropriate, preserving unrelated work. Record: + +```text +Repository / worktree / branch / app SHA: +Runner artifact platform / digest / advertised capabilities: +Ingress repository / SHA / relevant routing version: +App image / immutable digest: +Local host / port / data directory / process ownership: +Public origin / authorization origin / forwarding target: +Tenant / company / agent identifiers (private evidence only): +Sign-in entry and credential reference (no secret values): +Grant/binding status / subscription / pending code expiry (no code value): +Current case / run / assignment / lease: +Owner of each server, proxy, build, sandbox, and cleanup: +Environment retention agreement / next gate: +``` + +Use synthetic fixtures, not a production database clone containing private company +data or signing keys. Verify launcher arguments, selected instance, port mapping, +and data persistence after restart. A label such as `default` is not proof of +process ownership. Never kill unrelated databases or clear shared IPC globally. +If resource pressure blocks the fixture, inspect owned processes or use the +approved remote test path; preserve unrelated sessions. + +Verify sign-in is usable before handing over a URL. Deliver access/resume details +through an appropriate private path. If the user is still testing or the bot is +loading tools, keep the endpoint alive under the agreed bounds. Stopping it made +the first “tools unavailable” diagnosis premature. Announce an intentional expiry +and explain how to resume; do not assume a stopped shell process remains a service. + +## Coordinate all Cloud pieces + +Cloud qualification requires the tenant app, public ingress, managed provider +plugin/configuration, and packaged Runner artifact. The UI flag alone is not +enablement. Inspect guards in UI, scheduler, backend, remote launcher, and Cloud +front door. Use scoped route exceptions that retain endpoint authentication rather +than removing browser-session guards wholesale. + +Pairing and MCP must remain reachable without a managed job. If a sleeping tenant +returns a waking-up page, wait for the actual lifecycle transition, then test the +correct metadata response. Delay creating expiring pairing codes until the tenant +image is deployed and serving. A 3.4 GB image transfer consumed meaningful setup +time during Dot qualification; do not burn the code's lifetime watching a build. + +Record whether a test overrides the sandbox binary with a locally built artifact. +That proves the code can run, not that the published default image is correct. +Match platform and provider capability; a Mac controller's current binary cannot +stand in for a Linux sandbox artifact. Digest/capability checks should fail clearly +before an assignment is offered. + +For the historical private `paperclip-cloud` repository, the supported targeted +deployment CLI was `scripts/deploy-stack.mjs --plain`, with +`--resume ` for the same operation. Inspect current help/runbook before +using it. Do not invent or replay old tenant IDs, deployment requests, or secrets. +Preserve one request through prepare/build/check/deploy/verify; duplicated requests +make provenance and recovery harder. No fleet-default change is implied by a +single test-tenant deployment. + +## Classify a failure before retrying + +- **Product:** deterministic missing context, bad state transition, wrong endpoint, + lost receipt. Reproduce, fix, and rerun the affected contract. +- **Provider:** tool loading, wake delay, refusal, unsupported control. Capture the + actual provider result and test a supported recovery; don't manufacture success. +- **Infrastructure:** capacity, registry 429/504, image transfer, npm resolver, + provider allocation. Verify whether work/effects started; use bounded recovery. +- **Evidence/test:** controller didn't restart, stale build, nonce visible in prompt, + wrong image, scripted client presented as real. Correct the experiment. + +For an unchanged transient build failure, one targeted infrastructure retry is a +reasonable initial bound. If it fails again, report the precise remaining gate +or investigate the new cause; do not loop reruns or change registry/security +configuration without a task-grounded reason. Resume the existing deployment. +If code changes, re-evaluate which earlier evidence applies to the new candidate. + +Keep broad test runs tied to a frozen revision. Stop superseded owned runs when +safe, retain their results, and run the affected suites plus required final gates. +Do not claim a full local suite passed because failing tests passed individually; +state focused results and final CI separately. + +## Close the environment deliberately + +After the final task, inspect leases and actual provider resources. Expired lease +status alone is weaker than a successful sandbox deletion receipt. Preserve a +requested review instance with access and resumption information; release owned +disposable compute. For an authorized reset, account for provider account link, +Paperclip binding/grant, event subscription/automation, and cached configuration. +Disconnecting one layer can leave an enabled provider automation behind. + +Report qualified, review-ready, merged, deployed, and canary-verified separately. +Never say an instance received a fix until its serving revision and relevant +ordinary-use behavior establish it.