mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-22 16:51:28 +02:00
Integrate v33 runtime and saved-token setup for qualification
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed
commit
65a598f3a4
54 files changed
+5863
-3833
No files matched your search
@@ -1217,6 +1217,15 @@ stored event payloads, redacts and shortens log chunks, and caps stdout/stderr
|
||||
excerpts. `heartbeat.ts` keeps event writes, current-user redaction, and live
|
||||
event delivery. Existing public helpers remain available from `heartbeat.ts`.
|
||||
|
||||
Workspace preparation is in `server/src/services/heartbeat/workspaces.ts`. It
|
||||
owns managed checkout materialization, workspace validation and reuse, referenced
|
||||
project resolution, and session/workspace configuration freshness. Its
|
||||
`createHeartbeatWorkspaceResolver(db)` factory binds the run workspace resolvers
|
||||
to a service's database. The checkout single-flight map stays at module scope so
|
||||
all service instances share in-flight materialization. Existing public helpers
|
||||
and the workspace validation error class remain available from `heartbeat.ts`.
|
||||
Keep workspace policy changes separate from scheduling and run execution changes.
|
||||
|
||||
## Wake Context Delivery
|
||||
|
||||
Built-in adapters deliver wake context through the run prompt, including structured
|
||||
|
||||
@@ -83,6 +83,15 @@ membership approval. Existing members can self-link; disconnecting an identity
|
||||
does not permanently block that member from reconnecting. Guest access remains
|
||||
a separate, explicit setting for restricted work.
|
||||
|
||||
The person joining runs `/<saved slash command> connect` themselves (for example,
|
||||
`/maya connect`), without an `@person` argument. The command discovers their
|
||||
identity without starting agent work or granting access. Slack conversation
|
||||
agents receive this saved command on each fresh or resumed turn, alongside the
|
||||
confirmation and membership-approval instructions. If the saved command is
|
||||
unavailable, they refer the connection manager to **Access → Invite people**
|
||||
instead of guessing from the agent name. Inviting a person or bot to a Slack
|
||||
channel does not grant that person Paperclip access.
|
||||
|
||||
**Allowed Channels** updates every five seconds while Slack Settings is open.
|
||||
Inviting the bot to a new channel enables replies and writes there automatically;
|
||||
no additional Paperclip configuration is required. An explicit off switch stays
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
# Automatic idle sleep safety
|
||||
|
||||
`GET /api/instance/task-drain?idleSleepSafety=1&ownerId=<owner>` adds an
|
||||
instance-wide `idleSleepSafety` report for instance administrators and signed
|
||||
Cloud control callers. Ordinary task drains keep their existing behavior.
|
||||
Company members and agents cannot read this report.
|
||||
|
||||
## Hosting protocol
|
||||
|
||||
1. Hold external admission at the hosting layer. Queue user requests for wake or
|
||||
abort sleep when activity arrives. Do not send traffic into an instance that
|
||||
is being stopped.
|
||||
2. Send `POST /api/instance/task-drain` with
|
||||
`{ "purpose": "idle", "ttlMs": 120000 }`. Idle holds require a TTL from 5 to
|
||||
300 seconds. Save the returned `ownerId` and `expiresAt`. An existing drain
|
||||
returns 409. An unsupported purpose must leave automatic sleep disabled.
|
||||
3. Poll the report with that exact owner. Only
|
||||
`{ "version": 1, "backgroundWork": "none" }` permits sleep. Require the same
|
||||
`ownerId`, `draining: true`, zero run/wake counters, and enough remaining TTL
|
||||
for the provider stop. `present`, `unknown`, missing fields, or errors must
|
||||
keep the instance awake. Do not interpret `quiescent` alone as authorization.
|
||||
4. Revalidate the same hold immediately before stopping the instance. Keep
|
||||
external admission closed through the provider stop. Never use an old report
|
||||
after a restart, expired hold, changed owner, or failed validation.
|
||||
5. If sleep is abandoned, send `DELETE /api/instance/task-drain?ownerId=<owner>`.
|
||||
A missing or stale owner cannot release another controller's idle hold. The
|
||||
TTL clears an abandoned hold if the controller loses its response.
|
||||
|
||||
An idle hold rejects new tenant HTTP requests and WebSocket upgrades with 503
|
||||
and `Retry-After: 1`. Health checks and authenticated task-drain control remain
|
||||
available. Ordinary deploy drains do not enable this HTTP gate. The host must
|
||||
handle retry/wake at its ingress before enabling automatic sleep.
|
||||
Health and control mutations remain tracked through completion, including
|
||||
signed bootstrap writes. Only the task-drain read skips request tracking.
|
||||
Its authentication finishes under a separate work token before entering the
|
||||
report, so it counts concurrent writes without blocking on its own scan.
|
||||
|
||||
## Accepted work and cleanup
|
||||
|
||||
HTTP admission is installed before body parsing, authentication, webhooks and
|
||||
MCP ingress. The Express adapter tracks handler promises in nested routers and
|
||||
error middleware. Sending a response or losing the client does not complete an
|
||||
async handler. A callback-only handler that abandons its response without ending it remains a
|
||||
sleep blocker. An async route that settles after a disconnect releases its
|
||||
request token; normal cancellation does not permanently block sleep. New routes must be registered before this adapter is installed.
|
||||
Fire-and-forget work must have durable queue state or use `trackIdleWork`; a
|
||||
response is never a substitute for tracking that work.
|
||||
Accepted live-event WebSocket authentication remains counted until its promise
|
||||
settles, including after the raw socket closes. Detached startup reconciliation
|
||||
for built-in agents, managed homes and persisted runtime services also remains
|
||||
counted after readiness until its writes settle.
|
||||
|
||||
Scheduler work already in flight remains counted until its promise settles.
|
||||
Idle holds pause new scheduler admissions. Database backup promises remain
|
||||
counted through success or failure. Configured periodic backups also block
|
||||
sleep until a host owns a durable wake schedule for them. A generation counter invalidates a
|
||||
scan if tracked work both starts and finishes during inspection. Reports remain
|
||||
unknown until startup recovery and HTTP tracking are installed.
|
||||
|
||||
The report inspects accounting and sandbox-cleanup spools directly. Any entry,
|
||||
including malformed JSON, temporary files and failed write probes, blocks sleep.
|
||||
Read errors and invalid spool paths return unknown. Recovery readers that skip
|
||||
bad records cannot certify an empty spool.
|
||||
|
||||
An orphan cleanup token starts before its spool write and remains held through
|
||||
buffering, queue splices, flush retries and buffer overflow. It ends only after
|
||||
the database owns cleanup. Queued usage receipts stay counted until capture
|
||||
settles. If receipt capture and the database failure fence both fail, a token
|
||||
continues to block sleep until the failure fence is durably written. A query
|
||||
that updates zero rows does not clear that debt. Unrecoverable cleanup requires
|
||||
operator reconciliation; this API does not discard it to save compute.
|
||||
|
||||
## Persisted work and limits
|
||||
|
||||
The report checks all companies in one bounded read-only database transaction.
|
||||
Queued and orphaned runs, timers, retries, accounting debt, unfinished issues,
|
||||
cleanup, active routines, integrations, external API credentials, and plugin
|
||||
work block sleep. Completed ordinary run history does not. Less common work
|
||||
sources conservatively block on any retained records. Active issue watchdogs
|
||||
block sleep even when their watched issue is complete and no review has started.
|
||||
Missing migrations, malformed configuration and database errors produce unknown
|
||||
without exposing SQL or tenant data.
|
||||
|
||||
Every enabled plugin blocks sleep. A version label does not prove that arbitrary
|
||||
worker code has no background activity. No plugin approval or configuration
|
||||
bypass is supported. Scheduled and externally triggered work needs a durable
|
||||
host wake mechanism before this restriction can be relaxed.
|
||||
|
||||
This is a single-process admission protocol. A restart clears the owner. The
|
||||
host must serialize sleep with deploy/wake operations and bind its final check
|
||||
to the instance it stops. An unexpected restart after the final check can still
|
||||
race an external provider stop; a process-local report cannot fence that remote
|
||||
operation by itself. Multiple replicas require shared fencing. The protocol
|
||||
does not make existing in-memory-only debt survive an unplanned process crash.
|
||||
|
||||
Related public PR #13413 adds runtime-service controller requirements and uses
|
||||
the same `purpose: "idle"` / `ownerId` vocabulary. This report conservatively
|
||||
blocks all retained runtime-service state instead of approving such services.
|
||||
@@ -138,6 +138,9 @@ build and include them in the server's vendored Runner output before npm packing
|
||||
Assembly requires the provider pack's source revision to match `sourceRevision`
|
||||
and its ACPX profiles and Cursor distribution to match the current source pins.
|
||||
An independently rehashed older pack is rejected.
|
||||
Provider-pack builds replace the installed Copilot platform wrappers with relative
|
||||
launchers. These wrappers remain usable after the pack moves into an image. The
|
||||
build still rejects any wrapper that retains its temporary deployment path.
|
||||
Ordinary remote Cursor startup uses the packaged Linux daemon and verifies every
|
||||
image asset against that manifest. A mismatched image fails before the provider
|
||||
starts; install the matching package and image together.
|
||||
|
||||
@@ -32,12 +32,12 @@
|
||||
"requiresProviderPolicy": true
|
||||
},
|
||||
"copilot": {
|
||||
"agentProfileVersion": 32,
|
||||
"agentProfileVersion": 33,
|
||||
"agentServerPackage": "@github/copilot",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"agentRuntimePackage": null,
|
||||
"agentRuntimeVersion": null,
|
||||
"commandDigest": "sha256:3dcc8de3034e32001b0a819a614beca345d314266a70a09e0050a9f5f58275fc",
|
||||
"commandDigest": "sha256:7b459cc43771d4178898be8a362030ac2e4d8356c2a14a82334958959dd54e59",
|
||||
"requiresProviderPolicy": true
|
||||
},
|
||||
"claude": {
|
||||
|
||||
@@ -79,7 +79,7 @@
|
||||
"typecheck:browser": "tsc -p tsconfig.browser.json --noEmit",
|
||||
"test": "pnpm run test:typescript && pnpm run test:rust",
|
||||
"test:typescript": "pnpm run test:typescript:prep && vitest run",
|
||||
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs && pnpm run test:copilot:contracts",
|
||||
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/provider-pack-executable-shims.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs && pnpm run test:copilot:contracts",
|
||||
"test:typescript:vitest": "node ./scripts/run-pr-vitest-lane.mjs",
|
||||
"test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace",
|
||||
"test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider",
|
||||
|
||||
@@ -3620,7 +3620,7 @@ mod tests {
|
||||
descriptor_value["agentRuntimePackage"] = Value::Null;
|
||||
descriptor_value["agentRuntimeVersion"] = Value::Null;
|
||||
descriptor_value["commandDigest"] = serde_json::from_str::<Value>(include_str!(
|
||||
"../../../../test/fixtures/copilot-profile-v32-identity.json"
|
||||
"../../../../test/fixtures/copilot-profile-v33-identity.json"
|
||||
))
|
||||
.unwrap()["commandDigest"]
|
||||
.clone();
|
||||
|
||||
@@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option<AcpxReleaseProfile> {
|
||||
agent_runtime_package: None,
|
||||
agent_runtime_version: None,
|
||||
command_digest:
|
||||
"sha256:3dcc8de3034e32001b0a819a614beca345d314266a70a09e0050a9f5f58275fc",
|
||||
"sha256:7b459cc43771d4178898be8a362030ac2e4d8356c2a14a82334958959dd54e59",
|
||||
requires_provider_policy: true,
|
||||
},
|
||||
"claude" => AcpxReleaseProfile {
|
||||
|
||||
+1
-1
@@ -1029,7 +1029,7 @@ fn provider_death_expires_input_and_commits_failure_without_querying_the_dead_pr
|
||||
let mut config = acpx_config(&directory, mode);
|
||||
let mut payload = prepare_payload_with_mode(&directory, "codex", mode);
|
||||
let identity: Value = serde_json::from_str(include_str!(
|
||||
"../../../../test/fixtures/copilot-profile-v32-identity.json"
|
||||
"../../../../test/fixtures/copilot-profile-v33-identity.json"
|
||||
))
|
||||
.unwrap();
|
||||
let digest = identity["commandDigest"].as_str().unwrap();
|
||||
|
||||
@@ -7,7 +7,7 @@ import { COPILOT_VERSION, materializePinnedCopilotBinary, resolveCopilotDistribu
|
||||
|
||||
const MAX_ARCHIVE_BYTES = 128 * 1024 * 1024;
|
||||
const MAX_EXPANDED_BYTES = 384 * 1024 * 1024;
|
||||
const PROFILE_DIGEST = "sha256:3dcc8de3034e32001b0a819a614beca345d314266a70a09e0050a9f5f58275fc";
|
||||
const PROFILE_DIGEST = "sha256:7b459cc43771d4178898be8a362030ac2e4d8356c2a14a82334958959dd54e59";
|
||||
|
||||
/** Build-time only; outputRoot is the exact runner-owned platform asset directory. */
|
||||
export async function buildPinnedCopilotDistribution({ outputRoot, platform = process.platform, architecture = process.arch }, { fetchImpl = fetch } = {}) {
|
||||
|
||||
@@ -20,6 +20,7 @@ import { dirname, join, relative, resolve } from "node:path";
|
||||
import { createRequire } from "node:module";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { parseProviderPackArguments, materializeCandidateProviderPack, providerPackProviders, providerPackManifestFields } from "./candidate-provider-pack.mjs";
|
||||
import { writePortableCopilotShims, writePortableExecutableShim } from "./provider-pack-executable-shims.mjs";
|
||||
import { buildNodeStartupTimeout } from "./build-node-startup-timeout.mjs";
|
||||
|
||||
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
@@ -101,21 +102,6 @@ function writePortableNodeShim(name, entrypoint) {
|
||||
chmodSync(shimPath, 0o755);
|
||||
}
|
||||
|
||||
function writePortableExecutableShim(name, executable) {
|
||||
const shimPath = join(temporaryRoot, "node_modules", ".bin", name);
|
||||
writeFileSync(
|
||||
shimPath,
|
||||
[
|
||||
"#!/bin/sh",
|
||||
"set -eu",
|
||||
'basedir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)',
|
||||
`exec "$basedir/../${executable}" "$@"`,
|
||||
"",
|
||||
].join("\n"),
|
||||
);
|
||||
chmodSync(shimPath, 0o755);
|
||||
}
|
||||
|
||||
try {
|
||||
const deployed = spawnSync(
|
||||
"pnpm",
|
||||
@@ -212,22 +198,13 @@ try {
|
||||
`@anthropic-ai/claude-agent-sdk-${process.platform}-${process.arch}/claude`,
|
||||
);
|
||||
writePortableExecutableShim(
|
||||
temporaryRoot,
|
||||
"claude",
|
||||
relative(realpathSync(join(temporaryRoot, "node_modules")), realpathSync(claudeExecutable)),
|
||||
);
|
||||
writePortableExecutableShim("node", "node/bin/node");
|
||||
writePortableExecutableShim("opencode", "opencode-ai/bin/opencode.exe");
|
||||
// Optional Copilot platform packages expose native .bin entries too. Make
|
||||
// each installed entry portable without executing it during the build.
|
||||
for (const platform of ["darwin-arm64", "darwin-x64", "linux-x64"]) {
|
||||
const name = `copilot-${platform}`;
|
||||
if (!existsSync(join(temporaryRoot, "node_modules", ".bin", name))) continue;
|
||||
const executable = packRequire.resolve(`@github/${name}`);
|
||||
writePortableExecutableShim(
|
||||
name,
|
||||
relative(realpathSync(join(temporaryRoot, "node_modules")), realpathSync(executable)),
|
||||
);
|
||||
}
|
||||
writePortableExecutableShim(temporaryRoot, "node", "node/bin/node");
|
||||
writePortableExecutableShim(temporaryRoot, "opencode", "opencode-ai/bin/opencode.exe");
|
||||
writePortableCopilotShims(temporaryRoot);
|
||||
writePortableNodeShim("acpx", "acpx/dist/cli.js");
|
||||
writePortableNodeShim(
|
||||
"claude-agent-acp",
|
||||
|
||||
@@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url";
|
||||
import { resolve, dirname } from "node:path";
|
||||
|
||||
const root = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const fixture = resolve(root, "test/fixtures/copilot-profile-v32-identity.json");
|
||||
const fixture = resolve(root, "test/fixtures/copilot-profile-v33-identity.json");
|
||||
const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8"));
|
||||
const declaration = { ...historical.declaration, agentProfileVersion: 32 };
|
||||
const declaration = { ...historical.declaration, agentProfileVersion: 33 };
|
||||
delete declaration.permissionIdentitySourceSha256;
|
||||
const sources = {
|
||||
displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts",
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
import { chmodSync, existsSync, realpathSync, writeFileSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import { join, relative } from "node:path";
|
||||
|
||||
export function writePortableExecutableShim(packRoot, name, executable) {
|
||||
const shimPath = join(packRoot, "node_modules", ".bin", name);
|
||||
writeFileSync(
|
||||
shimPath,
|
||||
[
|
||||
"#!/bin/sh",
|
||||
"set -eu",
|
||||
'basedir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)',
|
||||
`exec "$basedir/../${executable}" "$@"`,
|
||||
"",
|
||||
].join("\n"),
|
||||
);
|
||||
chmodSync(shimPath, 0o755);
|
||||
}
|
||||
|
||||
export function writePortableCopilotShims(packRoot) {
|
||||
const packRequire = createRequire(join(packRoot, "package.json"));
|
||||
// Optional platform packages expose native .bin entries. Rewrite only the
|
||||
// installed entries, and resolve their pinned executable without running it.
|
||||
for (const platform of ["darwin-arm64", "darwin-x64", "linux-x64"]) {
|
||||
const name = `copilot-${platform}`;
|
||||
if (!existsSync(join(packRoot, "node_modules", ".bin", name))) continue;
|
||||
const executable = packRequire.resolve(`@github/${name}`);
|
||||
writePortableExecutableShim(
|
||||
packRoot,
|
||||
name,
|
||||
relative(realpathSync(join(packRoot, "node_modules")), realpathSync(executable)),
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import test from "node:test";
|
||||
import { writePortableCopilotShims } from "./provider-pack-executable-shims.mjs";
|
||||
|
||||
function fixture(t) {
|
||||
const root = mkdtempSync(join(tmpdir(), "provider-shims-"));
|
||||
t.after(() => rmSync(root, { recursive: true, force: true }));
|
||||
const pack = join(root, "temporary pack");
|
||||
mkdirSync(join(pack, "node_modules", ".bin"), { recursive: true });
|
||||
writeFileSync(join(pack, "package.json"), '{"type":"module"}');
|
||||
return { root, pack };
|
||||
}
|
||||
|
||||
for (const platform of ["darwin-arm64", "darwin-x64", "linux-x64"]) {
|
||||
test(`relocates the installed ${platform} Copilot command and preserves arguments and exit status`, (t) => {
|
||||
const { root, pack } = fixture(t);
|
||||
const name = `copilot-${platform}`;
|
||||
const installed = join(pack, "node_modules", "@github", name);
|
||||
mkdirSync(installed, { recursive: true });
|
||||
writeFileSync(join(installed, "package.json"), JSON.stringify({
|
||||
name: `@github/${name}`, exports: { ".": "./copilot" }, bin: { [name]: "copilot" },
|
||||
}));
|
||||
// A provider-free executable proves relocation without invoking a real CLI.
|
||||
writeFileSync(join(installed, "copilot"), '#!/bin/sh\nprintf "%s\\n" "$@"\nexit 17\n', { mode: 0o755 });
|
||||
const shim = join(pack, "node_modules", ".bin", name);
|
||||
writeFileSync(shim, `#!/bin/sh\nexport NODE_PATH="${pack}/node_modules"\nexit 99\n`);
|
||||
|
||||
writePortableCopilotShims(pack);
|
||||
assert.equal(readFileSync(shim, "utf8").includes(root), false);
|
||||
const relocated = join(root, "relocated pack");
|
||||
renameSync(pack, relocated);
|
||||
const args = ["argument with spaces", "", "$(not-a-command)"];
|
||||
const result = spawnSync(join(relocated, "node_modules", ".bin", name), args, {
|
||||
cwd: root, env: { PATH: "/usr/bin:/bin" }, encoding: "utf8", timeout: 5_000,
|
||||
});
|
||||
assert.equal(result.error, undefined);
|
||||
assert.equal(result.status, 17);
|
||||
assert.equal(result.stdout, args.join("\n") + "\n");
|
||||
assert.equal(result.stderr, "");
|
||||
});
|
||||
}
|
||||
|
||||
test("does not create commands for absent optional platform packages", (t) => {
|
||||
const { pack } = fixture(t);
|
||||
writePortableCopilotShims(pack);
|
||||
for (const platform of ["darwin-arm64", "darwin-x64", "linux-x64"]) {
|
||||
assert.equal(existsSync(join(pack, "node_modules", ".bin", `copilot-${platform}`)), false);
|
||||
}
|
||||
});
|
||||
|
||||
test("fails when an installed command has no matching package", (t) => {
|
||||
const { pack } = fixture(t);
|
||||
writeFileSync(join(pack, "node_modules", ".bin", "copilot-linux-x64"), "invalid old shim");
|
||||
assert.throws(() => writePortableCopilotShims(pack), { code: "MODULE_NOT_FOUND" });
|
||||
});
|
||||
@@ -1,6 +1,10 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import type { AcpPermissionRequest } from "acpx/runtime";
|
||||
import { normalizeAcpxPermission } from "./acp-permission-adapter.js";
|
||||
import { mkdtempSync, mkdirSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { createCopilotToolEvidence } from "./copilot-tool-evidence.js";
|
||||
|
||||
function request(kinds = ["allow_once", "allow_always", "reject_once"]): AcpPermissionRequest {
|
||||
return { sessionId: "session", inferredKind: "edit", raw: {
|
||||
@@ -42,4 +46,40 @@ describe("ACP permission normalization", () => {
|
||||
const duplicate = request(); duplicate.raw.options[1]!.optionId = duplicate.raw.options[0]!.optionId;
|
||||
expect(() => normalizeAcpxPermission(duplicate)).toThrow("ambiguous");
|
||||
});
|
||||
it("keeps a native absolute workspace alias answerable with the same projected target", () => {
|
||||
const root = mkdtempSync(join(tmpdir(), "copilot-permission-alias-"));
|
||||
try {
|
||||
const workspace = join(root, "workspace"), alias = join(root, "alias");
|
||||
mkdirSync(workspace); mkdirSync(join(workspace, "nested")); symlinkSync(workspace, alias);
|
||||
const path = join(alias, "nested", "new.txt"), value = request();
|
||||
value.raw.toolCall = { toolCallId: "patch-call", kind: "edit", rawInput: { fileName: path }, locations: [{ path }] };
|
||||
const cwd = realpathSync(workspace), normalized = normalizeAcpxPermission(value, { provider: "copilot", workingDirectory: cwd });
|
||||
expect(normalized.title).toBe("Change file: nested/new.txt");
|
||||
expect(normalized.resolve({ action: "accept" })).toEqual({ outcome: "allow_once" });
|
||||
const events: any[] = [];
|
||||
const projection = createCopilotToolEvidence({ sessionId: "session", turnId: "turn", workingDirectory: cwd, active: () => true, emit: event => events.push(event) });
|
||||
projection.permission(value, "permission", normalized.choices.map(choice => choice.key));
|
||||
expect(events[0].payload.details).toContainEqual({ name: "target", value: "nested/new.txt" });
|
||||
writeFileSync(join(workspace, "nested", "new.txt"), "existing");
|
||||
expect(normalizeAcpxPermission(value, { provider: "copilot", workingDirectory: cwd }).title).toBe(normalized.title);
|
||||
} finally { rmSync(root, { recursive: true, force: true }); }
|
||||
});
|
||||
it("refuses outside, dangling, conflicting and unresolvable alias targets", () => {
|
||||
const root = mkdtempSync(join(tmpdir(), "copilot-permission-alias-"));
|
||||
try {
|
||||
const workspace = join(root, "workspace"), alias = join(root, "alias"), outside = join(root, "outside");
|
||||
mkdirSync(workspace); mkdirSync(outside); symlinkSync(workspace, alias);
|
||||
symlinkSync(join(outside, "missing.txt"), join(workspace, "dangling.txt"));
|
||||
symlinkSync(outside, join(workspace, "escape"));
|
||||
writeFileSync(join(outside, "existing.txt"), "outside");
|
||||
for (const path of [join(alias, "dangling.txt"), join(alias, "escape", "new.txt"), join(alias, "escape", "existing.txt"), join(alias, "missing-parent", "new.txt"), join(outside, "new.txt")]) {
|
||||
const value = request(); value.raw.toolCall = { toolCallId: "patch-call", kind: "edit", rawInput: { fileName: path }, locations: [{ path }] };
|
||||
const normalized = normalizeAcpxPermission(value, { provider: "copilot", workingDirectory: realpathSync(workspace) });
|
||||
expect(normalized.title).toContain("target unavailable");
|
||||
expect(normalized.choices.map(choice => choice.key)).toEqual(["decline", "cancel"]);
|
||||
}
|
||||
const conflict = request(); conflict.raw.toolCall = { toolCallId: "patch-call", kind: "edit", rawInput: { fileName: join(alias, "one.txt") }, locations: [{ path: join(alias, "two.txt") }] };
|
||||
expect(normalizeAcpxPermission(conflict, { provider: "copilot", workingDirectory: realpathSync(workspace) }).title).toContain("target unavailable");
|
||||
} finally { rmSync(root, { recursive: true, force: true }); }
|
||||
});
|
||||
});
|
||||
@@ -70,7 +70,7 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi
|
||||
|
||||
describe("Copilot build-owned installation", () => {
|
||||
it("admits the current declaration and binds its source policy, receipts and patch hashes", () => {
|
||||
const identity = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v32-identity.json", import.meta.url), "utf8"));
|
||||
const identity = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v33-identity.json", import.meta.url), "utf8"));
|
||||
expect(identity.declaration.systemInstructionDelivery).toBe(COPILOT_SYSTEM_INSTRUCTION_DELIVERY);
|
||||
expect(identity.declaration.sharedRuntimeContract).toBe("paperclip.acpx-runtime-contract.v1");
|
||||
expect(identity.declaration.permissionContextContract).toBe(COPILOT_PERMISSION_CONTEXT_CONTRACT);
|
||||
@@ -101,7 +101,7 @@ describe("Copilot build-owned installation", () => {
|
||||
expect([...closure].filter(path => !bound.has(path))).toEqual(["negative-control:unbound-policy"]);
|
||||
});
|
||||
it.each([...PERMISSION_POLICY_SOURCES, ...COPILOT_DISPLAY_SOURCES, ...SEMANTIC_RECEIPT_SOURCES])("changing %s changes the candidate identity", (_path, field) => {
|
||||
const identity = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v32-identity.json", import.meta.url), "utf8"));
|
||||
const identity = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v33-identity.json", import.meta.url), "utf8"));
|
||||
identity.declaration[field] = "0".repeat(64);
|
||||
const sorted = Object.fromEntries(Object.entries(identity.declaration).sort(([a], [b]) => a.localeCompare(b)));
|
||||
expect(`sha256:${createHash("sha256").update(JSON.stringify(sorted)).digest("hex")}`).not.toBe(identity.commandDigest);
|
||||
|
||||
@@ -1,10 +1,35 @@
|
||||
import { redactPaperclipSemanticValue } from "../../semantic-tools/redaction.js";
|
||||
import { safeAcpxLocations } from "./safe-locations.js";
|
||||
import { lstatSync, realpathSync } from "node:fs";
|
||||
import { basename, dirname, isAbsolute, join } from "node:path";
|
||||
|
||||
export const COPILOT_PERMISSION_CONTEXT_CONTRACT = "copilot-edit-permission-context-v1";
|
||||
const record = (value: unknown): Record<string, unknown> =>
|
||||
value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
||||
|
||||
/** Native providers can spell an absolute workspace path through a host alias
|
||||
* (macOS /tmp and /private/tmp). Attest the real entry, or its existing parent
|
||||
* for a new file, before converting it to a display target. */
|
||||
function aliasedLocation(path: string, cwd: string): string | undefined {
|
||||
if (!isAbsolute(path)) return;
|
||||
try {
|
||||
const canonicalCwd = realpathSync(cwd);
|
||||
let canonicalPath: string;
|
||||
try {
|
||||
lstatSync(path);
|
||||
canonicalPath = realpathSync(path);
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== "ENOENT") return;
|
||||
// A dangling symlink must not become an apparently safe create target.
|
||||
try { lstatSync(path); return; } catch (missing) {
|
||||
if ((missing as NodeJS.ErrnoException).code !== "ENOENT") return;
|
||||
}
|
||||
canonicalPath = join(realpathSync(dirname(path)), basename(path));
|
||||
}
|
||||
return safeAcpxLocations([{ path: canonicalPath }], canonicalCwd, "edit")[0]?.path as string | undefined;
|
||||
} catch { return; }
|
||||
}
|
||||
|
||||
/** Display context only; this does not authorize filesystem access. */
|
||||
export function safeCopilotEditTarget(value: unknown, workingDirectory: string | undefined): string | undefined {
|
||||
const call = record(value);
|
||||
@@ -14,7 +39,12 @@ export function safeCopilotEditTarget(value: unknown, workingDirectory: string |
|
||||
if (Array.isArray(call.locations) && (call.locations.length > 16 || call.locations.some(x => typeof record(x).path !== "string"))) return;
|
||||
const paths = [input.path, input.fileName, ...(Array.isArray(call.locations) ? call.locations.map(x => record(x).path) : [])].filter(x => x !== undefined);
|
||||
if (!paths.length || paths.some(x => typeof x !== "string" || !x || x.trim() !== x || x.length > 2048 || /[\\\p{Cc}\p{Cf}\p{Zl}\p{Zp}]/u.test(x))) return;
|
||||
const safe = paths.flatMap(path => safeAcpxLocations([{ path }], workingDirectory, call.kind, call.title));
|
||||
const safe = paths.flatMap(path => {
|
||||
const locations = safeAcpxLocations([{ path }], workingDirectory, call.kind, call.title);
|
||||
if (locations.length) return locations;
|
||||
const aliased = aliasedLocation(path as string, workingDirectory);
|
||||
return aliased ? [{ path: aliased }] : [];
|
||||
});
|
||||
const names = safe.map(x => x.path).filter((x): x is string => typeof x === "string" && x.length <= 1024 && x.trim() === x && !x.includes(":"));
|
||||
if (names.length !== paths.length || new Set(names).size !== 1) return;
|
||||
const target = names[0];
|
||||
|
||||
@@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = {
|
||||
"protocolVersion": 1,
|
||||
"acpxVersion": "0.13.1",
|
||||
"agent": "copilot",
|
||||
"agentProfileVersion": 32,
|
||||
"agentProfileVersion": 33,
|
||||
"agentServerPackage": "@github/copilot",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"agentRuntimePackage": null,
|
||||
"agentRuntimeVersion": null,
|
||||
"commandDigest": "sha256:3dcc8de3034e32001b0a819a614beca345d314266a70a09e0050a9f5f58275fc",
|
||||
"commandDigest": "sha256:7b459cc43771d4178898be8a362030ac2e4d8356c2a14a82334958959dd54e59",
|
||||
"permissionPolicy": "interactive"
|
||||
},
|
||||
"claude": {
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
{
|
||||
"commandDigest": "sha256:7b459cc43771d4178898be8a362030ac2e4d8356c2a14a82334958959dd54e59",
|
||||
"declaration": {
|
||||
"schema": "paperclip.acpx_profile_declaration.v1",
|
||||
"agent": "copilot",
|
||||
"agentProfileVersion": 33,
|
||||
"acpxVersion": "0.13.1",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"protocolPolicyRevision": "copilot-agent-manual-v1",
|
||||
"acpxPatchSha256": "fedf1e498c1e19daf9c973d475234488aa9acabbf1b9de890245bfac7a534483",
|
||||
"policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3",
|
||||
"distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107",
|
||||
"agentFilesBinding": "registered-runtime-context-v1",
|
||||
"systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1",
|
||||
"sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1",
|
||||
"innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a",
|
||||
"upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0",
|
||||
"patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679",
|
||||
"messageIdentityContract": "copilot-native-message-id-v1",
|
||||
"ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1",
|
||||
"permissionContextContract": "copilot-edit-permission-context-v1",
|
||||
"permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b",
|
||||
"permissionAdapterSourceSha256": "eebcc1ec9bc8caa8d1edf0cbda74bd69f5c55cf8d65a0503e95798b9979a19dc",
|
||||
"permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9",
|
||||
"permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f",
|
||||
"permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4",
|
||||
"semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2",
|
||||
"semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791",
|
||||
"semanticBridgeSourceSha256": "cf136da18e2cdbefae8e01d23d2c93f64eb7aebccaec938e84c2887c888f87ee",
|
||||
"toolEvidenceSourceSha256": "4b517eed447615b3cc820af348ccc360978175149caa0bbc838d1f78332abe75",
|
||||
"semanticNormalizedInputContract": "validated-forwarded-input-commit-v1",
|
||||
"semanticSidecarSourceSha256": "49f7cb882fa09dcf879a2234d3e7b43fe72895d2c0f1e9b72fcbe18f012423bb",
|
||||
"semanticDirectDriverSourceSha256": "3cea77a03c8822d68f77f090457176a9190facbec7d88ddf8c583c0e42245f70",
|
||||
"semanticValidationSourceSha256": "3b0ac8d3f3cb92747d0bcea472ab0419a140e1cfdc43047e2c22542acc2d105f",
|
||||
"semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e",
|
||||
"semanticCompletionContractSourceSha256": "fa2b0c81cb9a62965839c905db81b930794774d8d557a01bce077dae5830b594",
|
||||
"semanticValidatorsSourceSha256": "328436d29148e0b79ce36f36c764b3d780913bdb90d174d1ea03cf1f59b5ef13",
|
||||
"semanticSchemaBundleSourceSha256": "d8cef4a9ebc136a3d37f3d0be7161986e6d65420e2ec734cfd0dfa468c665c97",
|
||||
"semanticSidecarProtocolSourceSha256": "907b2cdc1a5291c6fb570855bbcc66cb196b5cf64195d5dc201148940678460a",
|
||||
"semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1",
|
||||
"displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877",
|
||||
"displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e",
|
||||
"configuredEnvironmentSourceSha256": "a03575b3a737dea60da94cf0fa1feed80719a9aaa7f09456eb57a845d9e8e087",
|
||||
"configuredEnvironmentRustSourceSha256": "55d44e48e217385b5c64b1b7d8df16dda62319ca3fb524e452fd78c9ed804d3b",
|
||||
"profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282",
|
||||
"profileExtensionsSourceSha256": "0565d13d4064ed8c501d6ffda6d7bffca4493552de8aaa7fc12178dabb3ca435",
|
||||
"profileInstallationSourceSha256": "7c26545922fc85db75a5db6bf3f6e21e381b6e189f4f51243b19ee237b93f8b1",
|
||||
"attachmentPolicySourceSha256": "ce8027d1f8e2b7fd35df56a75ba162fd9696d1569c016b60bc9f0de1252cd958",
|
||||
"singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f",
|
||||
"nativeSessionRuntimeSourceSha256": "aac818c1cf50a25899ae95cafc70f093c8cd8af37d19f88cbb7847b4d202d2b6",
|
||||
"nativeRuntimeContextSourceSha256": "af192af3fa511347045dfdb86d443be9aa046a4c56bdb48198668cf4d30581b8",
|
||||
"nativeSessionContractSourceSha256": "93251b1eadc686e9d8339d4dfc50b995a3d0a9e6ad8aedc6f2bdf7e30c16138d",
|
||||
"harnessSessionContractSourceSha256": "75ae4607312ec13466dd2f4ac87f000302b7b5a952cb6d4b51573fa8b7c35d82",
|
||||
"harnessBackendSourceSha256": "3b11fd93b82ef13d5e6396217e3f102f900437f4d32efb00a9d75a9d2214e6ff",
|
||||
"harnessSessionSourceSha256": "a9d748b8530544342bbc9f59968816664872782a0865db76ca15f28a1705697b",
|
||||
"transportContractSourceSha256": "16f358c75dd4ae80af3ca921394c5899b15942b80b553e43ddfabb41c1bf31d3",
|
||||
"nativeBackendSourceSha256": "a23afeab7e7094cb7e3df05e7a9eac822d0144d8c44893dbcc8a0e7b2bce0c85",
|
||||
"nativeBackendFactorySourceSha256": "ce22825a53669f8c2db5aaa890bae94af19ede22295bda4ed7987a31d804cd8a",
|
||||
"nativeInstructionsSourceSha256": "ea85be2aca8dfbd6b4042dcd7b1ed9db9a57baf8c379649a3f5d7d80ca54a6a2",
|
||||
"runtimeRequestBridgeSourceSha256": "c449568f8abf7e90d5ae7a83b49809d9166461398f8b3046e4fe5f3367f3b052",
|
||||
"runtimeRequestFacadeSourceSha256": "3d70c6025169779f7886ddf363a1c6fda4add5d14fa3cd49aea46aa1f6a9d8ec",
|
||||
"runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211",
|
||||
"guardedInstallationSourceSha256": "8afd25df307772e484e179e58bf54ce1510fe92be694482111d589f9f0ae6e77",
|
||||
"nativeExecutionSourceSha256": "4cdd702c233bf0d086ff2aeea8f02c6db8817ba0c0bcb5e64e355ee7a1c2dd77",
|
||||
"profileCompatibilitySourceSha256": "fe521686dbd9884eec96ba0963ec00b6b58f4d891aa0292e0dc18b0d84770646",
|
||||
"installationSourceSha256": "9934653d6983905930099b1691b1f0c05461e04e950bb27dd0f92e64fc8e28e1",
|
||||
"assetOwnershipSourceSha256": "43005b7e896e0546c1f44cbe4ff3f34ad6d71f275f4b9a8113f73a71e77930c4",
|
||||
"assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f",
|
||||
"credentialBindingSourceSha256": "d5c200bea2e8ab730eb56e32ce2070c8a1b5407e5907108640f76a463d9b9c9c",
|
||||
"sidecarErrorSourceSha256": "23f3dbdc1ce488c690d218a854c820f4b4f3edd822bad574d67707314f8c512a",
|
||||
"profileSourceSha256": "baf0b64371fada6c92331f9f88e915292b2761c029fd86452f6de32e26dc7fd6",
|
||||
"metadataProbeSourceSha256": "826e74747a89b83a6f546ee1355de64ae5265eb920b50ae498a41dd561a493ee",
|
||||
"permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1",
|
||||
"runtimeHostSourceSha256": "83a5ae07c19681ab195f76363661c2381494b243343270ba363d4db2088932a5",
|
||||
"runtimeAdapterSourceSha256": "a1a8784f5883d2c93df6e3695d5499b83e226efb4f26e558639c4e7dbcc2d67a",
|
||||
"runtimeSandboxSourceSha256": "4acec4c9757fe5ef5f2788ce20ced8ca4fde8edcfb63b59d0effcfd4316c048d",
|
||||
"cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca",
|
||||
"agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba",
|
||||
"nativeDistributionSourceSha256": "9bbe8329064ee88418528394f8e1cdc716a0b2b35092dc70729b8850e46b7a68"
|
||||
}
|
||||
}
|
||||
@@ -129,8 +129,11 @@ export const patchInstanceSettingsSchema = z.object({
|
||||
export const MAX_TASK_DRAIN_TTL_MS = 24 * 60 * 60 * 1000;
|
||||
|
||||
export const startTaskDrainRequestSchema = z.object({
|
||||
purpose: z.literal("idle").optional(),
|
||||
ttlMs: z.number().int().positive().max(MAX_TASK_DRAIN_TTL_MS).nullable().optional(),
|
||||
}).strict();
|
||||
}).strict().refine(value => value.purpose !== "idle" ||
|
||||
(typeof value.ttlMs === "number" && value.ttlMs >= 5_000 && value.ttlMs <= 300_000),
|
||||
{ message: "Idle holds require ttlMs between 5000 and 300000", path: ["ttlMs"] });
|
||||
|
||||
export type InstanceGeneralSettings = z.infer<typeof instanceGeneralSettingsSchema>;
|
||||
// The patch schema removes each default so an absent key stays absent. Declare
|
||||
|
||||
@@ -18,6 +18,8 @@ import {
|
||||
import { routineWebhookUrl } from "../services/routines.js";
|
||||
import { paperclipCloudConnectorEnrollmentStatus } from "../services/paperclip-cloud-connector-enrollment.js";
|
||||
import { cloudRuntimeIdentityMiddleware } from "../middleware/cloud-runtime-identity.js";
|
||||
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "../middleware/idle-admission.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { healthRoutes } from "../routes/health.js";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
@@ -186,6 +188,37 @@ describeEmbeddedPostgres("Cloud runtime identity", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it.each([false, true])("counts exempt bootstrap writes until they settle (failure=%s)", async (fail) => {
|
||||
let enterWrite!: () => void, finishWrite!: () => void;
|
||||
const entered = new Promise<void>(resolve => { enterWrite = resolve; });
|
||||
const pending = new Promise<void>(resolve => { finishWrite = resolve; });
|
||||
const transaction = db.transaction.bind(db);
|
||||
vi.spyOn(db, "transaction").mockImplementationOnce(async (...args) => {
|
||||
enterWrite();
|
||||
await pending;
|
||||
if (fail) throw new Error("fixture bootstrap write failed");
|
||||
return transaction(...args);
|
||||
});
|
||||
const app = express();
|
||||
app.use(idleAdmissionMiddleware);
|
||||
app.use(cloudRuntimeIdentityMiddleware(db));
|
||||
app.get("/api/health", (_req, res) => res.sendStatus(204));
|
||||
trackIdleRequestHandlers(app);
|
||||
const requestTime = Math.floor(Date.now() / 1000);
|
||||
const accepted = request(app).get("/api/health")
|
||||
.set("x-paperclip-cloud-runtime-identity", assertion({ claims: { iat: requestTime, exp: requestTime + 300 } }))
|
||||
.then(response => response);
|
||||
try {
|
||||
await entered;
|
||||
expect(idleWorkSnapshot().active).toBe(1);
|
||||
} finally {
|
||||
finishWrite();
|
||||
const response = await accepted;
|
||||
expect(response.status).toBe(fail ? 401 : 204);
|
||||
}
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
});
|
||||
|
||||
it("keeps probes idle until a verified durable claim and stays active across restart", async () => {
|
||||
const env = { ...process.env, PAPERCLIP_CLOUD_WARM_STANDBY: "1" };
|
||||
const isStandby = await createCloudWarmStandby(db, env);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
@@ -2372,6 +2373,11 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
});
|
||||
|
||||
it("buffers the orphan in-process when the durable spool write also fails", async () => {
|
||||
const idleBefore = idleWorkSnapshot().active;
|
||||
let releaseFlush!: () => void;
|
||||
let enteredFlush!: () => void;
|
||||
const flushGate = new Promise<void>(resolve => { releaseFlush = resolve; });
|
||||
const flushEntered = new Promise<void>(resolve => { enteredFlush = resolve; });
|
||||
const { companyId, environment, runId } = await seedEnvironment({
|
||||
driver: "sandbox",
|
||||
name: "Foreign-bound Fake Sandbox Cleanup Spool Unwritable",
|
||||
@@ -2413,7 +2419,8 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
if (databaseDown) {
|
||||
return Promise.reject(new Error("pending-cleanup write failed; database down"));
|
||||
}
|
||||
return real.insertPendingCleanupLease(input);
|
||||
enteredFlush();
|
||||
return flushGate.then(() => real.insertPendingCleanupLease(input));
|
||||
},
|
||||
};
|
||||
});
|
||||
@@ -2454,11 +2461,18 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
"sandbox_orphan_cleanup_write_failed",
|
||||
);
|
||||
expect(fallbackLog?.[0]).toMatchObject({ persisted: false, buffered: true });
|
||||
expect(idleWorkSnapshot().active).toBe(idleBefore + 1);
|
||||
|
||||
// The same runtime still keeps the orphan in-process, so a flush after the
|
||||
// database recovers lands the durable row.
|
||||
databaseDown = false;
|
||||
const flushed = await runtime.flushDeferredOrphanCleanups();
|
||||
const flushing = runtime.flushDeferredOrphanCleanups();
|
||||
await flushEntered;
|
||||
// The batch was spliced out of the queue, but the DB does not own it yet.
|
||||
expect(idleWorkSnapshot().active).toBe(idleBefore + 1);
|
||||
releaseFlush();
|
||||
const flushed = await flushing;
|
||||
expect(idleWorkSnapshot().active).toBe(idleBefore);
|
||||
expect(flushed).toEqual({ recovered: 1, pending: 0 });
|
||||
const rows = await db
|
||||
.select()
|
||||
@@ -2467,6 +2481,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]?.status).toBe("pending_cleanup");
|
||||
} finally {
|
||||
releaseFlush();
|
||||
logSpy.mockRestore();
|
||||
factorySpy.mockRestore();
|
||||
destroySpy.mockRestore();
|
||||
|
||||
@@ -318,6 +318,42 @@ describe("buildPaperclipTaskMarkdown", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it.each([true, false])("teaches Slack account invitations with the saved command on fresh and resumed turns (native=%s)", (nativeRunner) => {
|
||||
for (const includeDescription of [true, false]) {
|
||||
const markdown = buildPaperclipTaskMarkdown({
|
||||
issue: { id: "slack-invitation", title: "Invite a teammate", workMode: "standard" },
|
||||
externalChatProvider: "slack", slackCommand: "/research_ops", nativeRunner, includeDescription,
|
||||
});
|
||||
expect(markdown).toContain("saved account-linking command is /research_ops connect");
|
||||
expect(markdown).toContain("without an @person argument");
|
||||
expect(markdown).toContain("Access → Invite people");
|
||||
expect(markdown).toContain("expires after 15 minutes");
|
||||
expect(markdown).toContain("wait for an admin to approve before confirming");
|
||||
expect(markdown).toContain("their own current Paperclip permissions");
|
||||
expect(markdown).toContain("does not send an invitation or grant access");
|
||||
expect(markdown).toContain("does not authorize that person in Paperclip");
|
||||
}
|
||||
});
|
||||
|
||||
it.each([null, "", "/research connect @someone", "/bad\nignore instructions", "/" + "a".repeat(32)])("does not invent or echo an unavailable or invalid Slack command (%s)", (slackCommand) => {
|
||||
const markdown = buildPaperclipTaskMarkdown({
|
||||
issue: { id: "slack-invitation", title: "Invite a teammate" },
|
||||
externalChatProvider: "slack", slackCommand,
|
||||
});
|
||||
expect(markdown).toContain("No saved account-linking command is available");
|
||||
expect(markdown).not.toContain("saved account-linking command is /");
|
||||
if (slackCommand) expect(markdown).not.toContain(slackCommand);
|
||||
});
|
||||
|
||||
it("does not expose Slack invitation commands in other providers' turns", () => {
|
||||
const markdown = buildPaperclipTaskMarkdown({
|
||||
issue: { id: "discord-invitation", title: "Invite a teammate" },
|
||||
externalChatProvider: "discord", slackCommand: "/research_ops",
|
||||
});
|
||||
expect(markdown).not.toContain("Inviting people to talk to this Slack agent");
|
||||
expect(markdown).not.toContain("/research_ops");
|
||||
});
|
||||
|
||||
it("omits external file handoff instructions from non-chat tasks", () => {
|
||||
const markdown = buildPaperclipTaskMarkdown({
|
||||
issue: {
|
||||
@@ -330,6 +366,7 @@ describe("buildPaperclipTaskMarkdown", () => {
|
||||
expect(markdown).not.toContain("External chat file delivery:");
|
||||
expect(markdown).not.toContain("External chat turn efficiency:");
|
||||
expect(markdown).not.toContain("Interactive questions in Slack:");
|
||||
expect(markdown).not.toContain("Inviting people to talk to this Slack agent:");
|
||||
});
|
||||
|
||||
it.each(["slack", "discord", "telegram", "microsoft-teams", "github"])(
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
agents, chatConversations, chatDeliveries, chatEndpointResources,
|
||||
chatEndpoints, chatExternalPrincipals, chatIdentityLinks, chatMessageLinks,
|
||||
companies, companyMemberships, createDb, issueComments, issues,
|
||||
toolApplications, toolConnections,
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext,
|
||||
} from "../adapters/index.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
|
||||
const postgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describePostgres = postgresSupport.supported ? describe : describe.skip;
|
||||
if (!postgresSupport.supported) {
|
||||
console.warn(`Skipping Slack invitation heartbeat tests: ${postgresSupport.reason ?? "embedded Postgres unavailable"}`);
|
||||
}
|
||||
|
||||
describePostgres("saved Slack invitation command in heartbeat prompts", () => {
|
||||
const adapterType = "slack_invitation_test";
|
||||
const captured = new Map<string, AdapterExecutionContext>();
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let heartbeat: ReturnType<typeof heartbeatService>;
|
||||
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("slack-invitation-");
|
||||
db = createDb(temporary.connectionString);
|
||||
heartbeat = heartbeatService(db);
|
||||
registerServerAdapter({
|
||||
type: adapterType,
|
||||
supportsLocalAgentJwt: false,
|
||||
execute: async (input) => {
|
||||
captured.set(input.runId, input);
|
||||
return { exitCode: 0, signal: null, timedOut: false, sessionId: "invitation-session", resultJson: {} };
|
||||
},
|
||||
testEnvironment: async () => ({
|
||||
adapterType, status: "pass", checks: [], testedAt: new Date().toISOString(),
|
||||
}),
|
||||
});
|
||||
}, 30_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await heartbeat?.drainActiveRunExecutions();
|
||||
unregisterServerAdapter(adapterType);
|
||||
await db?.$client.end({ timeout: 0 });
|
||||
await temporary?.cleanup();
|
||||
});
|
||||
|
||||
async function seed() {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID();
|
||||
const endpointId = randomUUID(), resourceId = randomUUID(), conversationId = randomUUID();
|
||||
const applicationId = randomUUID(), connectionId = randomUUID(), principalId = randomUUID();
|
||||
const userId = `invitation-user-${randomUUID()}`;
|
||||
await db.insert(companies).values({ id: companyId, name: "Invitation fixture", issuePrefix: `I${companyId.slice(0, 6)}` });
|
||||
await db.insert(agents).values({
|
||||
id: agentId, companyId, name: "Original researcher", adapterType, status: "idle",
|
||||
});
|
||||
await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });
|
||||
await db.insert(toolApplications).values({ id: applicationId, companyId, applicationKey: `chat:slack:${endpointId}`, name: "Slack", type: "chat", status: "active" });
|
||||
await db.insert(toolConnections).values({ id: connectionId, companyId, applicationId, name: "Slack", uid: endpointId, connectionPurpose: "channel", transport: "chat_sdk", status: "active", enabled: true });
|
||||
await db.insert(chatEndpoints).values({
|
||||
id: endpointId, companyId, connectionId, provider: "slack", publicId: randomUUID(),
|
||||
assignedAgentId: agentId, status: "active", providerAccountId: "T-INVITE",
|
||||
setup: { step: "complete", command: "/research_ops", providerUrl: "https://setup-canary-secret.invalid" },
|
||||
allowDirectMessages: true, allowUnlinkedPeople: false,
|
||||
});
|
||||
await db.insert(chatEndpointResources).values({ id: resourceId, companyId, endpointId, type: "direct_message", providerResourceId: "D-INVITE", label: "DM", availability: "available", enabled: true });
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Invite a teammate", description: "Help teammates connect their own Slack accounts.", status: "todo", assigneeAgentId: agentId, responsibleUserId: userId, issueNumber: 1, originKind: "chat_channel" });
|
||||
await db.insert(chatConversations).values({ id: conversationId, companyId, endpointId, resourceId, issueId, externalConversationId: "D-INVITE", externalThreadId: "slack:D-INVITE:1", externalLabel: "DM", sessionGeneration: 1, isDirectMessage: true, state: "active" });
|
||||
await db.insert(chatExternalPrincipals).values({ id: principalId, companyId, provider: "slack", providerAccountId: "T-INVITE", externalId: "U-INVITE", kind: "user" });
|
||||
await db.insert(chatIdentityLinks).values({ companyId, endpointId, principalId, paperclipUserId: userId, status: "linked" });
|
||||
// Renaming must not change the provider's registered command.
|
||||
await db.update(agents).set({ name: "Renamed analyst" }).where(eq(agents.id, agentId));
|
||||
return { companyId, agentId, issueId, endpointId, conversationId, principalId, userId };
|
||||
}
|
||||
|
||||
async function turn(f: Awaited<ReturnType<typeof seed>>) {
|
||||
const commentId = randomUUID(), deliveryId = randomUUID();
|
||||
await db.insert(issueComments).values({ id: commentId, companyId: f.companyId, issueId: f.issueId, authorType: "user", authorUserId: f.userId, body: "How can a teammate talk to you?" });
|
||||
await db.insert(chatDeliveries).values({ id: deliveryId, companyId: f.companyId, endpointId: f.endpointId, conversationId: f.conversationId, principalId: f.principalId, providerEventId: commentId, deduplicationKey: commentId, eventKind: "message", normalizedEvent: {}, state: "processed", attempts: 1, processedAt: new Date() });
|
||||
await db.insert(chatMessageLinks).values({ companyId: f.companyId, endpointId: f.endpointId, conversationId: f.conversationId, deliveryId, commentId, providerMessageId: commentId, direction: "inbound" });
|
||||
const run = await heartbeat.wakeup(f.agentId, {
|
||||
source: "automation", triggerDetail: "system", reason: "issue_commented",
|
||||
payload: { issueId: f.issueId, commentId },
|
||||
contextSnapshot: {
|
||||
issueId: f.issueId, taskId: f.issueId, source: "chat:slack", commentId,
|
||||
wakeCommentId: commentId, wakeCommentIds: [commentId],
|
||||
// Caller fields must not replace the durable endpoint command.
|
||||
slackCommand: "/forged", chatSlackCommand: "/forged",
|
||||
},
|
||||
requestedByActorType: "user", requestedByActorId: f.userId,
|
||||
});
|
||||
expect(run).not.toBeNull();
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
const input = captured.get(run!.id);
|
||||
expect(input).toBeDefined();
|
||||
return input!;
|
||||
}
|
||||
|
||||
it("loads the custom saved command for fresh and resumed prompts after an agent rename", async () => {
|
||||
const f = await seed();
|
||||
for (let index = 0; index < 2; index++) {
|
||||
const input = await turn(f);
|
||||
expect(input.runtime.sessionId).toBe(index === 0 ? null : "invitation-session");
|
||||
const context = input.context;
|
||||
for (const key of ["paperclipTaskMarkdown", "paperclipTaskMarkdownCompact", "paperclipTaskMarkdownAssignment", "paperclipTaskMarkdownAssignmentCompact"]) {
|
||||
const prompt = String(context[key]);
|
||||
expect(prompt).toContain("saved account-linking command is /research_ops connect");
|
||||
expect(prompt).not.toContain("/renamed");
|
||||
expect(prompt).not.toContain("/forged");
|
||||
expect(prompt).not.toContain("setup-canary-secret");
|
||||
}
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it("uses the Access page fallback when the saved command is missing", async () => {
|
||||
const f = await seed();
|
||||
await db.update(chatEndpoints).set({ setup: { step: "complete" } }).where(eq(chatEndpoints.id, f.endpointId));
|
||||
const { context } = await turn(f);
|
||||
expect(context.paperclipTaskMarkdown).toContain("No saved account-linking command is available");
|
||||
expect(context.paperclipTaskMarkdown).toContain("Access → Invite people");
|
||||
expect(context.paperclipTaskMarkdown).not.toContain("/forged");
|
||||
}, 30_000);
|
||||
|
||||
it.each(["inactive", "different-agent"])("excludes the command from an unauthorized endpoint (%s)", async (change) => {
|
||||
const f = await seed();
|
||||
if (change === "inactive") {
|
||||
await db.update(chatEndpoints).set({ status: "paused" }).where(eq(chatEndpoints.id, f.endpointId));
|
||||
} else {
|
||||
const otherId = randomUUID();
|
||||
await db.insert(agents).values({ id: otherId, companyId: f.companyId, name: "Other agent", adapterType, status: "idle" });
|
||||
await db.update(chatEndpoints).set({ assignedAgentId: otherId }).where(eq(chatEndpoints.id, f.endpointId));
|
||||
}
|
||||
const { context } = await turn(f);
|
||||
expect(context.paperclipTaskMarkdown).not.toContain("/research_ops");
|
||||
expect(context.paperclipTaskMarkdown).not.toContain("/forged");
|
||||
}, 30_000);
|
||||
});
|
||||
@@ -0,0 +1,76 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import { agents, companies, createDb, heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { createRunUsageRecorder } from "../services/usage-receipts.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("idle accounting work", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
const directories: string[] = [];
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("idle-accounting-");
|
||||
db = createDb(database.connectionString);
|
||||
}, 60_000);
|
||||
afterEach(async () => {
|
||||
await db.execute(sql`TRUNCATE companies CASCADE`);
|
||||
for (const directory of directories.splice(0)) await fs.rm(directory, { force: true, recursive: true });
|
||||
});
|
||||
afterAll(async () => { await database?.cleanup(); });
|
||||
|
||||
async function recorder() {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Accounting idle test", issuePrefix: "ACCT" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Test", role: "engineer", status: "idle", adapterType: "process" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", status: "running" });
|
||||
const directory = await fs.mkdtemp(path.join(os.tmpdir(), "idle-accounting-"));
|
||||
directories.push(directory);
|
||||
let offline = false;
|
||||
const proxy = new Proxy(db, { get(target, property) {
|
||||
if (property === "transaction" && offline) return () => Promise.reject(new Error("Database unavailable"));
|
||||
const value = Reflect.get(target, property);
|
||||
return typeof value === "function" ? value.bind(target) : value;
|
||||
} }) as Db;
|
||||
const recorder = await createRunUsageRecorder(proxy, { companyId, runId, adapterType: "process" }, directory);
|
||||
return { recorder, directory, runId, setOffline: (value: boolean) => { offline = value; } };
|
||||
}
|
||||
|
||||
it("counts queued receipts until capture and accounting writes settle", async () => {
|
||||
const before = idleWorkSnapshot().active;
|
||||
const { recorder: usage } = await recorder();
|
||||
const first = usage.capture({ complete: false, usage: { inputTokens: 1, outputTokens: 1 } });
|
||||
const second = usage.capture({ complete: true, usage: { inputTokens: 2, outputTokens: 2 } });
|
||||
expect(idleWorkSnapshot().active).toBe(before + 2);
|
||||
await Promise.all([first, second]);
|
||||
await usage.persistFailure();
|
||||
expect(idleWorkSnapshot().active).toBe(before);
|
||||
});
|
||||
|
||||
it("keeps unpersisted capture failure as a blocker until its failure fence reaches the database", async () => {
|
||||
const before = idleWorkSnapshot().active;
|
||||
const { recorder: usage, directory, runId, setOffline } = await recorder();
|
||||
await fs.rm(directory, { recursive: true });
|
||||
await fs.writeFile(directory, "unwritable spool");
|
||||
setOffline(true);
|
||||
await expect(usage.capture({ complete: true, usage: { inputTokens: 1, outputTokens: 1 } })).rejects.toThrow();
|
||||
await expect(usage.persistFailure()).rejects.toThrow();
|
||||
expect(idleWorkSnapshot().active).toBe(before + 1);
|
||||
setOffline(false);
|
||||
// A successful query which updates zero rows is not a durable fence.
|
||||
await db.update(heartbeatRuns).set({ costAccountedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
|
||||
await usage.persistFailure();
|
||||
expect(idleWorkSnapshot().active).toBe(before + 1);
|
||||
await db.update(heartbeatRuns).set({ costAccountedAt: null }).where(eq(heartbeatRuns.id, runId));
|
||||
await usage.persistFailure();
|
||||
expect(idleWorkSnapshot().active).toBe(before);
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(run.costAccountingPending).toBe(true);
|
||||
expect(run.usageJson?.accountingCaptureFailed).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,103 @@
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { request as httpRequest } from "node:http";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "../middleware/idle-admission.js";
|
||||
import { idleWorkSnapshot, startTaskDrain, stopTaskDrain, readTaskDrain, beginIdleTrackedWork } from "../services/task-admission.js";
|
||||
|
||||
function deferred() {
|
||||
let resolve!: () => void;
|
||||
const promise = new Promise<void>(done => { resolve = done; });
|
||||
return { promise, resolve };
|
||||
}
|
||||
function app() { const app = express(); app.use(idleAdmissionMiddleware); return app; }
|
||||
afterEach(() => stopTaskDrain());
|
||||
|
||||
describe("idle admission", () => {
|
||||
it("preserves legacy drains, but blocks all new work during an owned idle hold", async () => {
|
||||
const server = app();
|
||||
server.all("/{*path}", (_req, res) => res.sendStatus(204));
|
||||
trackIdleRequestHandlers(server);
|
||||
startTaskDrain({ ttlMs: 60_000 });
|
||||
await request(server).post("/api/issues").expect(204);
|
||||
startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
||||
for (const path of ["/api/issues", "/api/chat-webhooks/test", "/mcp", "/api/auth/callback"]) {
|
||||
await request(server).get(path).expect(503).expect("Retry-After", "1");
|
||||
await request(server).post(path).expect(503);
|
||||
}
|
||||
await request(server).get("/api/health").expect(204);
|
||||
await request(server).get("/api/instance/task-drain/").expect(204);
|
||||
await request(server).post("/api/instance/task-drain").expect(204);
|
||||
stopTaskDrain();
|
||||
await request(server).post("/api/issues").expect(204);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
});
|
||||
|
||||
it("counts a nested async handler after it sends a response", async () => {
|
||||
const server = app();
|
||||
const router = express.Router();
|
||||
const done = deferred();
|
||||
router.post("/work", async (_req, res) => { res.sendStatus(202); await done.promise; });
|
||||
server.use("/api", router);
|
||||
trackIdleRequestHandlers(server);
|
||||
await request(server).post("/api/work").expect(202);
|
||||
expect(idleWorkSnapshot().active).toBe(1);
|
||||
startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
||||
await request(server).post("/api/work").expect(503);
|
||||
done.resolve();
|
||||
await new Promise(resolve => setImmediate(resolve));
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
});
|
||||
|
||||
it("does not mistake a client disconnect for completed accepted work", async () => {
|
||||
const server = app();
|
||||
const entered = deferred(), done = deferred(), closed = deferred();
|
||||
server.post("/work", async (_req, res) => { res.once("close", closed.resolve); entered.resolve(); await done.promise; /* client has left */ });
|
||||
trackIdleRequestHandlers(server);
|
||||
const listener = server.listen(0, "127.0.0.1");
|
||||
await new Promise<void>(resolve => listener.once("listening", resolve));
|
||||
const address = listener.address() as { port: number };
|
||||
const client = httpRequest({ host: "127.0.0.1", port: address.port, path: "/work", method: "POST" });
|
||||
client.on("error", () => undefined);
|
||||
client.end();
|
||||
try {
|
||||
await entered.promise;
|
||||
client.destroy();
|
||||
await closed.promise;
|
||||
startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
||||
expect(idleWorkSnapshot().active).toBe(1);
|
||||
done.resolve();
|
||||
await new Promise(resolve => setImmediate(resolve));
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
} finally {
|
||||
done.resolve(); client.destroy();
|
||||
await new Promise<void>(resolve => listener.close(() => resolve()));
|
||||
}
|
||||
});
|
||||
|
||||
it("settles errors through async Express error middleware", async () => {
|
||||
const server = app(), done = deferred();
|
||||
server.get("/work", async () => { throw new Error("test"); });
|
||||
server.use(async (_err: Error, _req: express.Request, res: express.Response, _next: express.NextFunction) => {
|
||||
res.sendStatus(500); await done.promise;
|
||||
});
|
||||
trackIdleRequestHandlers(server);
|
||||
await request(server).get("/work").expect(500);
|
||||
expect(idleWorkSnapshot().active).toBe(1);
|
||||
done.resolve(); await new Promise(resolve => setImmediate(resolve));
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
});
|
||||
|
||||
it("uses unique owners even for simultaneous starts and expires the hold", () => {
|
||||
const first = startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
||||
const second = startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
||||
expect(second.ownerId).not.toBe(first.ownerId);
|
||||
expect(readTaskDrain(second.expiresAt!)).toBeNull();
|
||||
});
|
||||
|
||||
it("records even work that starts and finishes between observations", () => {
|
||||
const before = idleWorkSnapshot();
|
||||
const done = beginIdleTrackedWork(); done(); done();
|
||||
expect(idleWorkSnapshot()).toEqual({ active: before.active, generation: before.generation + 2 });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,51 @@
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { inspectIdleSpool, readIdleLocalWork, markIdleIngressTracked, markIdleStartupComplete, registerIdleSpoolDirectory } from "../services/idle-local-work.js";
|
||||
|
||||
let root: string;
|
||||
beforeEach(async () => { root = await fs.mkdtemp(path.join(os.tmpdir(), "idle-spool-")); });
|
||||
afterEach(async () => { vi.restoreAllMocks(); vi.unstubAllEnvs(); await fs.rm(root, { recursive: true, force: true }); });
|
||||
describe("idle cleanup inspection", () => {
|
||||
it("refuses to authorize sleep until startup and ingress tracking are ready", async () => {
|
||||
expect(await readIdleLocalWork()).toBe("unknown");
|
||||
});
|
||||
it("accepts absent and empty directories", async () => {
|
||||
expect(await inspectIdleSpool(path.join(root, "absent"))).toBe("none");
|
||||
expect(await inspectIdleSpool(root)).toBe("none");
|
||||
});
|
||||
it.each(["receipt.json", "malformed.json", "partial.tmp", "write.probe", "unexpected"])("retains every spool entry: %s", async name => {
|
||||
await fs.writeFile(path.join(root, name), "not valid JSON");
|
||||
expect(await inspectIdleSpool(root)).toBe("present");
|
||||
await fs.rm(path.join(root, name));
|
||||
expect(await inspectIdleSpool(root)).toBe("none");
|
||||
});
|
||||
it("fails closed on a non-directory, dangling symlink, or read failure", async () => {
|
||||
const file = path.join(root, "file"), link = path.join(root, "link");
|
||||
await fs.writeFile(file, "");
|
||||
await fs.symlink(path.join(root, "missing"), link);
|
||||
expect(await inspectIdleSpool(file)).toBe("unknown");
|
||||
expect(await inspectIdleSpool(path.join(file, "nested"))).toBe("unknown");
|
||||
expect(await inspectIdleSpool(link)).toBe("unknown");
|
||||
vi.spyOn(fs, "opendir").mockRejectedValue(Object.assign(new Error("private path"), { code: "EACCES" }));
|
||||
expect(await inspectIdleSpool(root)).toBe("unknown");
|
||||
});
|
||||
it("checks custom spool directories as well as the default paths", async () => {
|
||||
vi.stubEnv("PAPERCLIP_HOME", root);
|
||||
vi.stubEnv("SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR", path.join(root, "orphans"));
|
||||
const custom = path.join(root, "custom-accounting");
|
||||
registerIdleSpoolDirectory(custom);
|
||||
markIdleIngressTracked(); markIdleStartupComplete({ scheduledBackups: false });
|
||||
expect(await readIdleLocalWork()).toBe("none");
|
||||
markIdleStartupComplete({ scheduledBackups: true });
|
||||
expect(await readIdleLocalWork()).toBe("present");
|
||||
markIdleStartupComplete({ scheduledBackups: false });
|
||||
await fs.mkdir(custom);
|
||||
await fs.writeFile(path.join(custom, "receipt.tmp"), "incomplete");
|
||||
expect(await readIdleLocalWork()).toBe("present");
|
||||
await fs.rm(custom, { recursive: true });
|
||||
expect(await readIdleLocalWork()).toBe("none");
|
||||
});
|
||||
|
||||
});
|
||||
@@ -0,0 +1,253 @@
|
||||
import { beginIdleTrackedWork } from "../services/task-admission.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
agentInstructionWorkingCopies, agents, agentApiKeys, agentWakeupRequests, companies, companySecretProposals, createDb,
|
||||
adapterAuthSessions, environments, environmentLeases, executionWorkspaces,
|
||||
heartbeatRuns, issues, issueWatchdogs, projects, routines, type Db,
|
||||
} from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";
|
||||
|
||||
const now = Date.UTC(2020, 0, 1);
|
||||
const held = (): IdleSleepDrainStatus => ({
|
||||
draining: true, startedAt: new Date(now - 1000), expiresAt: new Date(now + 60_000),
|
||||
activeRuns: 0, pendingWakes: 0,
|
||||
});
|
||||
const unknown = { version: 1, backgroundWork: "unknown" };
|
||||
const present = { version: 1, backgroundWork: "present" };
|
||||
const none = { version: 1, backgroundWork: "none" };
|
||||
const ownerId = "d0b833f4-4098-42de-8420-1907f3aa4895";
|
||||
const owned = () => ({ ...held(), ownerId });
|
||||
const emptyLocal = async () => "none" as const;
|
||||
|
||||
|
||||
describe("idle sleep safety failure boundaries", () => {
|
||||
it.each([
|
||||
{ draining: false }, { activeRuns: 1 }, { pendingWakes: 1 },
|
||||
{ startedAt: null }, { expiresAt: new Date(now) },
|
||||
])("does not query durable work without a quiet admission hold: %j", async (change) => {
|
||||
const transaction = vi.fn();
|
||||
expect(await readIdleSleepSafety({ transaction } as unknown as Db, () => ({ ...held(), ...change }), () => now))
|
||||
.toEqual(unknown);
|
||||
expect(transaction).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("fails closed when the database is unavailable", async () => {
|
||||
const db = { transaction: vi.fn().mockRejectedValue(new Error("private connection detail")) } as unknown as Db;
|
||||
expect(await readIdleSleepSafety(db, held, () => now)).toEqual(unknown);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ draining: false }, { startedAt: new Date(now) }, { expiresAt: new Date(now) },
|
||||
{ activeRuns: 1 }, { pendingWakes: 1 },
|
||||
])("rejects a report when the hold changes during the durable scan: %j", async (change) => {
|
||||
const getStatus = vi.fn().mockReturnValueOnce(held()).mockReturnValue({ ...held(), ...change });
|
||||
const execute = vi.fn().mockResolvedValue([{ blocked: true }]);
|
||||
const transaction = vi.fn(async (fn: (tx: unknown) => Promise<unknown>) => fn({ execute }));
|
||||
expect(await readIdleSleepSafety({ transaction } as unknown as Db, getStatus, () => now)).toEqual(unknown);
|
||||
expect(transaction).toHaveBeenCalledWith(expect.any(Function), { isolationLevel: "repeatable read", accessMode: "read only" });
|
||||
});
|
||||
|
||||
it("reports persisted work when the admission hold remains unchanged", async () => {
|
||||
const execute = vi.fn().mockResolvedValue([{ blocked: true }]);
|
||||
const transaction = vi.fn(async (fn: (tx: unknown) => Promise<unknown>) => fn({ execute }));
|
||||
expect(await readIdleSleepSafety({ transaction } as unknown as Db, held, () => now)).toEqual(present);
|
||||
});
|
||||
});
|
||||
|
||||
describe("idle sleep admission and local work", () => {
|
||||
const emptyDb = () => ({ transaction: async (fn: (tx: unknown) => Promise<unknown>) => fn({ execute: async () => [{ blocked: false }] }) }) as unknown as Db;
|
||||
it("requires the exact current idle owner and a bounded expiry", async () => {
|
||||
expect(await readIdleSleepSafety(emptyDb(), held, () => now, undefined, emptyLocal)).toEqual(unknown);
|
||||
expect(await readIdleSleepSafety(emptyDb(), owned, () => now, "old-owner", emptyLocal)).toEqual(unknown);
|
||||
expect(await readIdleSleepSafety(emptyDb(), () => ({ ...owned(), expiresAt: null }), () => now, ownerId, emptyLocal)).toEqual(unknown);
|
||||
expect(await readIdleSleepSafety(emptyDb(), owned, () => now, ownerId, emptyLocal)).toEqual(none);
|
||||
});
|
||||
it.each(["present", "unknown"] as const)("retains local work reported as %s", async state => {
|
||||
expect(await readIdleSleepSafety(emptyDb(), owned, () => now, ownerId, async () => state))
|
||||
.toEqual({ version: 1, backgroundWork: state });
|
||||
});
|
||||
it("refuses sleep while accepted work remains in flight", async () => {
|
||||
const done = beginIdleTrackedWork();
|
||||
try { expect(await readIdleSleepSafety(emptyDb(), owned, () => now, ownerId, emptyLocal)).toEqual(unknown); }
|
||||
finally { done(); }
|
||||
expect(await readIdleSleepSafety(emptyDb(), owned, () => now, ownerId, emptyLocal)).toEqual(none);
|
||||
});
|
||||
it("invalidates a scan even when concurrent work finishes before the final check", async () => {
|
||||
const inspect = async () => { const done = beginIdleTrackedWork(); done(); return "none" as const; };
|
||||
expect(await readIdleSleepSafety(emptyDb(), owned, () => now, ownerId, inspect)).toEqual(unknown);
|
||||
});
|
||||
it("rechecks owner identity after disk inspection, including same-millisecond replacement", async () => {
|
||||
let status = owned();
|
||||
const inspect = async () => { status = { ...status, ownerId: "replacement" }; return "none" as const; };
|
||||
expect(await readIdleSleepSafety(emptyDb(), () => status, () => now, ownerId, inspect)).toEqual(unknown);
|
||||
});
|
||||
it("rechecks expiry after disk inspection and hides inspection errors", async () => {
|
||||
let clock = now;
|
||||
expect(await readIdleSleepSafety(emptyDb(), owned, () => clock, ownerId, async () => {
|
||||
clock += 60_000; return "none";
|
||||
})).toEqual(unknown);
|
||||
expect(await readIdleSleepSafety(emptyDb(), owned, () => now, ownerId, async () => { throw new Error("private spool path"); })).toEqual(unknown);
|
||||
});
|
||||
});
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${support.reason}`);
|
||||
(support.supported ? describe : describe.skip)("idle sleep durable work", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("idle-sleep-safety-");
|
||||
db = createDb(database.connectionString);
|
||||
}, 60_000);
|
||||
afterEach(async () => { await db.execute(sql`TRUNCATE companies, plugins, environments CASCADE`); });
|
||||
afterAll(async () => { await database?.cleanup(); });
|
||||
|
||||
const read = () => readIdleSleepSafety(db, owned, () => now, ownerId, emptyLocal);
|
||||
async function seed() {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
const runId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Idle test", issuePrefix: "IDLE" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "On-demand agent", role: "engineer", status: "idle", adapterType: "process" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", status: "succeeded" });
|
||||
return { companyId, agentId, runId };
|
||||
}
|
||||
|
||||
it("keeps saved agent files awake until deferred cleanup completes", async () => {
|
||||
const { companyId, agentId, runId } = await seed();
|
||||
await db.insert(agentInstructionWorkingCopies).values({
|
||||
runId, companyId, agentId, responsibleUserId: "test-user", entryFile: "AGENTS.md",
|
||||
baseHash: "test-hash", localRoot: "/tmp/idle-copy", executionRoot: "/tmp/idle-copy",
|
||||
location: "local", state: "saved", processStoppedAt: new Date(now),
|
||||
receipt: { schema: "paperclip.agent-files.v1", cleanupPending: true },
|
||||
});
|
||||
expect(await read()).toEqual(present);
|
||||
await db.update(agentInstructionWorkingCopies).set({ receipt: { schema: "paperclip.agent-files.v1" } })
|
||||
.where(eq(agentInstructionWorkingCopies.runId, runId));
|
||||
expect(await read()).toEqual(none);
|
||||
});
|
||||
|
||||
it("authorizes an empty database and completed history under a quiet owned hold", async () => {
|
||||
expect(await read()).toEqual(none);
|
||||
const { companyId } = await seed();
|
||||
await db.insert(issues).values({ companyId, title: "Finished work", status: "done" });
|
||||
await db.insert(routines).values({ companyId, title: "Paused schedule", status: "paused" });
|
||||
await db.insert(environmentLeases).values({ companyId, status: "released" });
|
||||
expect(await read()).toEqual(none);
|
||||
});
|
||||
|
||||
it("blocks a saved watchdog on a completed issue before its first review starts", async () => {
|
||||
const { companyId, agentId } = await seed();
|
||||
const [issue] = await db.insert(issues).values({ companyId, title: "Finished work", status: "done" }).returning();
|
||||
expect(await read()).toEqual(none);
|
||||
const [watchdog] = await db.insert(issueWatchdogs).values({
|
||||
companyId, issueId: issue!.id, watchdogAgentId: agentId, status: "active",
|
||||
}).returning();
|
||||
expect(watchdog!.watchdogIssueId).toBeNull();
|
||||
expect(watchdog!.lastTriggeredAt).toBeNull();
|
||||
expect(await read()).toEqual(present);
|
||||
await db.update(issueWatchdogs).set({ status: "disabled" }).where(eq(issueWatchdogs.id, watchdog!.id));
|
||||
expect(await read()).toEqual(none);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["queued run", { status: "queued" }],
|
||||
["orphan running run", { status: "running" }],
|
||||
["future retry", { scheduledRetryAt: new Date(now + 86_400_000) }],
|
||||
["unsettled accounting", { costAccountingPending: true }],
|
||||
] as const)("blocks %s even when process counters are zero", async (_name, change) => {
|
||||
const { runId } = await seed();
|
||||
await db.update(heartbeatRuns).set(change).where(eq(heartbeatRuns.id, runId));
|
||||
expect(await read()).toEqual(present);
|
||||
});
|
||||
|
||||
it("reports an enabled heartbeat timer and permits sleep after it is disabled", async () => {
|
||||
const { agentId } = await seed();
|
||||
await db.update(agents).set({ runtimeConfig: { heartbeat: { enabled: true, intervalSec: 86_400 } } }).where(eq(agents.id, agentId));
|
||||
expect(await read()).toEqual(present);
|
||||
await db.update(agents).set({ runtimeConfig: { heartbeat: { enabled: false, intervalSec: 86_400 } } }).where(eq(agents.id, agentId));
|
||||
expect(await read()).toEqual(none);
|
||||
});
|
||||
|
||||
it("blocks a durable deferred wake", async () => {
|
||||
const { companyId, agentId } = await seed();
|
||||
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "assignment", status: "deferred_issue_execution" });
|
||||
expect(await read()).toEqual(present);
|
||||
});
|
||||
|
||||
it("blocks an active routine without waiting for its next due time", async () => {
|
||||
const { companyId } = await seed();
|
||||
await db.insert(routines).values({ companyId, title: "Tomorrow", status: "active" });
|
||||
expect(await read()).toEqual(present);
|
||||
});
|
||||
|
||||
it("blocks a remote resource pending cleanup", async () => {
|
||||
const { companyId } = await seed();
|
||||
await db.insert(environmentLeases).values({ companyId, status: "pending_cleanup" });
|
||||
expect(await read()).toEqual(present);
|
||||
});
|
||||
|
||||
it("reports a login sandbox awaiting cleanup", async () => {
|
||||
const { companyId } = await seed();
|
||||
const [environment] = await db.insert(environments).values({ name: "Login fixture" }).returning();
|
||||
await db.insert(adapterAuthSessions).values({
|
||||
companyId, environmentId: environment!.id, adapterType: "codex_local",
|
||||
startedByUserId: "fixture-user", publicSessionId: "fixture-session", status: "cleanup_pending",
|
||||
});
|
||||
expect(await read()).toEqual(present);
|
||||
});
|
||||
|
||||
it("reports a pending secret proposal with future expiry", async () => {
|
||||
const { companyId, agentId, runId } = await seed();
|
||||
await db.insert(companySecretProposals).values({
|
||||
companyId, kind: "secret", proposedName: "Fixture", proposedKey: "FIXTURE",
|
||||
justification: "Test pending proposal", proposedByAgentId: agentId, originRunId: runId,
|
||||
expiresAt: new Date(now + 86_400_000),
|
||||
});
|
||||
expect(await read()).toEqual(present);
|
||||
});
|
||||
|
||||
it("reports retained execution workspaces that may need delayed cleanup", async () => {
|
||||
const { companyId } = await seed();
|
||||
const [project] = await db.insert(projects).values({ companyId, name: "Fixture project" }).returning();
|
||||
await db.insert(executionWorkspaces).values({
|
||||
companyId, projectId: project!.id, name: "Fixture workspace", mode: "isolated",
|
||||
strategyType: "git_worktree", status: "closed", cleanupEligibleAt: new Date(now + 86_400_000),
|
||||
});
|
||||
expect(await read()).toEqual(present);
|
||||
});
|
||||
|
||||
it("blocks externally usable agent keys until revoked", async () => {
|
||||
const { companyId, agentId } = await seed();
|
||||
const [key] = await db.insert(agentApiKeys).values({ companyId, agentId, name: "Fixture", keyHash: "fixture-not-a-key" }).returning();
|
||||
expect(await read()).toEqual(present);
|
||||
await db.update(agentApiKeys).set({ revokedAt: new Date(now) }).where(eq(agentApiKeys.id, key!.id));
|
||||
expect(await read()).toEqual(none);
|
||||
});
|
||||
|
||||
it("checks every company in the instance", async () => {
|
||||
await seed();
|
||||
const otherId = randomUUID();
|
||||
await db.insert(companies).values({ id: otherId, name: "Other company", issuePrefix: "OTHER" });
|
||||
await db.insert(issues).values({ companyId: otherId, title: "Pending work", status: "todo" });
|
||||
expect(await read()).toEqual(present);
|
||||
});
|
||||
|
||||
it("blocks enabled plugins even without declared jobs or webhooks", async () => {
|
||||
await db.execute(sql`INSERT INTO plugins (plugin_key, package_name, version, manifest_json)
|
||||
VALUES ('demo.on-demand', 'demo-plugin', '1.0.0', '{}'::jsonb)`);
|
||||
expect(await read()).toEqual(present);
|
||||
await db.execute(sql`UPDATE plugins SET status = 'disabled'`);
|
||||
expect(await read()).toEqual(none);
|
||||
});
|
||||
|
||||
it("fails closed when the installed schema is older than the report", async () => {
|
||||
await db.execute(sql`ALTER TABLE heartbeat_runs RENAME COLUMN cost_accounting_pending TO hidden_pending`);
|
||||
try { expect(await read()).toEqual(unknown); }
|
||||
finally { await db.execute(sql`ALTER TABLE heartbeat_runs RENAME COLUMN hidden_pending TO cost_accounting_pending`); }
|
||||
});
|
||||
});
|
||||
@@ -29,8 +29,10 @@ const mockCompanyService = vi.hoisted(() => ({
|
||||
}));
|
||||
const mockLogActivity = vi.hoisted(() => vi.fn());
|
||||
const mockPublishActivity = vi.hoisted(() => vi.fn());
|
||||
const mockReadIdleSleepSafety = vi.hoisted(() => vi.fn());
|
||||
|
||||
function registerModuleMocks() {
|
||||
vi.doMock("../services/idle-sleep-safety.js", () => ({ readIdleSleepSafety: mockReadIdleSleepSafety }));
|
||||
vi.doMock("../services/index.js", () => ({
|
||||
companyService: () => mockCompanyService,
|
||||
heartbeatService: () => mockHeartbeatService,
|
||||
@@ -89,6 +91,7 @@ describe("instance settings routes", () => {
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
mockReadIdleSleepSafety.mockReset();
|
||||
// vi.clearAllMocks() clears recorded calls only; it does not remove a
|
||||
// mockImplementation a prior test installed. Reinstall the default here
|
||||
// so a stateful implementation from one test can never leak into the
|
||||
@@ -960,6 +963,54 @@ describe("instance settings routes", () => {
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toEqual(idleStatus);
|
||||
expect(mockReadIdleSleepSafety).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("acquires an owned bounded idle hold and forwards its owner for inspection", async () => {
|
||||
const ownerId = "d0b833f4-4098-42de-8420-1907f3aa4895";
|
||||
mockHeartbeatService.computeTaskDrain.mockReturnValue({ ownerId, startedAt: new Date(), expiresAt: new Date(Date.now() + 60_000) });
|
||||
await request(createApp(adminActor)).post("/api/instance/task-drain").send({ purpose: "idle", ttlMs: 60_000 }).expect(200);
|
||||
expect(mockHeartbeatService.computeTaskDrain).toHaveBeenCalledWith({ purpose: "idle", ttlMs: 60_000 });
|
||||
await request(createApp(adminActor)).get(`/api/instance/task-drain?idleSleepSafety=1&ownerId=${ownerId}`).expect(200);
|
||||
expect(mockReadIdleSleepSafety).toHaveBeenLastCalledWith(mockDb, expect.any(Function), Date.now, ownerId);
|
||||
});
|
||||
|
||||
it.each([{}, { ttlMs: null }, { ttlMs: 1 }, { ttlMs: 300_001 }])("rejects an unbounded or invalid idle TTL: %j", async body => {
|
||||
await request(createApp(adminActor)).post("/api/instance/task-drain").send({ ...body, purpose: "idle" }).expect(400);
|
||||
expect(mockHeartbeatService.applyTaskDrain).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("rejects replacement and ownerless or stale release of an idle hold", async () => {
|
||||
mockHeartbeatService.getTaskDrainStatus.mockReturnValue({ ...idleStatus, draining: true, ownerId: "current-owner" });
|
||||
const app = createApp(adminActor);
|
||||
await request(app).post("/api/instance/task-drain").send({}).expect(409);
|
||||
await request(app).delete("/api/instance/task-drain").expect(409);
|
||||
await request(app).delete("/api/instance/task-drain?ownerId=previous-owner").expect(409);
|
||||
expect(mockHeartbeatService.applyTaskDrain).not.toHaveBeenCalled();
|
||||
expect(mockHeartbeatService.stopTaskDrain).not.toHaveBeenCalled();
|
||||
await request(app).delete("/api/instance/task-drain?ownerId=current-owner").expect(200);
|
||||
expect(mockHeartbeatService.stopTaskDrain).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("returns the opt-in instance-wide safety report to an instance admin", async () => {
|
||||
mockHeartbeatService.getTaskDrainStatus.mockReturnValue(idleStatus);
|
||||
const report = { version: 1, backgroundWork: "unknown" };
|
||||
mockReadIdleSleepSafety.mockResolvedValue(report);
|
||||
const res = await request(createApp(adminActor)).get("/api/instance/task-drain?idleSleepSafety=1");
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toEqual({ ...idleStatus, idleSleepSafety: report });
|
||||
expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, undefined);
|
||||
expect(mockReadIdleSleepSafety.mock.calls[0][1]()).toEqual(idleStatus);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["company member", nonAdminActor],
|
||||
["agent", { type: "agent", agentId: "agent-1", companyId: "company-1" }],
|
||||
["anonymous caller", { type: "none" }],
|
||||
])("does not expose the instance-wide work report to a %s", async (_name, actor) => {
|
||||
const res = await request(createApp(actor)).get("/api/instance/task-drain?idleSleepSafety=1");
|
||||
expect(res.status).toBe(403);
|
||||
expect(mockReadIdleSleepSafety).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("writes an activity record for every company, then applies the same drain values, in one transaction", async () => {
|
||||
|
||||
@@ -4,6 +4,8 @@ import type { Duplex } from "node:stream";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { setupLiveEventsWebSocketServer } from "../realtime/live-events-ws.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { idleWorkSnapshot, startTaskDrain, stopTaskDrain } from "../services/task-admission.js";
|
||||
import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
|
||||
|
||||
vi.mock("../middleware/logger.js", () => ({
|
||||
logger: {
|
||||
@@ -67,6 +69,43 @@ describe("setupLiveEventsWebSocketServer", () => {
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
it.each([false, true])("counts accepted upgrade authentication after disconnect (failure=%s)", async (fail) => {
|
||||
const server = new EventEmitter();
|
||||
const socket = new FakeUpgradeSocket();
|
||||
let finishAuth!: () => void;
|
||||
const pending = new Promise<void>(resolve => { finishAuth = resolve; });
|
||||
setupLiveEventsWebSocketServer(server as never, {} as never, {
|
||||
deploymentMode: "authenticated",
|
||||
resolveCloudActor: async () => {
|
||||
await pending;
|
||||
if (fail) throw new Error("fixture authentication write failed");
|
||||
return { userId: "fixture-user", companyIds: ["company-1"] };
|
||||
},
|
||||
});
|
||||
server.emit("upgrade", createUpgradeRequest(), socket as unknown as Duplex, Buffer.alloc(0));
|
||||
const hold = startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
||||
const emptyDb = { transaction: async (run: (tx: unknown) => Promise<unknown>) =>
|
||||
run({ execute: async () => [{ blocked: false }] }) };
|
||||
const report = () => readIdleSleepSafety(emptyDb as never,
|
||||
() => ({ ...hold, draining: true, activeRuns: 0, pendingWakes: 0 }), Date.now, hold.ownerId, async () => "none");
|
||||
try {
|
||||
socket.destroy();
|
||||
await flushPromises();
|
||||
expect(idleWorkSnapshot().active).toBe(1);
|
||||
expect((await report()).backgroundWork).toBe("unknown");
|
||||
finishAuth();
|
||||
await flushPromises();
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect((await report()).backgroundWork).toBe("none");
|
||||
expect(socket.endedChunks).toEqual([]);
|
||||
} finally {
|
||||
finishAuth();
|
||||
await flushPromises();
|
||||
stopTaskDrain();
|
||||
server.emit("close");
|
||||
}
|
||||
});
|
||||
|
||||
it("does not write a rejection response after the raw upgrade socket is already closed", async () => {
|
||||
const server = new EventEmitter();
|
||||
setupLiveEventsWebSocketServer(server as never, {} as never, { deploymentMode: "authenticated" });
|
||||
|
||||
@@ -1,3 +1,7 @@
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
|
||||
import { reconcileBuiltInAgentsOnStartup, reconcileCodexLocalManagedHomesOnStartup, reconcilePersistedRuntimeServicesOnStartup } from "../services/index.js";
|
||||
import { runDatabaseBackup } from "@paperclipai/db";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
@@ -437,6 +441,65 @@ describe("startServer feedback export wiring", () => {
|
||||
process.env.BETTER_AUTH_SECRET = "test-secret";
|
||||
});
|
||||
|
||||
it.each(([
|
||||
["built-in agents", reconcileBuiltInAgentsOnStartup],
|
||||
["managed homes", reconcileCodexLocalManagedHomesOnStartup],
|
||||
["runtime services", reconcilePersistedRuntimeServicesOnStartup],
|
||||
] as const).flatMap(([name, reconcile]) => [false, true].map(fail => ({ name, reconcile, fail }))))
|
||||
("counts detached $name startup writes after readiness (failure=$fail)", async ({ reconcile, fail }) => {
|
||||
let finishStartup!: () => void;
|
||||
const pending = new Promise<void>(resolve => { finishStartup = resolve; });
|
||||
vi.mocked(reconcile).mockImplementationOnce(async () => {
|
||||
await pending;
|
||||
if (fail) throw new Error("fixture startup write failed");
|
||||
return { reconciled: 0, seeded: 0, failed: 0 } as never;
|
||||
});
|
||||
const emptyDb = { transaction: async (run: (tx: unknown) => Promise<unknown>) =>
|
||||
run({ execute: async () => [{ blocked: false }] }) };
|
||||
const hold = { ownerId: "fixture-owner", draining: true, activeRuns: 0, pendingWakes: 0,
|
||||
startedAt: new Date(), expiresAt: new Date(Date.now() + 60_000) };
|
||||
const report = () => readIdleSleepSafety(emptyDb as never, () => hold, Date.now, hold.ownerId, async () => "none");
|
||||
try {
|
||||
await startServer();
|
||||
await new Promise(resolve => setImmediate(resolve));
|
||||
expect(idleWorkSnapshot().active).toBe(1);
|
||||
expect((await report()).backgroundWork).toBe("unknown");
|
||||
finishStartup();
|
||||
await new Promise(resolve => setImmediate(resolve));
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect((await report()).backgroundWork).toBe("none");
|
||||
} finally {
|
||||
finishStartup();
|
||||
await new Promise(resolve => setImmediate(resolve));
|
||||
}
|
||||
});
|
||||
|
||||
it.each([false, true])("counts an accepted backup until completion, including failure=%s", async fails => {
|
||||
await startServer();
|
||||
await new Promise(resolve => setImmediate(resolve));
|
||||
const before = idleWorkSnapshot().active;
|
||||
let resolveBackup!: (value: never) => void;
|
||||
let rejectBackup!: (error: Error) => void;
|
||||
vi.mocked(runDatabaseBackup).mockImplementationOnce(() => new Promise((resolve, reject) => {
|
||||
resolveBackup = resolve; rejectBackup = reject;
|
||||
}));
|
||||
const options = createAppMock.mock.calls[0]?.[1] as unknown as {
|
||||
databaseBackupService: { runManualBackup(): Promise<unknown> };
|
||||
};
|
||||
const backup = options.databaseBackupService.runManualBackup();
|
||||
await Promise.resolve();
|
||||
expect(idleWorkSnapshot().active).toBe(before + 1);
|
||||
if (fails) {
|
||||
const rejection = expect(backup).rejects.toThrow("backup failed");
|
||||
rejectBackup(new Error("backup failed"));
|
||||
await rejection;
|
||||
} else {
|
||||
resolveBackup({ backupFile: "backup.sql.gz", sizeBytes: 1, prunedCount: 0 } as never);
|
||||
await backup;
|
||||
}
|
||||
expect(idleWorkSnapshot().active).toBe(before);
|
||||
});
|
||||
|
||||
it("starts without PAPERCLIP_DECISION_SIGNING_SECRET by generating a persisted key", async () => {
|
||||
const originalHome = process.env.PAPERCLIP_HOME;
|
||||
const originalInstanceId = process.env.PAPERCLIP_INSTANCE_ID;
|
||||
|
||||
+16
-12
@@ -1,3 +1,5 @@
|
||||
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "./middleware/idle-admission.js";
|
||||
import { isIdleTaskDrainActive, trackIdleWork } from "./services/task-admission.js";
|
||||
import { customerSuccessRoutes } from "./routes/customer-success.js";
|
||||
import { cloudWarmStandbyMiddleware } from "./middleware/cloud-warm-standby.js";
|
||||
import type { CloudWarmStandby } from "./services/cloud-warm-standby.js";
|
||||
@@ -520,6 +522,7 @@ export async function createApp(
|
||||
},
|
||||
) {
|
||||
const app = express();
|
||||
app.use(idleAdmissionMiddleware);
|
||||
const staticUi = express.Router();
|
||||
app.locals.paperclipDb = db;
|
||||
const isWarmStandby = opts.cloudWarmStandby ?? (() => false);
|
||||
@@ -629,7 +632,7 @@ export async function createApp(
|
||||
// mutation guard. The Chat SDK adapter verifies the provider signature
|
||||
// before Paperclip persists or acts on any event.
|
||||
const emailChannels = emailChannelService(db, {
|
||||
isBackgroundWorkEnabled: () => !isWarmStandby(),
|
||||
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
|
||||
heartbeat: connectionIntentHeartbeat,
|
||||
storage: opts.storageService,
|
||||
publicBaseUrl: opts.chatWebhookPublicBaseUrl ?? opts.authPublicBaseUrl,
|
||||
@@ -841,7 +844,7 @@ export async function createApp(
|
||||
const jobStore = pluginJobStore(db);
|
||||
const lifecycle = pluginLifecycleManager(db, { workerManager });
|
||||
const scheduler = createPluginJobScheduler({
|
||||
isBackgroundWorkEnabled: () => !isWarmStandby(),
|
||||
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
|
||||
db,
|
||||
jobStore,
|
||||
workerManager,
|
||||
@@ -1000,14 +1003,14 @@ export async function createApp(
|
||||
if (publicMcpOAuth) {
|
||||
const dispatch = createMcpApiDispatch(api);
|
||||
publicMcpEvents = createPublicMcpEvents(db, publicMcpOAuth, dispatch, {
|
||||
isBackgroundWorkEnabled: () => !isWarmStandby(),
|
||||
isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive(),
|
||||
});
|
||||
publicMcpEvents.start();
|
||||
const transfers = createPublicMcpTransfers(db, publicMcpOAuth, dispatch, opts.storageService);
|
||||
publicMcpIngress.use(transfers.router);
|
||||
publicMcpIngress.use(publicMcpIngressRoutes(publicMcpOAuth, createPublicMcpExecutor(db, publicMcpOAuth, dispatch, transfers), publicMcpEvents));
|
||||
const dotOAuth = createPublicMcpOAuth(db, { ...publicMcpOAuth.config, resource: publicMcpOAuth.config.origin + "/mcp/runner" });
|
||||
dotMcpEvents = createPublicMcpEvents(db, dotOAuth, dispatch, { enableDotRunner: true, isBackgroundWorkEnabled: () => !isWarmStandby() });
|
||||
dotMcpEvents = createPublicMcpEvents(db, dotOAuth, dispatch, { enableDotRunner: true, isBackgroundWorkEnabled: () => !isWarmStandby() && !isIdleTaskDrainActive() });
|
||||
dotMcpEvents.start();
|
||||
publicMcpIngress.use(publicMcpIngressRoutes(dotOAuth, createPublicMcpExecutor(db, dotOAuth, dispatch), dotMcpEvents, createDotRunnerMcpTools(db)));
|
||||
api.use(publicMcpManagementRoutes(publicMcpOAuth, dotOAuth));
|
||||
@@ -1194,7 +1197,7 @@ export async function createApp(
|
||||
}
|
||||
};
|
||||
const flushPendingFeedbackExports = async () => {
|
||||
if (feedbackExportShuttingDown || isWarmStandby()) return;
|
||||
if (feedbackExportShuttingDown || isWarmStandby() || isIdleTaskDrainActive()) return;
|
||||
try {
|
||||
await opts.feedbackExportService?.flushPendingFeedbackTraces();
|
||||
} catch (err) {
|
||||
@@ -1243,25 +1246,25 @@ export async function createApp(
|
||||
});
|
||||
const unsubscribeChatPublicationSignals = subscribeAllCompanyLiveEvents(
|
||||
(event) => {
|
||||
if (!isWarmStandby() && isChatPublicationCommitSignal(event))
|
||||
if (!isWarmStandby() && !isIdleTaskDrainActive() && isChatPublicationCommitSignal(event))
|
||||
chatReconciliation.notifyPublications();
|
||||
},
|
||||
);
|
||||
let chatPublicationTimer: ReturnType<typeof setInterval> | null = setInterval(
|
||||
() => {
|
||||
if (!isWarmStandby()) chatReconciliation.reconcile();
|
||||
if (!isWarmStandby() && !isIdleTaskDrainActive()) chatReconciliation.reconcile();
|
||||
},
|
||||
CHAT_PUBLICATION_FLUSH_INTERVAL_MS,
|
||||
);
|
||||
chatPublicationTimer.unref?.();
|
||||
if (!isWarmStandby()) chatReconciliation.reconcile();
|
||||
if (!isWarmStandby() && !isIdleTaskDrainActive()) chatReconciliation.reconcile();
|
||||
// Abandoned chunked-import spool sweep: hourly (plus once at startup),
|
||||
// deleting spool dirs whose transfer saw no activity for 24h and cancelling
|
||||
// their still-open ledger runs. Same setInterval + unref + shutdown-clear
|
||||
// shape as the feedback export flush above.
|
||||
const importTransferSpoolRoot = resolveDefaultImportTransferSpoolRoot();
|
||||
const sweepImportTransferSpools = () => {
|
||||
if (isWarmStandby()) return;
|
||||
if (isWarmStandby() || isIdleTaskDrainActive()) return;
|
||||
sweepAbandonedImportTransferSpools(db, importTransferSpoolRoot)
|
||||
.then((result) => {
|
||||
if (result.swept > 0) {
|
||||
@@ -1276,11 +1279,11 @@ export async function createApp(
|
||||
});
|
||||
};
|
||||
const browserUseTimer = setInterval(() => {
|
||||
if (isWarmStandby()) return;
|
||||
if (isWarmStandby() || isIdleTaskDrainActive()) return;
|
||||
void browserUse.sweep().catch(() => logger.warn("Browser Use reconciliation failed; retrying."));
|
||||
}, 3000);
|
||||
browserUseTimer.unref?.();
|
||||
if (!isWarmStandby()) void browserUse.sweep().catch(() => logger.warn("Browser Use startup reconciliation failed; retrying."));
|
||||
if (!isWarmStandby() && !isIdleTaskDrainActive()) void browserUse.sweep().catch(() => logger.warn("Browser Use startup reconciliation failed; retrying."));
|
||||
let importTransferSweepTimer: ReturnType<typeof setInterval> | null =
|
||||
setInterval(
|
||||
sweepImportTransferSpools,
|
||||
@@ -1369,7 +1372,7 @@ export async function createApp(
|
||||
.catch((err) => {
|
||||
logger.error({ err }, "Failed to load ready plugins on startup");
|
||||
});
|
||||
app.locals.bundledPluginsStartup = bundledPluginsStartup;
|
||||
app.locals.bundledPluginsStartup = trackIdleWork(bundledPluginsStartup);
|
||||
// The shutdown hook runs at most once. It caches the in-flight promise, so a
|
||||
// second caller (for example the `exit` handler) awaits the same completion
|
||||
// instead of starting a second teardown.
|
||||
@@ -1427,5 +1430,6 @@ export async function createApp(
|
||||
void flushPluginLogBuffer();
|
||||
});
|
||||
|
||||
trackIdleRequestHandlers(app);
|
||||
return app;
|
||||
}
|
||||
+36
-27
@@ -1,3 +1,5 @@
|
||||
import { isIdleTaskDrainActive, beginIdleTrackedWork, trackIdleWork } from "./services/task-admission.js";
|
||||
import { markIdleStartupComplete } from "./services/idle-local-work.js";
|
||||
import { cloudWarmStandbyServerOptions } from "./middleware/cloud-warm-standby.js";
|
||||
import { createCloudWarmStandby } from "./services/cloud-warm-standby.js";
|
||||
import { subscribeAllCompanyLiveEvents } from "./services/live-events.js";
|
||||
@@ -830,6 +832,7 @@ async function startServerWithDatabaseTeardown(
|
||||
}
|
||||
|
||||
databaseBackupInFlight = true;
|
||||
const finishIdleBackup = beginIdleTrackedWork();
|
||||
const startedAt = new Date();
|
||||
const startedAtMs = Date.now();
|
||||
const label = trigger === "scheduled" ? "Automatic" : "Manual";
|
||||
@@ -873,6 +876,7 @@ async function startServerWithDatabaseTeardown(
|
||||
throw err;
|
||||
} finally {
|
||||
databaseBackupInFlight = false;
|
||||
finishIdleBackup();
|
||||
}
|
||||
};
|
||||
const pluginWorkerManager = createPluginWorkerManager();
|
||||
@@ -928,7 +932,7 @@ async function startServerWithDatabaseTeardown(
|
||||
managedPluginAutoInstall,
|
||||
});
|
||||
// Upgrade admission runs before every WebSocket listener, outside Express.
|
||||
const server = createServer(cloudWarmStandbyServerOptions(isWarmStandby), app as unknown as RequestListener);
|
||||
const server = createServer(cloudWarmStandbyServerOptions(() => isWarmStandby() || isIdleTaskDrainActive()), app as unknown as RequestListener);
|
||||
|
||||
// Increase keep-alive timeouts to safely outlive default idle timeouts
|
||||
// of common reverse proxies and load balancers (like AWS ALB, Nginx, or Traefik).
|
||||
@@ -1021,7 +1025,9 @@ async function startServerWithDatabaseTeardown(
|
||||
logger.error({ err }, "startup reconciliation of managed runtime control operations failed");
|
||||
}
|
||||
|
||||
void reconcilePersistedRuntimeServicesOnStartup(db as any)
|
||||
// These jobs need not delay readiness, but their writes must finish before
|
||||
// an idle report can authorize sleep. Count the whole detached promise.
|
||||
void trackIdleWork(reconcilePersistedRuntimeServicesOnStartup(db as any)
|
||||
.then((result) => {
|
||||
if (
|
||||
result.reconciled > 0
|
||||
@@ -1046,12 +1052,12 @@ async function startServerWithDatabaseTeardown(
|
||||
})
|
||||
.catch((err) => {
|
||||
logger.error({ err }, "startup reconciliation of persisted runtime services failed");
|
||||
});
|
||||
}));
|
||||
|
||||
// Backfill auth.json into any already-isolated codex_local managed home that
|
||||
// was created by the #8272 isolation guard before the Phase 1 seeding fix.
|
||||
// Idempotent; the Phase 1 execute-time seeding covers new strandings.
|
||||
void reconcileCodexLocalManagedHomesOnStartup(db)
|
||||
void trackIdleWork(reconcileCodexLocalManagedHomesOnStartup(db)
|
||||
.then((result) => {
|
||||
if (result.seeded > 0 || result.failed > 0) {
|
||||
logger.warn(
|
||||
@@ -1068,9 +1074,9 @@ async function startServerWithDatabaseTeardown(
|
||||
})
|
||||
.catch((err) => {
|
||||
logger.error({ err }, "startup reconciliation of codex_local managed homes failed");
|
||||
});
|
||||
}));
|
||||
|
||||
void reconcileBuiltInAgentsOnStartup(db as any)
|
||||
void trackIdleWork(reconcileBuiltInAgentsOnStartup(db as any)
|
||||
.then((result) => {
|
||||
if (
|
||||
result.reconciled > 0
|
||||
@@ -1087,7 +1093,7 @@ async function startServerWithDatabaseTeardown(
|
||||
})
|
||||
.catch((err) => {
|
||||
logger.error({ err }, "startup reconciliation of built-in agents failed");
|
||||
});
|
||||
}));
|
||||
|
||||
// Force the instance onto the Kubernetes sandbox provider when configured via
|
||||
// env (PAPERCLIP_EXECUTION_MODE=kubernetes). Runs BEFORE the heartbeat resumes
|
||||
@@ -1148,11 +1154,13 @@ async function startServerWithDatabaseTeardown(
|
||||
let heartbeatSchedulerInterval: ReturnType<typeof setInterval> | null = null;
|
||||
const heartbeatSchedulerInFlight = new Set<Promise<void>>();
|
||||
const trackHeartbeatSchedulerWork = (work: Promise<unknown>) => {
|
||||
const finishIdleWork = beginIdleTrackedWork();
|
||||
let tracked: Promise<void>;
|
||||
tracked = Promise.resolve(work)
|
||||
.then(() => undefined, () => undefined)
|
||||
.finally(() => {
|
||||
heartbeatSchedulerInFlight.delete(tracked);
|
||||
finishIdleWork();
|
||||
});
|
||||
heartbeatSchedulerInFlight.add(tracked);
|
||||
};
|
||||
@@ -1171,7 +1179,7 @@ async function startServerWithDatabaseTeardown(
|
||||
["local_ai_login_cleanup", () => localAiLoginService(db).reapExpired()],
|
||||
] as const;
|
||||
const sweepExecutionControl = () => {
|
||||
if (heartbeatSchedulerStopped || isWarmStandby()) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped || isWarmStandby()) return;
|
||||
// Independent durable queues must not block one another. Each queue remains
|
||||
// single-flight; a later sweep observes committed transitions from its peers.
|
||||
for (const [queue, work] of executionControlSweeps) {
|
||||
@@ -1187,7 +1195,7 @@ async function startServerWithDatabaseTeardown(
|
||||
sweepExecutionControl();
|
||||
const startHeartbeatSchedulerInterval = (callback: () => void) => {
|
||||
heartbeatSchedulerInterval = setInterval(() => {
|
||||
if (!isWarmStandby()) callback();
|
||||
if (!isWarmStandby() && !isIdleTaskDrainActive()) callback();
|
||||
}, config.heartbeatSchedulerIntervalMs);
|
||||
heartbeatSchedulerInterval?.unref?.();
|
||||
};
|
||||
@@ -1196,7 +1204,7 @@ async function startServerWithDatabaseTeardown(
|
||||
enabled: async () => (await instanceSettingsService(db).getExperimental()).enableExternalObjects === true,
|
||||
});
|
||||
const scheduleExternalObjectRefreshSweep = (now = new Date()) => {
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork(externalObjects
|
||||
.refreshDueObjectsForActiveCompanies(50, now)
|
||||
.then((result) => {
|
||||
@@ -1236,7 +1244,7 @@ async function startServerWithDatabaseTeardown(
|
||||
// Activity publication happens after the status transaction commits. This is
|
||||
// a best-effort fast path; the durable outbox and sweeps remain authoritative.
|
||||
const unsubscribeChatCompletions = subscribeAllCompanyLiveEvents(event => {
|
||||
if (heartbeatSchedulerStopped || event.type !== "activity.logged" ||
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped || event.type !== "activity.logged" ||
|
||||
event.payload.action !== "issue.updated" || typeof event.payload.entityId !== "string") return;
|
||||
trackHeartbeatSchedulerWork(chatCompletionDeliveries.sweepPending({ companyId: event.companyId, taskId: event.payload.entityId })
|
||||
.catch(err => logger.error({ err }, "post-commit chat completion delivery failed")));
|
||||
@@ -1259,7 +1267,7 @@ async function startServerWithDatabaseTeardown(
|
||||
logger.error({ err }, "environment lease cleanup sweep failed");
|
||||
});
|
||||
const scheduleEnvironmentLeaseCleanupSweep = () => {
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork(runEnvironmentLeaseCleanupSweep(ENVIRONMENT_LEASE_CLEANUP_SWEEP_BACKOFF_MS));
|
||||
};
|
||||
const githubConnectionEvents = githubConnectionEventService(db as any, {
|
||||
@@ -1273,7 +1281,7 @@ async function startServerWithDatabaseTeardown(
|
||||
?? null,
|
||||
});
|
||||
const scheduleGitHubConnectionEventPoll = () => {
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork(githubConnectionEvents.pollOnce()
|
||||
.then((result) => {
|
||||
if (result.leased > 0 || result.failed > 0) {
|
||||
@@ -1285,7 +1293,7 @@ async function startServerWithDatabaseTeardown(
|
||||
}));
|
||||
};
|
||||
const scheduleGitHubConnectionContinuitySweep = () => {
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork(tools.sweepGitHubConnectionContinuity()
|
||||
.then((result) => {
|
||||
if (result.due > 0 || result.failed > 0) {
|
||||
@@ -1335,7 +1343,7 @@ async function startServerWithDatabaseTeardown(
|
||||
workspaceReaperCooldownDays: config.workspaceReaperCooldownDays,
|
||||
});
|
||||
const scheduleMergedPullRequestConfirmationSweep = () => {
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork(mergedPullRequestConfirmations
|
||||
.sweepMergedPullRequestConfirmations()
|
||||
.then((result) => {
|
||||
@@ -1353,7 +1361,7 @@ async function startServerWithDatabaseTeardown(
|
||||
let lastTerminalWorkspaceSkipLogAt = 0;
|
||||
const terminalWorkspaceSkipLogIntervalMs = 10 * 60 * 1000;
|
||||
const scheduleTerminalWorkspaceSweep = () => {
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork(terminalWorkspaces
|
||||
.sweepTerminalWorkspaces()
|
||||
.then((result) => {
|
||||
@@ -1404,7 +1412,7 @@ async function startServerWithDatabaseTeardown(
|
||||
}
|
||||
};
|
||||
const scheduleAdapterLoginReaperSweep = () => {
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork(adapterLoginReaper
|
||||
.sweep()
|
||||
.then(logAdapterLoginReaperResult)
|
||||
@@ -1430,7 +1438,7 @@ async function startServerWithDatabaseTeardown(
|
||||
}
|
||||
};
|
||||
const scheduleSetupTokenReaperSweep = () => {
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork(setupTokenReaper
|
||||
.sweep()
|
||||
.then(logSetupTokenReaperResult)
|
||||
@@ -1649,7 +1657,7 @@ async function startServerWithDatabaseTeardown(
|
||||
// can then wait through an already-running suppression check before it
|
||||
// captures the authoritative set of running heartbeat rows.
|
||||
trackHeartbeatSchedulerWork((async () => {
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork(decisionExecutor.sweepExpired().catch((err: unknown) => {
|
||||
logger.error({ err }, "decision expiry sweep failed");
|
||||
}));
|
||||
@@ -1677,10 +1685,10 @@ async function startServerWithDatabaseTeardown(
|
||||
}));
|
||||
}
|
||||
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
scheduleExternalObjectRefreshSweep(new Date());
|
||||
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
scheduleMergedPullRequestConfirmationSweep();
|
||||
scheduleGitHubConnectionEventPoll();
|
||||
scheduleGitHubConnectionContinuitySweep();
|
||||
@@ -1689,7 +1697,7 @@ async function startServerWithDatabaseTeardown(
|
||||
scheduleSetupTokenReaperSweep();
|
||||
scheduleEnvironmentLeaseCleanupSweep();
|
||||
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork(routines
|
||||
.tickScheduledTriggers(new Date())
|
||||
.then((result) => {
|
||||
@@ -1701,7 +1709,7 @@ async function startServerWithDatabaseTeardown(
|
||||
logger.error({ err }, "routine scheduler tick failed");
|
||||
}));
|
||||
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork((async () => {
|
||||
const experimental = await instanceSettingsService(db).getExperimental();
|
||||
if (experimental.enableStatusCards !== true) return;
|
||||
@@ -1730,7 +1738,7 @@ async function startServerWithDatabaseTeardown(
|
||||
logger.error({ err }, "status-card scheduler tick failed");
|
||||
}));
|
||||
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork(environmentCustomImages
|
||||
.cleanupExpiredSetupSessions()
|
||||
.then((result) => {
|
||||
@@ -1742,7 +1750,7 @@ async function startServerWithDatabaseTeardown(
|
||||
logger.error({ err }, "environment customImage setup cleanup failed");
|
||||
}));
|
||||
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
trackHeartbeatSchedulerWork(tools
|
||||
.sweepConnectionHealth()
|
||||
.then((swept) => {
|
||||
@@ -1780,7 +1788,7 @@ async function startServerWithDatabaseTeardown(
|
||||
logger.error({ err }, "periodic question-response delivery sweep failed");
|
||||
}));
|
||||
|
||||
if (heartbeatSchedulerStopped) return;
|
||||
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
|
||||
if (!(await heartbeat.resolveSchedulingSuppression()).suppressed) {
|
||||
// Periodically reap orphaned runs (5-min staleness threshold) and make sure
|
||||
// persisted queued work is still being driven forward.
|
||||
@@ -1865,7 +1873,7 @@ async function startServerWithDatabaseTeardown(
|
||||
"Automatic database backups enabled",
|
||||
);
|
||||
setInterval(() => {
|
||||
if (isWarmStandby()) return;
|
||||
if (isWarmStandby() || isIdleTaskDrainActive()) return;
|
||||
void runServerDatabaseBackup("scheduled").catch(() => {
|
||||
// runServerDatabaseBackup already logs the failure with context.
|
||||
});
|
||||
@@ -1890,6 +1898,7 @@ async function startServerWithDatabaseTeardown(
|
||||
throw err;
|
||||
}
|
||||
|
||||
markIdleStartupComplete({ scheduledBackups: config.databaseBackupEnabled });
|
||||
setStartupRecoveryPhase("ready");
|
||||
logger.info(`Server startup recovery complete on ${config.host}:${listenPort}`);
|
||||
void systemdNotify(["--ready", `--status=Listening on ${config.host}:${listenPort}`]).then((notified) => {
|
||||
|
||||
@@ -28,6 +28,7 @@ import { logger } from "./logger.js";
|
||||
import { captureRunIdentity } from "../services/run-identity.js";
|
||||
import { boardAuthService } from "../services/board-auth.js";
|
||||
import { retryIdempotentDatabaseOperation } from "../database-retry.js";
|
||||
import { beginIdleTrackedWork } from "../services/task-admission.js";
|
||||
|
||||
export {
|
||||
isTransientDbConnectionError,
|
||||
@@ -226,7 +227,7 @@ const publicMcpGatewayProtocolPath = /^\/mcp\/gateways\/gw_[a-f0-9]{32}\/?$/i;
|
||||
|
||||
export function actorMiddleware(db: Db, opts: ActorMiddlewareOptions): RequestHandler {
|
||||
const boardAuth = boardAuthService(db);
|
||||
return async (req, _res, next) => {
|
||||
const authenticate: RequestHandler = async (req, _res, next) => {
|
||||
req.actor =
|
||||
opts.deploymentMode === "local_trusted"
|
||||
? {
|
||||
@@ -501,6 +502,22 @@ export function actorMiddleware(db: Db, opts: ActorMiddlewareOptions): RequestHa
|
||||
|
||||
next();
|
||||
};
|
||||
return async (req, res, next) => {
|
||||
// Health and task-drain requests bypass tenant admission, but resolving
|
||||
// their actor can still write users, companies, memberships and key usage.
|
||||
// Finish all authentication before entering the next handler: the report
|
||||
// must count concurrent authentication, without counting its own auth.
|
||||
const finish = beginIdleTrackedWork();
|
||||
let continueRequest: (() => void) | undefined;
|
||||
try {
|
||||
await authenticate(req, res, (error?: unknown) => {
|
||||
continueRequest = () => next(error);
|
||||
});
|
||||
} finally {
|
||||
finish();
|
||||
}
|
||||
continueRequest?.();
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -1,9 +1,13 @@
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import type { Request } from "express";
|
||||
import express, { type Request } from "express";
|
||||
import request from "supertest";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { authUsers, companies, companyMemberships, instanceSettings, instanceUserRoles } from "@paperclipai/db";
|
||||
import { cloudActorHeaderSourceFromHeaders, resolveCloudTenantActor } from "./auth.js";
|
||||
import { actorMiddleware, cloudActorHeaderSourceFromHeaders, resolveCloudTenantActor } from "./auth.js";
|
||||
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "./idle-admission.js";
|
||||
import { idleWorkSnapshot, startTaskDrain, stopTaskDrain } from "../services/task-admission.js";
|
||||
import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
|
||||
|
||||
// Minimal fake Drizzle Db: records every table passed to .insert() / .delete() and
|
||||
// supports the chained call shapes used by resolveCloudTenantActor (values /
|
||||
@@ -18,6 +22,7 @@ function createFakeDb(options: {
|
||||
membershipQueryRows?: Array<{ companyId: string; membershipRole: string | null; status: string }>;
|
||||
settingsRow?: Record<string, unknown> | null;
|
||||
selectThrows?: boolean;
|
||||
beforeWrite?: () => Promise<void>;
|
||||
} = {}) {
|
||||
const membershipRow =
|
||||
options.membershipRow ?? { companyId: "company-x", membershipRole: "owner", status: "active" };
|
||||
@@ -41,8 +46,9 @@ function createFakeDb(options: {
|
||||
chain.onConflictDoUpdate = () => chain;
|
||||
chain.onConflictDoNothing = () => chain;
|
||||
chain.where = () => chain;
|
||||
chain.returning = async () => [membershipRow];
|
||||
chain.then = (resolve: (v: unknown) => unknown) => Promise.resolve(undefined).then(resolve);
|
||||
chain.returning = async () => { await options.beforeWrite?.(); return [membershipRow]; };
|
||||
chain.then = (resolve: (v: unknown) => unknown, reject?: (error: unknown) => unknown) =>
|
||||
Promise.resolve().then(options.beforeWrite).then(resolve, reject);
|
||||
const db = {
|
||||
insert: (table: unknown) => {
|
||||
insertedTables.push(table);
|
||||
@@ -122,6 +128,7 @@ describe("resolveCloudTenantActor (shared-pool hardening)", () => {
|
||||
process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = "test-server-token";
|
||||
});
|
||||
afterEach(() => {
|
||||
stopTaskDrain();
|
||||
delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
|
||||
delete process.env.PAPERCLIP_MANAGED_CONFIG;
|
||||
});
|
||||
@@ -159,6 +166,52 @@ describe("resolveCloudTenantActor (shared-pool hardening)", () => {
|
||||
expect(insertedTables).toContain(companyMemberships);
|
||||
});
|
||||
|
||||
it.each(["/api/health", "/api/instance/task-drain"])("counts unfinished authentication on exempt %s requests", async (path) => {
|
||||
let enterWrite!: () => void, finishWrite!: () => void;
|
||||
const entered = new Promise<void>(resolve => { enterWrite = resolve; });
|
||||
const pending = new Promise<void>(resolve => { finishWrite = resolve; });
|
||||
const { db, insertedTables } = createFakeDb({ beforeWrite: async () => { enterWrite(); await pending; } });
|
||||
const server = express();
|
||||
server.use(idleAdmissionMiddleware);
|
||||
server.use(actorMiddleware(db, { deploymentMode: "authenticated", resolveSession: async () => null }));
|
||||
const hold = startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
||||
const emptyDb = { transaction: async (run: (tx: unknown) => Promise<unknown>) =>
|
||||
run({ execute: async () => [{ blocked: false }] }) } as unknown as Db;
|
||||
server.get("/api/health", (_req, res) => res.sendStatus(204));
|
||||
server.get("/api/instance/task-drain", async (_req, res) => {
|
||||
res.json(await readIdleSleepSafety(emptyDb, () => ({ ...hold, draining: true, activeRuns: 0, pendingWakes: 0 }),
|
||||
Date.now, hold.ownerId, async () => "none"));
|
||||
});
|
||||
trackIdleRequestHandlers(server);
|
||||
const accepted = request(server).get(path).set(VALID_HEADERS).then(response => response);
|
||||
try {
|
||||
await entered;
|
||||
expect(idleWorkSnapshot().active).toBeGreaterThan(0);
|
||||
const during = await request(server).get("/api/instance/task-drain").expect(200);
|
||||
expect(during.body.backgroundWork).toBe("unknown");
|
||||
} finally {
|
||||
finishWrite();
|
||||
await accepted;
|
||||
}
|
||||
expect(insertedTables).toEqual(expect.arrayContaining([authUsers, companies, companyMemberships]));
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
const after = await request(server).get("/api/instance/task-drain").expect(200);
|
||||
expect(after.body.backgroundWork).toBe("none");
|
||||
});
|
||||
|
||||
it("releases failed control authentication and preserves Express error handling", async () => {
|
||||
const { db } = createFakeDb({ beforeWrite: async () => { throw new Error("fixture write failed"); } });
|
||||
const server = express();
|
||||
server.use(idleAdmissionMiddleware);
|
||||
server.use(actorMiddleware(db, { deploymentMode: "authenticated", resolveSession: async () => null }));
|
||||
server.get("/api/health", (_req, res) => res.sendStatus(204));
|
||||
server.use((_err: Error, _req: express.Request, res: express.Response, _next: express.NextFunction) => res.sendStatus(500));
|
||||
trackIdleRequestHandlers(server);
|
||||
await request(server).get("/api/health").set(VALID_HEADERS).expect(500);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
await request(server).get("/api/health").expect(204);
|
||||
});
|
||||
|
||||
it("resyncs an A to B to A context transition inside the debounce window", async () => {
|
||||
const { db, insertedTables } = createFakeDb();
|
||||
const contextA = VALID_HEADERS;
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
import { markIdleIngressTracked } from "../services/idle-local-work.js";
|
||||
import type { Application, Request, RequestHandler } from "express";
|
||||
import { beginIdleTrackedWork, isIdleTaskDrainActive } from "../services/task-admission.js";
|
||||
|
||||
type RequestWork = { pending: number; ended: boolean; closed: boolean; asyncRoute: boolean; routeSequence: number; active: boolean; finish: () => void };
|
||||
const requests = new WeakMap<Request, RequestWork>();
|
||||
function settle(work: RequestWork) {
|
||||
if (work.active && (work.ended || (work.closed && work.asyncRoute)) && work.pending === 0) {
|
||||
work.active = false;
|
||||
work.finish();
|
||||
}
|
||||
}
|
||||
|
||||
// Only the exact control endpoint and health probes bypass admission. The
|
||||
// control route still authenticates each operation. GET is otherwise tracked:
|
||||
// OAuth callbacks and tool streams can create work too.
|
||||
// Reachability does not exempt health/bootstrap or control mutations from
|
||||
// tracking. Only the task-drain read skips request tracking to avoid counting
|
||||
// its own scan; actorMiddleware counts its preceding authentication separately.
|
||||
function isControlRequest(req: Request) {
|
||||
const path = req.path.replace(/\/$/, "");
|
||||
return (path === "/api/instance/task-drain" && ["GET", "POST", "DELETE"].includes(req.method)) ||
|
||||
(path === "/api/health" && ["GET", "HEAD"].includes(req.method));
|
||||
}
|
||||
|
||||
/** Install before body parsers, auth, webhooks and tool ingress. */
|
||||
export const idleAdmissionMiddleware: RequestHandler = (req, res, next) => {
|
||||
const controlRequest = isControlRequest(req);
|
||||
if (!controlRequest && isIdleTaskDrainActive()) {
|
||||
res.set("Retry-After", "1").status(503).json({ error: "instance_preparing_to_sleep" });
|
||||
return;
|
||||
}
|
||||
if (controlRequest && req.method === "GET" && req.path.replace(/\/$/, "") === "/api/instance/task-drain") {
|
||||
next();
|
||||
return;
|
||||
}
|
||||
const work: RequestWork = { pending: 0, ended: false, closed: false, asyncRoute: false, routeSequence: 0, active: true, finish: beginIdleTrackedWork() };
|
||||
requests.set(req, work);
|
||||
res.once("close", () => { work.closed = true; settle(work); });
|
||||
// close/aborted are NOT completion: an async handler can still be committing
|
||||
// a mutation after its client leaves. Wait for the application's end and all
|
||||
// returned handler promises. Callback-only handlers abandoned without end
|
||||
// stay blockers; an async route proves completion when its promise settles.
|
||||
const end = res.end;
|
||||
res.end = function (this: typeof res, ...args: Parameters<typeof end>) {
|
||||
try { return end.apply(this, args); }
|
||||
finally { work.ended = true; settle(work); }
|
||||
} as typeof end;
|
||||
next();
|
||||
};
|
||||
|
||||
// Express 5 exposes its router stack. Walk it once after route registration,
|
||||
// preserving router objects and error-handler arity. Waiting only for finish
|
||||
// loses async work after res.json(), including the disconnected-client case.
|
||||
// Keep this adapter isolated and exercise it against real Express in tests.
|
||||
type Layer = { handle: Function & { stack?: Layer[] }; route?: { stack: Layer[] } };
|
||||
export function trackIdleRequestHandlers(app: Application): void {
|
||||
const seen = new Set<Layer>();
|
||||
const visit = (stack: Layer[], route = false) => {
|
||||
for (const layer of stack) {
|
||||
if (seen.has(layer)) continue;
|
||||
seen.add(layer);
|
||||
if (layer.route) { visit(layer.route.stack, true); continue; }
|
||||
if (layer.handle.stack) { visit(layer.handle.stack); continue; }
|
||||
const original = layer.handle;
|
||||
if (original === idleAdmissionMiddleware) continue;
|
||||
const invoke = (req: Request, args: unknown[]) => {
|
||||
const work = requests.get(req);
|
||||
if (!work) return original(...args);
|
||||
if (!work.active) { work.active = true; work.finish = beginIdleTrackedWork(); }
|
||||
work.pending++;
|
||||
const routeSequence = route ? ++work.routeSequence : undefined;
|
||||
if (route) work.asyncRoute = false;
|
||||
let calledNext = false;
|
||||
const originalNext = args[args.length - 1] as (...args: unknown[]) => unknown;
|
||||
args[args.length - 1] = (...nextArgs: unknown[]) => { calledNext = true; return originalNext(...nextArgs); };
|
||||
const finish = () => { work.pending--; settle(work); };
|
||||
try {
|
||||
const result = original(...args);
|
||||
if (result && typeof result.then === "function") {
|
||||
return Promise.resolve(result).finally(() => {
|
||||
// An async validation middleware which calls next() does not
|
||||
// prove completion of a later callback-only route handler.
|
||||
if (route && !calledNext && work.routeSequence === routeSequence) work.asyncRoute = true;
|
||||
finish();
|
||||
});
|
||||
}
|
||||
finish();
|
||||
return result;
|
||||
} catch (error) { finish(); throw error; }
|
||||
};
|
||||
layer.handle = original.length === 4
|
||||
? function (error: unknown, req: Request, res: unknown, next: unknown) { return invoke(req, [error, req, res, next]); }
|
||||
: function (req: Request, res: unknown, next: unknown) { return invoke(req, [req, res, next]); };
|
||||
}
|
||||
};
|
||||
visit(app.router.stack as Layer[]);
|
||||
markIdleIngressTracked();
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import type { DeploymentMode, LiveEvent } from "@paperclipai/shared";
|
||||
import type { BetterAuthSessionResult } from "../auth/better-auth.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { subscribeCompanyLiveEvents } from "../services/live-events.js";
|
||||
import { trackIdleWork } from "../services/task-admission.js";
|
||||
|
||||
interface WsSocket {
|
||||
readyState: number;
|
||||
@@ -384,7 +385,9 @@ export function setupLiveEventsWebSocketServer(
|
||||
return;
|
||||
}
|
||||
|
||||
void authorizeUpgrade(db, req, companyId, url, {
|
||||
// Upgrade admission precedes this async authentication. An idle hold or
|
||||
// socket close must not make its accepted database writes disappear.
|
||||
void trackIdleWork(authorizeUpgrade(db, req, companyId, url, {
|
||||
deploymentMode: opts.deploymentMode,
|
||||
resolveSessionFromHeaders: opts.resolveSessionFromHeaders,
|
||||
resolveCloudActor: opts.resolveCloudActor,
|
||||
@@ -411,7 +414,7 @@ export function setupLiveEventsWebSocketServer(
|
||||
.catch((err) => {
|
||||
logger.error({ err, path: req.url }, "failed websocket upgrade authorization");
|
||||
rejectUpgrade(socket, "500 Internal Server Error", "upgrade failed");
|
||||
});
|
||||
}));
|
||||
});
|
||||
|
||||
return wss;
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { trackIdleWork } from "../services/task-admission.js";
|
||||
import { Router, type Request } from "express";
|
||||
import { companies, type Db } from "@paperclipai/db";
|
||||
import {
|
||||
@@ -6,13 +7,14 @@ import {
|
||||
patchInstanceGeneralSettingsSchema,
|
||||
startTaskDrainRequestSchema,
|
||||
} from "@paperclipai/shared";
|
||||
import { forbidden } from "../errors.js";
|
||||
import { conflict, forbidden } from "../errors.js";
|
||||
import {
|
||||
cloudTenantPrimaryCompanyId,
|
||||
getCloudStackContext,
|
||||
isCloudManagedInstance,
|
||||
} from "../services/cloud-instance.js";
|
||||
import { getHiddenSettings } from "../services/settings-visibility.js";
|
||||
import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
|
||||
import { validate } from "../middleware/validate.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import {
|
||||
@@ -106,7 +108,7 @@ function assertCanManageInstanceSettings(req: Request) {
|
||||
let taskDrainTransitionQueue: Promise<void> = Promise.resolve();
|
||||
|
||||
function withTaskDrainTransition<T>(run: () => Promise<T>): Promise<T> {
|
||||
const turn = taskDrainTransitionQueue.then(run);
|
||||
const turn = trackIdleWork(taskDrainTransitionQueue.then(run));
|
||||
// Normalize to a settled void promise for the next caller in line, so a
|
||||
// rejected transition (a failed audit write, for example) cannot wedge
|
||||
// every later transition behind it.
|
||||
@@ -297,6 +299,15 @@ export function instanceSettingsRoutes(db: Db) {
|
||||
|
||||
router.get("/instance/task-drain", async (req, res) => {
|
||||
assertBoardOrgAccess(req);
|
||||
if (req.query.idleSleepSafety === "1") {
|
||||
// The report covers every company in this process. Ordinary company
|
||||
// members may read process counters, but not instance-wide work state.
|
||||
assertCanManageInstanceSettings(req);
|
||||
const idleSleepSafety = await readIdleSleepSafety(db, () => heartbeat.getTaskDrainStatus(), Date.now,
|
||||
typeof req.query.ownerId === "string" ? req.query.ownerId : undefined);
|
||||
res.json({ ...heartbeat.getTaskDrainStatus(), idleSleepSafety });
|
||||
return;
|
||||
}
|
||||
res.json(heartbeat.getTaskDrainStatus());
|
||||
});
|
||||
|
||||
@@ -315,7 +326,12 @@ export function instanceSettingsRoutes(db: Db) {
|
||||
// startedAt reflects the moment this request actually took effect,
|
||||
// not the moment it arrived and was queued behind another transition.
|
||||
const drain = await withTaskDrainTransition(async () => {
|
||||
const computed = heartbeat.computeTaskDrain({ ttlMs });
|
||||
const prior = heartbeat.getTaskDrainStatus();
|
||||
if (prior?.ownerId || (req.body.purpose === "idle" && prior?.draining)) {
|
||||
throw conflict("Another task drain is already active");
|
||||
}
|
||||
const computed = heartbeat.computeTaskDrain({ ttlMs,
|
||||
...(req.body.purpose === "idle" ? { purpose: "idle" as const } : {}) });
|
||||
// One transaction for every company's audit row, so a write that
|
||||
// succeeds for one company and fails for another never leaves a
|
||||
// partial activity history behind — either every company gets the
|
||||
@@ -339,6 +355,7 @@ export function instanceSettingsRoutes(db: Db) {
|
||||
details: {
|
||||
startedAt: computed.startedAt,
|
||||
expiresAt: computed.expiresAt,
|
||||
...(computed.ownerId ? { purpose: "idle", ownerId: computed.ownerId } : {}),
|
||||
},
|
||||
}, postCommitActivityPublications),
|
||||
),
|
||||
@@ -369,6 +386,10 @@ export function instanceSettingsRoutes(db: Db) {
|
||||
// queued transition.
|
||||
const wasActive = await withTaskDrainTransition(async () => {
|
||||
const priorStatus = heartbeat.getTaskDrainStatus();
|
||||
if ((priorStatus.ownerId || req.query.ownerId !== undefined) &&
|
||||
(typeof req.query.ownerId !== "string" || req.query.ownerId !== priorStatus.ownerId)) {
|
||||
throw conflict("Task drain ownership changed");
|
||||
}
|
||||
// Read wasActive once, here, and use this same value for the audit
|
||||
// detail and the response body below. A TTL that expires between two
|
||||
// separate reads would otherwise make the two values disagree.
|
||||
|
||||
@@ -6611,7 +6611,8 @@ registry.registerPath({
|
||||
tags: ["instance"],
|
||||
summary:
|
||||
"Get the task-drain status for this process only; quiescent counts in-process work, and a process restart clears it even when the database still holds running rows",
|
||||
responses: { 200: r.ok(), 401: r.unauthorized },
|
||||
request: { query: z.object({ ownerId: z.string().uuid().optional(), idleSleepSafety: z.literal("1").optional().describe("Instance admins can request a conservative durable-work report while admission and ingress are held. Unknown or present work must prevent automatic sleep.") }) },
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
@@ -6626,6 +6627,7 @@ registry.registerPath({
|
||||
400: r.badRequest,
|
||||
401: r.unauthorized,
|
||||
403: r.forbidden,
|
||||
409: r.conflict,
|
||||
},
|
||||
});
|
||||
|
||||
@@ -6634,7 +6636,8 @@ registry.registerPath({
|
||||
path: "/api/instance/task-drain",
|
||||
tags: ["instance"],
|
||||
summary: "End a task drain and restore run admission",
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden },
|
||||
request: { query: z.object({ ownerId: z.string().uuid().optional() }) },
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden, 409: r.conflict },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
import { slackAppConfigurationSchema } from "@paperclipai/shared";
|
||||
|
||||
// Per-turn routing guidance, including resumed CLI sessions. This describes
|
||||
// existing capabilities; it never grants tools or overrides action policies.
|
||||
export function slackChatAgentGuidance(nativeRunner: boolean): string[] {
|
||||
export function slackChatAgentGuidance(nativeRunner: boolean, savedCommand?: string | null): string[] {
|
||||
const command = slackAppConfigurationSchema.shape.command.safeParse(savedCommand);
|
||||
return [
|
||||
"",
|
||||
"Slack replies and actions:",
|
||||
@@ -8,6 +11,13 @@ export function slackChatAgentGuidance(nativeRunner: boolean): string[] {
|
||||
"For a requested file in this conversation, follow the external-chat file-delivery contract below. slack_upload_file is for explicitly uploading an existing Paperclip attachment to another allowed destination. Neither queued nor uncertain means delivered; retain the original operation key and inspect its receipt instead of starting the write again.",
|
||||
"Use only the tools assigned to this run and honor their action policies. Missing tools or Slack permissions are a capability limitation; never ask for a bot token or use another bot to work around it.",
|
||||
"",
|
||||
"Inviting people to talk to this Slack agent:",
|
||||
command.success
|
||||
? `This bot's saved account-linking command is ${command.data} connect. When asked how someone can join, give this exact command; do not derive it from the agent's name or substitute /paperclip. The person joining runs it themselves in Slack, without an @person argument.`
|
||||
: "No saved account-linking command is available in this turn. Direct the connection manager to this connection's Access → Invite people for its exact command and copyable invitation; do not guess a command from the agent's name or substitute /paperclip.",
|
||||
"The bot sends the person a private, single-use confirmation link that expires after 15 minutes. They open it, sign in to Paperclip, and confirm their own Slack account. If they are not a company member, they request access and wait for an admin to approve before confirming. Linked people use their own current Paperclip permissions.",
|
||||
"Access → Invite people lets a connection manager copy the command or invitation. Sharing instructions does not send an invitation or grant access. Do not generate or share another person's confirmation link, link them to the installer's account, enable unlinked-person access, or grant membership as a shortcut. Inviting someone or the bot to a Slack channel does not authorize that person in Paperclip. Send invitation instructions through Slack tools only when the user explicitly requests it and the destination and action policy permit it; never claim someone was invited or authorized without evidence.",
|
||||
"",
|
||||
"Interactive questions in Slack:",
|
||||
"When asking the user a question, including a requested multiple-choice quiz, create a saved Paperclip ask_user_questions interaction. Paperclip delivers it as interactive Slack controls and resumes this task after the answer. Plain-text lettered options do not create those controls. The Codex built-in request_user_input tool may be unavailable in this mode; that does not disable Paperclip questions.",
|
||||
nativeRunner
|
||||
|
||||
@@ -7,7 +7,7 @@ individual action descriptions are in
|
||||
|
||||
## Coverage and gaps
|
||||
|
||||
[`cases.json`](cases.json) defines eleven neutral user prompts with fixtures,
|
||||
[`cases.json`](cases.json) defines twelve neutral user prompts with fixtures,
|
||||
independent expected outcomes, required evidence, and related deterministic
|
||||
regressions. These are **manual model acceptance probes**, not registered Runner
|
||||
or Product E2E workflows. Running the regressions does not run those prompts,
|
||||
@@ -26,10 +26,12 @@ contract; its pass does not establish the probe's complete outcome.
|
||||
| Destructive changes and invitations | Honor the saved action policy and approval decision. | `approval-decline` |
|
||||
| Canvases and lists | Use advertised capabilities; explain plan/scope limitations without inventing success. | `canvas-list-availability` |
|
||||
| Task links | Use the server-provided public URL. | `task-link` |
|
||||
| People joining | Give the saved bot command, personal confirmation flow, and company-access approval. Channel invitations do not grant Paperclip access. | `invite-person` |
|
||||
|
||||
Pins, bookmarks, channel topics, invitations, list edits, and channel creation
|
||||
have tool descriptions and transport fixtures but no dedicated model probe here
|
||||
yet. The canvas probe does not qualify list editing. Automatic app creation,
|
||||
yet. Those provider channel invitations are distinct from the `invite-person`
|
||||
account-linking guidance probe. The canvas probe does not qualify list editing. Automatic app creation,
|
||||
OAuth, setup navigation, and installation recovery are a separate onboarding
|
||||
surface covered by the existing setup and browser tests.
|
||||
|
||||
|
||||
@@ -3,6 +3,15 @@
|
||||
"version": 1,
|
||||
"automatedModelCoverage": "not_registered",
|
||||
"cases": [
|
||||
{
|
||||
"id": "invite-person",
|
||||
"fixture": "A linked requester using a bot with a customized saved slash command that differs from the current agent name. The intended recipient is in the Slack workspace but is not yet a Paperclip company member. Repeat on a resumed turn.",
|
||||
"prompt": "How can my teammate start talking to you here? They're not in our Paperclip organization yet.",
|
||||
"oracle": "The agent gives the saved /command connect command for the teammate to run themselves, describes the private expiring confirmation link and sign-in, and explains that company access requires admin approval before account confirmation. Access → Invite people provides copyable instructions. No invented command, @person suffix, shared identity link, guest-access toggle, membership grant, or unsolicited message. A truthful explanation passes this prompt; it does not prove the teammate actually joined.",
|
||||
"regressionFiles": ["server/src/__tests__/heartbeat-context-summary.test.ts", "server/src/__tests__/heartbeat-slack-invitation.test.ts"],
|
||||
"additionalRegression": "server/src/__tests__/chat-channels.integration.test.ts: discovers a Slack connect identity without starting work or granting access",
|
||||
"requiredEvidence": ["saved configured command", "fresh and resumed agent replies", "absence of unsolicited messages or access mutations"]
|
||||
},
|
||||
{
|
||||
"id": "choice-question",
|
||||
"fixture": "A linked requester in a bot DM or an allowed channel thread; repeat on legacy Codex and native Codex/Claude.",
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { beginIdleTrackedWork } from "./task-admission.js";
|
||||
import { hasStopOnlyCleanup, prepareSandboxStopAndRetain, readStopOnlyCleanup, settleStopOnlyCleanup, stopOnlyCleanupKey } from "./sandbox-stop-and-retain.js";
|
||||
import { readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk";
|
||||
import { preserveEnvironmentSyncOutErrorDiagnostic } from "./environment-sync-out-error.js";
|
||||
@@ -1327,6 +1328,16 @@ function createSandboxEnvironmentDriver(
|
||||
// write, so a flushed row and a synchronous row are indistinguishable to the
|
||||
// sweep.
|
||||
const deferredOrphanCleanups: DeferredOrphanCleanupRecord[] = [];
|
||||
// Keep the token through queue splices, failed flushes and buffer overflow.
|
||||
// Release only when the database durably owns cleanup of this exact orphan.
|
||||
const orphanIdleWork = new Map<string, () => void>();
|
||||
const orphanIdleKey = (record: DeferredOrphanCleanupRecord) => JSON.stringify([
|
||||
record.companyId, record.provider, record.providerLeaseId ?? record,
|
||||
]);
|
||||
const trackOrphanIdleWork = (record: DeferredOrphanCleanupRecord) => {
|
||||
const key = orphanIdleKey(record);
|
||||
if (!orphanIdleWork.has(key)) orphanIdleWork.set(key, beginIdleTrackedWork());
|
||||
};
|
||||
const DEFERRED_ORPHAN_CLEANUP_FAILURE_REASON = "acquire_rejected_teardown_failed";
|
||||
|
||||
// The in-process buffer alone loses an orphan on a restart. So the driver also
|
||||
@@ -1345,6 +1356,7 @@ function createSandboxEnvironmentDriver(
|
||||
// Add one orphan record to the in-process buffer. Report `false` only when the
|
||||
// buffer is full, so the caller keeps the error log as the last durable handle.
|
||||
const enqueueDeferredOrphanCleanup = (record: DeferredOrphanCleanupRecord): boolean => {
|
||||
trackOrphanIdleWork(record);
|
||||
// Dedup by the provider lease id, so a repeated failure for the same orphan
|
||||
// never buffers it twice. Each acquire mints a unique provider lease id, so
|
||||
// two distinct orphans never collide. Skip the dedup for a null lease id,
|
||||
@@ -1406,6 +1418,9 @@ function createSandboxEnvironmentDriver(
|
||||
// leaves the spooled copy for a later flush, which re-inserts a duplicate
|
||||
// the idempotent teardown handles. This order never loses the orphan.
|
||||
await orphanCleanupSpool.remove(record);
|
||||
const idleKey = orphanIdleKey(record);
|
||||
orphanIdleWork.get(idleKey)?.();
|
||||
orphanIdleWork.delete(idleKey);
|
||||
recovered += 1;
|
||||
} catch {
|
||||
// The database is still down. Re-queue the record for a later flush,
|
||||
@@ -1544,6 +1559,7 @@ function createSandboxEnvironmentDriver(
|
||||
cause: unknown,
|
||||
cleanupWriteError: unknown,
|
||||
): Promise<never> => {
|
||||
trackOrphanIdleWork(record);
|
||||
const persisted = await orphanCleanupSpool.append(record);
|
||||
const buffered = enqueueDeferredOrphanCleanup(record);
|
||||
logger.error(
|
||||
|
||||
+129
-3715
File diff suppressed because it is too large.
Load diff
@@ -62,6 +62,7 @@ export function buildPaperclipTaskMarkdown(input: {
|
||||
body: string;
|
||||
} | null;
|
||||
externalChatProvider?: string | null;
|
||||
slackCommand?: string | null;
|
||||
nativeRunner?: boolean;
|
||||
// false builds the compact variant used for resume deltas, where the session
|
||||
// already received the description with the assignment.
|
||||
@@ -140,7 +141,7 @@ export function buildPaperclipTaskMarkdown(input: {
|
||||
);
|
||||
}
|
||||
if (input.externalChatProvider === "slack") {
|
||||
lines.push(...slackChatAgentGuidance(input.nativeRunner === true));
|
||||
lines.push(...slackChatAgentGuidance(input.nativeRunner === true, input.slackCommand));
|
||||
}
|
||||
if (input.externalChatProvider && input.nativeRunner) {
|
||||
lines.push(
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
import type { agents, Db, projectWorkspaces } from "@paperclipai/db";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
ensureManagedProjectWorkspace as legacyEnsureManagedProjectWorkspace,
|
||||
WorkspaceValidationFailure as LegacyWorkspaceValidationFailure,
|
||||
} from "../heartbeat.js";
|
||||
import { resolveDefaultAgentWorkspaceDir } from "../../home-paths.js";
|
||||
import {
|
||||
createHeartbeatWorkspaceResolver,
|
||||
ensureManagedProjectWorkspace,
|
||||
WorkspaceValidationFailure,
|
||||
} from "./workspaces.js";
|
||||
|
||||
const runGit = promisify(execFile);
|
||||
let testRoot: string;
|
||||
|
||||
beforeEach(async () => {
|
||||
testRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-module-"));
|
||||
vi.stubEnv("PAPERCLIP_HOME", testRoot);
|
||||
vi.stubEnv("PAPERCLIP_INSTANCE_ID", "workspace-test");
|
||||
vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "false");
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
vi.unstubAllEnvs();
|
||||
await fs.rm(testRoot, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function agent(id = "agent-one") {
|
||||
return { id, companyId: "company-one" } as typeof agents.$inferSelect;
|
||||
}
|
||||
|
||||
function workspace(id: string, projectId: string, cwd: string) {
|
||||
return {
|
||||
id, projectId, companyId: "company-one", cwd,
|
||||
repoUrl: null, repoRef: null, sourceType: "local_path",
|
||||
} as typeof projectWorkspaces.$inferSelect;
|
||||
}
|
||||
|
||||
// The existing service integration suites cover the SQL and authorization rules.
|
||||
// These query results exercise the new factory's database binding and call order.
|
||||
function databaseWithResults(...results: unknown[][]) {
|
||||
const select = vi.fn(() => {
|
||||
const rows = results.shift() ?? [];
|
||||
const query = {
|
||||
from: () => query,
|
||||
where: () => query,
|
||||
orderBy: async () => rows,
|
||||
then: (resolve: (value: unknown[]) => unknown) => Promise.resolve(rows).then(resolve),
|
||||
};
|
||||
return query;
|
||||
});
|
||||
return { db: { select } as unknown as Db, select };
|
||||
}
|
||||
|
||||
function deferred() {
|
||||
let resolve!: () => void;
|
||||
const promise = new Promise<void>(complete => { resolve = complete; });
|
||||
return { promise, resolve };
|
||||
}
|
||||
|
||||
async function createRepository(parent: string, text: string) {
|
||||
const cwd = path.join(testRoot, parent, "repo");
|
||||
await fs.mkdir(cwd, { recursive: true });
|
||||
await runGit("git", ["init", cwd]);
|
||||
await runGit("git", ["-C", cwd, "config", "user.email", "workspace-test@example.com"]);
|
||||
await runGit("git", ["-C", cwd, "config", "user.name", "Workspace Test"]);
|
||||
await fs.writeFile(path.join(cwd, "README.md"), text);
|
||||
await runGit("git", ["-C", cwd, "add", "README.md"]);
|
||||
await runGit("git", ["-C", cwd, "commit", "-m", "initial"]);
|
||||
return cwd;
|
||||
}
|
||||
|
||||
describe("heartbeat workspace module", () => {
|
||||
it("preserves the legacy error class and checkout entry point", () => {
|
||||
expect(LegacyWorkspaceValidationFailure).toBe(WorkspaceValidationFailure);
|
||||
expect(legacyEnsureManagedProjectWorkspace).toBe(ensureManagedProjectWorkspace);
|
||||
const failure = new WorkspaceValidationFailure("invalid checkout", { reason: "test" });
|
||||
expect(failure).toBeInstanceOf(LegacyWorkspaceValidationFailure);
|
||||
expect(failure.code).toBe("workspace_validation_failed");
|
||||
});
|
||||
|
||||
it("binds each resolver to its own database", async () => {
|
||||
const firstCwd = path.join(testRoot, "first");
|
||||
const secondCwd = path.join(testRoot, "second");
|
||||
await Promise.all([fs.mkdir(firstCwd), fs.mkdir(secondCwd)]);
|
||||
const first = databaseWithResults([workspace("workspace-one", "project-one", firstCwd)]);
|
||||
const second = databaseWithResults([workspace("workspace-two", "project-two", secondCwd)]);
|
||||
const [firstResult, secondResult] = await Promise.all([
|
||||
createHeartbeatWorkspaceResolver(first.db).resolveWorkspaceForRun(agent(), { projectId: "project-one" }, null),
|
||||
createHeartbeatWorkspaceResolver(second.db).resolveWorkspaceForRun(agent(), { projectId: "project-two" }, null),
|
||||
]);
|
||||
expect(firstResult).toMatchObject({ cwd: firstCwd, projectId: "project-one", workspaceId: "workspace-one", source: "project_primary", additionalWorkspaces: [] });
|
||||
expect(secondResult).toMatchObject({ cwd: secondCwd, projectId: "project-two", workspaceId: "workspace-two", source: "project_primary", additionalWorkspaces: [] });
|
||||
expect(first.select).toHaveBeenCalledTimes(1);
|
||||
expect(second.select).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("uses the current issue's project and selected workspace over stale context", async () => {
|
||||
const selectedCwd = path.join(testRoot, "selected");
|
||||
await fs.mkdir(selectedCwd);
|
||||
const database = databaseWithResults(
|
||||
[{ projectId: "current-project", projectWorkspaceId: "selected-workspace" }],
|
||||
[workspace("default-workspace", "current-project", path.join(testRoot, "missing")), workspace("selected-workspace", "current-project", selectedCwd)],
|
||||
);
|
||||
const result = await createHeartbeatWorkspaceResolver(database.db).resolveWorkspaceForRun(
|
||||
agent(),
|
||||
{ issueId: "issue-one", projectId: "stale-project", projectWorkspaceId: "stale-workspace" },
|
||||
null,
|
||||
);
|
||||
expect(result).toMatchObject({ cwd: selectedCwd, projectId: "current-project", workspaceId: "selected-workspace", warnings: [], baseCwdFallback: false });
|
||||
expect(database.select).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("retains a valid prior session workspace and falls back when it disappears", async () => {
|
||||
const priorCwd = path.join(testRoot, "prior-session");
|
||||
await fs.mkdir(priorCwd);
|
||||
const database = databaseWithResults();
|
||||
const resolver = createHeartbeatWorkspaceResolver(database.db);
|
||||
const previousSession = { cwd: priorCwd, workspaceId: "previous-workspace" };
|
||||
expect(await resolver.resolveWorkspaceForRun(agent(), {}, previousSession)).toMatchObject({ cwd: priorCwd, source: "task_session", workspaceId: "previous-workspace", warnings: [] });
|
||||
await fs.rm(priorCwd, { recursive: true });
|
||||
const fallback = await resolver.resolveWorkspaceForRun(agent(), {}, previousSession);
|
||||
expect(fallback).toMatchObject({ cwd: resolveDefaultAgentWorkspaceDir(agent().id), source: "agent_home", workspaceId: null });
|
||||
expect(fallback.warnings).toEqual([expect.stringContaining("is not available")]);
|
||||
expect((await fs.stat(fallback.cwd)).isDirectory()).toBe(true);
|
||||
expect(database.select).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("shares one in-flight checkout across legacy and direct module callers", async () => {
|
||||
const repoUrl = await createRepository("source", "shared checkout\n");
|
||||
const started = deferred();
|
||||
const release = deferred();
|
||||
const resolveGitAuth = vi.fn(async () => {
|
||||
started.resolve();
|
||||
await release.promise;
|
||||
return null;
|
||||
});
|
||||
const input = { companyId: "company-one", projectId: "shared-project", repoUrl, resolveGitAuth };
|
||||
const attempts = Array.from({ length: 8 }, (_, index) =>
|
||||
(index % 2 ? legacyEnsureManagedProjectWorkspace : ensureManagedProjectWorkspace)(input),
|
||||
);
|
||||
await started.promise;
|
||||
release.resolve();
|
||||
const results = await Promise.all(attempts);
|
||||
expect(resolveGitAuth).toHaveBeenCalledTimes(1);
|
||||
expect(new Set(results.map(result => result.cwd)).size).toBe(1);
|
||||
expect(await fs.readFile(path.join(results[0]!.cwd, "README.md"), "utf8")).toBe("shared checkout\n");
|
||||
expect((await runGit("git", ["-C", results[0]!.cwd, "remote", "get-url", "origin"])).stdout.trim()).toBe(repoUrl);
|
||||
});
|
||||
|
||||
it("keeps concurrent repositories with the same basename in separate checkouts", async () => {
|
||||
const [firstRepo, secondRepo] = await Promise.all([
|
||||
createRepository("first-source", "first repository\n"),
|
||||
createRepository("second-source", "second repository\n"),
|
||||
]);
|
||||
const [first, second] = await Promise.all([firstRepo, secondRepo].map(repoUrl =>
|
||||
ensureManagedProjectWorkspace({ companyId: "company-one", projectId: "shared-project", repoUrl }),
|
||||
));
|
||||
expect(first!.cwd).not.toBe(second!.cwd);
|
||||
expect(await fs.readFile(path.join(first!.cwd, "README.md"), "utf8")).toBe("first repository\n");
|
||||
expect(await fs.readFile(path.join(second!.cwd, "README.md"), "utf8")).toBe("second repository\n");
|
||||
});
|
||||
|
||||
it("releases a failed checkout attempt so the next caller can retry", async () => {
|
||||
const repoUrl = await createRepository("retry-source", "retry succeeded\n");
|
||||
const resolveGitAuth = vi.fn()
|
||||
.mockRejectedValueOnce(new Error("credential lookup failed"))
|
||||
.mockResolvedValue(null);
|
||||
const input = { companyId: "company-one", projectId: "retry-project", repoUrl, resolveGitAuth };
|
||||
await expect(ensureManagedProjectWorkspace(input)).rejects.toThrow("credential lookup failed");
|
||||
const result = await legacyEnsureManagedProjectWorkspace(input);
|
||||
expect(resolveGitAuth).toHaveBeenCalledTimes(2);
|
||||
expect(await fs.readFile(path.join(result.cwd, "README.md"), "utf8")).toBe("retry succeeded\n");
|
||||
});
|
||||
});
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,43 @@
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
|
||||
import { idleWorkSnapshot } from "./task-admission.js";
|
||||
|
||||
export type IdleLocalWork = "none" | "present" | "unknown";
|
||||
const spoolDirectories = new Set<string>();
|
||||
export function registerIdleSpoolDirectory(directory: string): void { spoolDirectories.add(path.resolve(directory)); }
|
||||
export const idleAccountingSpoolPath = () => path.join(resolvePaperclipInstanceRoot(), "accounting-receipts");
|
||||
export const idleOrphanSpoolPath = () => process.env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR ??
|
||||
path.join(resolvePaperclipInstanceRoot(), "data", "sandbox-orphan-cleanup");
|
||||
let startupComplete = false;
|
||||
let ingressTracked = false;
|
||||
let scheduledBackups = false;
|
||||
export function markIdleIngressTracked() { ingressTracked = true; }
|
||||
export function markIdleStartupComplete(options: { scheduledBackups: boolean }) {
|
||||
scheduledBackups = options.scheduledBackups;
|
||||
startupComplete = true;
|
||||
}
|
||||
|
||||
/** Any entry, including malformed JSON, a temp file or a failed probe, is
|
||||
* evidence to retain. Recovery readers intentionally skip some such files;
|
||||
* they cannot certify that stopping the process is safe. */
|
||||
export async function inspectIdleSpool(directory: string): Promise<IdleLocalWork> {
|
||||
try {
|
||||
const info = await fs.lstat(directory);
|
||||
if (!info.isDirectory() || info.isSymbolicLink()) return "unknown";
|
||||
const dir = await fs.opendir(directory);
|
||||
try { return (await dir.read()) === null ? "none" : "present"; }
|
||||
finally { await dir.close(); }
|
||||
} catch (error) {
|
||||
return (error as NodeJS.ErrnoException).code === "ENOENT" ? "none" : "unknown";
|
||||
}
|
||||
}
|
||||
|
||||
export async function readIdleLocalWork(): Promise<IdleLocalWork> {
|
||||
if (!startupComplete || !ingressTracked) return "unknown";
|
||||
if (scheduledBackups || idleWorkSnapshot().active !== 0) return "present";
|
||||
const directories = new Set([...spoolDirectories, idleAccountingSpoolPath(), idleOrphanSpoolPath()]);
|
||||
const results = await Promise.all([...directories].map(inspectIdleSpool));
|
||||
if (results.includes("unknown")) return "unknown";
|
||||
return results.includes("present") ? "present" : "none";
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
import { idleWorkSnapshot } from "./task-admission.js";
|
||||
import { readIdleLocalWork, type IdleLocalWork } from "./idle-local-work.js";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { sql } from "drizzle-orm";
|
||||
|
||||
export interface IdleSleepDrainStatus {
|
||||
ownerId?: string;
|
||||
draining: boolean;
|
||||
startedAt: Date | null;
|
||||
expiresAt: Date | null;
|
||||
activeRuns: number;
|
||||
pendingWakes: number;
|
||||
}
|
||||
|
||||
export interface IdleSleepSafety {
|
||||
version: 1;
|
||||
backgroundWork: "none" | "present" | "unknown";
|
||||
}
|
||||
|
||||
// These are deliberately conservative. Completed agent runs and workspace
|
||||
// operations are history, not reasons to keep an otherwise idle instance up.
|
||||
// Other background features remain awake until their work and inbound events
|
||||
// have a durable wake path. In particular, an empty in-memory run set is not
|
||||
// evidence that a restarted process has no persisted work to recover.
|
||||
const WORK_CHECKS = [
|
||||
`SELECT 1 FROM agents WHERE status = 'running' OR
|
||||
(status NOT IN ('terminated', 'paused', 'pending_approval') AND
|
||||
runtime_config->'heartbeat'->>'enabled' = 'true' AND
|
||||
COALESCE((runtime_config->'heartbeat'->>'intervalSec')::numeric, 0) > 0)`,
|
||||
`SELECT 1 FROM heartbeat_runs WHERE
|
||||
status NOT IN ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted')
|
||||
OR scheduled_retry_at IS NOT NULL OR cost_accounting_pending`,
|
||||
`SELECT 1 FROM agent_instruction_working_copies WHERE
|
||||
state NOT IN ('saved', 'unchanged', 'resolved', 'unavailable', 'conflict')
|
||||
OR process_stopped_at IS NULL OR next_attempt_at IS NOT NULL
|
||||
OR receipt ? 'baseline' OR receipt->>'cleanupPending' = 'true'`,
|
||||
`SELECT 1 FROM budget_reservations WHERE state NOT IN ('settled', 'released')`,
|
||||
// Persistent machine credentials can receive work without a human opening
|
||||
// the instance. They need an ingress wake contract before we can sleep.
|
||||
`SELECT 1 FROM agent_api_keys WHERE revoked_at IS NULL`,
|
||||
`SELECT 1 FROM board_api_keys WHERE revoked_at IS NULL
|
||||
AND (expires_at IS NULL OR expires_at > now())`,
|
||||
`SELECT 1 FROM agent_wakeup_requests WHERE
|
||||
status NOT IN ('completed', 'failed', 'cancelled', 'skipped', 'timed_out')`,
|
||||
`SELECT 1 FROM issues WHERE status NOT IN ('done', 'cancelled')`,
|
||||
// A completed issue can still need its first watchdog review, including a
|
||||
// retry after immediate evaluation failed before creating a review/run.
|
||||
`SELECT 1 FROM issue_watchdogs WHERE status <> 'disabled'`,
|
||||
`SELECT 1 FROM workspace_operations WHERE status NOT IN ('succeeded', 'failed', 'cancelled')`,
|
||||
`SELECT 1 FROM environment_leases WHERE status NOT IN ('released', 'expired')`,
|
||||
`SELECT 1 FROM routines WHERE status NOT IN ('paused', 'archived')`,
|
||||
`SELECT 1 FROM routine_runs WHERE status NOT IN ('completed', 'failed', 'skipped', 'cancelled')`,
|
||||
`SELECT 1 FROM plugin_jobs WHERE status <> 'paused'`,
|
||||
`SELECT 1 FROM plugin_job_runs WHERE status NOT IN ('succeeded', 'failed', 'cancelled')`,
|
||||
`SELECT 1 FROM plugin_webhook_deliveries WHERE status NOT IN ('succeeded', 'failed')`,
|
||||
`SELECT 1 FROM tool_connections WHERE status <> 'archived'`,
|
||||
`SELECT 1 FROM issue_recovery_actions WHERE status NOT IN ('resolved', 'cancelled')`,
|
||||
`SELECT 1 FROM status_cards WHERE archived_at IS NULL`,
|
||||
`SELECT 1 FROM external_objects WHERE NOT is_terminal OR next_refresh_at IS NOT NULL`,
|
||||
// These less common work sources fail closed on any retained state. Their
|
||||
// terminal-state exceptions can be added with tests for the owning service.
|
||||
...[
|
||||
"chat_endpoints", "chat_deliveries", "chat_publications", "chat_actions",
|
||||
"chat_completion_deliveries", "connection_event_deliveries",
|
||||
"connection_intent_deliveries", "tool_action_requests", "tool_action_deliveries",
|
||||
"issue_question_response_deliveries", "workspace_runtime_services",
|
||||
"execution_workspace_runtime_leases", "pipeline_automation_executions",
|
||||
"native_run_finalizations", "company_transfer_runs", "decisions", "decision_effect_executions",
|
||||
"decision_archive_notification_outbox", "browser_use_sessions", "browser_use_runs",
|
||||
"browser_use_browsers", "environment_custom_image_setup_sessions", "feedback_exports",
|
||||
"adapter_auth_sessions", "company_secret_proposals", "execution_workspaces",
|
||||
"mcp_oauth_grants", "mcp_mutation_receipts", "mcp_event_subscriptions",
|
||||
"mcp_event_deliveries", "mcp_attachment_uploads", "dot_agent_bindings",
|
||||
"dot_runner_assignments", "dot_runner_operations", "dot_mailbox_items",
|
||||
].map((table) => `SELECT 1 FROM ${table}`),
|
||||
] as const;
|
||||
|
||||
function sameQuietHold(before: IdleSleepDrainStatus, after: IdleSleepDrainStatus, now: number): boolean {
|
||||
return before.ownerId === after.ownerId && before.draining && after.draining && before.activeRuns === 0 && before.pendingWakes === 0
|
||||
&& after.activeRuns === 0 && after.pendingWakes === 0
|
||||
&& before.startedAt !== null && after.startedAt?.getTime() === before.startedAt.getTime()
|
||||
&& (after.expiresAt === null || after.expiresAt.getTime() > now);
|
||||
}
|
||||
|
||||
/** Inventory persisted work after the task-drain admission hold lands.
|
||||
* Only a caller holding the exact idle owner may receive an empty report.
|
||||
*/
|
||||
export async function readIdleSleepSafety(
|
||||
db: Db,
|
||||
getDrainStatus: () => IdleSleepDrainStatus,
|
||||
now: () => number = Date.now,
|
||||
ownerId?: string,
|
||||
inspectLocalWork: () => Promise<IdleLocalWork> = readIdleLocalWork,
|
||||
): Promise<IdleSleepSafety> {
|
||||
const unknown: IdleSleepSafety = { version: 1, backgroundWork: "unknown" };
|
||||
const before = getDrainStatus();
|
||||
const localBefore = idleWorkSnapshot();
|
||||
if (!sameQuietHold(before, before, now())) return unknown;
|
||||
try {
|
||||
const blocked = await db.transaction(async (tx) => {
|
||||
await tx.execute(sql`SET LOCAL statement_timeout = '5s'`);
|
||||
const checks = WORK_CHECKS.map((query) => sql`EXISTS (${sql.raw(query)})`);
|
||||
// Version labels and manifest declarations do not prove arbitrary worker
|
||||
// code has no background activity. No plugin approvals are supported.
|
||||
checks.push(sql`EXISTS (SELECT 1 FROM plugins WHERE status <> 'disabled')`);
|
||||
const rows = await tx.execute<{ blocked: boolean }>(sql`SELECT ${sql.join(checks, sql` OR `)} AS blocked`);
|
||||
return rows.length === 1 && typeof rows[0]?.blocked === "boolean" ? rows[0].blocked : undefined;
|
||||
}, { isolationLevel: "repeatable read", accessMode: "read only" });
|
||||
if (blocked === undefined || !sameQuietHold(before, getDrainStatus(), now())) return unknown;
|
||||
if (blocked) return { version: 1, backgroundWork: "present" };
|
||||
if (!ownerId || ownerId !== before.ownerId || before.expiresAt === null || localBefore.active !== 0) return unknown;
|
||||
const local = await inspectLocalWork();
|
||||
const after = getDrainStatus();
|
||||
const localAfter = idleWorkSnapshot();
|
||||
if (!sameQuietHold(before, after, now()) || localAfter.active !== 0 ||
|
||||
localBefore.generation !== localAfter.generation) return unknown;
|
||||
return { version: 1, backgroundWork: local };
|
||||
|
||||
} catch {
|
||||
// Missing migrations, unknown state, malformed configuration, timeouts and
|
||||
// database failures never authorize stopping the application. Do not leak
|
||||
// SQL, configuration or tenant data through this control response.
|
||||
return unknown;
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { createHash } from "node:crypto";
|
||||
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
|
||||
import { idleOrphanSpoolPath, registerIdleSpoolDirectory } from "./idle-local-work.js";
|
||||
|
||||
/**
|
||||
* One orphan pending-cleanup record. The acquire builds it when the conditional
|
||||
@@ -72,13 +72,6 @@ const RECORD_FILE_SUFFIX = ".json";
|
||||
// without a torn write.
|
||||
let tempFileCounter = 0;
|
||||
|
||||
function resolveDefaultSpoolDir(): string {
|
||||
return (
|
||||
process.env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR ??
|
||||
path.resolve(resolvePaperclipInstanceRoot(), "data", "sandbox-orphan-cleanup")
|
||||
);
|
||||
}
|
||||
|
||||
// Build the stable file name for one record. Key on the provider lease id, the
|
||||
// value the teardown needs, so a repeated append for the same orphan reuses the
|
||||
// same file. A null provider lease id carries no identity, so fall back to a
|
||||
@@ -147,7 +140,8 @@ async function writeFileDurable(absPath: string, data: string): Promise<void> {
|
||||
* directory for a test.
|
||||
*/
|
||||
export function createSandboxOrphanCleanupSpool(spoolDir?: string): SandboxOrphanCleanupSpool {
|
||||
const dir = spoolDir ?? resolveDefaultSpoolDir();
|
||||
const dir = spoolDir ?? idleOrphanSpoolPath();
|
||||
registerIdleSpoolDirectory(dir);
|
||||
|
||||
return {
|
||||
async append(record) {
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
|
||||
export type TaskDrainState = { startedAt: Date; expiresAt: Date | null; ownerId?: string };
|
||||
let taskDrainState: TaskDrainState | null = null;
|
||||
|
||||
export function readTaskDrain(now: Date): TaskDrainState | null {
|
||||
if (taskDrainState?.expiresAt && taskDrainState.expiresAt.getTime() <= now.getTime()) taskDrainState = null;
|
||||
return taskDrainState;
|
||||
}
|
||||
|
||||
export function computeTaskDrain(opts: { ttlMs?: number | null; purpose?: "idle" } = {}): TaskDrainState {
|
||||
const startedAt = new Date();
|
||||
return { startedAt, expiresAt: opts.ttlMs == null ? null : new Date(startedAt.getTime() + opts.ttlMs),
|
||||
...(opts.purpose === "idle" ? { ownerId: randomUUID() } : {}) };
|
||||
}
|
||||
|
||||
export function applyTaskDrain(drain: TaskDrainState): void { taskDrainState = drain; }
|
||||
export function startTaskDrain(opts: { ttlMs?: number | null; purpose?: "idle" } = {}): TaskDrainState {
|
||||
const drain = computeTaskDrain(opts);
|
||||
applyTaskDrain(drain);
|
||||
return drain;
|
||||
}
|
||||
export function stopTaskDrain(): { wasActive: boolean } {
|
||||
const wasActive = readTaskDrain(new Date()) !== null;
|
||||
taskDrainState = null;
|
||||
return { wasActive };
|
||||
}
|
||||
export function isIdleTaskDrainActive(): boolean { return Boolean(readTaskDrain(new Date())?.ownerId); }
|
||||
|
||||
// Tokens count work until its actual promise settles, including after HTTP
|
||||
// disconnects and after a cleanup batch has been removed from its input queue.
|
||||
// The generation also catches work which starts AND finishes during a scan.
|
||||
let generation = 0;
|
||||
let active = 0;
|
||||
export function beginIdleTrackedWork(): () => void {
|
||||
generation++;
|
||||
active++;
|
||||
let finished = false;
|
||||
return () => {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
active--;
|
||||
generation++;
|
||||
};
|
||||
}
|
||||
export function trackIdleWork<T>(work: Promise<T>): Promise<T> {
|
||||
const finish = beginIdleTrackedWork();
|
||||
return work.finally(finish);
|
||||
}
|
||||
export function idleWorkSnapshot() { return { generation, active }; }
|
||||
@@ -1,3 +1,4 @@
|
||||
import { beginIdleTrackedWork } from "./task-admission.js";
|
||||
import { priceAnthropicReceipt } from "./anthropic-pricing.js";
|
||||
import { priceCodexReceipt } from "./codex-pricing.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
@@ -8,7 +9,7 @@ import { and, eq, isNull, sql } from "drizzle-orm";
|
||||
import { heartbeatRuns, runUsageReceipts, type Db } from "@paperclipai/db";
|
||||
import type { AdapterExecutionResult, AdapterUsageCheckpoint } from "@paperclipai/adapter-utils";
|
||||
import { pricingProvenanceSchema, usdToUnits } from "@paperclipai/shared";
|
||||
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
|
||||
import { idleAccountingSpoolPath, registerIdleSpoolDirectory } from "./idle-local-work.js";
|
||||
import { conflict, notFound } from "../errors.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { receiptFingerprint } from "./receipt-fingerprint.js";
|
||||
@@ -37,9 +38,10 @@ const envelopeSchema = z.object({
|
||||
adapterType: z.string().max(100), receipt: checkpointSchema,
|
||||
});
|
||||
export type UsageReceiptEnvelope = z.infer<typeof envelopeSchema>;
|
||||
export const usageReceiptSpoolPath = () => path.join(resolvePaperclipInstanceRoot(), "accounting-receipts");
|
||||
export const usageReceiptSpoolPath = idleAccountingSpoolPath;
|
||||
|
||||
export async function spoolUsageReceipt(envelope: UsageReceiptEnvelope, directory = usageReceiptSpoolPath()) {
|
||||
registerIdleSpoolDirectory(directory);
|
||||
const parsed = envelopeSchema.parse(envelope);
|
||||
await ensureDurableDirectory(directory);
|
||||
const target = path.join(directory, `${parsed.id}.json`);
|
||||
@@ -185,6 +187,7 @@ export async function recoverPendingRunUsageReceipts(db: Db, input: { companyId:
|
||||
}
|
||||
|
||||
export async function createRunUsageRecorder(db: Db, input: { companyId: string; runId: string; adapterType: string }, directory = usageReceiptSpoolPath()) {
|
||||
registerIdleSpoolDirectory(directory);
|
||||
const sourceId = randomUUID();
|
||||
// Verify durable storage before starting paid work, including a directory
|
||||
// sync. The database binding fences checkpoints from replaced controllers.
|
||||
@@ -211,26 +214,31 @@ export async function createRunUsageRecorder(db: Db, input: { companyId: string;
|
||||
}
|
||||
let chain = Promise.resolve();
|
||||
let captureFailed = bound[0].usageJson?.accountingCaptureFailed === true;
|
||||
let finishFailedCapture: (() => void) | undefined;
|
||||
async function persistFailure() {
|
||||
if (!captureFailed) return;
|
||||
await withAccountingTransaction(db, input.companyId, async tx => {
|
||||
await tx.update(heartbeatRuns).set({
|
||||
const persisted = await withAccountingTransaction(db, input.companyId, async tx => {
|
||||
const rows = await tx.update(heartbeatRuns).set({
|
||||
costAccountingPending: true, accountingLastError: "usage_capture_failed",
|
||||
usageJson: sql`coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || '{"accountingCaptureFailed":true,"accountingReceiptReady":false}'::jsonb`,
|
||||
}).where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId),
|
||||
isNull(heartbeatRuns.costAccountedAt), sql`${heartbeatRuns.usageJson}->>'accountingReceiptSourceId' = ${sourceId}`));
|
||||
isNull(heartbeatRuns.costAccountedAt), sql`${heartbeatRuns.usageJson}->>'accountingReceiptSourceId' = ${sourceId}`)).returning({ id: heartbeatRuns.id });
|
||||
return rows.length === 1;
|
||||
});
|
||||
if (persisted) { finishFailedCapture?.(); finishFailedCapture = undefined; }
|
||||
}
|
||||
function enqueue(action: () => Promise<AdapterUsageCheckpoint>) {
|
||||
const finishQueuedCapture = beginIdleTrackedWork();
|
||||
const result = chain.then(action).catch(async error => {
|
||||
captureFailed = true;
|
||||
finishFailedCapture ??= beginIdleTrackedWork();
|
||||
// Persist before exposing rejection: the adapter's flush may throw before
|
||||
// complete(), and failure finalization must not settle an older receipt.
|
||||
try { await persistFailure(); }
|
||||
catch (failure) { logger.error({ err: failure, runId: input.runId }, "Usage capture failure fence requires retry before finalization"); }
|
||||
throw error;
|
||||
});
|
||||
chain = result.then(() => undefined, () => undefined);
|
||||
chain = result.then(() => undefined, () => undefined).finally(finishQueuedCapture);
|
||||
return result;
|
||||
}
|
||||
async function capture(raw: AdapterUsageCheckpoint) {
|
||||
|
||||
@@ -37,6 +37,7 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [
|
||||
"packages/paperclip-runner/runner/crates",
|
||||
"packages/paperclip-runner/scripts/acpx-sidecar-contract.mjs",
|
||||
"packages/paperclip-runner/scripts/build-provider-pack.mjs",
|
||||
"packages/paperclip-runner/scripts/provider-pack-executable-shims.mjs",
|
||||
"packages/paperclip-runner/scripts/build-node-startup-timeout.mjs",
|
||||
"packages/paperclip-runner/scripts/candidate-provider-pack.mjs",
|
||||
"packages/paperclip-runner/scripts/build-copilot-distribution.mjs",
|
||||
|
||||
@@ -369,9 +369,11 @@ describe("runner E2E Daytona image contract", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("invalidates the image when the Cursor runtime isolation patch changes", async () => {
|
||||
const root = await mkdtemp(path.join(tmpdir(), "paperclip-cursor-image-id-"));
|
||||
const patchPath = "packages/paperclip-runner/scripts/cursor-runtime-patch.mjs";
|
||||
it.each([
|
||||
"packages/paperclip-runner/scripts/cursor-runtime-patch.mjs",
|
||||
"packages/paperclip-runner/scripts/provider-pack-executable-shims.mjs",
|
||||
])("invalidates the image when %s changes", async (patchPath) => {
|
||||
const root = await mkdtemp(path.join(tmpdir(), "paperclip-provider-image-id-"));
|
||||
expect(DAYTONA_IMAGE_INPUT_PATHS).toContain(patchPath);
|
||||
const options = {
|
||||
repositoryRoot: root,
|
||||
|
||||
Reference in new issue
Block a user