mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 22:24:16 +01:00
## Thinking Path > - Paperclip manages agents, tasks and their durable native results. > - Changed evidence, task authority or review retirement can require a new status assessment. > - Periodic orphan recovery repeatedly reads history to find these changes. > - This change gives reassessment an owner driven by committed changes and saved deadlines. > - Existing result, assessment, decision and task rows remain durable truth. > - The status committer still verifies current sources and authority before writing. > - Settled or insufficient evidence becomes quiet until a supported producer changes it. ## Linked Issues or Issue Description Refs #17702 (merged), #17679 and #17680. This is slice 50, now based on master. It owns native status reassessment; workspace settlement, retained sessions and other orphan duties remain separate. **What happened?** Periodic recovery repeatedly inspects completed native results. Removing that scan without complete writer coverage or current authority checks could strand new evidence or commit an old result over newer work. **Expected behavior** Reassess from existing durable facts after relevant outer transactions commit. Retain real retry/lease deadlines and restart/drain recovery. Preserve current authority, quality-review and supersession checks. Make no recurring SQL when work is settled or dormant. **Steps to reproduce** Complete a native result awaiting evidence or review, leave it idle, then approve a real work product or retire the selected automatic review. Its saved result should advance once when authorized, or record the existing nonterminal business gate, without a periodic scan. ## What Changed - Extract reassessment and its pure status policy from finalization recovery into their owning modules. Add a scoped worker using the existing delivery coordinator and scheduler; no queue table or migration is added. - Retain the three durable discovery triggers: changed work-product history, authoritative task status/version, or retirement of the selected automatic completion review. Preserve selected-source, ownership, effects, retry and lease fences under the existing committer. - Notify from actual task/work-product, claim release, approval, interaction, OAuth, event evidence and workspace-settlement owners before their transaction writes. Keep provider calls outside short transactions and nested writes behind outer commit. - Scope ordinary milestone/finalization/coalesced-input wakes to an issue/run. Keep genuine company authority restoration explicit. Real Photon and coalescer regressions cover all three response/status queues. - Move the current parent's required quality policy into the owner and apply it to every reassessment that could complete a task. An unmet review records the existing in-progress decision and next action, avoiding an operational retry loop. Preserve conversation/session reset locks and the workspace omission-barrier signal. - Skip reassessment in periodic recovery while preserving explicit/startup recovery and the historical supersession guard before independent workspace work. Document the source contract, producer inventory, failures and qualifications in the linked audit. ## Verification - Broad source `9fddb3f0b1157df32b64fb1f22e9d6e9624470e8` passes **2731 distinct cases in 28 complete files**, with no skips: 2458 producer/owner/authority/workspace/quality cases in 394.43 seconds and all 273 governed-response cases in 53.74 seconds. Nineteen dedicated fresh PostgreSQL 17 databases migrate through 0413; eighteen embedded-only invocations preserve fresh-per-call isolation through the existing recovery helper. All 22 temporary imports restore exactly and all 28 hashes match. - Final runtime source `5ea3b0fd016bf7e63ba82a7044825a46a0cd2082` passes **1211 cases in five complete affected files** in 108.23 seconds after the narrowly reproduced evidence-trigger fix: all 43 worker, 145 corpus, 32 finalization-recovery, 968 native-session and 23 quality-review cases. Four fresh databases plus the quality helper's own fresh fixture use PostgreSQL 17. Its one adapted import restores exactly; all five hashes match. The other 27 broad-run test files are unchanged. These qualifications are source-scoped and not cumulatively double-counted. - Required/optional real regressions reproduce retry loops on retired-review and evidence-approval paths before repair; both become SQL quiet afterward with unchanged assertions/time limits. Controlled removal of only ordinary scope guards reproduces two coarse-hint SQL failures; exact guard restoration passes. Real producer, outer transaction, claim, authority, 103-item backlog, retry and drain cases pass. Photon transport is mocked; callbacks, locks and observed SQL are real. - Normal merges preserve 12661 parent/unrelated entries, 35 owned entries and 21 automatic shared results, with five original conflicts reviewed explicitly. The parent's actual squash reconciliation preserves the entire held child tree. The final quality fix changes only its owner and tests. Fourteen added comments are removed with identical nontrivia tokens; no directives are removed and the final comment check is empty. - Database build, server types/build, boundaries and exact-head migration order pass. The database tree is identical to merged #17702: no new SQL, queue table or snapshot is generated. Earlier benchmarks, host limitations, failed assertions and their repairs remain in `doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md` and `doc/idle-native-status-reassessment.md`. - This qualifies targeted external PostgreSQL behavior, not embedded bootstrap on the semaphore-exhausted host, a full local repository suite, deployed sleep or measured fleet savings. Fresh exact-head CI and review remain required. ## Risks - Notifications retain the process-local supported-writer contract. Independent SQL writers and uncoordinated replicas are not covered by a new transport. - Reassessment competes with current authority changes. Existing row/source locks and retries remain required; arbitrary missing identities have no row to lock, so this is not a general predicate-lock guarantee. - Discovery pages 100 rows but retains outstanding candidates in memory. It remains proportional to backlog size, and cancellation occurs between queries. - Other orphan/workspace/session duties and Cloud sleep eligibility remain unchanged. Missing supported producer coverage could strand dormant work, so the producer audit and real callback/transaction tests remain part of acceptance. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution and PostgreSQL tests. The runtime does not expose a more specific serving snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted scope is stated above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>
259 lines
11 KiB
TypeScript
259 lines
11 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { and, desc, eq, sql } from "drizzle-orm";
|
|
|
|
import type { Db } from "@paperclipai/db";
|
|
import { completionContracts } from "@paperclipai/db";
|
|
import type { NativeCompletionSource, StrictCompletionContractInput } from "../../vendor/paperclip-runner/index.js";
|
|
|
|
import { nativeSha256 } from "./canonical.js";
|
|
|
|
export const NATIVE_COMPLETION_CONTRACT_SCHEMA = "paperclip.completion-contract.v1";
|
|
export const NATIVE_COMPLETION_POLICY_VERSION = "phase6-v4";
|
|
|
|
/** Canonical persisted receipt includes the policy that governs the contract. */
|
|
export function nativeCompletionContractSha256(row: Pick<typeof completionContracts.$inferSelect,
|
|
"schemaVersion" | "policyVersion" | "risk" | "completionAuthority" | "contractJson">): string {
|
|
return nativeSha256({
|
|
schemaVersion: row.schemaVersion,
|
|
policyVersion: row.policyVersion,
|
|
risk: row.risk,
|
|
completionAuthority: row.completionAuthority,
|
|
contract: row.contractJson,
|
|
});
|
|
}
|
|
|
|
type CompletionComment = {
|
|
id?: string;
|
|
body: string;
|
|
attachments?: readonly unknown[];
|
|
};
|
|
|
|
export function nativeCompletionSource(kind: NativeCompletionSource["kind"], id: string, content: string): NativeCompletionSource {
|
|
return { kind, id, revision: createHash("sha256").update(content.trim()).digest("hex") };
|
|
}
|
|
|
|
/** Select only the server-owned description source for the ordinary objective path. */
|
|
export function nativeImmediateObjectiveSource(input: {
|
|
issueId: string;
|
|
objectiveSource?: {
|
|
kind: "comment" | "description" | "title";
|
|
id: string;
|
|
revision: string | null;
|
|
} | null;
|
|
excluded: boolean;
|
|
}): NativeCompletionSource | null {
|
|
const source = input.objectiveSource;
|
|
return !input.excluded && source?.kind === "description" && source.id === input.issueId && source.revision
|
|
? { kind: "description", id: source.id, revision: source.revision }
|
|
: null;
|
|
}
|
|
|
|
/** The same source operation produces both the requirement and its provenance. */
|
|
export function nativeCompletionRequestsWithSources(
|
|
comments: readonly CompletionComment[],
|
|
options: { requiredFullWakeCommentCount?: number } = {},
|
|
): { requests: string[]; sources: Array<NativeCompletionSource | null> } {
|
|
if (Number.isSafeInteger(options.requiredFullWakeCommentCount) && (options.requiredFullWakeCommentCount ?? 0) > 0) {
|
|
return { requests: [
|
|
`Read every server-bound pending external-chat comment with read_current_wake_comments until complete=true, then answer all ${options.requiredFullWakeCommentCount} accepted comments in order without omitting a request. Report any unavailable attachment honestly; metadata alone is not its content.`,
|
|
], sources: [null] };
|
|
}
|
|
const entries = comments.flatMap((comment, index) => {
|
|
const body = comment.body.trim();
|
|
if (body) return [{ request: body, source: comment.id ? nativeCompletionSource("comment", comment.id, body) : null }];
|
|
// A file-only message has an independent inspection requirement.
|
|
return comment.attachments?.length
|
|
? [{ request: `Inspect and respond to the attached file(s) on pending comment ${index + 1}.`, source: null }]
|
|
: [];
|
|
});
|
|
return { requests: entries.map((entry) => entry.request), sources: entries.map((entry) => entry.source) };
|
|
}
|
|
|
|
export function nativeCompletionRequestsForComments(
|
|
comments: readonly CompletionComment[],
|
|
options: { requiredFullWakeCommentCount?: number } = {},
|
|
): string[] {
|
|
return nativeCompletionRequestsWithSources(comments, options).requests;
|
|
}
|
|
|
|
export function resolveNativeCompletionPolicy(issue: {
|
|
reviewPolicy?: string | null;
|
|
}) {
|
|
const externalReviewRequired =
|
|
issue.reviewPolicy === "human_only" || issue.reviewPolicy === "not_creator";
|
|
return externalReviewRequired
|
|
? { risk: "standard", completionAuthority: "server_arbiter" } as const
|
|
: { risk: "low", completionAuthority: "agent_claim_policy" } as const;
|
|
}
|
|
|
|
export function buildNativeCompletionContract(
|
|
issue: { title: string; description: string | null },
|
|
options: {
|
|
readonly revision?: number;
|
|
readonly immediateRequest?: string | null;
|
|
readonly immediateRequests?: readonly string[] | null;
|
|
/** ID of this wake's server-verified humanResponses entry; its content is already in task context. */
|
|
readonly humanResponseId?: string | null;
|
|
} = {},
|
|
): StrictCompletionContractInput {
|
|
const immediateRequests = (
|
|
options.immediateRequests ??
|
|
(options.immediateRequest == null ? [] : [options.immediateRequest])
|
|
)
|
|
.map((request) => request.trim())
|
|
.filter((request) => request.length > 0);
|
|
const humanResponseId = options.humanResponseId?.trim();
|
|
const hasFollowUp = immediateRequests.length > 0 || Boolean(humanResponseId);
|
|
// Reference existing context rather than copying the brief/history into every
|
|
// follow-up contract. Keep this guidance stable across comments and resumes.
|
|
const followUpObjective = "Complete the current authorized stage using the task brief and current user direction in the supplied context. Later human direction replaces conflicting scope; preserve other requirements, assigned-skill instructions, and approval gates. Apply authenticated humanResponses only to their question or decision. Clarification is not approval. If acceptance is required, propose or save the requested plan and wait before executing.";
|
|
return {
|
|
revision: String(options.revision ?? 1),
|
|
objective: hasFollowUp
|
|
? followUpObjective
|
|
: issue.title,
|
|
criteria: hasFollowUp
|
|
? [
|
|
...immediateRequests.map((request, index) => ({
|
|
id: immediateRequests.length === 1 ? "objective" : `pending_comment_${index + 1}`,
|
|
requirement: request,
|
|
})),
|
|
...(humanResponseId ? [{
|
|
id: "human_response",
|
|
requirement: `Apply the server-verified humanResponses entry with id ${JSON.stringify(humanResponseId)} in the supplied current request context, within its question or decision scope and subject to later user direction.`,
|
|
}] : []),
|
|
]
|
|
: [
|
|
{
|
|
id: "objective",
|
|
requirement:
|
|
issue.description?.trim() || `Complete: ${issue.title}`,
|
|
},
|
|
],
|
|
};
|
|
}
|
|
|
|
function nativeCompletionContractLockKey(companyId: string, issueId: string): string {
|
|
return `paperclip:native-completion-contract:${companyId}:${issueId}`;
|
|
}
|
|
|
|
export async function lockCurrentNativeCompletionContract(tx: Db, input: {
|
|
companyId: string; issueId: string; contractId: string;
|
|
}): Promise<boolean> {
|
|
const [lock] = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(
|
|
${nativeCompletionContractLockKey(input.companyId, input.issueId)}, 0)) as acquired`) as unknown as Array<{ acquired: boolean }>;
|
|
if (!lock.acquired) return false;
|
|
const [latest] = await tx.select({ id: completionContracts.id }).from(completionContracts)
|
|
.where(and(eq(completionContracts.companyId, input.companyId), eq(completionContracts.issueId, input.issueId)))
|
|
.orderBy(desc(completionContracts.revision)).limit(1);
|
|
return latest?.id === input.contractId;
|
|
}
|
|
|
|
export async function ensureNativeCompletionContract(input: {
|
|
db: Db;
|
|
companyId: string;
|
|
issue: {
|
|
id: string;
|
|
title: string;
|
|
description: string | null;
|
|
reviewPolicy?: string | null;
|
|
};
|
|
actorId: string;
|
|
immediateRequest?: string | null;
|
|
/** Provenance for a singular server-selected request, when explicitly known. */
|
|
immediateRequestSource?: NativeCompletionSource | null;
|
|
immediateRequests?: readonly string[] | null;
|
|
immediateRequestSources?: readonly (NativeCompletionSource | null)[];
|
|
humanResponseId?: string | null;
|
|
}) {
|
|
return input.db.transaction(async (tx) => {
|
|
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(
|
|
${nativeCompletionContractLockKey(input.companyId, input.issue.id)}, 0))`);
|
|
const policy = resolveNativeCompletionPolicy(input.issue);
|
|
const latest = await tx
|
|
.select()
|
|
.from(completionContracts)
|
|
.where(and(
|
|
eq(completionContracts.companyId, input.companyId),
|
|
eq(completionContracts.issueId, input.issue.id),
|
|
))
|
|
.orderBy(desc(completionContracts.revision))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
const latestRevision = latest?.revision ?? 1;
|
|
const latestCandidate = buildNativeCompletionContract(input.issue, {
|
|
revision: latestRevision,
|
|
immediateRequest: input.immediateRequest,
|
|
immediateRequests: input.immediateRequests,
|
|
humanResponseId: input.humanResponseId,
|
|
});
|
|
const latestCandidateSha256 = nativeCompletionContractSha256({
|
|
schemaVersion: NATIVE_COMPLETION_CONTRACT_SCHEMA,
|
|
policyVersion: NATIVE_COMPLETION_POLICY_VERSION,
|
|
...policy,
|
|
contractJson: latestCandidate as unknown as Record<string, unknown>,
|
|
});
|
|
if (latest?.canonicalSha256 === latestCandidateSha256) {
|
|
return { row: latest, contract: latestCandidate, sources: buildNativeCompletionContractSources(input) };
|
|
}
|
|
|
|
const nextRevision = latest ? latest.revision + 1 : 1;
|
|
const contract = buildNativeCompletionContract(input.issue, {
|
|
revision: nextRevision,
|
|
immediateRequest: input.immediateRequest,
|
|
immediateRequests: input.immediateRequests,
|
|
humanResponseId: input.humanResponseId,
|
|
});
|
|
const canonicalSha256 = nativeCompletionContractSha256({
|
|
schemaVersion: NATIVE_COMPLETION_CONTRACT_SCHEMA,
|
|
policyVersion: NATIVE_COMPLETION_POLICY_VERSION,
|
|
...policy,
|
|
contractJson: contract as unknown as Record<string, unknown>,
|
|
});
|
|
const [row] = await tx.insert(completionContracts).values({
|
|
companyId: input.companyId,
|
|
issueId: input.issue.id,
|
|
revision: nextRevision,
|
|
schemaVersion: NATIVE_COMPLETION_CONTRACT_SCHEMA,
|
|
policyVersion: NATIVE_COMPLETION_POLICY_VERSION,
|
|
...policy,
|
|
incompleteCriteriaPolicy: "preserve_non_terminal",
|
|
contractJson: contract as unknown as Record<string, unknown>,
|
|
canonicalSha256,
|
|
createdByActorType: "system",
|
|
createdByActorId: input.actorId,
|
|
supersedesContractId: latest?.id ?? null,
|
|
}).returning();
|
|
if (!row) throw new Error("native_completion_contract_not_persisted");
|
|
return { row, contract, sources: buildNativeCompletionContractSources(input) };
|
|
});
|
|
}
|
|
|
|
|
|
export function buildNativeCompletionContractSources(input: {
|
|
issue: { id: string; description: string | null };
|
|
immediateRequest?: string | null;
|
|
immediateRequestSource?: NativeCompletionSource | null;
|
|
immediateRequests?: readonly string[] | null;
|
|
immediateRequestSources?: readonly (NativeCompletionSource | null)[];
|
|
humanResponseId?: string | null;
|
|
}): Array<{ id: string; source: NativeCompletionSource }> {
|
|
const requests = (input.immediateRequests ?? (input.immediateRequest == null ? [] : [input.immediateRequest]))
|
|
.map((body, index) => ({
|
|
body: body.trim(),
|
|
source: input.immediateRequests
|
|
? input.immediateRequestSources?.[index]
|
|
: input.immediateRequestSource,
|
|
}))
|
|
.filter((entry) => entry.body.length > 0);
|
|
if (requests.length > 0) {
|
|
return requests.flatMap((entry, index) => entry.source ? [{
|
|
id: requests.length === 1 ? "objective" : `pending_comment_${index + 1}`,
|
|
source: entry.source,
|
|
}] : []);
|
|
}
|
|
return !input.humanResponseId?.trim() && input.issue.description?.trim()
|
|
? [{ id: "objective", source: nativeCompletionSource("description", input.issue.id, input.issue.description) }]
|
|
: [];
|
|
}
|