Registry↗Dashboard↗

Validation

How validation works in UluOps — agents, scoring, decisions, and runs.

How Validation Works

ℹ
Note

"Validation" in UluOps has three distinct senses:

  1. Schema validation — checking YAML definitions against ADL/CDL/WDL schemas (ulu definitions validate)
  2. Run validation — executing agents against code to produce scored findings (this page)
  3. Pre-save validation — previewing a tracker save without committing (validate_run)

This page covers sense 2 — the core validation pipeline.

Validation in UluOps follows a pipeline: define what to check, run agents against code, correlate findings with previous runs, and track results over time.

Define→
Run→
Score→
Decide→
Track
←Improve validators←

Each validation run produces:

  • Scores — One per agent (0–100)
  • Decisions — A status per agent (e.g., PASS/FAIL, DEPLOY/REVISE) based on configured decision presets
  • Recommendations — Specific findings with file paths, line numbers, and severity
  • Correlation — Fingerprint matching against prior runs (new, recurring, regression)

Agents

A validation agent is defined by an ADL spec. It reads code, evaluates it against criteria, and produces structured output. Validation agents are one of six agent types — each focuses on one quality dimension:

AgentFocus
code-validatorCode quality, patterns, error handling
type-safety-validatorTypeScript strictness, type assertions
test-architectTest coverage, test quality, edge cases
security-analystOWASP top 10, injection, auth
docs-validatorDocumentation completeness, accuracy

Scoring Model

Each agent produces a numeric score from 0 to 100 and a decision based on configurable thresholds. An agent's thresholds live in its decisions block — set explicitly via decisions.thresholds, or named by a decision preset. Conventional preset configurations:

PresetPositive (≥)ConditionalNegative (<)
quality_gate8570–8470
high_stakes9075–8975
low_risk6040–5940

security and critical complete the five preset names (low_risk, quality_gate, high_stakes, security, critical). The numbers above are conventional examples, not a fixed mapping — an agent's effective thresholds come from its own decisions.thresholds (a pass score of 75 applies when none are set), so check the agent's resolved decisions block rather than assuming a preset's values.

Each agent also defines its own decision vocabulary — the labels used for each threshold tier. Common vocabularies include PASS/FAIL, DEPLOY/REVISE, and SAFE/UNSAFE. Check the decisions block in the agent's YAML to see its specific vocabulary and preset.

ℹ
Note

Decision labels and thresholds vary by agent and workflow. For example, a ship workflow may require PASS ≥ 85 across all agents, while a post-implementation workflow may use a softer gate. The ADL spec for each agent defines its specific thresholds. See the ADL Specification for the full list of presets.

Gate Conditions

Workflows define gates between phases. A gate evaluates whether the pipeline should continue:

  • all_pass — Every agent in the phase must reach its threshold
  • average_above — Average score across agents must exceed threshold
  • any_pass — At least one agent must pass

When a gate fails, the pipeline stops and reports all findings collected so far.

Validation Runs

A run is a single execution of a validation workflow. Each run is stored with:

  • Project name and workflow type
  • Timestamp and run number (auto-incremented per project+workflow)
  • Agent scores and statuses
  • All recommendations with fingerprints
  • Correlation summary (new issues, regressions, resolved)
typescript
const result = await client.runs.save({
  project: 'my-project',
  workflowType: 'post-implementation',
  agents: [
    { name: 'code-validator', score: 85, decision: 'PASS' },
  ],
  recommendations: [
    {
      agent: 'code-validator',
      title: 'Missing error handling in API client',
      priority: 'suggested',
      filePath: 'src/api/client.ts',
      lineNumber: 42,
    },
  ],
});

Agent Performance in Practice

The validation tracker records performance metrics across all agents and projects. Here are the most-used agents from the current ecosystem:

AgentRunsAvg ScorePass RateIssues Found
workflow-synthesis13585.097.0%365
code-validator11286.380.4%373
public-interface-validator11282.067.0%466
test-architect9280.361.5%339
type-safety-validator9086.993.3%244
docs-validator8479.348.8%437
security-analyst8484.981.0%289
pre-implementation-architect7784.983.1%363
assumption-excavator7381.684.7%490
code-auditor4989.098.0%141

Source: UluOps Platform API get_analytics({ metric: "agent_performance" }), 90-day window as of July 2026 (238 agents active in-window). Run get_analytics for current data.

Notable patterns:

  • test-architect and docs-validator have the lowest pass rates among the core validators (61.5% / 48.8%) — test quality and documentation completeness are the hardest dimensions to satisfy, and both produce many findings per run
  • type-safety-validator and code-auditor have the highest pass rates (93.3% / 98.0%) — their domains are highly verifiable, so findings tend to be clear-cut
  • Cognitive-lens and synthesis agents (workflow-synthesis, assumption-excavator) now rank among the most-run agents — the ecosystem has broadened well beyond the original code-quality validators

Query your own project's agent performance with get_agent_reliability (MCP) or client.analytics.getAgentReliability() (SDK).

Workflow Types

UluOps supports multiple workflow types for different stages of development:

WorkflowWhen to RunPurpose
pre-implementationBefore codingValidate design and architecture
post-implementationAfter each phaseIterative code quality checks
shipBefore releaseFinal gate with all agents
chaosStress testingResilience and edge case validation

Each workflow type maintains its own run counter and issue history per project.

Resolution Rates Across Projects

The tracker correlates findings across runs using fingerprint-based matching, enabling resolution tracking per project:

ProjectTotal IssuesResolvedResolution Rate
uluops-registry-api2,0971,78485.1%
ops-uluops-api1,9931,73787.2%
ops-uluops-registry94289895.3%
ops-uluops-dashboard88582393.0%
uluops-specifications62854586.8%
-uluops-core60748579.9%
ops-sdk54951092.9%
registry-sdk43241395.6%

Source: UluOps Platform API get_analytics({ metric: "resolution_rates" }), 90-day window as of July 2026. Showing top projects by volume. Run get_analytics for current data.

Resolution rates for high-volume projects cluster in the 80–95% band. Projects under active heavy development (like the registry API and execution core) sit at the lower end because new issues are filed faster than old ones close; libraries that shipped and stabilized sit at the top.

Across the ecosystem, fixes are durable: of 5,229 issues resolved in the 90-day window, only 5 regressed (0.10% issue-level regression rate; 194 recurred without having been marked resolved — a recurrence, not a regression). (July 2026, under the earlier definition that counted any resolved status reappearing; since October 2026 a regression is a completed fix that reappeared.)