Validation
How validation works in UluOps — agents, scoring, decisions, and runs.
How Validation Works
"Validation" in UluOps has three distinct senses:
- Schema validation — checking YAML definitions against ADL/CDL/WDL schemas (
ulu definitions validate) - Run validation — executing agents against code to produce scored findings (this page)
- Pre-save validation — previewing a tracker save without committing (
validate_run)
This page covers sense 2 — the core validation pipeline.
Validation in UluOps follows a pipeline: define what to check, run agents against code, correlate findings with previous runs, and track results over time.
Each validation run produces:
- Scores — One per agent (0–100)
- Decisions — A status per agent (e.g., PASS/FAIL, DEPLOY/REVISE) based on configured decision presets
- Recommendations — Specific findings with file paths, line numbers, and severity
- Correlation — Fingerprint matching against prior runs (new, recurring, regression)
Agents
A validation agent is defined by an ADL spec. It reads code, evaluates it against criteria, and produces structured output. Validation agents are one of six agent types — each focuses on one quality dimension:
| Agent | Focus |
|---|---|
code-validator | Code quality, patterns, error handling |
type-safety-validator | TypeScript strictness, type assertions |
test-architect | Test coverage, test quality, edge cases |
security-analyst | OWASP top 10, injection, auth |
docs-validator | Documentation completeness, accuracy |
Scoring Model
Each agent produces a numeric score from 0 to 100 and a decision based on configurable thresholds. An agent's thresholds live in its decisions block — set explicitly via decisions.thresholds, or named by a decision preset. Conventional preset configurations:
| Preset | Positive (≥) | Conditional | Negative (<) |
|---|---|---|---|
quality_gate | 85 | 70–84 | 70 |
high_stakes | 90 | 75–89 | 75 |
low_risk | 60 | 40–59 | 40 |
security and critical complete the five preset names (low_risk, quality_gate, high_stakes, security, critical). The numbers above are conventional examples, not a fixed mapping — an agent's effective thresholds come from its own decisions.thresholds (a pass score of 75 applies when none are set), so check the agent's resolved decisions block rather than assuming a preset's values.
Each agent also defines its own decision vocabulary — the labels used for each threshold tier. Common vocabularies include PASS/FAIL, DEPLOY/REVISE, and SAFE/UNSAFE. Check the decisions block in the agent's YAML to see its specific vocabulary and preset.
Decision labels and thresholds vary by agent and workflow. For example, a ship workflow may require PASS ≥ 85 across all agents, while a post-implementation workflow may use a softer gate. The ADL spec for each agent defines its specific thresholds. See the ADL Specification for the full list of presets.
Gate Conditions
Workflows define gates between phases. A gate evaluates whether the pipeline should continue:
all_pass— Every agent in the phase must reach its thresholdaverage_above— Average score across agents must exceed thresholdany_pass— At least one agent must pass
When a gate fails, the pipeline stops and reports all findings collected so far.
Validation Runs
A run is a single execution of a validation workflow. Each run is stored with:
- Project name and workflow type
- Timestamp and run number (auto-incremented per project+workflow)
- Agent scores and statuses
- All recommendations with fingerprints
- Correlation summary (new issues, regressions, resolved)
const result = await client.runs.save({
project: 'my-project',
workflowType: 'post-implementation',
agents: [
{ name: 'code-validator', score: 85, decision: 'PASS' },
],
recommendations: [
{
agent: 'code-validator',
title: 'Missing error handling in API client',
priority: 'suggested',
filePath: 'src/api/client.ts',
lineNumber: 42,
},
],
});Agent Performance in Practice
The validation tracker records performance metrics across all agents and projects. Here are the most-used agents from the current ecosystem:
| Agent | Runs | Avg Score | Pass Rate | Issues Found |
|---|---|---|---|---|
workflow-synthesis | 135 | 85.0 | 97.0% | 365 |
code-validator | 112 | 86.3 | 80.4% | 373 |
public-interface-validator | 112 | 82.0 | 67.0% | 466 |
test-architect | 92 | 80.3 | 61.5% | 339 |
type-safety-validator | 90 | 86.9 | 93.3% | 244 |
docs-validator | 84 | 79.3 | 48.8% | 437 |
security-analyst | 84 | 84.9 | 81.0% | 289 |
pre-implementation-architect | 77 | 84.9 | 83.1% | 363 |
assumption-excavator | 73 | 81.6 | 84.7% | 490 |
code-auditor | 49 | 89.0 | 98.0% | 141 |
Source: UluOps Platform API get_analytics({ metric: "agent_performance" }), 90-day window as of July 2026 (238 agents active in-window). Run get_analytics for current data.
Notable patterns:
test-architectanddocs-validatorhave the lowest pass rates among the core validators (61.5% / 48.8%) — test quality and documentation completeness are the hardest dimensions to satisfy, and both produce many findings per runtype-safety-validatorandcode-auditorhave the highest pass rates (93.3% / 98.0%) — their domains are highly verifiable, so findings tend to be clear-cut- Cognitive-lens and synthesis agents (
workflow-synthesis,assumption-excavator) now rank among the most-run agents — the ecosystem has broadened well beyond the original code-quality validators
Query your own project's agent performance with get_agent_reliability (MCP) or client.analytics.getAgentReliability() (SDK).
Workflow Types
UluOps supports multiple workflow types for different stages of development:
| Workflow | When to Run | Purpose |
|---|---|---|
pre-implementation | Before coding | Validate design and architecture |
post-implementation | After each phase | Iterative code quality checks |
ship | Before release | Final gate with all agents |
chaos | Stress testing | Resilience and edge case validation |
Each workflow type maintains its own run counter and issue history per project.
Resolution Rates Across Projects
The tracker correlates findings across runs using fingerprint-based matching, enabling resolution tracking per project:
| Project | Total Issues | Resolved | Resolution Rate |
|---|---|---|---|
| uluops-registry-api | 2,097 | 1,784 | 85.1% |
| ops-uluops-api | 1,993 | 1,737 | 87.2% |
| ops-uluops-registry | 942 | 898 | 95.3% |
| ops-uluops-dashboard | 885 | 823 | 93.0% |
| uluops-specifications | 628 | 545 | 86.8% |
| -uluops-core | 607 | 485 | 79.9% |
| ops-sdk | 549 | 510 | 92.9% |
| registry-sdk | 432 | 413 | 95.6% |
Source: UluOps Platform API get_analytics({ metric: "resolution_rates" }), 90-day window as of July 2026. Showing top projects by volume. Run get_analytics for current data.
Resolution rates for high-volume projects cluster in the 80–95% band. Projects under active heavy development (like the registry API and execution core) sit at the lower end because new issues are filed faster than old ones close; libraries that shipped and stabilized sit at the top.
Across the ecosystem, fixes are durable: of 5,229 issues resolved in the 90-day window, only 5 regressed (0.10% issue-level regression rate; 194 recurred without having been marked resolved — a recurrence, not a regression). (July 2026, under the earlier definition that counted any resolved status reappearing; since October 2026 a regression is a completed fix that reappeared.)