Run a Validation
How to run validation agents against your codebase and interpret results.
Overview
There are three ways to run validation agents, depending on your environment:
| Method | Token Cost | Best For |
|---|---|---|
| Claude Code (Agent tool + MCP) | Included in subscription | Interactive development, iteration, day-to-day use |
UluOps CLI (ulu exec) | BYOK — your own API key (Anthropic, OpenAI, or Google) | CI/CD pipelines, headless environments, automation |
Programmatic (@uluops/core + @uluops/ops-sdk) | BYOK | Custom integrations, embedded validation, result tracking |
Don't have an agent yet? See Build Your First Agent to create one from scratch before running validation.
Using Claude Code (Recommended)
In Claude Code, agents run on your subscription — no separate API credits needed. Agents are invoked via the Agent tool and results are tracked through the ops MCP server.
Run a Single Agent
Agent(subagent_type: "code-validator", prompt: "Validate ./src")This works with any agent installed as a Claude Code agent definition or published in the Registry.
Agent discovery: Claude Code loads agents from ~/.claude/agents/ (global) and .claude/agents/ (project-level, takes priority). Each agent is a markdown file with YAML frontmatter. Published agents can be rendered from the Registry via render_definition and placed in either directory.
Run via Slash Commands
Individual agents and multi-agent pipelines are available as slash commands:
# Individual agents
/agents:code-validate ./src
/agents:type-safety ./src
/agents:public-interface ./src
/agents:security ./src
# Multi-agent pipelines (gated phases)
/pipelines:ship ./src
/pipelines:post-implementation ./srcPipelines chain agents in sequence (or parallel groups), stop on gate failures, and save all results to the tracker automatically.
Example Output
After a validation run, the agent produces a structured report:
AGENT REPORT
Files Reviewed:
- src/api/users.ts
- src/services/auth.ts
- src/utils/errors.ts
━━━━━━━━━━━━━━━━━━━━━━━━━━
VALIDATION RESULTS
━━━━━━━━━━━━━━━━━━━━━━━━━━
Score: 78/100
Async Safety: 25/30
Error Propagation: 22/25
Input Validation: 18/25
Error Messages: 13/20
━━━━━━━━━━━━━━━━━━━━━━━━━━
ISSUES FOUND
━━━━━━━━━━━━━━━━━━━━━━━━━━
CRITICAL (Must Fix):
- Empty catch block swallows database error: src/api/users.ts:45 [PRA-EFF/H]
Empty catch block silently discards connection failures
WARNINGS (Should Fix):
- Missing null check on .find() result: src/services/auth.ts:23 [SEM-COM/M]
user.role accessed without verifying user exists
- Error message not actionable: src/utils/errors.ts:78 [STR-OMI/M]
"Invalid input" — should specify which field and expected format
━━━━━━━━━━━━━━━━━━━━━━━━━━
DECISION
━━━━━━━━━━━━━━━━━━━━━━━━━━
PASS — Score 78 meets threshold (≥70)
1 critical issue should be addressed before shippingThe report includes per-category scores, file:line references for each finding, and a failure taxonomy code classifying the type of issue.
Save Results to Tracker
Pipelines save results to the tracker automatically. For individual agent runs or custom validations, save manually via the ops MCP server:
save_run({
project: "my-project",
workflow_type: "post-implementation",
agents: [
{ name: "code-validator", score: 85, decision: "PASS" },
{ name: "test-architect", score: 72, decision: "APPROVED" }
],
recommendations: [
{
agent: "code-validator",
title: "Missing error boundary in React component",
priority: "suggested",
severity: "medium",
failure_code: "SEM-COM/M",
file_path: "src/components/Dashboard.tsx",
line_number: 15
}
]
})Check Project Health
get_project_summary({ project: "my-project" })Using the CLI
The ulu CLI runs agents outside Claude Code using your own API key (BYOK). This is the right choice for CI/CD pipelines and automated workflows.
# Run a single agent (defaults to Claude Sonnet)
ulu exec agent code-validator -t ./src --project my-project
# Run a workflow (multiple agents, gated phases)
ulu exec workflow ship ./src --project my-project
# Use a different model provider
ulu exec agent code-validator -t ./src --project my-project --model gpt-5
ulu exec agent code-validator -t ./src --project my-project --model gemini-3Results are automatically saved to the Platform API. Each run is numbered sequentially per project and workflow type.
Supported Models
The CLI supports three model providers via BYOK:
| Provider | Models | API Key Variable |
|---|---|---|
| Anthropic | haiku, sonnet, opus | ANTHROPIC_API_KEY |
| OpenAI | gpt-5, gpt-5-mini | OPENAI_API_KEY |
gemini-3, gemini-3-flash | GOOGLE_API_KEY |
Use --model to override the agent's default. Model aliases (e.g., sonnet) resolve to the latest version via the Registry's model catalog.
Common Flags
| Flag | Description |
|---|---|
--project | Project name for tracking (required) |
--model | Model to use (e.g., sonnet, gpt-5, gemini-3) |
--local-definitions | Path to local YAML definitions (skip Registry lookup) |
--no-tracking | Skip saving results to the tracker |
The CLI requires ULUOPS_API_KEY for tracker access and at least one model provider key (ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_API_KEY) for agent execution. See the CLI reference for setup.
Programmatic Execution
For custom integrations, two packages work together:
@uluops/core— The execution engine. Runs agents locally against your codebase using your API key. MIT open-source.@uluops/ops-sdk— The tracking SDK. Saves results, manages issues, queries analytics via the Platform API.
Running Agents with @uluops/core
@uluops/core runs agents through an LLM, so it needs two kinds of key: your ULUOPS_API_KEY (platform), and at least one model-provider key — ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_API_KEY. Provider keys are auto-detected from the environment; the client constructs fine without one, but runAgent() will throw at call time if no provider is configured.
import { UluOpsClient } from '@uluops/core';
const client = new UluOpsClient({
apiKey: process.env.ULUOPS_API_KEY,
defaultProject: 'my-project',
});
const result = await client.runAgent('code-validator', './src', {
model: 'sonnet',
});
console.log(result.score); // 85
console.log(result.decision); // "PASS"
console.log(result.recommendations); // [{ title, priority, failure_code, ... }]See the @uluops/core SDK reference for the full client API.
Saving Results with @uluops/ops-sdk
import { OpsClient } from '@uluops/ops-sdk';
const client = new OpsClient({ apiKey: process.env.ULUOPS_API_KEY });
const result = await client.runs.save({
project: 'my-project',
workflowType: 'post-implementation',
agents: [
{ name: 'code-validator', score: 85, decision: 'PASS' },
{ name: 'test-architect', score: 72, decision: 'APPROVED' },
],
recommendations: [
{
agent: 'code-validator',
title: 'Missing error boundary in React component',
priority: 'suggested',
severity: 'medium',
failureCode: 'SEM-COM/M',
filePath: 'src/components/Dashboard.tsx',
lineNumber: 15,
},
],
});
// Check if all gates passed
if (result.run.allGatesPassed) {
console.log('All agents passed — safe to deploy');
} else {
console.log(`${result.correlation.newIssues} new issues found`);
process.exit(1);
}CI/CD Integration Pattern
# .github/workflows/validate.yml
- name: Run validation
run: npx tsx scripts/validate.ts
env:
ULUOPS_API_KEY: ${{ secrets.ULUOPS_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}Have validate.ts exit non-zero on a failed gate — as in the process.exit(1) branch above — so the CI job fails when validation does. Wrap the run in a try/catch to also fail on thrown execution errors (e.g. a workflow gate failure).
Interpreting Results
Scores
Scores range from 0 to 100. Decision thresholds are configured per-agent in the ADL definition using presets or custom values. Common presets:
| Preset | Positive | Conditional | Negative |
|---|---|---|---|
quality_gate | 85+ | 70-84 | Below 70 |
high_stakes | 90+ | 75-89 | Below 75 |
low_risk | 60+ | 40-59 | Below 40 |
security and critical complete the five preset names (low_risk, quality_gate, high_stakes, security, critical). The values above are conventional examples, not a fixed mapping — an agent's effective thresholds come from its own decisions.thresholds, defaulting to a pass score of 75 when unset.
Each agent's threshold is defined in its YAML — check the decisions block to see what score triggers PASS vs FAIL for a specific agent.
Correlation
Every run includes correlation data showing how findings relate to previous runs:
| Field | Meaning |
|---|---|
newIssues | Fingerprints not seen before |
recurring | Known issues still present |
regressions | Previously resolved issues that reappeared |
resolved | Previous issues no longer detected |
A healthy project shows newIssues trending down and resolved trending up over time. See Fingerprinting for how issues are tracked across runs.
Troubleshooting
| Problem | Cause | Fix |
|---|---|---|
| Agent not found | Not in ~/.claude/agents/ or Registry | Render from Registry with render_definition and save to ~/.claude/agents/ |
ULUOPS_API_KEY not set | Environment variable missing | Set in shell profile or .env file |
| Low scores on good code | Model too fast for nuance | Try --model sonnet or --model opus for more accurate scoring |
| Slash command not found | Command file not in commands/ | Check that .md file exists in project's commands/ directory |
| CLI model error | Missing provider API key | Set ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_API_KEY for the model you're using |
Next Steps
- Track Issues & Regressions — Manage issue lifecycle across runs
- Improve Your Agents — Use results to refine scoring
- Build Your First Agent — Create a custom agent for your codebase