Run a Validation

How to run validation agents against your codebase and interpret results.

Overview

There are three ways to run validation agents, depending on your environment:

MethodToken CostBest For
Claude Code (Agent tool + MCP)Included in subscriptionInteractive development, iteration, day-to-day use
UluOps CLI (ulu exec)BYOK — your own API key (Anthropic, OpenAI, or Google)CI/CD pipelines, headless environments, automation
Programmatic (@uluops/core + @uluops/ops-sdk)BYOKCustom integrations, embedded validation, result tracking
💡
Tip

Don't have an agent yet? See Build Your First Agent to create one from scratch before running validation.

In Claude Code, agents run on your subscription — no separate API credits needed. Agents are invoked via the Agent tool and results are tracked through the ops MCP server.

Run a Single Agent

text
Agent(subagent_type: "code-validator", prompt: "Validate ./src")

This works with any agent installed as a Claude Code agent definition or published in the Registry.

Note

Agent discovery: Claude Code loads agents from ~/.claude/agents/ (global) and .claude/agents/ (project-level, takes priority). Each agent is a markdown file with YAML frontmatter. Published agents can be rendered from the Registry via render_definition and placed in either directory.

Run via Slash Commands

Individual agents and multi-agent pipelines are available as slash commands:

text
# Individual agents
/agents:code-validate ./src
/agents:type-safety ./src
/agents:public-interface ./src
/agents:security ./src

# Multi-agent pipelines (gated phases)
/pipelines:ship ./src
/pipelines:post-implementation ./src

Pipelines chain agents in sequence (or parallel groups), stop on gate failures, and save all results to the tracker automatically.

Example Output

After a validation run, the agent produces a structured report:

text
AGENT REPORT

Files Reviewed:
- src/api/users.ts
- src/services/auth.ts
- src/utils/errors.ts

━━━━━━━━━━━━━━━━━━━━━━━━━━
VALIDATION RESULTS
━━━━━━━━━━━━━━━━━━━━━━━━━━

Score: 78/100

Async Safety:          25/30
Error Propagation:     22/25
Input Validation:      18/25
Error Messages:        13/20

━━━━━━━━━━━━━━━━━━━━━━━━━━
ISSUES FOUND
━━━━━━━━━━━━━━━━━━━━━━━━━━

CRITICAL (Must Fix):
- Empty catch block swallows database error: src/api/users.ts:45 [PRA-EFF/H]
  Empty catch block silently discards connection failures

WARNINGS (Should Fix):
- Missing null check on .find() result: src/services/auth.ts:23 [SEM-COM/M]
  user.role accessed without verifying user exists
- Error message not actionable: src/utils/errors.ts:78 [STR-OMI/M]
  "Invalid input" — should specify which field and expected format

━━━━━━━━━━━━━━━━━━━━━━━━━━
DECISION
━━━━━━━━━━━━━━━━━━━━━━━━━━

PASS — Score 78 meets threshold (≥70)
1 critical issue should be addressed before shipping

The report includes per-category scores, file:line references for each finding, and a failure taxonomy code classifying the type of issue.

Save Results to Tracker

Pipelines save results to the tracker automatically. For individual agent runs or custom validations, save manually via the ops MCP server:

text
save_run({
  project: "my-project",
  workflow_type: "post-implementation",
  agents: [
    { name: "code-validator", score: 85, decision: "PASS" },
    { name: "test-architect", score: 72, decision: "APPROVED" }
  ],
  recommendations: [
    {
      agent: "code-validator",
      title: "Missing error boundary in React component",
      priority: "suggested",
      severity: "medium",
      failure_code: "SEM-COM/M",
      file_path: "src/components/Dashboard.tsx",
      line_number: 15
    }
  ]
})

Check Project Health

text
get_project_summary({ project: "my-project" })

Using the CLI

The ulu CLI runs agents outside Claude Code using your own API key (BYOK). This is the right choice for CI/CD pipelines and automated workflows.

bash
# Run a single agent (defaults to Claude Sonnet)
ulu exec agent code-validator -t ./src --project my-project

# Run a workflow (multiple agents, gated phases)
ulu exec workflow ship ./src --project my-project

# Use a different model provider
ulu exec agent code-validator -t ./src --project my-project --model gpt-5
ulu exec agent code-validator -t ./src --project my-project --model gemini-3

Results are automatically saved to the Platform API. Each run is numbered sequentially per project and workflow type.

Supported Models

The CLI supports three model providers via BYOK:

ProviderModelsAPI Key Variable
Anthropichaiku, sonnet, opusANTHROPIC_API_KEY
OpenAIgpt-5, gpt-5-miniOPENAI_API_KEY
Googlegemini-3, gemini-3-flashGOOGLE_API_KEY

Use --model to override the agent's default. Model aliases (e.g., sonnet) resolve to the latest version via the Registry's model catalog.

Common Flags

FlagDescription
--projectProject name for tracking (required)
--modelModel to use (e.g., sonnet, gpt-5, gemini-3)
--local-definitionsPath to local YAML definitions (skip Registry lookup)
--no-trackingSkip saving results to the tracker
Note

The CLI requires ULUOPS_API_KEY for tracker access and at least one model provider key (ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_API_KEY) for agent execution. See the CLI reference for setup.

Programmatic Execution

For custom integrations, two packages work together:

  • @uluops/core — The execution engine. Runs agents locally against your codebase using your API key. MIT open-source.
  • @uluops/ops-sdk — The tracking SDK. Saves results, manages issues, queries analytics via the Platform API.

Running Agents with @uluops/core

Note

@uluops/core runs agents through an LLM, so it needs two kinds of key: your ULUOPS_API_KEY (platform), and at least one model-provider key — ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_API_KEY. Provider keys are auto-detected from the environment; the client constructs fine without one, but runAgent() will throw at call time if no provider is configured.

typescript
import { UluOpsClient } from '@uluops/core';

const client = new UluOpsClient({
  apiKey: process.env.ULUOPS_API_KEY,
  defaultProject: 'my-project',
});

const result = await client.runAgent('code-validator', './src', {
  model: 'sonnet',
});

console.log(result.score);           // 85
console.log(result.decision);        // "PASS"
console.log(result.recommendations); // [{ title, priority, failure_code, ... }]

See the @uluops/core SDK reference for the full client API.

Saving Results with @uluops/ops-sdk

typescript
import { OpsClient } from '@uluops/ops-sdk';

const client = new OpsClient({ apiKey: process.env.ULUOPS_API_KEY });

const result = await client.runs.save({
  project: 'my-project',
  workflowType: 'post-implementation',
  agents: [
    { name: 'code-validator', score: 85, decision: 'PASS' },
    { name: 'test-architect', score: 72, decision: 'APPROVED' },
  ],
  recommendations: [
    {
      agent: 'code-validator',
      title: 'Missing error boundary in React component',
      priority: 'suggested',
      severity: 'medium',
      failureCode: 'SEM-COM/M',
      filePath: 'src/components/Dashboard.tsx',
      lineNumber: 15,
    },
  ],
});

// Check if all gates passed
if (result.run.allGatesPassed) {
  console.log('All agents passed — safe to deploy');
} else {
  console.log(`${result.correlation.newIssues} new issues found`);
  process.exit(1);
}

CI/CD Integration Pattern

yaml
# .github/workflows/validate.yml
- name: Run validation
  run: npx tsx scripts/validate.ts
  env:
    ULUOPS_API_KEY: ${{ secrets.ULUOPS_API_KEY }}
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Have validate.ts exit non-zero on a failed gate — as in the process.exit(1) branch above — so the CI job fails when validation does. Wrap the run in a try/catch to also fail on thrown execution errors (e.g. a workflow gate failure).

Interpreting Results

Scores

Scores range from 0 to 100. Decision thresholds are configured per-agent in the ADL definition using presets or custom values. Common presets:

PresetPositiveConditionalNegative
quality_gate85+70-84Below 70
high_stakes90+75-89Below 75
low_risk60+40-59Below 40

security and critical complete the five preset names (low_risk, quality_gate, high_stakes, security, critical). The values above are conventional examples, not a fixed mapping — an agent's effective thresholds come from its own decisions.thresholds, defaulting to a pass score of 75 when unset.

Each agent's threshold is defined in its YAML — check the decisions block to see what score triggers PASS vs FAIL for a specific agent.

Correlation

Every run includes correlation data showing how findings relate to previous runs:

FieldMeaning
newIssuesFingerprints not seen before
recurringKnown issues still present
regressionsPreviously resolved issues that reappeared
resolvedPrevious issues no longer detected

A healthy project shows newIssues trending down and resolved trending up over time. See Fingerprinting for how issues are tracked across runs.

Troubleshooting

ProblemCauseFix
Agent not foundNot in ~/.claude/agents/ or RegistryRender from Registry with render_definition and save to ~/.claude/agents/
ULUOPS_API_KEY not setEnvironment variable missingSet in shell profile or .env file
Low scores on good codeModel too fast for nuanceTry --model sonnet or --model opus for more accurate scoring
Slash command not foundCommand file not in commands/Check that .md file exists in project's commands/ directory
CLI model errorMissing provider API keySet ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_API_KEY for the model you're using

Next Steps