Improve Your Agents

How to use validation results to improve your agents over time.

Overview

Validation data tells you not just about your code, but about your agents. False positives, missed issues, and calibration drift are signals that an agent's ADL definition needs refinement. This guide covers the practical workflow for analyzing patterns, updating definitions, and measuring improvement.

There are two complementary approaches:

Manual refinement — Analyze issue patterns, update YAML, re-test (this guide)
Prompt-audit pipeline — Run meta-validators against the agent's own definition for automated quality assessment (see Recursive Appreciation)

Both approaches use the same loop: analyze → update YAML → re-render → test → compare.

Analyzing Patterns

Separate Accuracy from Relevance

Not all rejected findings mean the same thing. The tracker distinguishes two types of rejection:

false-positive — The agent was factually wrong. It claimed something about the code that isn't true. This is an accuracy failure.
wontfix — The agent's finding was correct, but it's not worth acting on right now. This is a lifecycle mismatch, not an error.

These require different fixes. False positives need more specific criteria. Wontfix findings may need the agent's scope narrowed, or they may be fine — some agents intentionally surface observations that aren't always actionable.

Identify False Positives

Query issues where the agent was factually wrong:

text

query_issues({
  project: "my-project",
  status: "false-positive",
  agent: "code-validator"
})

Group results by title to find the most common false-positive patterns. If the same type of finding keeps getting marked false-positive, the criterion that produces it needs more specific verification checks.

Check Agent Reliability

text

get_agent_reliability({ project: "my-project" })

This returns false-positive rates, resolution rates, and reliability scores per agent. A high false-positive rate means the agent is making factual errors. A high wontfix rate means the agent's findings are correct but not actionable — different problem, different fix.

You can also filter to a specific agent:

text

get_agent_reliability({ project: "my-project", agent: "code-validator" })

The agent matrix shows which failure taxonomy domains each agent covers:

text

get_agent_matrix({ project: "my-project" })

This identifies blind spots (domains with no coverage), single points of failure (only one agent detects a mode), and high overlap (3+ agents detect the same mode).

Check Registry Health

If the agent is published, the Registry tracks effectiveness metrics across all runs:

text

get_effectiveness({ type: "agent", name: "code-validator" })
get_health({ type: "agent", name: "code-validator" })

Health grades (A-F) combine pass rate, score trends, and issue distribution. Effectiveness shows taxonomy breakdown and composition lift when used in workflows.

ℹ

Note

SDK alternative:

typescript

const reliability = await client.analytics.getAgentReliability({
  project: 'my-project',
});

const matrix = await client.analytics.getAgentMatrix({
  project: 'my-project',
});

Updating Definitions

Refine Criteria

The most common fix for false positives is making criteria more specific. Vague criteria produce inconsistent results:

yaml

# Before — too vague
criteria:
  - id: async_try_catch
    name: "Functions handle errors properly"
    points: 10

# After — specific and measurable
criteria:
  - id: async_try_catch
    name: "Async functions have try/catch or explicit propagation"
    points: 10
    verification:
      method: manual
      checks:
        - "Async functions in request handlers have try/catch"
        - "Functions returning Promise have .catch() or are awaited in try/catch"
        - "Intentional propagation via typed error returns is acceptable"

Adding verification.checks gives the model concrete things to look for instead of making subjective judgments.

Adjust Thresholds

If the agent consistently scores 65-70 on code that your team considers acceptable, the threshold may be too high. You can either use a preset or define explicit thresholds:

yaml

# Using a preset
decisions:
  vocabulary:
    positive: PASS
    negative: FAIL
  preset: low_risk  # 60/40 thresholds instead of quality_gate's 85/70

# Or define explicit thresholds
decisions:
  vocabulary:
    positive: PASS
    conditional: REVIEW
    negative: FAIL
  thresholds:
    - decision: positive
      min_score: 70
      label: "Quality meets standards"
    - decision: conditional
      min_score: 50
      max_score: 69
      label: "Issues found, review recommended"
    - decision: negative
      max_score: 49
      label: "Critical problems, must fix"

Available presets: low_risk, quality_gate, high_stakes, security, critical.

Add Domain Knowledge

Agents perform better with explicit knowledge about what patterns to expect. Use the knowledge_base section to encode domain expertise:

yaml

knowledge_base:
  sections:
    - id: async-patterns
      title: "Async Error Handling Patterns"
      content: >
        Common mistakes: fire-and-forget promises in Express route handlers,
        missing await in test assertions, empty catch blocks.

        Safe patterns: intentional void for logging calls,
        Promise.allSettled for batch operations.

        Red flags: catching Error but only logging message (not stack),
        swallowing errors in middleware chains.

This gives the model concrete examples of what to flag and what to accept, reducing false positives from ambiguous criteria.

Add Auto-Fail Conditions

For critical issues that should fail the validation regardless of score, add auto_fail conditions:

yaml

auto_fail:
  enabled: true
  conditions:
    - id: hardcoded-secrets
      description: "Hardcoded API keys, passwords, or tokens in source files"
    - id: sql-injection
      description: "String concatenation used to build SQL queries"

Auto-fail findings bypass the scoring system entirely — if the condition is detected, the agent's decision is FAIL regardless of the numeric score.

The Improvement Loop

After making changes to the YAML, the loop is: validate → render → install → test → compare.

CLI path:

bash

ulu definitions validate agent --file code-validator.agent.yaml
ulu def render agent --file code-validator.agent.yaml
cp code-validator.md ~/.claude/agents/
# Test in Claude Code, then publish when satisfied

MCP path — update and publish in one step:

text

update_and_publish(type: "agent", name: "code-validator",
  version: "3.2.0", file_path: "code-validator.agent.yaml")

Compare Versions

After publishing, run both versions against the same codebase and diff the results:

text

diff_runs({
  project: "my-project",
  base_run: 12,
  compare_run: 14
})

The diff shows:

Fixed — false positives that are now gone
New — new findings from refined criteria
Unchanged — stable findings across both versions

You can also compare definitions structurally:

text

diff_versions({
  type: "agent", name: "code-validator",
  from: "3.1.0", to: "3.2.0"
})

And check the metric impact of changes:

text

get_diff_impact({
  type: "agent", name: "code-validator",
  from_version: "3.1.0", to_version: "3.2.0"
})

Running a Prompt Audit

For systematic improvement beyond manual pattern analysis, run the prompt-audit pipeline against your agent's rendered definition:

text

/pipelines:prompt-audit code-validator.md

This chains three meta-validators:

prompt-pattern-analyzer — Checks against ecosystem-wide conventions
prompt-engineer — Scores clarity, effectiveness, and specificity
prompt-quality-validator — Catches contradictions, ambiguities, and structural issues

A typical agent improves 8-15 points on the first prompt-audit cycle and converges in 2-3 cycles. See Recursive Appreciation for the full methodology and empirical results.

Measuring Improvement

Score Trends

Track how scores evolve after updating an agent:

text

get_project_trends({ project: "my-project", days: 30 })

A good agent update shows: scores stay stable or improve slightly, while issue count decreases (fewer false positives).

Version Evolution

Track metrics across definition versions in the Registry:

text

get_evolution({ type: "agent", name: "code-validator" })

This shows version-over-version trends with confidence levels (improving, declining, or stable).

Burndown

Monitor open issue count over time by failure domain:

text

get_burndown({ project: "my-project", days: 30 })

A downward burndown trend with stable scores means the team is resolving real issues, not just lowering the bar.

Velocity

Track the rate of change per failure mode:

text

get_velocity({ project: "my-project", days: 30 })

This shows which failure modes are being resolved fastest and which are stalling — useful for identifying where agent criteria might be too strict or too vague.

ℹ

Note

SDK alternative:

typescript

const trends = await client.analytics.getTrends({
  project: 'my-project',
  days: 30,
});

const burndown = await client.analytics.getBurndown({
  project: 'my-project',
  days: 30,
});

Next Steps

Recursive Appreciation — The deeper theory behind why agents improve through iteration, with empirical evidence from 7,100+ tracked issues
ADL Specification — Full schema reference for all definition fields
Definitions — How the two-stage pipeline works
Failure Taxonomy — Classification system for agent findings
Fingerprinting — How issue correlation works

Improve Your Agents

Overview

Analyzing Patterns

Separate Accuracy from Relevance

Identify False Positives

Check Agent Reliability

Find Blind Spots

Check Registry Health

Updating Definitions

Refine Criteria

Adjust Thresholds

Add Domain Knowledge

Add Auto-Fail Conditions

The Improvement Loop

Compare Versions

Running a Prompt Audit

Measuring Improvement

Score Trends

Version Evolution

Burndown

Velocity

Next Steps