Skip to main content

​
Overview

Adversarial training is a key component of evolveRL that helps create more robust and capable agents. The process involves:
  1. Creating adversary agents that generate challenging scenarios
  2. Testing variant agents against these scenarios
  3. Evaluating performance and selecting the best performers

​
Creating Effective Adversaries

​
Basic Adversary

# Create adversary for math problems
adversary = await evolution.build_adversary(
    domain="mathematics",
    description="""Generate challenging math problems that test:
    1. Understanding of core concepts
    2. Edge cases and corner conditions
    3. Common misconceptions
    4. Problem-solving strategies"""
)

​
Advanced Adversary

# Create adversary with specific focus
adversary = await evolution.build_adversary(
    domain="mathematics",
    description="""Generate challenging calculus problems that:
    1. Combine multiple concepts (e.g., integration + trigonometry)
    2. Require creative problem-solving approaches
    3. Include subtle traps or common pitfalls
    4. Test understanding of theoretical foundations
    5. Challenge numerical computation skills
    
    Focus on:
    - Limits and continuity
    - Integration techniques
    - Series and sequences
    - Differential equations"""
)

​
Testing Strategies

​
1. Progressive Difficulty

Start with simpler challenges and gradually increase difficulty: 
# Stage 1: Basic concepts
basic_adversary = await evolution.build_adversary(
    domain="mathematics",
    description="Generate basic algebra problems"
)

# Stage 2: Intermediate challenges
intermediate_adversary = await evolution.build_adversary(
    domain="mathematics",
    description="Generate problems combining algebra and calculus"
)

# Stage 3: Advanced scenarios
advanced_adversary = await evolution.build_adversary(
    domain="mathematics",
    description="Generate complex multi-step problems"
)

# Test variants against each stage
for variant in variants:
    # Basic testing
    history = await evolution.run_interaction(variant, basic_adversary)
    basic_score = await evolution.get_agent_score(history, judge)
    
    # Continue if basic score is good
    if basic_score > 800:
        history = await evolution.run_interaction(variant, intermediate_adversary)
        intermediate_score = await evolution.get_agent_score(history, judge)
        
        # Test advanced if intermediate score is good
        if intermediate_score > 700:
            history = await evolution.run_interaction(variant, advanced_adversary)
            advanced_score = await evolution.get_agent_score(history, judge)

​
2. Diverse Scenarios

Test across different types of challenges: 
# Create multiple adversaries
adversaries = [
    await evolution.build_adversary(
        domain="mathematics",
        description="Generate algebra problems"
    ),
    await evolution.build_adversary(
        domain="mathematics",
        description="Generate calculus problems"
    ),
    await evolution.build_adversary(
        domain="mathematics",
        description="Generate geometry problems"
    )
]

# Test variants against all adversaries
for variant in variants:
    scores = []
    for adversary in adversaries:
        history = await evolution.run_interaction(variant, adversary)
        score = await evolution.get_agent_score(history, judge)
        scores.append(score)
    
    # Use average score
    variant.update_score(sum(scores) / len(scores))

​
Evaluation Criteria

Create comprehensive judging criteria: 
# Create detailed judge
judge = await evolution.build_judge(
    domain="mathematics",
    description="""Evaluate solutions based on:
    1. Correctness (40%):
       - Accurate final answer
       - Valid mathematical steps
       - Proper use of notation
    
    2. Methodology (30%):
       - Clear problem-solving strategy
       - Efficient approach
       - Mathematical rigor
    
    3. Explanation (20%):
       - Clear step-by-step explanation
       - Proper justification
       - Use of mathematical terminology
    
    4. Edge Cases (10%):
       - Handling of special cases
       - Consideration of constraints
       - Error checking"""
)

​
Best Practices

  1. Diverse Testing: Create multiple adversaries with different focuses
  2. Progressive Difficulty: Start simple and increase complexity
  3. Comprehensive Evaluation: Use detailed judging criteria
  4. Balanced Scoring: Consider multiple aspects of performance
  5. Iterative Improvement: Regularly update adversaries based on results

​
Common Pitfalls

  1. Over-specialization: Testing only specific types of problems
  2. Unrealistic Scenarios: Creating challenges that are too artificial
  3. Inconsistent Scoring: Using different criteria across evaluations
  4. Insufficient Testing: Not testing enough edge cases
  5. Poor Feedback: Not providing detailed performance analysis