# {{Project Name}} LLM Agent Evaluation Suite

[[LLM: Initial Setup

1. Replace {{Project Name}} with actual project name
2. Define comprehensive test scenarios covering all agent capabilities
3. Include edge cases, adversarial inputs, and performance benchmarks
4. Configure for automated execution with PromptFoo or similar tools

Output file location: `docs/ai-agents/evaluation-suite.md`]]

## Evaluation Overview

- **Project**: {{project-name}}
- **Agent(s) Under Test**: {{agent-names}}
- **Evaluation Framework**: PromptFoo {{version}}
- **Last Updated**: {{date}}
- **Test Coverage**: {{percentage}}%

## Test Categories

### Functional Tests

[[LLM: Define tests that verify core functionality works as expected.]]

<<REPEAT: functional_test>>

#### {{Test Name}}

- **Test ID**: `{{test-id}}`
- **Category**: {{functional-category}}
- **Priority**: {{High/Medium/Low}}
- **Automated**: {{Yes/No}}

**Test Scenario**:

````yaml
description: '{{test-description}}'
vars:
  input: '{{test-input}}'
  context: '{{optional-context}}'
expected:
  - output: '{{expected-output}}'
  - contains:
      - '{{keyword1}}'
      - '{{keyword2}}'
  - not_contains:
      - '{{forbidden1}}'
      - '{{forbidden2}}'
  - format: '{{json/markdown/plain}}'
assertions:
  - type: equals
    value: '{{exact-match}}'
  - type: contains
    value: '{{partial-match}}'
  - type: javascript
    value: |
      output.length > {{min-length}} &&
      output.length < {{max-length}}
```text

**Success Criteria**:
- {{criterion-1}}
- {{criterion-2}}
<</REPEAT>>

### Edge Case Tests

[[LLM: Test boundary conditions and unusual inputs.]]

<<REPEAT: edge_case>>
#### {{Edge Case Name}}
- **Test ID**: `{{test-id}}`
- **Risk Level**: {{High/Medium/Low}}
- **Failure Impact**: {{impact-description}}

**Scenarios**:
1. **Empty Input**: {{empty-behavior}}
2. **Oversized Input**: {{large-input-behavior}}
3. **Special Characters**: {{special-char-handling}}
4. **Language Mix**: {{multilingual-handling}}
5. **Concurrent Requests**: {{concurrency-behavior}}

**Test Configuration**:
```yaml
edge_cases:
  - name: '{{case-name}}'
    input: '{{edge-input}}'
    expected_behavior: '{{graceful-handling}}'
    error_type: '{{expected-error}}'
````

<</REPEAT>>

### Safety and Alignment Tests

[[LLM: Verify safety measures and alignment constraints.]]

#### Content Safety

````yaml
safety_tests:
  harmful_content:
    - prompt: '{{harmful-prompt}}'
      expected: refusal
      message_contains: cannot assist
  bias_detection:
    - prompt: '{{biased-prompt}}'
      expected: neutral
      bias_score: < 0.3
  privacy_protection:
    - prompt: '{{pii-prompt}}'
      expected: redacted
      pii_detected: false
```text

#### Jailbreak Attempts
<<REPEAT: jailbreak>>
- **Technique**: {{jailbreak-type}}
- **Test Prompt**: `{{jailbreak-prompt}}`
- **Expected Response**: {{safe-response}}
- **Detection Method**: {{how-detected}}
<</REPEAT>>

### Performance Benchmarks

[[LLM: Define performance tests with specific metrics.]]

#### Latency Tests
```yaml
performance:
  latency:
    - scenario: simple_query
      max_latency_ms:
        '[object Object]': null
      percentiles:
        p50:
          '[object Object]': null
        p95:
          '[object Object]': null
        p99:
          '[object Object]': null
    - scenario: complex_analysis
      max_latency_ms:
        '[object Object]': null
      timeout_ms:
        '[object Object]': null
````

#### Throughput Tests

````yaml
throughput:
  - concurrent_users: {{number}}
    duration_seconds: {{duration}}
    target_rps: {{requests-per-second}}
    error_rate_threshold: {{percentage}}%
```text

#### Resource Usage
```yaml
resources:
  - metric: token_usage
    limit_per_request:
      '[object Object]': null
    average_target:
      '[object Object]': null
  - metric: memory_usage
    max_mb:
      '[object Object]': null
  - metric: cost_per_request
    max_usd:
      '[object Object]': null
````

### Quality Evaluation

[[LLM: Define quality metrics and evaluation criteria.]]

#### Accuracy Tests

<<REPEAT: accuracy_test>>

##### {{Domain}} Accuracy

- **Test Set**: {{dataset-name}}
- **Sample Size**: {{n-samples}}
- **Target Accuracy**: {{percentage}}%

**Evaluation Method**:

````python
def evaluate_{{domain}}_accuracy(response, ground_truth):
    # {{evaluation-logic}}
    return accuracy_score
```text

**Results**:
| Model | Accuracy | Precision | Recall | F1 Score |
|-------|----------|-----------|--------|----------|
| {{model}} | {{acc}}% | {{prec}}% | {{rec}}% | {{f1}}% |
<</REPEAT>>

#### Consistency Tests
```yaml
consistency:
  - test: "deterministic_responses"
    runs: {{number-of-runs}}
    temperature: 0
    expected_variance: < {{threshold}}

  - test: "semantic_consistency"
    paraphrased_queries: {{number}}
    similarity_threshold: > {{threshold}}
````

### Integration Tests

[[LLM: Test agent integration with other systems.]]

<<REPEAT: integration>>

#### {{System}} Integration

- **Integration Type**: {{API/Database/Service}}
- **Test Coverage**: {{percentage}}%

**Test Scenarios**:

1. **Connection Test**: {{connectivity-check}}
2. **Data Flow Test**: {{data-validation}}
3. **Error Handling**: {{failure-recovery}}
4. **Performance Impact**: {{latency-addition}}

**Mock Configuration**:

````yaml
mocks:
  '[object Object]':
    response_time_ms:
      '[object Object]': null
    success_rate:
      '[object Object]': null
    data_format:
      '[object Object]': null
```text
<</REPEAT>>

### Regression Tests

[[LLM: Ensure new changes don't break existing functionality.]]

#### Test Suite Version Control
```yaml
regression_suite:
  version: {{suite-version}}
  baseline_date: {{date}}
  test_count: {{number}}
  coverage_delta: {{+/-percentage}}%
````

#### Critical Path Tests

<<REPEAT: critical_path>>

- **Path**: {{user-journey}}
- **Steps**: {{number-of-steps}}
- **Must Pass**: {{Yes/No}}
- **Last Verified**: {{date}}
  <</REPEAT>>

## Evaluation Automation

### PromptFoo Configuration

````yaml
providers:
  - id:
      '[object Object]': null
    config:
      apiKey: ${API_KEY}
      model:
        '[object Object]': null
      temperature:
        '[object Object]': null
      max_tokens:
        '[object Object]': null
prompts:
  - file://prompts/{{prompt-file}}.txt
tests:
  - file://tests/functional/*.yml
  - file://tests/safety/*.yml
  - file://tests/performance/*.yml
  - file://tests/quality/*.yml
outputPath: ./results/{{timestamp}}/
assertions:
  - type: python
    file: ./evaluators/custom_evaluator.py
```text

### Continuous Evaluation

#### Scheduled Runs
```yaml
schedule:
  daily:
    - time: 02:00 UTC
      suite: full
      notify:
        - '{{email}}'
  hourly:
    - time: '*/1 * * * *'
      suite: smoke
      alert_on_failure: true
  on_deploy:
    - trigger: deployment
      suite: critical_path
      block_on_failure: true
````

#### Results Tracking

````yaml
metrics_storage:
  backend:
    '[object Object]': null
  retention_days:
    '[object Object]': null
  dashboards:
    - name: LLM Agent Health
      url: '{{dashboard-url}}'
```text

## Evaluation Reports

### Report Generation
```python
# Generate evaluation report
def generate_report(test_results):
    report = {
        "summary": calculate_summary(test_results),
        "passed": filter_passed_tests(test_results),
        "failed": filter_failed_tests(test_results),
        "performance": calculate_performance_metrics(test_results),
        "recommendations": generate_recommendations(test_results)
    }
    return format_report(report)
````

### Key Metrics Dashboard

| Metric               | Target       | Current      | Trend     | Status  |
| -------------------- | ------------ | ------------ | --------- | ------- |
| Functional Pass Rate | >{{target}}% | {{current}}% | {{↑/↓/→}} | {{✓/✗}} |
| Safety Score         | >{{target}}  | {{current}}  | {{↑/↓/→}} | {{✓/✗}} |
| Avg Latency (ms)     | <{{target}}  | {{current}}  | {{↑/↓/→}} | {{✓/✗}} |
| Error Rate           | <{{target}}% | {{current}}% | {{↑/↓/→}} | {{✓/✗}} |
| Cost per 1K requests | <${{target}} | ${{current}} | {{↑/↓/→}} | {{✓/✗}} |

## Test Data Management

### Test Dataset

- **Location**: {{dataset-path}}
- **Format**: {{json/csv/parquet}}
- **Size**: {{number-of-examples}}
- **Categories**: {{list-categories}}
- **Update Frequency**: {{how-often}}

### Synthetic Data Generation

```python
# Generate test data
def generate_test_data(category, count):
    generator = TestDataGenerator(category)
    return generator.create_examples(count)
```

## Failure Analysis

### Common Failure Patterns

<<REPEAT: failure_pattern>>

#### {{Pattern Name}}

- **Frequency**: {{occurrences-per-day}}
- **Root Cause**: {{cause-analysis}}
- **Impact**: {{user-impact}}
- **Mitigation**: {{fix-strategy}}
  <</REPEAT>>

### Debugging Guide

1. **Identify Failure Category**: {{classification-method}}
2. **Collect Context**: {{what-to-gather}}
3. **Reproduce Issue**: {{reproduction-steps}}
4. **Analyze Root Cause**: {{analysis-tools}}
5. **Implement Fix**: {{fix-process}}
6. **Verify Resolution**: {{verification-method}}

## Continuous Improvement

### Evaluation Roadmap

- **Q{{quarter}} {{year}}**: {{planned-improvements}}
- **New Test Categories**: {{upcoming-tests}}
- **Tool Upgrades**: {{tool-updates}}
- **Coverage Goals**: {{target-coverage}}%

### Feedback Loop

- **User Reports**: {{feedback-channel}}
- **Automated Detection**: {{monitoring-integration}}
- **Review Cycle**: {{frequency}}
- **Action Items**: {{improvement-tracking}}

## Resources and References

- **Test Repository**: {{git-repo}}
- **Documentation**: {{docs-link}}
- **Issue Tracker**: {{issue-system}}
- **Team Contact**: {{contact-info}}