# Approval Workflow Engine - Documentation

**Version:** 1.0.0
**Status:** Production Ready
**Last Updated:** 2025-11-15

---

## Table of Contents

1. [Overview](#overview)
2. [State Machine Architecture](#state-machine-architecture)
3. [Components](#components)
4. [Database Schema](#database-schema)
5. [Usage Guide](#usage-guide)
6. [SLA Management](#sla-management)
7. [Audit Trail](#audit-trail)
8. [Integration Guide](#integration-guide)
9. [Troubleshooting](#troubleshooting)

---

## Overview

The Approval Workflow Engine manages the lifecycle of auto-generated skills from detection through deployment, ensuring human expert oversight through a robust state machine with audit trails and SLA tracking.

### Key Features

- **State Machine:** Enforces valid state transitions with transaction safety
- **Expert Review:** CLI for approve/reject/correct actions
- **SLA Tracking:** 48h for high priority, 7 days for medium/low
- **Audit Logging:** Complete history of all state changes and expert actions
- **Notifications:** Email and Slack templates for expert alerts
- **Rollback Support:** Safe rollback to previous states
- **Concurrent Safety:** PostgreSQL row-level locking prevents race conditions

### Business Value

- **60-80% cost reduction** for codified workflows
- **95% faster execution** (scripts vs AI agents)
- **Expert oversight** maintains quality standards
- **Complete audit trail** for compliance and debugging

---

## State Machine Architecture

### States

```
DETECTED → GENERATING → PENDING_REVIEW → {APPROVED, REJECTED, NEEDS_CORRECTION} → DEPLOYED
```

#### State Descriptions

| State | Description | Next States |
|-------|-------------|-------------|
| **DETECTED** | Workflow pattern detected by analyzer | GENERATING |
| **GENERATING** | AI agent generating skill code | PENDING_REVIEW, DETECTED (rollback) |
| **PENDING_REVIEW** | Awaiting expert review | APPROVED, REJECTED, NEEDS_CORRECTION |
| **NEEDS_CORRECTION** | Expert requested changes | GENERATING, REJECTED |
| **APPROVED** | Expert approved, ready for deployment | DEPLOYED, REJECTED (rollback) |
| **REJECTED** | Expert or system rejected | (terminal state) |
| **DEPLOYED** | Deployed to production | APPROVED (rollback) |

### Valid Transitions

```bash
DETECTED → GENERATING              # Pattern detection complete
GENERATING → PENDING_REVIEW        # Skill generation complete
GENERATING → DETECTED              # Generation failed, retry
PENDING_REVIEW → APPROVED          # Expert approved
PENDING_REVIEW → REJECTED          # Expert rejected
PENDING_REVIEW → NEEDS_CORRECTION  # Expert wants changes
NEEDS_CORRECTION → GENERATING      # Re-generate with feedback
NEEDS_CORRECTION → REJECTED        # Give up after multiple corrections
APPROVED → DEPLOYED                # Deploy to production
APPROVED → REJECTED                # Rollback approval (rare)
DEPLOYED → APPROVED                # Rollback deployment
```

### Concurrency Control

The state machine uses PostgreSQL row-level locking to prevent race conditions:

```sql
-- Lock pattern row for update
SELECT status FROM workflow_patterns WHERE id = 'pattern-id' FOR UPDATE;

-- Verify current state matches expected state
-- Update only if state matches
-- Commit transaction
```

This ensures that concurrent state transitions are serialized and conflicting updates fail gracefully.

---

## Components

### 1. approval-workflow.sh

**Purpose:** Core state machine implementation

**Commands:**
- `transition` - Perform state transition with validation
- `get-state` - Query current state
- `rollback` - Safe rollback to previous state
- `history` - View state transition history
- `init` - Initialize database schema

**Features:**
- Transaction safety (BEGIN/COMMIT/ROLLBACK)
- Row-level locking for concurrency
- Validation of state transitions
- Audit logging for all changes
- Metadata support for context

**Example:**
```bash
# Transition skill from PENDING_REVIEW to APPROVED
./approval-workflow.sh transition \
  --pattern-id "123e4567-e89b-12d3-a456-426614174000" \
  --from-state PENDING_REVIEW \
  --to-state APPROVED \
  --metadata '{"approved_by": "expert@example.com"}'

# Rollback deployment due to bug
./approval-workflow.sh rollback \
  --pattern-id "123e4567-e89b-12d3-a456-426614174000" \
  --to-state APPROVED \
  --reason "Critical bug found in production"
```

---

### 2. review-skill.sh

**Purpose:** Expert review CLI for skill approval workflow

**Commands:**
- `--action approve` - Approve skill for deployment
- `--action reject` - Reject skill with reason
- `--action correct` - Request corrections with feedback
- `--list-pending` - List all pending reviews
- `--check-sla` - Check SLA compliance status
- `--init` - Initialize skill_approvals table

**Features:**
- Three-way review actions (approve/reject/correct)
- Mandatory feedback for reject/correct
- SLA status tracking
- Team-based filtering
- Automatic audit logging

**Example:**
```bash
# Approve a skill
./review-skill.sh \
  --skill-id "123e4567-e89b-12d3-a456-426614174000" \
  --action approve \
  --feedback "Code looks good, tests pass"

# Reject with reason
./review-skill.sh \
  --skill-id "123e4567-e89b-12d3-a456-426614174000" \
  --action reject \
  --feedback "Security vulnerability: SQL injection in line 45"

# Request corrections
./review-skill.sh \
  --skill-id "123e4567-e89b-12d3-a456-426614174000" \
  --action correct \
  --feedback "Add input validation for email parameter. Use regex pattern."

# List pending reviews for frontend team
./review-skill.sh --list-pending --team frontend

# Check SLA status
./review-skill.sh --check-sla
```

---

### 3. Notification Templates

#### Email Template (templates/email-notification.txt)

**Purpose:** Notify experts of new skills awaiting review

**Placeholders:**
- `{{SKILL_NAME}}` - Skill pattern name
- `{{PATTERN_ID}}` - UUID of the pattern
- `{{PRIORITY}}` - high/medium/low
- `{{PRIMARY_TEAM}}` - Team responsible for review
- `{{ESTIMATED_SAVINGS}}` - Monthly cost savings estimate
- `{{OCCURRENCE_COUNT}}` - Number of workflow occurrences
- `{{TEAMS_AFFECTED}}` - Comma-separated team list
- `{{CONFIDENCE_SCORE}}` - Pattern detection confidence
- `{{SIMILARITY_SCORE}}` - Workflow similarity score
- `{{WORKFLOW_STEPS}}` - Formatted workflow steps
- `{{SKILL_ID}}` - UUID for review commands
- `{{TEST_COUNT}}` - Number of test cases
- `{{EDGE_CASE_COUNT}}` - Number of edge cases
- `{{SLA_DEADLINE}}` - Review deadline
- `{{SLA_HOURS}}` - Hours until SLA breach

#### Slack Template (templates/slack-notification.md)

**Purpose:** Send Slack notifications for skill reviews

**Format:** Markdown with emoji support

**Additional Placeholders:**
- `{{PRIORITY_EMOJI}}` - Visual priority indicator (🔴/🟡/🟢)
- `{{EXPERT_SLACK_HANDLE}}` - Slack @mention for expert
- `{{WORKFLOW_STEP_N}}` - Individual workflow steps
- `{{TEAMS_AFFECTED_COUNT}}` - Number of teams affected

---

## Database Schema

### workflow_patterns

**Purpose:** Core table for workflow patterns and skill metadata

```sql
CREATE TABLE workflow_patterns (
    id UUID PRIMARY KEY,
    pattern_name VARCHAR(255) NOT NULL,
    workflow_steps JSONB NOT NULL,
    occurrence_count INTEGER NOT NULL,
    teams_affected TEXT[] NOT NULL,
    similarity_score DECIMAL(3,2) NOT NULL,
    deterministic BOOLEAN DEFAULT FALSE,
    confidence_score DECIMAL(3,2) NOT NULL,
    estimated_savings_usd DECIMAL(10,2),
    priority VARCHAR(20),
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW(),
    status VARCHAR(50) DEFAULT 'DETECTED'
);

CREATE INDEX idx_pattern_status ON workflow_patterns(status);
```

### pattern_state_history

**Purpose:** Track all state transitions for audit trail

```sql
CREATE TABLE pattern_state_history (
    id SERIAL PRIMARY KEY,
    pattern_id UUID REFERENCES workflow_patterns(id) ON DELETE CASCADE,
    from_state VARCHAR(50) NOT NULL,
    to_state VARCHAR(50) NOT NULL,
    metadata JSONB DEFAULT '{}',
    timestamp TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_pattern_state_history_pattern_id ON pattern_state_history(pattern_id);
```

### skill_approvals

**Purpose:** Log all expert review actions

```sql
CREATE TABLE skill_approvals (
    id SERIAL PRIMARY KEY,
    skill_id UUID REFERENCES workflow_patterns(id) ON DELETE CASCADE,
    expert_id VARCHAR(255) NOT NULL,
    action VARCHAR(50) NOT NULL,
    feedback TEXT,
    timestamp TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_skill_approvals_skill_id ON skill_approvals(skill_id);
CREATE INDEX idx_skill_approvals_expert_id ON skill_approvals(expert_id);
CREATE INDEX idx_skill_approvals_timestamp ON skill_approvals(timestamp);
```

### workflow_audit_log

**Purpose:** General audit log for all workflow events

```sql
CREATE TABLE workflow_audit_log (
    id SERIAL PRIMARY KEY,
    pattern_id UUID,
    event_type VARCHAR(100) NOT NULL,
    description TEXT,
    metadata JSONB DEFAULT '{}',
    timestamp TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_audit_log_pattern_id ON workflow_audit_log(pattern_id);
```

---

## Usage Guide

### Setup

#### 1. Configure Database Connection

Create `.env` file in project root:

```bash
# PostgreSQL Configuration
CFN_DB_HOST=localhost
CFN_DB_PORT=5432
CFN_DB_NAME=cfn_workflow
CFN_DB_USER=postgres
CFN_DB_PASSWORD=your_secure_password

# Expert Configuration
CFN_EXPERT_ID=$(whoami)
CFN_EXPERT_EMAIL=expert@example.com
```

#### 2. Initialize Database Schema

```bash
# Initialize workflow_patterns and related tables
./approval-workflow.sh init

# Initialize skill_approvals table
./review-skill.sh --init
```

#### 3. Verify Setup

```bash
# Check database connection
psql -h localhost -U postgres -d cfn_workflow -c "SELECT COUNT(*) FROM workflow_patterns;"

# Verify schema
psql -h localhost -U postgres -d cfn_workflow -c "\dt"
```

---

### Workflow Example

#### Scenario: Auto-generated skill needs review

**Step 1: Pattern Detected**
```bash
# System detects pattern and creates record
# Status: DETECTED
```

**Step 2: Skill Generation**
```bash
# Transition to GENERATING
./approval-workflow.sh transition \
  --pattern-id "abc123..." \
  --from-state DETECTED \
  --to-state GENERATING

# AI agent generates skill...

# Transition to PENDING_REVIEW
./approval-workflow.sh transition \
  --pattern-id "abc123..." \
  --from-state GENERATING \
  --to-state PENDING_REVIEW
```

**Step 3: Expert Notification**
```bash
# System sends email using templates/email-notification.txt
# System sends Slack message using templates/slack-notification.md
```

**Step 4: Expert Review**
```bash
# Expert lists pending reviews
./review-skill.sh --list-pending

# Expert reviews code, runs tests
cd .claude/skills/staging/codified-abc123/
./test.sh

# Expert approves
./review-skill.sh \
  --skill-id "abc123..." \
  --action approve \
  --feedback "All tests pass, code looks secure"
```

**Step 5: Deployment**
```bash
# Transition to DEPLOYED
./approval-workflow.sh transition \
  --pattern-id "abc123..." \
  --from-state APPROVED \
  --to-state DEPLOYED
```

---

### Rollback Example

#### Scenario: Bug found in production skill

```bash
# Step 1: Rollback deployment
./approval-workflow.sh rollback \
  --pattern-id "abc123..." \
  --to-state APPROVED \
  --reason "Critical bug: fails on empty input"

# Step 2: Request correction
./review-skill.sh \
  --skill-id "abc123..." \
  --action correct \
  --feedback "Add null check before processing input parameter"

# Step 3: Re-generate skill
./approval-workflow.sh transition \
  --pattern-id "abc123..." \
  --from-state NEEDS_CORRECTION \
  --to-state GENERATING

# (AI regenerates with feedback)

# Step 4: Re-review and re-deploy
./approval-workflow.sh transition \
  --pattern-id "abc123..." \
  --from-state GENERATING \
  --to-state PENDING_REVIEW

./review-skill.sh \
  --skill-id "abc123..." \
  --action approve

./approval-workflow.sh transition \
  --pattern-id "abc123..." \
  --from-state APPROVED \
  --to-state DEPLOYED
```

---

## SLA Management

### SLA Thresholds

| Priority | SLA | Description |
|----------|-----|-------------|
| High | 48 hours | Critical workflows, high savings potential |
| Medium | 7 days | Standard workflows, moderate savings |
| Low | 7 days | Nice-to-have workflows, low savings |

### SLA Tracking

```bash
# Check current SLA status
./review-skill.sh --check-sla

# Example output:
# High Priority Breaches (>48h): 2
# Medium/Low Priority Breaches (>7d): 1
# Total Pending Reviews: 15
```

### SLA Breach Handling

1. **Automated Alerts:** System sends reminder emails at 75% of SLA
2. **Escalation:** Product Owner notified at 100% SLA breach
3. **Metrics:** SLA compliance tracked in dashboard

### Monitoring Query

```sql
SELECT
    wp.id,
    wp.pattern_name,
    wp.priority,
    ROUND(EXTRACT(EPOCH FROM (NOW() - wp.created_at))/3600, 1) as hours_pending,
    CASE
        WHEN wp.priority = 'high' AND EXTRACT(EPOCH FROM (NOW() - wp.created_at))/3600 > 48 THEN 'BREACH'
        WHEN wp.priority IN ('medium', 'low') AND EXTRACT(EPOCH FROM (NOW() - wp.created_at))/3600 > 168 THEN 'BREACH'
        ELSE 'OK'
    END as sla_status
FROM workflow_patterns wp
WHERE wp.status IN ('PENDING_REVIEW', 'NEEDS_CORRECTION')
ORDER BY hours_pending DESC;
```

---

## Audit Trail

### Complete History Tracking

Every action in the approval workflow is logged:

1. **State Transitions:** `pattern_state_history` table
2. **Expert Actions:** `skill_approvals` table
3. **System Events:** `workflow_audit_log` table

### Querying Audit Trail

#### Get full history for a skill
```sql
SELECT
    psh.from_state,
    psh.to_state,
    psh.timestamp,
    psh.metadata
FROM pattern_state_history psh
WHERE psh.pattern_id = 'abc123...'
ORDER BY psh.timestamp ASC;
```

#### Get expert actions for a skill
```sql
SELECT
    sa.expert_id,
    sa.action,
    sa.feedback,
    sa.timestamp
FROM skill_approvals sa
WHERE sa.skill_id = 'abc123...'
ORDER BY sa.timestamp ASC;
```

#### Get all audit events for a skill
```sql
SELECT
    wal.event_type,
    wal.description,
    wal.metadata,
    wal.timestamp
FROM workflow_audit_log wal
WHERE wal.pattern_id = 'abc123...'
ORDER BY wal.timestamp ASC;
```

### Compliance Reports

Generate compliance reports for audits:

```bash
# Export audit trail for date range
psql -h localhost -U postgres -d cfn_workflow -c "
SELECT
    wp.pattern_name,
    sa.expert_id,
    sa.action,
    sa.feedback,
    sa.timestamp
FROM skill_approvals sa
JOIN workflow_patterns wp ON wp.id = sa.skill_id
WHERE sa.timestamp BETWEEN '2025-01-01' AND '2025-12-31'
ORDER BY sa.timestamp DESC;
" -o audit_report_2025.csv -A -F ','
```

---

## Integration Guide

### Email Integration

**Setup:**
1. Configure SMTP server in `.env`
2. Update `send_approval_notification()` in `review-skill.sh`
3. Replace placeholders in `templates/email-notification.txt`

**Example Integration (using sendmail):**
```bash
send_email() {
    local to="$1"
    local subject="$2"
    local body="$3"

    echo -e "Subject: ${subject}\n\n${body}" | sendmail "$to"
}
```

### Slack Integration

**Setup:**
1. Create Slack webhook URL
2. Update `send_approval_notification()` in `review-skill.sh`
3. Format message using `templates/slack-notification.md`

**Example Integration (using curl):**
```bash
send_slack() {
    local webhook_url="$SLACK_WEBHOOK_URL"
    local message="$1"

    curl -X POST "$webhook_url" \
        -H 'Content-Type: application/json' \
        -d "{\"text\": \"${message}\"}"
}
```

### CI/CD Integration

**GitHub Actions Example:**
```yaml
name: Skill Approval Workflow

on:
  schedule:
    - cron: '0 */6 * * *'  # Check every 6 hours

jobs:
  check-sla:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Check SLA Status
        run: |
          ./.claude/skills/workflow-codification/review-skill.sh --check-sla
          if [ $? -ne 0 ]; then
            echo "SLA breaches detected!"
            exit 1
          fi
```

---

## Troubleshooting

### Common Issues

#### 1. Database Connection Failed

**Symptom:**
```
ERROR: Database connection failed
```

**Solution:**
```bash
# Verify .env configuration
cat .env | grep CFN_DB_

# Test connection
psql -h $CFN_DB_HOST -p $CFN_DB_PORT -U $CFN_DB_USER -d $CFN_DB_NAME -c "SELECT 1;"

# Check PostgreSQL is running
systemctl status postgresql
```

#### 2. Invalid State Transition

**Symptom:**
```
ERROR: Invalid state transition: PENDING_REVIEW → DEPLOYED. Allowed transitions: APPROVED,REJECTED,NEEDS_CORRECTION
```

**Solution:**
- Cannot skip states in the workflow
- Must transition through intermediate states
- Use `get-state` to verify current state
- Example: PENDING_REVIEW → APPROVED → DEPLOYED

#### 3. SLA Breach Alert

**Symptom:**
```
⚠ SLA breaches detected! Please review pending skills urgently.
High Priority Breaches (>48h): 3
```

**Solution:**
```bash
# List pending reviews by priority
./review-skill.sh --list-pending

# Prioritize high-priority reviews
# Escalate to Product Owner if necessary
```

#### 4. Concurrent Modification Error

**Symptom:**
```
ERROR: State mismatch: expected PENDING_REVIEW, got APPROVED
```

**Solution:**
- Another expert already reviewed the skill
- Check state history: `./approval-workflow.sh history --pattern-id "abc123..."`
- Verify current state before retrying

---

## Best Practices

### For Experts

1. **Review Code Thoroughly:**
   - Check for security vulnerabilities
   - Verify input validation
   - Test edge cases
   - Run shellcheck validation

2. **Provide Clear Feedback:**
   - Be specific about what needs correction
   - Reference line numbers when applicable
   - Suggest solutions, not just problems

3. **Meet SLA Deadlines:**
   - Check pending reviews daily
   - Prioritize high-priority skills
   - Escalate blockers early

4. **Use Meaningful Approval Comments:**
   - Document why you approved/rejected
   - Note any concerns for future reference
   - Suggest improvements for next iteration

### For System Administrators

1. **Monitor SLA Compliance:**
   - Run `--check-sla` daily
   - Set up automated alerts
   - Track trends over time

2. **Review Audit Logs:**
   - Weekly audit of expert actions
   - Identify patterns in rejections
   - Improve skill generation based on feedback

3. **Backup Database:**
   - Daily backups of workflow database
   - Test restore procedures
   - Retain audit trail for compliance

4. **Optimize Performance:**
   - Monitor database query performance
   - Add indexes as needed
   - Archive old patterns (>90 days)

---

## Metrics and KPIs

### Success Metrics

- **Approval Rate:** % of skills approved on first review
- **SLA Compliance:** % of reviews completed within SLA
- **Cost Savings:** Monthly savings from deployed skills
- **Cycle Time:** Average time from DETECTED to DEPLOYED
- **Correction Rate:** % of skills requiring corrections

### Monitoring Queries

```sql
-- Approval rate (last 30 days)
SELECT
    COUNT(*) FILTER (WHERE action = 'approve') * 100.0 / COUNT(*) as approval_rate
FROM skill_approvals
WHERE timestamp > NOW() - INTERVAL '30 days';

-- SLA compliance (last 30 days)
SELECT
    COUNT(*) FILTER (WHERE
        (priority = 'high' AND EXTRACT(EPOCH FROM (completed_at - created_at))/3600 <= 48) OR
        (priority IN ('medium', 'low') AND EXTRACT(EPOCH FROM (completed_at - created_at))/3600 <= 168)
    ) * 100.0 / COUNT(*) as sla_compliance
FROM workflow_patterns
WHERE created_at > NOW() - INTERVAL '30 days' AND status IN ('APPROVED', 'DEPLOYED');

-- Total cost savings
SELECT
    SUM(estimated_savings_usd) as total_monthly_savings
FROM workflow_patterns
WHERE status = 'DEPLOYED';
```

---

## Version History

- **1.0.0** (2025-11-15): Initial release
  - State machine implementation
  - Expert review CLI
  - Email and Slack templates
  - Complete audit trail
  - SLA tracking

---

## Support

For issues or questions:
- **Documentation:** This file
- **Bug Reports:** Create issue in project repository
- **Feature Requests:** Submit via project backlog
- **Emergency:** Contact CFN System Administrator

---

**End of Documentation**