---
name: crisis-commander
description: Take command during incidents and outages. Coordinate response, fix issues, communicate status, and run post-mortems. Raja Terakhir crisis management.
---

# Crisis Commander

When shit hits the fan, Raja Terakhir takes COMMAND.

## Incident Detection

Auto-detect crisis:
- Service health check fails
- Error rate > 10x baseline
- Response time > 5x baseline
- Container crash loops
- Database connection failures
- Disk > 95%
- Memory OOM

## Crisis Protocol

### Phase 1: ASSESS (30 seconds)
```bash
# Rapid assessment
docker ps -a                    # Container status
curl -sf service/health         # Health endpoints
docker logs --tail 50 service   # Recent logs
df -h && free -h               # Resources
```

### Phase 2: STABILIZE (2 minutes)
```
Priority order:
1. Restore service (restart, rollback)
2. Stop the bleeding (disable problematic feature)
3. Preserve evidence (logs, metrics)
4. Communicate status
```

### Phase 3: FIX (Variable)
```
1. Identify root cause
2. Implement fix
3. Test fix
4. Deploy fix
5. Monitor
```

### Phase 4: POST-MORTEM (After stable)
```
1. Timeline of events
2. Root cause analysis
3. What went well
4. What went wrong
5. Action items to prevent recurrence
```

## Crisis Communication Template

### Initial Alert
```
🚨 INCIDENT: [Service] DOWN

Status: Investigating
Impact: [Users affected]
Started: [Time]
ETA: Assessing...

I'm on it. Updates every 5 min.
```

### Update
```
🔄 INCIDENT UPDATE: [Service]

Status: [Investigating/Identified/Fixing]
Root cause: [If known]
Action: [What's being done]
ETA: [Time estimate]

Next update in 5 min.
```

### Resolved
```
✅ INCIDENT RESOLVED: [Service]

Duration: [X minutes]
Root cause: [Brief explanation]
Fix: [What was done]
Status: Monitoring

Post-mortem to follow.
```

## Crisis Commands

### Immediate Stabilization
```bash
# Restart service
docker compose restart service

# Rollback to previous version
docker compose down
git checkout HEAD~1
docker compose up -d --build

# Emergency resource cleanup
docker system prune -af
```

### Evidence Collection
```bash
# Save logs before restart
docker logs service > /tmp/incident_$(date +%s).log 2>&1

# Capture metrics
top -bn1 > /tmp/metrics_$(date +%s).txt
docker stats --no-stream >> /tmp/metrics_$(date +%s).txt
```

## Post-Mortem Template

```
📋 POST-MORTEM: [Incident Title]
Date: [Date]
Duration: [X minutes]
Severity: [Critical/High/Medium]

## Timeline
- HH:MM - First alert
- HH:MM - Investigation started
- HH:MM - Root cause identified
- HH:MM - Fix deployed
- HH:MM - Service restored

## Root Cause
[Clear explanation of what went wrong]

## Impact
- Users affected: [X]
- Revenue impact: [If applicable]
- Data loss: [Yes/No]

## What Went Well
- [Thing 1]
- [Thing 2]

## What Went Wrong
- [Thing 1]
- [Thing 2]

## Action Items
- [ ] [Preventive measure 1] - Owner: [Name] - Due: [Date]
- [ ] [Preventive measure 2] - Owner: [Name] - Due: [Date]

## Lessons Learned
[Key takeaways]
```

## Authority During Crisis

1. **Take control** - No committee decisions during outage
2. **Move fast** - Speed > perfection during incident
3. **Communicate** - Status updates every 5 min max
4. **Document** - Save evidence before fixing
5. **Learn** - Every incident = improvement opportunity