# Post-Mortem: [INCIDENT TITLE]

**Hotfix ID**: hotfix-###
**Incident Date**: [YYYY-MM-DD]
**Post-Mortem Date**: [YYYY-MM-DD] (within 48 hours of resolution)
**Participants**: [Names of people involved in incident and post-mortem]
**Incident Commander**: [Who led the response]

---

## Executive Summary
[2-3 paragraphs for non-technical stakeholders]
- What happened
- Impact (users, revenue, reputation)
- Root cause in simple terms
- What we're doing to prevent it

---

## Incident Details

### What Happened (Detailed)
[Comprehensive technical explanation of the incident]
- What functionality broke
- What error messages users saw
- What logs showed
- What monitoring detected

### Impact Analysis

#### Users
- **Total Affected**: [number]
- **User Segments**: [which types of users]
- **Geographic Distribution**: [if relevant]
- **Duration of Impact**: [how long users were affected]

#### Business
- **Revenue Loss**: [$amount or estimate]
- **Support Tickets**: [number created]
- **Cancellations**: [if any]
- **SLA Credits**: [$amount if applicable]
- **Reputation**: [media coverage, social sentiment]

#### Engineering
- **Teams Involved**: [which teams responded]
- **Engineer Hours**: [total time spent on incident]
- **Opportunity Cost**: [what work was delayed]

### Timeline (Detailed)
**All times in UTC**

| Time | Event | Who | Notes |
|------|-------|-----|-------|
| HH:MM | Incident began | System | [What triggered it] |
| HH:MM | First detection | [Alerting/User] | [How detected] |
| HH:MM | Investigation started | [Person] | [What they checked first] |
| HH:MM | Root cause identified | [Person] | [What was found] |
| HH:MM | Fix implemented | [Person] | [What change made] |
| HH:MM | Fix deployed | [Person] | [To which environment] |
| HH:MM | Verification complete | [Person] | [How verified] |
| HH:MM | Incident closed | [Person] | [Resolution confirmed] |
| HH:MM | Post-incident monitoring | Team | [Watching for recurrence] |

---

## Root Cause Analysis

### Technical Root Cause
[Deep technical explanation - code, architecture, deployment, etc.]

**Code Issue**:
[Specific code that caused the problem, with examples]

**Why It Existed**:
[How did this code/config/deployment get into production?]

**Why Tests Didn't Catch It**:
[What was the gap in test coverage?]

### Contributing Factors
[Factors that made the incident worse or more likely]

1. **Factor 1**: [e.g., Missing monitoring for this scenario]
   - **How it contributed**: [Made detection slower]
   - **Why it existed**: [Monitoring not prioritized]

2. **Factor 2**: [e.g., Tight coupling in code]
   - **How it contributed**: [Made fix more complex]
   - **Why it existed**: [Technical debt]

3. **Factor 3**: [e.g., Deployment at high-traffic time]
   - **How it contributed**: [Magnified user impact]
   - **Why it existed**: [No deployment policy]

### Five Whys Analysis
**Problem Statement**: [The incident in one sentence]

1. **Why did [problem] happen?**
   - [Answer]

2. **Why did [answer from #1] happen?**
   - [Answer]

3. **Why did [answer from #2] happen?**
   - [Answer]

4. **Why did [answer from #3] happen?**
   - [Answer]

5. **Why did [answer from #4] happen?**
   - [Root cause revealed]

---

## What Went Well

### Detection
[How we found out about the issue - what worked]

### Response
[What went smoothly during the incident response]

### Communication
[How internal/external communication worked]

### Remediation
[What made the fix process effective]

---

## What Went Wrong

### Detection Issues
[What delayed detection or made it harder to find]

### Response Issues
[What slowed down or complicated the response]

### Communication Issues
[Where communication broke down]

### Remediation Issues
[What made the fix harder than it should have been]

---

## Action Items

### Immediate Actions (This Week)
Priority actions to prevent immediate recurrence:

- [ ] **AI-001**: [Action item]
  - **Owner**: [Person responsible]
  - **Due**: [Date]
  - **Why**: [Prevents what]

- [ ] **AI-002**: [Action item]
  - **Owner**: [Person responsible]
  - **Due**: [Date]
  - **Why**: [Prevents what]

### Short-Term Actions (This Month)
Important but not urgent:

- [ ] **AI-003**: [Action item]
  - **Owner**: [Person responsible]
  - **Due**: [Date]
  - **Why**: [Improves what]

### Long-Term Actions (This Quarter)
Structural improvements:

- [ ] **AI-004**: [Action item]
  - **Owner**: [Person responsible]
  - **Due**: [Date]
  - **Why**: [Systemic improvement]

---

## Prevention Measures

### Tests Added
- [ ] **Test 1**: [Regression test for this specific bug]
  - **File**: [path to test]
  - **Covers**: [specific scenario]

- [ ] **Test 2**: [Related edge case]
  - **File**: [path to test]
  - **Covers**: [specific scenario]

### Monitoring Added
- [ ] **Alert 1**: [New alert to catch this earlier]
  - **Metric**: [what it watches]
  - **Threshold**: [when it fires]

- [ ] **Dashboard Update**: [New metrics visible]
  - **Purpose**: [what it helps identify]

### Process Changes
- [ ] **Process 1**: [e.g., deployment checklist updated]
  - **What changed**: [specific change]
  - **Why**: [prevents what]

- [ ] **Process 2**: [e.g., code review requirements]
  - **What changed**: [specific change]
  - **Why**: [catches what]

### Architecture Changes
- [ ] **Change 1**: [e.g., circuit breaker added]
  - **Where**: [system component]
  - **Why**: [limits blast radius]

### Documentation Updates
- [ ] **Doc 1**: [Runbook updated]
  - **What added**: [new troubleshooting steps]

- [ ] **Doc 2**: [Architecture doc updated]
  - **What added**: [failure mode documented]

---

## Lessons Learned

### Technical Lessons
[What we learned about our systems]
1. [Lesson 1]
2. [Lesson 2]

### Process Lessons
[What we learned about our processes]
1. [Lesson 1]
2. [Lesson 2]

### Communication Lessons
[What we learned about our communication]
1. [Lesson 1]
2. [Lesson 2]

---

## Acknowledgments
[Recognize people who helped resolve the incident]

---

## Appendix

### Related Incidents
[Link to similar past incidents if any]

### External References
[Links to relevant documentation, tickets, monitoring dashboards]

### Metrics and Logs
[Attach relevant data - error rates, response times, log samples]

---
*Post-mortem completed [date] - See hotfix-###/hotfix.md for incident log*