# Analyze Validation Override Patterns ## Purpose Analyze patterns in validation override decisions to improve the validation rules and provide data-driven insights for process improvement. ## Context Teams override validation warnings for various reasons. By analyzing these patterns, we can distinguish between rules that are genuinely helpful versus those that create unnecessary friction. ## Task Instructions ### 1. Load Override Logs ```yaml [[LLM: Load validation override logs from the configured location]] Log location: .bmad/parallel-dev/validation-overrides.log Time period: Last 30 days (configurable) Each log entry contains: - story_id - timestamp - user_id - team_id - rule_id overridden - justification provided - story_type - outcome (if tracked) ``` ### 2. Aggregate Override Data #### By Rule ```yaml [[LLM: Count overrides by rule_id]] Override frequency by rule: NO_DEV_NOTES: 45 overrides (32%) NO_ACCEPTANCE_CRITERIA: 38 overrides (27%) UNMAPPED_DEPENDENCIES: 25 overrides (18%) MISSING_TEST_SPECS: 20 overrides (14%) HIGH_COMPLEXITY_NO_REVIEW: 12 overrides (9%) Insights: - NO_DEV_NOTES most frequently overridden - May indicate rule is too strict or not applicable to all contexts ``` #### By Story Type ```yaml [[LLM: Analyze overrides by story type]] Override patterns by story type: spike: - NO_DEV_NOTES: 85% override rate - NO_ACCEPTANCE_CRITERIA: 90% override rate - Insight: Spike validation rules need adjustment feature: - NO_DEV_NOTES: 15% override rate - UNMAPPED_DEPENDENCIES: 25% override rate - Insight: Feature rules appropriately calibrated bug: - MISSING_TEST_SPECS: 5% override rate - Insight: Bug validation working well ``` #### By Team ```yaml [[LLM: Identify team-specific patterns]] Team override rates: Team Alpha: 45% average override rate - Frequent justification: "Discussed in planning" - Suggestion: Team may benefit from different process Team Beta: 12% average override rate - Mostly follows standard process - Few overrides are well-justified Team Gamma: 78% average override rate - Pattern: Overrides all spike validations - Recommendation: Custom rules for research team ``` ### 3. Analyze Justifications ```yaml [[LLM: Extract patterns from justification text]] Common justification themes: 1. "Discussed verbally in planning" (25%) - Indicates good communication, documentation gap 2. "Will add details during implementation" (20%) - Suggests just-in-time documentation preference 3. "Not applicable to spike/research" (35%) - Clear need for story-type-specific rules 4. "Legacy system - details in wiki" (10%) - External documentation exists 5. "Mob programming session planned" (10%) - Alternative collaboration method ``` ### 4. Correlate with Outcomes ```yaml [[LLM: If outcome tracking available, correlate overrides with results]] Override outcome analysis: Stories with overridden warnings: - 70% completed successfully - 20% had minor conflicts (resolved quickly) - 10% had major issues Breakdown by rule: NO_DEV_NOTES override outcomes: - 60% successful (lower than average) - Conflicts usually in shared code NO_ACCEPTANCE_CRITERIA override outcomes: - 85% successful (higher than average) - Issues mainly in QA phase, not development ``` ### 5. Generate Recommendations ```yaml [[LLM: Create actionable recommendations based on analysis]] ## Validation Rule Recommendations ### Immediate Actions 1. **Downgrade spike rules** - Change NO_DEV_NOTES from "warning" to "suggestion" for spikes - Change NO_ACCEPTANCE_CRITERIA to "suggestion" for spikes - Justification: 87% override rate indicates misalignment 2. **Add team-specific configs** - Create Team Gamma research profile with relaxed rules - Document Team Alpha's verbal planning process 3. **Refine warning messages** - Add "unless discussed in planning" to relevant warnings - Include mob programming as valid alternative ### Process Improvements 1. **Just-in-time documentation** - Allow "dev notes during implementation" workflow - Add checkpoint at Wave 1 completion 2. **External documentation references** - Allow links to wiki/confluence as alternative - Validate links are accessible 3. **Success pattern sharing** - Teams with low override + high success share practices - Create "validation best practices" guide ### Rule Evolution Based on 30-day data, propose these rule changes: rules_to_downgrade: - rule_id: NO_ACCEPTANCE_CRITERIA for_story_types: [spike, research] from: warning to: suggestion - rule_id: NO_DEV_NOTES for_teams: [team_gamma] from: warning to: suggestion rules_to_add: - rule_id: EXTERNAL_DOCS_REFERENCE description: "Story should reference design docs" severity: suggestion applies_to: [feature] ``` ### 6. Create Insight Report ```yaml [[LLM: Generate executive summary for SM agent and team leads]] ## Validation Override Insights Report **Period**: {{start_date}} to {{end_date}} **Total Overrides**: {{total_count}} ### Key Findings 1. Spike stories have fundamentally different validation needs 2. Verbal planning is common, creating documentation gaps 3. Most overrides (70%) don't lead to problems 4. Team-specific working styles require flexible rules ### Recommended Changes 1. Implement story-type-specific validation profiles 2. Allow team customization of warning levels 3. Add "planning discussion" as valid justification category 4. Track outcomes automatically for better correlation ### Success Stories - Team Beta's low override rate (12%) with high success - Mob programming as alternative to detailed dev notes - Just-in-time documentation working for Team Alpha ### Next Steps 1. Review recommendations with team leads 2. Update validation-rules.yml with approved changes 3. Re-analyze in 30 days to measure improvement ``` ## Success Criteria - Meaningful patterns identified from override data - Actionable recommendations generated - Validation rules evolve based on real usage - Teams feel heard and supported - Override rates decrease as rules improve ## Notes - Run monthly or after significant sprints - Share insights with all teams for transparency - Use data to drive process improvements, not blame - Consider A/B testing rule changes with volunteer teams