# Astermind Pro - Premium Features

This document lists all premium features extracted from the `astermind-kelm-elm-demo` that are **not included** in the base `@astermind/astermind-elm` package.

---

## Overview

The demo showcases **AsterMind Omega**, an advanced RAG (Retrieval-Augmented Generation) system built on top of Astermind ELM. These premium features extend the base ELM capabilities with:

- Advanced retrieval and reranking
- Sophisticated summarization
- Information flow analysis
- Production-grade numerical methods
- Advanced text processing

---

## 1. Omega RAG System

### 1.1 Omega (`Omega.ts`)
**Advanced RAG answer composition with recursive compression**

**Features:**
- Recursive sentence compression using online ridge regression
- Multi-round summarization with weighted sentence selection
- Query-aligned sentence scoring with cosine similarity
- Lexical bonus for overlapping query terms
- Personality modes (neutral, teacher, scientist)
- Deterministic, context-locked summarization

**Not in base ELM:** Base ELM has no RAG or summarization capabilities.

---

### 1.2 OmegaRR (`OmegaRR.ts`)
**Production-grade reranking system with engineered features**

**Features:**
- **Rich Feature Engineering:**
  - TF-IDF and BM25 sparse similarity
  - Heading-query match scores
  - Jaccard token overlap
  - Code block detection flags
  - Structural signals (Go code, links, etc.)
  - Random projection dense hints
  - Length heuristics
  - Prior score integration

- **Weak Supervision:**
  - Automatic label generation from heuristics
  - Per-query ridge model training
  - Relevance probability estimation

- **MMR (Maximal Marginal Relevance) Filtering:**
  - Diversity-aware selection
  - Character budget constraints
  - Coverage optimization

- **Feature Exposure:**
  - Optional feature vector export
  - Feature name mapping
  - Diagnostic utilities

**Not in base ELM:** Base ELM has no reranking, feature engineering, or MMR capabilities.

---

### 1.3 OmegaSumDet (`OmegaSumDet.ts`)
**Deterministic, intent-aware summarization**

**Features:**
- **Intent Detection:**
  - Function, variable, constant detection
  - Concurrency pattern recognition
  - Loop detection

- **Code-Aware Processing:**
  - Atomic code block handling
  - Intent-aware code gating
  - Code relevance scoring
  - Query-aligned code inclusion

- **Advanced Text Processing:**
  - Stemmed Dice coefficient for heading alignment
  - Stopword-aware tokenization
  - Jaccard deduplication
  - Section diversity capping

- **Deterministic Scoring:**
  - Normalized feature weights
  - Explicit tie-breakers
  - Stable ordering guarantees
  - Context-locked (no leakage)

- **Output Shaping:**
  - Character budget management
  - Bullet point formatting
  - Citation generation
  - Footer with sources

**Not in base ELM:** Base ELM has no summarization, intent detection, or code-aware processing.

---

## 2. Advanced Numerical Methods

### 2.1 KRR - Kernel Ridge Regression (`krr.ts`)
**Production-grade ridge regression solver**

**Features:**
- Cholesky decomposition with adaptive jitter
- Conjugate Gradient (CG) fallback for ill-conditioned systems
- Symmetry enforcement
- Matrix validation (NaN/Inf detection)
- Abort signal support for cancellation
- Comprehensive diagnostics

**Not in base ELM:** Base ELM uses simpler ridge solving without these production features.

---

### 2.2 RFF - Random Fourier Features (`rff.ts`)
**RBF kernel approximation via random projections**

**Features:**
- Deterministic RFF construction
- Box-Muller Gaussian sampling
- L2 normalization for stability
- Efficient cosine/sine feature mapping

**Not in base ELM:** Base ELM's KernelELM uses exact kernels or Nyström, but not RFF.

---

### 2.3 OnlineRidge (`online_ridge.ts`)
**Online ridge regression with rank-1 updates**

**Features:**
- Incremental updates via Sherman-Morrison formula
- Efficient inverse maintenance
- Multi-output support (stacked heads)
- No full retraining required

**Not in base ELM:** Base OnlineELM uses RLS but not this specific online ridge implementation.

---

### 2.4 Advanced Math Utilities (`math.ts`)
**Production-grade numerical operations**

**Features:**
- Robust vector operations (dot, add, scale, Hadamard)
- In-place variants for GC efficiency
- Safe exp/log/sigmoid with overflow guards
- Stable softmax via log-sum-exp trick
- Hyperbolic distance proxy
- Vector normalization and clamping
- Statistical functions (mean, variance, standardization)
- Top-K selection
- Formatting utilities

**Not in base ELM:** Base ELM has basic math but not these production-grade utilities.

---

## 3. Information Flow Analysis

### 3.1 Transfer Entropy (`infoflow/TransferEntropy.ts`)
**Information-theoretic causal analysis**

**Features:**
- Streaming Transfer Entropy (TE) estimation
- Linear-Gaussian approximation
- Configurable lag windows
- Ridge-regularized regression
- Bits/nats reporting
- Multi-variable monitoring
- InfoFlow graph construction

**Use Cases:**
- Monitor information flow between RAG components
- Query → Score influence tracking
- Feature → Relevance analysis
- Kept chunks → Summary grounding verification

**Not in base ELM:** Base ELM has no information flow or causal analysis capabilities.

---

### 3.2 Transfer Entropy PWS (`infoflow/TransferEntropyPWS.ts`)
**Phase-Weighted Stacking variant with importance sampling**

**Features:**
- **Importance Sampling:**
  - Rare event detection via tail quantile thresholds
  - Tail boost weighting for important samples
  - Time decay for recency weighting

- **Path-Weight Sampling (PWS):**
  - Jittered context histories
  - Multiple perturbed path averaging
  - Kernel Density Estimation (KDE) for conditional entropy
  - Silverman's rule-of-thumb bandwidth selection

- **KDE-Based Estimation:**
  - Product Gaussian kernels
  - Joint and marginal density estimation
  - Ridge floor to avoid log(0) issues
  - Importance-weighted averaging

**Not in base ELM:** Advanced variant with importance sampling and PWS not available in base.

---

### 3.3 TE Controller (`infoflow/TEController.ts`)
**Closed-loop adaptive control via Transfer Entropy**

**Features:**
- **Target Band Management:**
  - Configurable TE target bands per channel
  - Query→Score, Feature→Score, Kept→Summary monitoring
  - Optional loop stability guards

- **Adaptive Parameter Tuning:**
  - EMA (Exponential Moving Average) smoothing
  - Single-knob adjustments per step
  - Cooldown periods between adjustments
  - Hard caps on session adjustments

- **Knob Control:**
  - Alpha (sparse/dense mix)
  - Sigma (kernel bandwidth)
  - Ridge (regularization)
  - ProbThresh (reranker threshold)
  - MMR Lambda (diversity tradeoff)
  - BudgetChars (answer length)

- **Safety Features:**
  - Minimum sample requirements
  - Parameter limits enforcement
  - Step size constraints
  - Adjustment history tracking

**Use Cases:**
- Automatic hyperparameter tuning based on information flow
- Maintaining optimal TE ranges for system health
- Preventing information leakage or overfitting
- Adaptive system optimization

**Not in base ELM:** No closed-loop control or TE-based adaptation in base.

---

## 4. Advanced Retrieval

**NEW:** All retrieval functionality is now available as **standalone, reusable modules** outside of workers! Use `buildIndex()`, `hybridRetrieve()`, and related functions directly in your applications.

### 4.1 Hybrid Retrieval System
**Combining sparse and dense methods**

**Features:**
- **TF-IDF Sparse Retrieval:**
  - Heading-weighted tokenization
  - Custom vocabulary management
  - Stemming support

- **Dense Kernel Retrieval:**
  - Nyström approximation for scalability
  - Multiple kernel types (RBF, cosine, polynomial)
  - Landmark-based projection

- **Hybrid Scoring:**
  - Ridge-regularized combination
  - Alpha/beta mixing parameters
  - Keyword bonus integration
  - Tanh clipping for stability

**Not in base ELM:** Base ELM has EmbeddingStore but not this hybrid retrieval pipeline.

**Available Modules:**
- `buildIndex()` - Build vocabulary, IDF, and Nyström landmarks from documents
- `hybridRetrieve()` - Perform hybrid retrieval (sparse + dense + keyword bonus)
- `toTfidf()` - Compute TF-IDF vectors
- `cosineSparse()` - Sparse vector cosine similarity
- `projectToDense()` - Project sparse vectors to dense using Nyström landmarks
- `parseMarkdownToSections()` - Parse markdown into hierarchical sections
- `flattenSections()` - Flatten sections into chunks

---

### 4.2 Nyström Approximation
**Efficient kernel computation for large datasets**

**Features:**
- Landmark selection strategies
- Sparse-to-dense projection
  - Kernel similarity computation
  - Normalized feature vectors
  - Configurable landmark count

**Not in base ELM:** Base KernelELM has Nyström but not integrated into a full retrieval system.

---

## 5. Advanced Text Processing

### 5.1 Tree-Aware Markdown Parsing
**Hierarchical section extraction**

**Features:**
- Multi-level heading parsing (##, ###, etc.)
- Parent-child relationship tracking
- Empty parent backfilling
- Rich + plain text retention
- Code fence preservation
- Link handling

**Not in base ELM:** Base ELM has basic text encoding but not markdown parsing.

---

### 5.2 Advanced Tokenization
**Production-grade text preprocessing**

**Features:**
- **Stemming:**
  - Plural folding (ies → y, etc.)
  - Suffix removal (ization → ize)
  - Conservative rule-based stemming
  - Memoization for performance

- **Query Expansion:**
  - Domain-specific term expansion
  - Context-aware keyword addition
  - Go-specific expansions

- **Code-Aware Splitting:**
  - Fenced code block detection
  - Atomic code block handling
  - Mixed text/code processing

**Not in base ELM:** Base ELM has UniversalEncoder but not this advanced preprocessing.

---

### 5.3 ELM Scorer (`elm_scorer.ts`)
**Custom ELM-based relevance scoring**

**Features:**
- Random hidden layer initialization
- GELU activation approximation
- Online ridge output layer
- Partial fit support
- Deterministic seeding

**Not in base ELM:** This is a custom implementation, though base ELM could be used similarly.

---

## 6. Auto-Tuning System

**NEW:** Auto-tuning is now available as a **standalone function** outside of workers! Use `autoTune()` directly in your applications.

### 6.1 Hyperparameter Optimization
**Automated configuration search**

**Features:**
- Random search with refinement
- Jaccard-based evaluation metric
- Ridge-aware parameter exploration
- Caching for efficiency
- Penalty functions for complexity
- Real-time progress reporting

**Parameters Tuned:**
- Alpha (sparse/dense mix)
- Beta (keyword bonus)
- Sigma (kernel bandwidth)
- Kernel type
- Vocabulary size
- Landmark count
- Prefilter size
- Top-K selection
- Heading weights
- Chunk/overlap sizes
- Ridge regularization

**Not in base ELM:** Base ELM has no auto-tuning capabilities.

**Available Functions:**
- `autoTune()` - Automated hyperparameter optimization
- `sampleQueriesFromCorpus()` - Generate synthetic queries for tuning
- `penalty()` - Compute complexity penalty for configurations
- `jaccard()` - Calculate Jaccard similarity between index sets

---

## 7. Model Persistence

**NEW:** Model serialization is now available as **standalone functions** outside of workers! Use `exportModel()` and `importModel()` directly in your applications.

### 7.1 Serialized Model Format
**Complete model export/import**

**Features:**
- Full state serialization
- Version tracking
- Optional dense vector storage
- Checksum generation
- Settings snapshot
- Vocabulary + IDF preservation
- Nyström landmark storage
- Chunk metadata retention

**Not in base ELM:** Base ELM has JSON import/export but not this comprehensive format.

**Available Functions:**
- `exportModel()` - Export complete model state to serialized format
- `importModel()` - Import model from serialized format
- `quickHash()` - Generate deterministic hash for model verification

---

## 8. Worker Architecture

### 8.1 Web Worker Pipeline
**Background processing for RAG**

**Features:**
- Non-blocking retrieval
- Async model loading
- Progress reporting
- Error handling
- Message-based API
- State management

**Not in base ELM:** Base ELM has ELMWorker but not this full RAG pipeline.

---

## 9. Advanced ELM Variants

### 9.1 Deep ELM Pro (`elm/deep-elm-pro.ts`)
**Improved multi-layer ELM with advanced training strategies**

**Improvements over Base DeepELM:**

| Feature | Base DeepELM | DeepELMPro |
|---------|--------------|------------|
| **Pretraining** | ❌ None | ✅ Autoencoder pretraining for each layer |
| **Training Strategy** | Joint only | ✅ Layer-wise or Joint (configurable) |
| **Regularization** | ❌ None | ✅ L1/L2/Elastic Net regularization |
| **Batch Normalization** | ❌ None | ✅ Optional batch normalization between layers |
| **Dropout** | ❌ None | ✅ Optional dropout with configurable rate |
| **Training Flexibility** | Basic | ✅ Advanced with multiple strategies |

**Key Features:**
- **Autoencoder Pretraining**: Each layer can be pretrained as an autoencoder to learn better feature representations before supervised training
- **Layer-wise Training**: Train layers sequentially for more stable learning and better feature extraction
- **Regularization**: L1, L2, and Elastic Net regularization to prevent overfitting and improve generalization
- **Batch Normalization**: Normalize activations between layers for faster convergence and training stability
- **Dropout**: Randomly drop neurons during training to reduce overfitting
- **Flexible Training Modes**: Choose between layer-wise sequential training or joint training based on your data

**Not in base ELM:** Base DeepELM has basic multi-layer support but lacks these advanced training strategies, regularization, and normalization techniques.

**Use Cases:**
- Complex pattern recognition requiring deep feature hierarchies
- High-dimensional data classification
- When base DeepELM overfits or doesn't converge well
- Production systems requiring robust, generalizable models

---

### 9.2 Multi-Kernel ELM (`elm/multi-kernel-elm.ts`)
**Combines multiple kernel types for improved accuracy**

**Features:**
- Weighted combination of multiple kernels (RBF, linear)
- Automatic kernel weight learning based on validation performance
- Multiple kernel types support
- Ridge-regularized combination

**Not in base ELM:** Base KernelELM uses a single kernel type. Multi-Kernel ELM combines multiple kernels for better performance.

---

### 9.3 Online Kernel ELM (`elm/online-kernel-elm.ts`)
**Real-time learning for streaming data**

**Features:**
- Incremental kernel matrix updates
- Sliding window with forgetting mechanisms
- Adaptive landmark selection
- Real-time prediction

**Not in base ELM:** Base KernelELM requires batch training. Online Kernel ELM supports streaming updates.

---

### 9.4 Multi-Task ELM (`elm/multi-task-elm.ts`)
**Joint learning across related tasks**

**Features:**
- Shared feature extraction layer
- Task-specific output layers
- Task weighting for importance
- Joint optimization

**Not in base ELM:** Base ELM trains one task at a time. Multi-Task ELM learns multiple related tasks simultaneously.

---

### 9.5 Sparse ELM (`elm/sparse-elm.ts`)
**Efficiency and interpretability for high-dimensional data**

**Features:**
- L1/L2/Elastic net regularization
- Weight pruning for sparsity
- Feature importance ranking
- Interpretable models

**Not in base ELM:** Base ELM has no built-in sparsity mechanisms. Sparse ELM provides regularization and feature selection.

---

## Summary: Premium Feature Categories

### Core Premium Features:
1. ✅ **Omega RAG System** - Complete RAG pipeline
2. ✅ **OmegaRR Reranking** - Production reranking with MMR
3. ✅ **OmegaSumDet** - Intent-aware summarization
4. ✅ **Transfer Entropy** - Information flow analysis
5. ✅ **Hybrid Retrieval** - Sparse + dense combination (**NEW: Standalone modules!**)
6. ✅ **Auto-Tuning** - Hyperparameter optimization (**NEW: Standalone function!**)
7. ✅ **Advanced Math** - Production-grade numerics
8. ✅ **Tree-Aware Parsing** - Hierarchical markdown processing (**NEW: Standalone functions!**)
9. ✅ **Model Serialization** - Export/import models (**NEW: Standalone functions!**)
10. ✅ **Tokenization & Utilities** - Reusable text processing (**NEW: Standalone functions!**)
11. ✅ **Advanced ELM Variants** - 5 premium variants (**NEW!**)

### Advanced Algorithms:
- KRR (Kernel Ridge Regression with CG fallback)
- RFF (Random Fourier Features)
- OnlineRidge (Rank-1 updates)
- Nyström approximation
- MMR filtering
- Weak supervision

### Production Features:
- Robust error handling
- Matrix validation
- Abort signals
- Comprehensive diagnostics
- Model versioning
- Checksums

---

## Integration Notes

These premium features are designed to work **on top of** `@astermind/astermind-elm`:

- Uses base ELM's `Tokenizer`, `TFIDFVectorizer`, `ELM`, `OnlineELM`
- Extends with premium retrieval, reranking, and summarization
- Adds information flow monitoring
- Provides production-grade numerical methods

---

## License & Usage

These premium features are part of **Astermind Pro** and are not included in the MIT-licensed Community Edition (`@astermind/astermind-elm`).

---

*Last updated: 2025-01-16*