# 🚀 Star-Vault **Deterministic data engine that eliminates query-time joins and enables normalized data execution.** Architect secure, scalable, real-time systems with integrated sharding, encryption, and event-driven data flows. Manage hierarchical structures, execute advanced queries, and power enterprise systems, real-time dashboards, collaborative platforms, IoT infrastructures, and complex data-driven applications. Traditional systems force a fundamental tradeoff: - relational databases preserve structure but depend on runtime joins - NoSQL systems remove joins but sacrifice integrity through denormalization **Star-Vault introduces Hyper-Normalization™ — a model that resolves relationships deterministically within the storage and cache layers instead of reconstructing them during queries.** This model delivers: - near-constant-time access across normalized data structures - zero join planning or query optimization overhead - deterministic traversal across collections - real-time consistency without duplication Star-Vault integrates sharding, WAL, caching, and real-time events into a unified execution engine for high-performance, structure-driven systems. > ⚡ In Star-Vault, normalization stops costing performance and starts driving it. ## 📘 Table of Contents - [🌌 Features](#-features) - [⚙️ System Requirements](#️-system-requirements) ### API Overview - [📜 API Specifications](#-api-specifications) - [🔧 Constructor](#-constructor) - [🔐 authOptions (StarAuth Configuration)](#-authoptions-starauth-configuration) - [🔒 Lockout Configuration](#-lockout-configuration) - [Timing Heuristics](#-timing-heuristics) - [Password Management](#-password-management) - [🌍 Star Locator Configuration](#-star-locator-configuration) - [Stellar Login](#stellar-login) - [Hooks & Extensions](#hooks--extensions) - [Guest Session Configurations](#guest-session-configurations) - [Cleanup & Locking](#cleanup--locking) ### Core Engine - [📦 Core Methods](#-core-methods) - [📥 Import & Snapshot Methods](#-import--snapshot-methods) - [🧾 StarTransactions (WAL + Idempotency)](#-startransactions-wal--idempotency) ### Query System - [🔍 Query Engine](#-query-engine) - [⚙️ Return Modes](#️-return-modes) - [Execution Modes](#execution-modes) - [Methods](#methods) - [Examples](#examples) - [⚙️ Advanced Controls](#️-advanced-controls) - [📄 Cursor-Based Pagination](#️-cursor-based-pagination) - [Performance & Behavior](#performance--behavior) - [Developer Note](#developer-note) ### Hyper Normalization - [⚡️ Hyper-Normalization™](#️-hyper-normalization) - [🧬 Origin of Hyper-Normalization™](#-origin-of-hyper-normalization) - [🌌 Why Hyper-Normalization Matters](#-why-hyper-normalization-matters) - [🧩 User Display Example](#-example--the-user-display-case) - [Benchmark & Reproducibility](#benchmark--reproducibility) - [🚀 Performance Implications](#-performance-implications) ### Authentication - [🛡️ Authentication](#️-authentication) - [👤 User Registration & Authentication](#-user-registration--authentication) - [🛰️ Session Management](#-session-management) - [🔑 Password & Magic Link Recovery](#-password--magic-link-recovery) - [👥 Guest Account Management](#-guest-account-management) - [🔁 User Lifecycle Management](#-user-lifecycle-management) - [📊 User & Session Querying](#-user--session-querying) - [🧩 Internal Validation & Utility Methods](#-internal-validation--utility-methods) ### History + Logging - [🧭 Record History & Timeline](#-record-history--timeline) - [🧾 Auditing](#-auditing-history--timeline-reads) - [🪶 Debug & Developer Logging Options](#-debug--developer-logging-options) ### Logger - [🛰️ StarLogger](#-starlogger) - [🔧 Core Logger Methods](#-core-logger-methods) - [Write, System, and Audit Events](#write-system-and-audit-events) - [Recovery, Query & Maintenance](#recovery-query--maintenance) - [Rotation & Retention](#rotation--retention-built-in) ### Storage Layer - [📂 Starchive (Storage Layer)](#-starchive-storage-layer) - [⚙️ Starchive Configuration Modes](#️-starchive-configuration-modes) - [⚙️ Starchive Configurations](#️-starchive-configurations) - [📦 Starchive Core Methods](#-starchive-core-methods) - [✍️ Starchive Convenience Methods](#️-starchive-convenience-methods) - [🛰️ Remote Management Helpers](#-remote-management-helpers) - [🔏 Starchive Signing & Presigning](#-starchive-signing--presigning) - [🔏 Starchive Downloading](#-starchive-downloading) - [📜 Starchive Records](#-starchive-records) ### Event System - [🎧 Event Listening](#-event-listening) - [🌌 Collection & Path Matching](#-collection--path-matching) - [🌌 StarVault Events](#-starvault-events) ### Vacuum System - [🧽 Vacuum & Auto-Maintenance System](#-vacuum--auto-maintenance-system) - [⚙️ Automatic Vacuum Policy](#️-automatic-vacuum-policy) - [🪄 Methods](#-methods-1) - [🪶 Example](#-example) - [⚡ Auto-Vacuum Triggers](#️-auto-vacuum-triggers) - [📊 Metrics Example](#-metrics-example) ### Getting Started - [✨ Getting Started](#-getting-started) - [📦 Installation](#-installation) - [Basic Usage](#basic-usage) - [📈 Querying](#-querying) - [🎧 Listening to Changes](#-listening-to-changes) - [🌐 Wildcard Collection Queries](#-wildcard-collection-queries) - [👥 Registering Users](#-registering-users-guest-and-normal) - [📋 Listing Users and Sessions](#-listing-users-and-sessions) - [🔍 Quick Starchive Examples](#-quick-starchive-examples) - [✨ License](#-license) - [🚀 Transform Data into Action](#-transform-data-into-action) --- ## 🌌 Features - 🔐 **Optional Encryption** – Secure sensitive data at rest - ⚙️ **Sharded Storage Engine** – Efficiently scales writes across shards - 🧠 **In-Memory Caching** – High-speed read layer with `StarCache` - 📜 **Write-Ahead Logging** – Resilient logs with rotation and retention policies - 🔍 **Advanced Query Engine** – Chainable and expressive queries with filtering, search, sorting, and spatial support - 🚀 **Real-Time Event Emission** – Listen to data changes with fine-grained control - 🛡️ **Authentication Layer** – Optional handler to authorize every operation - 🌠 **Collection Wildcards** – Seamlessly operate across multiple collections --- ## ⚙️ System Requirements | Requirement | Version | |----------------|--------------------| | **Node.js** | ≥ 19.x | | **npm** | ≥ 9.x (recommended)| | **OS** | Windows, macOS, Linux | --- # 📜 API Specifications ## 🔧 Constructor Use these parameters when initializing `StarVault`. ```js new StarVault(dbPath, logPath, shardCount, maxLogSize, logRetention, options) ``` ### Core Parameters | Key | Description | Default | |---------------|------------------------------------------------------------------------------|--------------| | `dbPath` | Root directory for all collection data; each collection is stored in sharded files. | **Required** | | `logPath` | Directory where write-ahead logs (WAL) are stored for durability and replay. | **Required** | | `shardCount` | Number of shards used per collection. More shards improve concurrency. | `4` | | `maxLogSize` | Max size of each WAL file before rotation. Accepts values like `"100MB"`. | `"869MB"` | | `logRetention`| Duration to retain old WAL files. Accepts values like `"1w"` or `"30d"`. | `"1w"` | ### `options` Object | Key | Description | Default | |-------------------|-----------------------------------------------------------------------------|------------------------| | `enableEncryption`| Enables encryption for data at rest. | `false` | | `vaultPath` | Path to the vault metadata file used for encryption. | `"./star-vault.json"` | | `masterKey` | Master encryption key for vault encryption. Must securely generate/store.| `null` | | `authHandler` | Custom function that receives `clientAuth` and returns `true`/`false`. | `null` | | `authOptions` | Configuration object passed to internal `StarAuth` for authentication. | `{}` | | `auditHistory` | Turn on default auditing for history/timeline reads (writes entries to `audit.log`). Also toggle per call. | `false` | | `enableMesh` | Turn on StarMesh sync layer. | `false` | | `storageOptions` | Configuration object passed to the main storage engine for controlling low-level storage behavior, metrics flushing, shard hashing, and the automatic, idle-aware background compaction scheduler. | `{}` | | `meshOptions` | Options passed to `StarMesh` when `enableMesh` = `true`. | `{}` | | `starchive` | Configuration for the `Starchive` subsystem (local and remote file management). | `{}` | | `queryMetrics` | Query metrics configuration object or `false` to disable metrics. | `{ enabled : true }` | | `debugOptions` | Fine-grained developer logging controls for internal tracing (see below). | `{ all flags default to true }` | | `dirMode` | UNIX permission mode for directories (e.g., `0o700`). | `0o700` | | `fileMode` | UNIX permission mode for files (e.g., `0o600`). | `0o600` | --- ## storageOptions (Main Storage Configuration) These options control low-level storage behavior, metrics flushing, shard hashing, and the automatic, idle-aware background compaction scheduler. | Key | Type | Description | Default | | ----------------------------------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | | `idxFlushIntervalMs` | `number` | Interval (ms) for flushing pending `.idx` writes when `idxSyncMode` is set to `interval`. Lower values improve durability and cross-process visibility; higher values reduce IO overhead. Minimum enforced value is `10`. | `50` | | `idxSyncMode` | `"interval" \| Controls how `.idx` files are synced to disk. `"interval"` batches fsync calls based on `idxFlushIntervalMs`; `"always"` fsyncs on every index write for maximum durability at the cost of IO. | `"always"` | | `onWriteError` | `function` | Optional callback invoked when a write fails prior to persistence (for example, due to serialization failure). Receives the error and contextual metadata (`collection`, `id`, `value`). The write skips persistence and creates no pending or on-disk state. | | `metricsFlushIntervalMs` | `number` | Interval (ms) for flushing in-memory storage metrics (`fileBytes`, `liveBytes`, update counters) to disk. Lower values increase durability; higher values reduce IO. | `50` | | `shardHashFn` | `(id : string) => number` | Optional custom hash function used to determine shard placement. Return value is modulo-ed by shard count. | `undefined` | | `compactionEnabled` | `boolean` | Enable or disable background compaction entirely. Manual `vacuum*()` APIs still work when disabled. | `true` | | `compactionIdleMs` | `number` | Minimum time (ms) a collection must be idle (no writes) before it becomes eligible for background compaction. | `2500` | | `compactionCooldownMs` | `number` | Cooldown period (ms) after a collection is compacted before it may be compacted again by the scheduler. | `60000` | | `compactionTickMs` | `number` | Scheduler tick interval (ms). Determines how often pending compaction work is evaluated. | `200` | | `compactionMaxCollectionsPerTick` | `number` | Maximum number of collections that may be compacted in a single scheduler tick. | `1` | | `compactionMaxShardsPerCollectionPerTick` | `number` | Maximum number of shards compacted per collection per tick. Keeps pauses small and predictable. | `1` | --- ## 🔐 authOptions (StarAuth Configuration) Pass this object under the `options.authOptions` key when creating your `StarVault` instance. ### Collection Settings | Key | Description | Default | |----------------------|--------------------------------------------|---------------------| | `starAuthEnabled` | Global switch that controls whether the StarAuth subsystem activates. Set to `false` to disable authentication features entirely while keeping vault access open. | `true` | | `collection` | User collection name | `"auth-users"` | | `sessionCollection` | Session tracking collection | `"auth-sessions"` | | `resetCollection` | Password reset token collection | `"auth-resets"` | | `lockCollection` | Lockout state tracking collection (per account/IP/UA). | `"auth-locks"` | | `stellarCollection` | Magic link / code token collection | `"stellar-auths"` | ### Session Behavior These options control the creation, validation, and protection of sessions against brute-force attacks. | Key | Description | Default | |----------------------------|------------------------------------------------------------------------------|--------------------------| | `tokenExpiry` | Lifetime of each session token in seconds. | `3600` (1 hour) | | `sessionValidationFields` | Fields compared on every request to detect hijacking (e.g. IP, fingerprint). | `["ip", "fingerprint"]` | | `strictSessionValidation` | If `true`, any mismatch in validation fields invalidates the session. | `true` | | `enableSuspiciousCheck` | If `true`, compares geo/IP against recent sessions to flag anomalies. | `true` | | `sessionPolicy` | Session strategy: `"default"` (allow many), `"strict"` (1 active max), `"replace"` (replace old). | `"default"` | --- ### 🔒 Lockout Configuration Controls the tracking, weighting, and escalation behavior of failed login attempts. All options go inside `options.authOptions.lockout`. | Key | Description | Default | |--------------------|-------------------------------------------------------------------------------------------------|------------------------| | `strategy` | Lockout algorithm: `"fixed"`, `"slidingWindow"`, or `"exponential"`. | `"fixed"` | | `scopes` | Dimensions to track failures against: `["account", "ip", "ipua"]`. | `["account", "ipua"]` | | `maxAttempts` | Failures allowed before triggering lock. | `5` | | `baseDuration` | Base lockout duration in ms. | `15 * 60 * 1000` | | `maxDuration` | Max lockout duration in ms (for exponential strategy). | `24 * 60 * 60 * 1000` | | `minDuration` | **Absolute minimum** lock duration in ms (applies to all strategies; jitter won’t go below this). | `30 * 1000` | | `jitterDuration` | Random jitter ± ms added to lock duration. | `15 * 1000` | | `decayDuration` | Auto-decay failures if no attempts within this duration. | `30 * 60 * 1000` | | `windowDuration` | For `slidingWindow`, the rolling time window in ms. | `10 * 60 * 1000` | | `windowThreshold` | For `slidingWindow`, number of failures in the window that triggers lock. | `7` | | `windowGrowth` | For `slidingWindow`, how lock duration grows when over threshold: `"none" \| "linear" \| "exponential"`. | `"none"` | | `windowGrowthFactor` | For `slidingWindow: "exponential"` — multiplier per extra failure over threshold. | `2` | | `windowGrowthStep` | For `slidingWindow: "linear"` — extra ms per extra failure over threshold. | `baseDuration / 2` | | `captchaAfter` | Number of failures after which a CAPTCHA challenge is required. | `null` | | `otpAfter` | Number of failures after which an OTP challenge is required. | `null` | | `captchaVerifier` | Function `(req) => boolean` to validate CAPTCHA. | `null` | | `otpVerifier` | Function `({ email, code }) => boolean` to validate OTP. | `null` | | `onLock` | Callback `(info) => void` fired when a lock is applied. | `null` | | `onUnlock` | Callback `(info) => void` fired when a lock is cleared. | `null` | | `onChallenge` | Callback `(info) => void` fired when a CAPTCHA/OTP challenge is required. | `null` | | `notifyLock` | Optional notifier `(info) => void` for alerts/logging. | `null` | | `bypassed` | Function `(req) => boolean` to completely bypass lockout logic for trusted requests. | `null` | --- #### Timing Heuristics `options.authOptions.lockout.timingHeuristics` adds intelligence by classifying failures as bursts, normal, or human-like delays. | Key | Description | Default | |-----------------------|-----------------------------------------------------------|----------| | `enabled` | Enable timing heuristics. | `true` | | `burstDetectionDuration` | Attempts within this ms count as a burst (weighted more). | `3000` | | `humanDetectionDuration` | Attempts slower than this ms count as human/grace. | `45000` | | `maxConsiderDuration` | Ignore intervals longer than this ms. | `600000` | | `burstWeight` | Weight applied to burst failures. | `2.0` | | `normalWeight` | Weight applied to normal failures. | `1.0` | | `graceWeight` | Weight applied to grace (slow) failures. | `0.0` | | `emaAlpha` | Alpha for exponential moving average of attempt intervals. | `0.3` | | `decayOnGrace` | Whether to decay counters when a grace attempt is detected.| `true` | | `decayStep` | How many failures to subtract during a grace decay. | `1` | | `minFailures` | Clamp: minimum number of failures tracked. | `0` | | `maxFailures` | Clamp: maximum number of failures tracked. | `9999` | --- #### Example Fixed Lockout Config ```js authOptions: { lockout: { strategy: "fixed", scopes: ["account", "ipua"], maxAttempts: 5, // lock after N total failures baseDuration: 15 * 60 * 1000, // fixed lock duration (with jitter) minDuration: 30 * 1000, jitterDuration: 15 * 1000, // desync scripted retries decayDuration: 30 * 60 * 1000, // clear counters after idle period // Optional human gates captchaAfter: 3, otpAfter: null, captchaVerifier: async (req) => verifyCaptcha(req.body.token), // Heuristics amplify bursts, forgive human pace timingHeuristics: { enabled: true, burstDetectionDuration: 3000, humanDetectionDuration: 45000, burstWeight: 2.0, normalWeight: 1.0, graceWeight: 0.0, emaAlpha: 0.3, decayOnGrace: true, decayStep: 1, minFailures: 0, maxFailures: 9999 }, onLock: (info) => console.log("LOCKED", info), onUnlock: (info) => console.log("UNLOCKED", info), onChallenge: (info) => console.log("CHALLENGE", info), bypassed: (req) => false } } ``` 👉 When to use: Simple setups where predictability matters. Good for smaller apps, prototypes, or admin dashboards where you want consistent enforcement. --- #### Example Sliding Window Lockout Config ```js authOptions: { lockout: { strategy: "slidingWindow", scopes: ["account", "ipua"], windowDuration: 10 * 60 * 1000, windowThreshold: 7, baseDuration: 60 * 1000, minDuration: 30 * 1000, maxDuration: 24 * 60 * 60 * 1000, jitterDuration: 15 * 1000, windowGrowth: "exponential", windowGrowthFactor: 2, // windowGrowthStep: 30 * 1000, // if using "linear" // Optional Human gates captchaAfter: 3, otpAfter: null, captchaVerifier: async (req) => verifyCaptcha(req.body.token), // Heuristics amplify bursts, forgive human pace timingHeuristics: { enabled: true, burstDetectionDuration: 2000, humanDetectionDuration: 45000, burstWeight: 2.5, normalWeight: 1.0, graceWeight: 0.0, emaAlpha: 0.35, decayOnGrace: true, decayStep: 1, minFailures: 0, maxFailures: 9999 }, onLock: (info) => console.log("LOCKED", info), onUnlock: (info) => console.log("UNLOCKED", info), onChallenge: (info) => console.log("CHALLENGE", info), bypassed: (req) => false } } ``` 👉 When to use: Ideal for games and consumer apps. It’s forgiving to legit players (counters decay naturally) but still stops brute-force bursts quickly. --- #### Example Exponential Lockout Config ```js authOptions: { lockout: { strategy: "exponential", scopes: ["account", "ipua"], maxAttempts: 5, baseDuration: 30 * 1000, minDuration: 30 * 1000, maxDuration: 24 * 60 * 60 * 1000, jitterDuration: 10 * 1000, captchaAfter: 3, otpAfter: 5, captchaVerifier: async (req) => verifyCaptcha(req.body.token), otpVerifier: async ({ email, code }) => verifyOtp(email, code), onLock: (info) => console.log("LOCKED", info), onUnlock: (info) => console.log("UNLOCKED", info), timingHeuristics: { enabled: true, burstWeight: 3.0, graceWeight: 0.0, emaAlpha: 0.4 } } } ``` 👉 When to use: Best for high-security apps, admin portals, or payment flows. Makes repeated brute-forcing impractical, but can be harsh for end-users if tuned too aggressively. --- #### 🧭 Legacy Lockout These highlight the original “simple” lockout knobs. They remain in place for backward compatibility and apply **in addition** to the modern `authOptions.lockout` logic. | Key | Description | Default | |---------------------|----------------------------------------------------------------------------------------------------------|----------------------------| | `lockoutDuration` | **(Legacy)** Fixed lock duration (ms) after too many failed logins. | `900000` (15 minutes) | | `maxLoginAttempts` | **(Legacy)** Number of failed attempts before the legacy lockout triggers. | `5` | **How they interact:** - On every failed login, the modern lockout mechanism (if configured) evaluates first. - The legacy counters also increment and, if `failedAttempts >= maxLoginAttempts`, a fixed lock of `lockoutDuration` is applied. - This ensures older deployments keep their original behavior while you migrate to the new model. **Recommendation:** - Prefer the new `authOptions.lockout` settings for fine-grained control. - If you want the legacy lockout to *rarely* affect users, set a **very large** `maxLoginAttempts` (e.g., `1_000_000`) and leave modern lockout to do the real work. **Example (modern lockout active, legacy effectively neutralized):** ```js authOptions: { // Modern lockout (recommended) lockout: { strategy: "exponential", scopes: ["account", "ipua"], maxAttempts: 5, baseDuration: 15 * 60 * 1000, // 15 minutes maxDuration: 24 * 60 * 60 * 1000 // 24 hours // ...captcha/otp/timingHeuristics, etc. }, maxLoginAttempts: 5, lockoutDuration: 15 * 60 * 1000 } ``` --- ### Password Management | Key | Description | Default | |----------------------|----------------------------------------------------------------------------------------------|--------------------------------------------| | `passwordRequirements` | Object defining the password policy | `{ minLength: 8, requireLetter: true, requireNumber: true, requireSymbol: false }` | | └ `minLength` | Minimum number of characters required | `8` | | └ `requireLetter` | Requires at least one letter (a-z or A-Z) | `true` | | └ `requireNumber` | Requires at least one numeric digit (0-9) | `true` | | └ `requireSymbol` | Requires at least one symbol (e.g., `!@#$%^&*`) | `false` | | └ `customValidator` | Optional custom function `(password) => boolean` for advanced password checks | `null` | | `lockingCombinations` | Password hash complexity | `10` | ### 🌍 Star Locator Configuration | Key | Description | Default | |---|---|---| | `enableGeo` | Enable geo lookups. | `false` | | `enableReverseGeo` | Reverse geocode coordinates. | `false` | | `enableGeoDebug` | Log provider results. | `false` | | `geoProviders` | Extra providers `(ip, {signal}) => result`. | `[]` | | `ipgeolocationKey` | ipgeolocation.io key. | `null` | | `ipinfoToken` | ipinfo.io token. | `null` | | `nominatimUserAgent` | UA for Nominatim reverse geo. | `"StarAuth/1.0 (star-vault@sclpowerful.com)"` | | `googleMapsKey` | Google Geocoding key. | `null` | | `mapboxToken` | Mapbox token. | `null` | | `geoCacheTtlMs` | Cache TTL. | `600000` | | `geoDeadlineMs` | Deadline per lookup. | `3000` | | `geoTimeoutMs` | Timeout per provider. | `2500` | | `geoScoreOk` | Early-accept score threshold. | `5` | | `geoMaxConcurrency` | Max concurrent calls. | `8` | #### Example `geo` Object in Sessions ```js geo: { requestIP: "203.0.113.45", ip: "203.0.113.45", city: "City", region: "Region", country: "Country", continent: "Continent", org: "Org / ASN", isp: "ISP", loc: "12.3456,-98.7654", timezone: "Area/City", postal: "ZIP", flag: "🌍", confidence: 0.83, geoAddress: "123 Main St, Example City, Country" } ``` #### Custom Provider Registration ```js const vault = new StarVault(dataDir, logDir, 4, "869MB", "1w", { authOptions: { enableGeo: true, enableReverseGeo: true, ipinfoToken: process.env.IPINFO_TOKEN, geoProviders: [ async function customProvider(ip, { signal }) { return { city: "Custom City", country: "Customland" }; } ] } }); ``` ### Stellar Login | Key | Description | Default | |---------------------------|-------------------------------------------------|-----------------| | `stellarRequestCooldown`| Minimum delay between stellar requests | `60000` (1 min) | | `generateStellarCode` | Function to generate a stellar numeric code | 6-digit default | ### Hooks & Extensions | Key | Description | |----------------------|------------------------------------------------------| | `onSuspiciousSession`| `(currentSession, pastSession) => void` | | `handleHijack` | `(session, field, expected, actual) => void` | | `onCleanup` | `({ result, timestamp, vaultID }) => void` | | `tagSession` | `(session, userData) => string[]` | ### Guest Session Configurations | Key | Description | Default | |-----------------------------|-----------------------------------------------------------------------------|----------------------------------| | `allowGuestSessions` | Enables or disables guest session support. | `true` | | `guestInactivityThreshold` | Duration of inactivity (in ms) before considering a guest account stale. | `7 * 24 * 60 * 60 * 1000` (1w) | | `cleanupGuestInterval` | Interval (in ms) to check for inactive guest accounts. | `5 * 60 * 1000` (5m) | | `guestActivityTrackers` | Array of custom functions to execute on guest activity updates. | `[]` | | `generateGuestID` | Function to generate guest user IDs. | `() => "guest-" + UUID` | ### Cleanup & Locking | Key | Description | Default | |--------------------------------|------------------------------------------------------|-------------| | `vaultID` | ID for this vault instance | `null` | | `autoCleanupInterval` | How often to auto-clean expired tokens | `null` | | `expiredSessionCleanupInterval` | Interval for cleaning expired sessions | `null` | | `detachedTimers` | Controls whether background cleanup timers remain linked to the process. Set to `false` to keep timers active until completion (useful for long-running or graceful shutdown scenarios). | `true` | | `cleanupExpiredTokensActionInfo` | Metadata for cleanup activity | `{}` | | `cleanupExpiredTokensClientAuth` | Auth context used for cleanup calls | `null` | | `cleanupExpiredSessionsActionInfo` | Metadata for session cleanup | `{}` | | `cleanupExpiredSessionsClientAuth` | Auth context used during session cleanup | `null` | --- ## 📦 Core Methods The primary database interaction methods. | Method | Description | Sync/Async | |--------|-------------|------------| | `create(collection, data, actionInfo = {}, clientAuth = null)` | Creates a new record in the specified collection. Returns the created record object. | ✅ Sync | | `update(collection, id, updates, actionInfo = {}, clientAuth = null)` | Updates an existing record by ID. Setting record data properties to undefined removes them respectively. Returns the updated record. Throws error if not found. | ✅ Sync | | `unset(collection, id, dotPaths = [], actionInfo = {}, clientAuth = null)` | Removes specific nested fields using dot notation (e.g., "profile.stats.xp"). Returns updated record. | ✅ Sync | | `deleteCollection(collection, actionInfo = {}, clientAuth = null)` | Deletes an entire collection. Returns `{ wholeCollection: true, deleted: true/false }`. | ✅ Sync | | `deleteRecord(collection, id, actionInfo = {}, clientAuth = null)` | Deletes a specific record by ID. Returns `{ id, deleted: true/false }`. | ✅ Sync | | `deleteMany(collection, ids = [], actionInfo = {}, clientAuth = null)` | Deletes many records efficiently (shard-grouped delete). Returns `{ requested, deleted, shardsTouched }`. | ✅ Sync | | `delete(collection, id, actionInfo = {}, clientAuth = null)` | Soft-deletes a record by filtering it out and overwriting the collection. Returns `{ id, deleted: true }`. | ✅ Sync | | `softDelete(collection, id, { reason = "soft-delete", deletedBy = null } = {}, actionInfo = {}, clientAuth = null)` | Mark a record as deleted **without removing it**. Sets flags: `deleted: true`, `deletedAt`, `deletedReason`, `deletedBy`. Emits `update`. Returns `{ id, softDeleted: true, at, reason, deletedBy }`. | ✅ Sync | | `restore(collection, id, { restoredBy = null } = {}, actionInfo = {}, clientAuth = null)` | Reverse a prior soft delete. Clears `deleted*` flags and sets `restoredAt`, `restoredBy`. Emits `update`. Returns `{ id, restored: true, at, restoredBy }`. | ✅ Sync | --- ### 🧾 StarTransactions (WAL + Idempotency) Star-Vault handles **multi-record transactions** with staged commits, WAL logging, and per-step idempotency. #### Core Traits - Logs each `TX-BEGIN` and `TX-COMMIT` before writing data. - Replays committed groups safely after crash or restart. - Keeps uncommitted changes isolated until commit. #### Transaction Methods ```js // High-level helper const { transactionID, result } = await vault.transact({ type: "transfer" }, async (t) => { await t.update("accounts", "A", { balance: v => v - 100 }); await t.update("accounts", "B", { balance: v => v + 100 }); await t.create("ledger", { from: "A", to: "B", amount: 100, ts: Date.now() }); return "ok"; }); // Manual flow const tx = await vault.begin({ type: "batch-import" }); await tx.create("users", { id: "u1", name: "Nova" }); await tx.softDelete("posts", "p-123", { reason: "policy", deletedBy: "mod-9" }); await tx.restore("posts", "p-123", { restoredBy: "mod-9" }); await tx.commit(); // or await tx.abort("reason"); ``` | Context | Method | Purpose | Sync/Async | |----------|---------|----------|-------------| | vault | `transact(actionInfo, fn)` | Open txn, run `fn(tx)`, commit on success, abort on failure. Returns `{ transactionID, result }`. | ⚙️ Async | | vault | `begin(actionInfo?, clientAuth?)` | Start txn and return handle `{ id, create, update, unset, deleteRecord, softDelete, restore, commit, abort }`. | ⚙️ Async | | tx | `create(collection, data, actionInfo?, clientAuth?)` | Stage record creation. | ⚙️ Async | | tx | `update(collection, id, updates, actionInfo?, clientAuth?)` | Stage record update; supports function deltas `v => v - 50`. | ⚙️ Async | | tx | `unset(collection, id, dotPaths = [], actionInfo?, clientAuth?)` | Stage removal of nested fields. | ⚙️ Async | | tx | `deleteRecord(collection, id, actionInfo?, clientAuth?)` | Stage hard deletion. | ⚙️ Async | | tx | `softDelete(collection, id, { reason?, deletedBy? } = {}, actionInfo?, clientAuth?)` | Stage soft deletion (flags only). | ⚙️ Async | | tx | `restore(collection, id, { restoredBy? } = {}, actionInfo?, clientAuth?)` | Stage record restoration. | ⚙️ Async | | tx | `commit(actionInfo?)` | Apply staged ops, refresh cache, emit events, seal WAL. | ⚙️ Async | | tx | `abort(reason = "manual")` | Discard staging and log `TX-ABORT`. | ⚙️ Async | #### Idempotency and WAL - Each step logs internal `transactionID` and deterministic `idempotencyKey`. - Steps replay safely on crash recovery; duplicates ignored. #### Example Flow ```js await vault.transact({ actor: "u42" }, async (tx) => { await tx.create("users", { id: "42", name: "Orion" }); await tx.update("profiles", "42", { visits: v => v + 1 }); await tx.softDelete("sessions", "s-23", { reason: "expired", deletedBy: "system" }); }); ``` --- ## 🔍 Query Engine The **Query Engine** powers Star-Vault’s adaptive data traversal and filtering system — providing a fully chainable, composable query builder with deterministic caching and automatic mode optimization. ### Overview ```js vault.query("collection") .whereRecord("id", "value") // Filter by root record metadata .where({ key : "value" }) // Filter by record data fields .search("field", "text") // Text search within a field .recent("timestamp", "7d") // Filter records from last 7 days .near("location", { lat : 0, lng : 0 }, 50) // Geospatial radius filter .sort({ name : 1 }) // Sort ascending (1) or descending (-1) .select(["id", "name"]) // Return specific fields only .limit(10) // Limit number of results .offset(0) // Skip first N results .page({ limit : 10, after : null }) // Cursor-based pagination alternative .filterBy(record => record.active) // Apply custom JS-level filter .callback(row => { row._scanned = true }) // Optional pre-execution side effect .return("data") // (Optional) Return only the record data .execute(false, "auto"); // Execute with collection matching and mode control ``` --- ### ⚙️ Return Modes | Method | Description | Default | |---------|-------------|----------| | **`return("record")`** | Return the full record object including metadata (`{ id, data, timestamp, ... }`). | ✅ Default | | **`return("data")`** | Return only the user data (`record.data`). | | | **`returnFull()`** | Shorthand for `.return("record")`. | ✅ | --- ### Execution Modes | Mode | Description | Ideal Use Case | |------|--------------|----------------| | **auto** | Automatically selects the best execution path depending on query shape. Defaults to `hyper` when possible. | General queries | | **hyper** | Direct single-record retrieval path. Bypasses iteration and uses direct cache lookup for deterministic key-value equality filters. | Exact key lookups (`id`, `userID`, etc.) | | **default** | Standard streaming scan with limit/offset early cutoffs. Supports progressive paging and partial reductions. | Sequential or paginated scans | > ⚙️ Star-Vault automatically promotes `auto` queries to `hyper` when the filter is deterministic (single key-value pair + limit ≤ 1). --- ### Methods | Method | Description | Sync/Async | |--------|--------------|------------| | `query(collection, actionInfo?)` | Begin a query on the specified collection. Returns a chainable QueryBuilder. | ✅ Sync | | `whereRecord(criteria)` | Filter by root record metadata (`record.id`, etc.). | ✅ Sync | | `where(criteria)` | Filter by record data key-value pairs. | ✅ Sync | | `search(field, text)` | Text search (substring or tokenized) in a specific field. | ✅ Sync | | `recent(field, duration)` | Filter by recency, e.g. `"7d"`, `"30m"`. | ✅ Sync | | `near(field, center, radius)` | Filter spatially near a `{ lat, lng }` point. | ✅ Sync | | `sort(criteria)` | Sort results ascending (`1`) or descending (`-1`). | ✅ Sync | | `select(fields)` | Choose fields to include in the output. | ✅ Sync | | `limit(number)` | Restrict result count. | ✅ Sync | | `offset(number)` | Skip first N results for pagination. | ✅ Sync | | `page(options)` | Enables cursor-based pagination with deterministic ordering. Returns `{ items, nextCursor }` when calling execute(). | ✅ Sync | | `filterBy(fn)` | Apply a custom filter function to each record. | ✅ Sync | | `callback(fn)` | Add a hook or side effect before result finalization. | ✅ Sync | | `configureAutoIndex(options)` | Enables or tunes adaptive auto-indexing for queries that repeatedly miss fast paths. | ✅ Sync | | `return(mode)` | Control whether queries return `"record"` (full record) or `"data"` (just data). | ✅ Sync | | `returnFull()` | Explicitly reset return mode to `"record"`. | ✅ Sync | | `execute(exactCollection = false, executionMode = "auto")` | Execute query. If `exactCollection = true`, restrict to the specified collection only. | ✅ Sync | | `getByID(collection, id, actionInfo?)` | Retrieve a record directly by primary key. | ✅ Sync | | `getManyByID(collection, ids, options?)` | Retrieve multiple records by an array of primary keys, **preserving input order**. | ✅ Sync | | `range(min, max)` | Static helper to produce inclusive numeric ranges. | ✅ Sync | --- ### Examples #### 🔸 Full Records (Default) const orders = vault.query("orders") .where({ status : "shipped" }) .execute(); // Returns [{ id, data, timestamp, ... }] #### 🔸 Data-Only View const orders = vault.query("orders") .where({ status : "shipped" }) .return("data") .execute(); // Returns only record.data objects #### 🔸 Revert to Full Records const orders = vault.query("orders") .where({ status : "shipped" }) .returnFull() .execute(); --- #### 🔸 Auto Mode (Adaptive) ```js const user = vault.query("users") .where({ id : "u-1" }) .select(["name", "email"]) .execute(false, "auto")[0]; ``` Automatically selects `hyper` mode due to deterministic equality filter and small result limit. --- #### 🔸 Hyper Mode (Accelerated) ```js const user = vault.query("users") .where({ id : "u-1" }) .limit(1) .execute(false, "hyper")[0]; ``` Uses a direct reference lookup path through **StarCache** for **O(1)** deterministic retrieval. --- #### 🔸 Default Mode (Streaming Scan) ```js const users = vault.query("users") .where({ active : true }) .offset(100) .limit(50) .sort({ createdAt : -1 }) .execute(false, "default"); ``` Uses a sequential scan with **early cutoff** logic for predictable pagination over large datasets. --- #### 🔸 **Many-by-ID (order-preserving)** ```js const [ u1, u5, u3 ] = vault.getManyByID("users", ["u-1", "u-5", "u-3"]); ``` --- #### 🔸 Advanced Example (Custom Logic & Hooks) ```js vault.query("orders") .where({ status : "pending" }) .filterBy(order => order.total > 100) .callback(order => { order.checked = true }) .select(["id", "total", "status"]) .limit(5) .execute(false, "auto"); ``` > The callback() hook works best for small, customized post-processing operations or live UI effects before final emission. --- ### ⚙️ Advanced Controls #### `configureAutoIndex(options)` Enable or tune adaptive auto-indexing for queries that repeatedly miss fast paths. **Options** - `enabled : boolean` — Turn auto-indexing on/off. - `missThreshold : number` — How many cache/scan “misses” trigger an index build. - `windowMs : number` — Rolling time window for counting misses. - `maxConcurrentBuilds : number` — Cap background index builds. - `lazyMaxScan : number` — Upper bound for a “lazy” scan before promoting to an index. **Example** ```js vault.query("orders") .configureAutoIndex({ enabled : true, missThreshold : 8, windowMs : 60_000, maxConcurrentBuilds : 2, lazyMaxScan : 50_000 }) .where({ status : "pending" }) .sort({ createdAt : -1 }) .limit(100) .execute(false, "auto"); ``` **Notes** - `configureAutoIndex()` affects only the current query builder chain. - The adaptive indexer monitors repetitive scans and builds background indexes once thresholds reach their limits. - Star-Vault manages concurrency and I/O impact internally—no manual worker orchestration required. - Auto-indexing integrates with StarCache and never blocks live queries. - You can disable it in lightweight or memory-constrained deployments without affecting correctness. --- ### 📄 Cursor-Based Pagination Star-Vault provides **deterministic, cursor-based pagination** for stable traversal across large datasets. Unlike offset-based pagination, cursor pagination: - avoids skipped/duplicated records under concurrent writes - maintains stable ordering - scales efficiently for large collections --- #### Basic Usage ```js const page1 = vault.query("users") .where({ country : "US" }) .page({ limit : 5 }) .execute(); console.log(page1.items); console.log(page1.nextCursor); ``` --- #### Fetching the Next Page ```js const page2 = vault.query("users") .where({ country : "US" }) .page({ limit : 5, after : page1.nextCursor }) .execute(); ``` --- #### Return Shape ```js { items : [ /* records */ ], nextCursor : "cursor-string-or-null" } ``` | Field | Description | |--------------|-------------| | `items` | Array of records (or data depending on return mode) | | `nextCursor` | Cursor string for the next page, or `null` if no more results | > ℹ️ Pagination changes the return type of `.execute()` from an array to a `{ items, nextCursor }` object. --- #### Cursor Behavior - `nextCursor = null` → no more pages - Passing `after : cursor` resumes from the last record of the previous page - Cursor encoding is **opaque and versioned internally** --- #### Example — Full Pagination Flow ```js let cursor = null; do { const page = vault.query("users") .where({ country : "US" }) .page({ limit : 5, after : cursor }) .execute(); console.log(page.items); cursor = page.nextCursor; } while (cursor); ``` --- #### ⚙️ Deterministic Ordering Pagination relies on Star-Vault’s **deterministic ID ordering**, ensuring: - stable page boundaries - no duplication across pages - no missing records during traversal > ⚠️ Important: Do not mutate record IDs or manually inject non-deterministic IDs if you rely on pagination stability. --- ### Performance & Behavior - **Adaptive Execution Pathing** — chooses optimal mode based on filter complexity and limit count. - **Cache-Aware Traversal** — leverages in-memory record handles for active collections. - **Early-Cutoff Scanning** — truncates scan loops when limits are reached to reduce latency. - **Geospatial and Temporal Filtering** — supports proximity, duration, and time-based conditions natively. - **Lightweight Chaining** — each query stage mutates minimal state, maintaining sub-millisecond builder overhead. --- ### Developer Note Query Engine provides direct low-level access to **StarVault's compositional access model**. When combined with **Hyper-Normalization™**, queries can achieve sub-millisecond lookups even across multi-collection normalized schemas. --- ## ⚡️ Hyper-Normalization™ Star-Vault introduces **Hyper-Normalization™**, a revolutionary data-modeling paradigm that unlocks the **highest logical purity** of data — with **near-zero performance penalties**. ### Definition Hyper-Normalization™ (noun)  /ˈhaɪ.pər ˌnɔr.mə.laɪˈzeɪ.ʃən/ A data-modeling paradigm introduced by Steven Compton in 2025 (Star-Vault) that implements physical-path normalization, eliminating runtime join planning through deterministic reference resolution and achieving near-constant-time cross-collection traversal under defined consistency and locality models. _First published: 2025-11-04_ _Coined and defined by Steven Compton, creator of Star-Vault._ --- ### 🧬 Origin of Hyper-Normalization™ **Before Star-Vault**, no database truly achieved runtime-level normalization without sacrificing speed.   **Relational databases** (PostgreSQL, MySQL, Oracle) implemented normalization theoretically through 3NF to 5NF schemas—but still required runtime joins and query planners to reconstruct relationships. Each query carried planner latency, driver overhead, and redundant I/O. **NoSQL systems** (MongoDB, Couchbase) took the opposite path—denormalizing data for speed at the cost of consistency and integrity. Developers duplicated or embedded related data, breaking the very principles of normalization. **Columnar and graph databases** (ClickHouse, Neo4j, ArangoDB) came close by optimizing joins or reference traversals, yet still relied on runtime evaluation paths or bulk vectorized execution. They never fused structural purity with constant-time resolution. **Star-Vault extends that evolution by redefining how normalization operates in real time.** Instead of chasing faster joins or heavier optimizers, Star-Vault redefined the storage and access model itself. This engine makes normalization a **native physical characteristic** of data organization, embedding relational integrity into the storage layer itself. It forms a unified data graph that resolves relationships instantly with zero planner cost. Every relationship resolves in near-constant time through StarCache, Star-Vault's in-memory reference engine optimized for deterministic cross-collection access. This design yields a system where **perfect normalization** and **constant-time performance coexist**, eliminating the trade-off between relational purity and real-time scalability. > ⚙️ **In short:** Star-Vault did not just optimize normalized data—it turned normalization itself into the fastest query path in real workloads. --- ### 🌌 Why Hyper-Normalization Matters - 🔗 **No denormalization trade-offs** — each concept lives in its own collection. - ⚡ **Near-zero-cost references** — cross-collection lookups resolve deterministically without runtime joins. - 🧠 **Near constant-time traversal** — each relationship resolves through deterministic reference pathways optimized for direct in-memory access, yielding O(1) access. - 🔐 **MVCC-safe snapshots** — all reads operate on immutable snapshots for perfect consistency. - 💾 **Physically normalized storage** — data and indexes align for locality at the shard and log levels, keeping access ultra-fast. --- ### 🧩 Example — The “User Display” Case Traditional SQL joins: ```sql SELECT users.id, first_names.name, last_names.name, emails.email, usernames.username FROM users JOIN first_names ON users.first_name_id = first_names.id JOIN last_names ON users.last_name_id = last_names.id JOIN emails ON users.email_id = emails.id JOIN usernames ON users.username_id = usernames.id WHERE users.id = 7865789; ``` Star-Vault’s Hyper-Normalized equivalent: ```js const user = vault.query("users").where({ userID : 7865789 }).first(); const first = vault.query("first-names").where({ firstNameID : user.firstNameID }).select(["firstName"]).first(); const last = vault.query("last-names").where({ lastNameID : user.lastNameID }).select(["lastName"]).first(); const email = vault.query("emails").where({ emailID : user.emailID }).select(["email"]).first(); const uname = vault.query("usernames").where({ usernameID : user.usernameID }).select(["username"]).first(); ``` Each lookup executes in **microseconds** - directly from in-memory StarCache. No planner. No join reordering. No network round-trips in embedded setups. Just raw, deterministic access. --- ### Benchmark & Reproducibility #### Test Configuration To verify Star-Vault’s retrieval performance under normalized, encrypted conditions, a reproducible benchmark evaluated an **8-collection user display composition scenario**. Each iteration reconstructs a normalized user profile composed of `users`, `first-names`, `last-names`, `emails`, `usernames`, `locales`, `permission-sets`, and `profiles` — all under identical encryption, indexing, and cache settings. | Parameter | Configuration | |------------|---------------| | Runtime | Node.js 22.0.0 (Windows 11) | | Dataset | 8 facet collections (User composite) | | Access Method | Star-Vault compositional queries and direct retrievals | | Runs | 20 warm runs after 5 warm-ups | | Environment | NVMe SSD, 32 GB RAM | | Encryption | Enabled | | Audit History | Disabled | | Execution Mode | Hyper (automatic optimization) | > **Table 1.** Test configuration used for Star-Vault performance benchmarks. --- #### Reproducible Benchmark Code ```js const { performance } = require("node:perf_hooks"); const StarVault = require("@trap_stevo/star-vault"); async function buildUserDisplay_query(vault, userID) { const t0 = performance.now(); const user = vault.query("users").where({ userID }).select([ "firstNameID","lastNameID","emailID","usernameID", "avatarFileID","localeID","permissionSetID","profileID" ]).execute(true)[0]; if (!user) { return null; } const [ first,last,email,uname, avatar,locale,perms,profile ] = await Promise.all([ vault.query("first-names").where({ id : user.firstNameID }).select(["firstName"]).execute(true)[0], vault.query("last-names").where({ id : user.lastNameID }).select(["lastName"]).execute(true)[0], vault.query("emails").where({ id : user.emailID }).select(["email"]).execute(true)[0], vault.query("usernames").where({ id : user.usernameID }).select(["username"]).execute(true)[0], vault.starchive ? vault.starchive.records.getByID(user.avatarFileID) : null, vault.query("locales").where({ id : user.localeID }).select(["code"]).execute(true)[0], vault.query("permission-sets").where({ id : user.permissionSetID }).select(["roles"]).execute(true)[0], vault.query("profiles").where({ id : user.profileID }).select(["bio","links"]).execute(true)[0] ]); const display = { name : `${first?.firstName || ""} ${last?.lastName || ""}`.trim(), email : email?.email || null, username : uname?.username || null, avatarFile : avatar || null, locale : locale?.code || "en", roles : perms?.roles || [], profile : profile || {} }; display._ms = (performance.now() - t0).toFixed(3); return display; } async function buildUserDisplay_getByID(vault, userID) { const t0 = performance.now(); const user = vault.query("users").where({ userID }).select([ "firstNameID","lastNameID","emailID","usernameID", "avatarFileID","localeID","permissionSetID","profileID" ]).execute(true)[0]; if (!user) { return null; } const [ first,last,email,uname, avatar,locale,perms,profile ] = await Promise.all([ vault.getByID("first-names", user.firstNameID), vault.getByID("last-names", user.lastNameID), vault.getByID("emails", user.emailID), vault.getByID("usernames", user.usernameID), vault.starchive ? vault.starchive.records.getByID(user.avatarFileID) : null, vault.getByID("locales", user.localeID), vault.getByID("permission-sets", user.permissionSetID), vault.getByID("profiles", user.profileID) ]); const display = { name : `${first?.data?.firstName || ""} ${last?.data?.lastName || ""}`.trim(), email : email?.data?.email || null, username : uname?.data?.username || null, avatarFile : avatar || null, locale : locale?.data?.code || "en", roles : perms?.data?.roles || [], profile : profile?.data || {} }; display._ms = (performance.now() - t0).toFixed(3); return display; } (async function main() { const sV = new StarVault("./data","./logs",4,"869MB","1w",{ enableEncryption : true, masterKey : "starvault-demo", auditHistory : false, debugOptions : { displayEventEmissions : false, displayRecordQueryErrors : false, displayCollectionQueryErrors : false } }); const first = sV.create("first-names",{ id : "fn-1", firstName : "Nova" }); const last = sV.create("last-names",{ id : "ln-1", lastName : "Orion" }); const email = sV.create("emails",{ id : "em-1", email : "nova@example.com" }); const uname = sV.create("usernames",{ id : "un-1", username : "nova" }); const locale = sV.create("locales",{ id : "lc-1", code : "en" }); const perms = sV.create("permission-sets",{ id : "ps-1", roles : ["user","beta"] }); const profile = sV.create("profiles",{ id : "pr-1", bio : "Explorer", links : ["https://example.com/nova"] }); sV.create("users",{ userID : "u-1", firstNameID : first.id, lastNameID : last.id, emailID : email.id, usernameID : uname.id, avatarFileID : "file-001", localeID : locale.id, permissionSetID : perms.id, profileID : profile.id }); // Warm-up await buildUserDisplay_query(sV,"u-1"); await buildUserDisplay_getByID(sV,"u-1"); // Benchmarks const runBench = async (label, fn) => { const runs = 20; let best = Infinity, worst = 0, total = 0; for (let i=0;ibuildUserDisplay_query(sV,"u-1")); await runBench("Warm compose (getByID)",()=>buildUserDisplay_getByID(sV,"u-1")); })(); ``` #### Results | Method | Best (ms) | Avg (ms) | Worst (ms) | |---------|-----------|-----------|------------| | Warm compose (query) | 3.285 | 4.384 | 5.922 | | Warm compose (getByID) | 0.094 | 0.124 | 0.202 | > **Table 2.** Warm benchmark results comparing compositional queries and direct primary-key retrievals in Star-Vault. --- #### Analysis Star-Vault demonstrates **sub-5 ms** average performance for compositional (multi-facet) reads and **sub-0.1 ms** primary-key retrievals — both fully encrypted and cache-synchronized. These results confirm the effectiveness of Star-Vault’s **Hyper Normalization™** model, achieving consistent, near-instant retrieval performance across normalized datasets. --- #### Interpretation Even in a **cold state**, Star-Vault composes a fully normalized, eight-collection user display in milliseconds. Once warmed, the deterministic cache layer maintains micro-scale latency across repeated traversals — a direct realization of Hyper Normalization™ in practice. --- #### Cross-Hardware Consistency Star-Vault’s traversal and caching models operate deterministically, scale linearly with CPU frequency, and maintain architecture neutrality across device classes. Performance remains stable on laptops, servers, and embedded systems, with variance driven primarily by traversal depth rather than hardware architecture. --- #### Result Variability Notice Actual timings may vary slightly depending on: - CPU frequency and thermal scaling - Storage device performance and I/O queue depth - Background processes and concurrent file access However, Star-Vault’s normalized, cache-aware execution model ensures that **relative performance ratios remain consistent** across hardware configurations — preserving the same latency hierarchy between compositional and direct retrieval paths. ##### Key Takeaway Star-Vault achieves ≈ **4.4 ms** multi-facet reads and ≈ **0.094 ms** direct retrievals on standard NVMe hardware — establishing a new benchmark for encrypted, normalized, cache-optimized persistence systems. --- #### Benchmark Notes & Scope The benchmark evaluates **deterministic multi-record composition and retrieval** under encrypted, normalized, and cache-warmed conditions - demonstrating **Star-Vault's optimized read-path architecture**. The test focuses on **real-world access behavior** rather than synthetic concurrency stress, emphasizing **traversal consistency, normalization efficiency, and encryption overhead neutrality**. --- ### 🚀 Performance Implications - ✅ Achieves O(1) cross-collection resolution for defined reference paths; depth and topology influence total latency. - ✅ Near-zero integrity overhead - deterministic reference synchronization without relational cost. - ✅ Linear scaling with CPU, memory, and shard count. - ✅ Deterministic read and write paths for predictable real-time performance. --- > 💡 In essence: > Hyper-Normalization transforms normalization from a theoretical ideal into a physical performance advantage - proving that **data integrity and speed can coexist with minimal compromise for the targeted access patterns.** --- ## 🛡️ Authentication All methods below require `authOptions` configuration and are used for session/user management. --- ### 👤 User Registration & Authentication | Method | Description | Sync/Async | |--------|-------------|------------| | `registerUser(email, password, actionInfo, clientAuth)` | Registers a new user. `. | ⚙️ Async | | `registerGuestUser(req, metadata, actionInfo, clientAuth)` | Registers a temporary guest user. `. | ⚙️ Async | | `loginGuestUser(accountID, req, actionInfo, clientAuth)` | Logs in a guest user and creates a session. `. | ⚙️ Async | | `loginUser(email, password, req, actionInfo, clientAuth)` | Logs in a user and creates a session. `. | ⚙️ Async | | `resumeUserSession(accountID, req, actionInfo, clientAuth)` | Resumes session from account ID (used for guest linking or session carryover). | ⚙️ Async | | `upgradeGuestUser(accountID, email, password, actionInfo, clientAuth)` | Converts a guest account to a full registered user. | ⚙️ Async | --- ### 🛰️ Session Management | Method | Description | Sync/Async | |--------|-------------|------------| | `logoutUser(token, actionInfo, clientAuth)` | Logs out a user by invalidating their session token. | ✅ Sync | | `extendUserSession(token, ms, actionInfo, clientAuth)` | Extends an active session's duration. | ✅ Sync | | `cleanupExpiredAuthTokens(actionInfo, clientAuth)` | Cleans up expired tokens and invalid sessions. | ✅ Sync | | `cleanupInactiveGuests(actionInfo, clientAuth)` | Removes or deactivates inactive guest accounts. | ⚙️ Async | --- ### 🔑 Password & Magic Link Recovery | Method | Description | Sync/Async | |--------|-------------|------------| | `updateUserPassword(userID, newPassword, actionInfo, clientAuth)` | Updates a user's password. | ⚙️ Async | | `requestUserPasswordReset(email, actionInfo, clientAuth)` | Sends a password reset token. | ⚙️ Async | | `resetUserPassword(token, newPassword, actionInfo, clientAuth)` | Resets password using a valid token. | ⚙️ Async | | `requestUserStellarLink(email, type, actionInfo, clientAuth)` | Sends a Stellar link (magic link or code) for login or verification. | ⚙️ Async | | `consumeUserStellarToken(token, type, req, actionInfo, clientAuth)` | Consumes a Stellar link/token to verify user and return session. | ⚙️ Async | --- ### 👥 Guest Account Management | Method | Description | Sync/Async | |--------|-------------|------------| | `registerGuestUser(req, metadata, actionInfo, clientAuth)` | Registers a guest session with optional request/metadata. | ⚙️ Async | | `trackGuestActivity(userID, timestamp, actionInfo, clientAuth)` | Updates guest user's `lastActiveAt` timestamp. | ⚙️ Async | | `upgradeGuestUser(accountID, email, password, actionInfo, clientAuth)` | Promotes a guest to a full registered account. | ⚙️ Async | | `cleanupInactiveGuests(actionInfo, clientAuth)` | Cleans up inactive or stale guest users based on threshold. | ⚙️ Async | --- ### 🔁 User Lifecycle Management | Method | Description | Sync/Async | |--------|-------------|------------| | `updateUser(accountID, updatedMetadata, accountType, actionInfo, clientAuth)` | Updates a user by Account ID. Does not allow for password updates. | ⚙️ Async | | `deleteUser(userID, reason, actionInfo, clientAuth)` | Soft-deletes a user by ID. | ⚙️ Async | | `deleteUserByAccountID(accountID, reason, actionInfo, clientAuth)` | Soft-deletes a user using their account ID. | ⚙️ Async | | `reactivateUser(userID, actionInfo, clientAuth)` | Reactivates a previously deleted user. | ⚙️ Async | | `reactivateUserByAccountID(accountID, actionInfo, clientAuth)` | Reactivates a deleted user by account ID. | ⚙️ Async | --- ### 📊 User & Session Querying | Method | Description | Sync/Async | |--------|-------------|------------| | `getUserByAccountID(accountID, filter, accountType, actionInfo)` | Retrieves a user by account id. If a certain account type (default ~ guest). | ⚙️ Async | | `getUser(userID, filter, accountType, actionInfo)` | Retrieves a user by id. If a certain account type (default ~ guest). | ⚙️ Async | | `getUserSession(sessionID, actionInfo)` | Retrieves a session by id. | ⚙️ Async | | `listSessions(userID, filters, includeInactive, actionInfo)` | Lists sessions for a specific user. | ⚙️ Async | | `listAllSessions(filters, includeInactive, actionInfo)` | Lists all sessions across users. | ⚙️ Async | | `listUsers(filters, options, actionInfo)` | Lists users based on filters and options (pagination, includes). | ⚙️ Async | | `listUserActivity(options, actionInfo)` | Lists recent activity, including filtering by active, inactive, or recently seen users. | ⚙️ Async | ### 🧩 Internal Validation & Utility Methods The following utility methods get utilized internally across authentication-related flows. You may call them directly for validation, diagnostics, or extension purposes. | Method | Description | Sync/Async | |--------------------------------|-----------------------------------------------------------------------------------------------|-------------------------------------------------------------------------| | `validEmail(email)` | Determines if an input email follows proper email address format. | ✅ Sync | | `satisfiesPasswordStrength(password)` | Validates whether the password meets the defined `passwordRequirements`. | ✅ Sync | | `extractSessionMetadata(req)` | Extracts metadata (IP, fingerprint, user agent, etc.) from the request object . | ✅ Sync | | `lookupUserGeo(ip)` | Performs geo-IP lookup using built-in and custom geo services . | ⚙️ Async | --- ## 🧭 Record History & Timeline Star-Vault reconstructs **record history** and a human-readable **timeline** from write-ahead logs (WAL). These calls support filtering, integrity checks, and encryption-aware output. ### Methods | Method | Description | Sync/Async | |---|---|---| | `getRecordHistory(collection, id, options = {}, actionInfo = {}, clientAuth = null)` | Returns detailed event entries for a single record in time order. | ✅ Sync | | `getRecordTimeline(collection, id, options = {}, actionInfo = {}, clientAuth = null)` | Returns a compact timeline view (action, timestamp, brief summary). | ✅ Sync | | `getCollectionHistory(collection, options = {}, actionInfo = {}, clientAuth = null)` | Returns all WAL entries for a collection, including multi-record activity. | ✅ Sync | | `getCollectionTimeline(collection, options = {}, actionInfo = {}, clientAuth = null)` | Returns a summarized timeline for collection-level changes. | ✅ Sync | | `getVaultHistory(options = {}, actionInfo = {}, clientAuth = null)` | Returns all WAL entries across the vault, with filters for collections and actions. | ✅ Sync | | `syncHistory(actionInfo = {}, clientAuth = null)` | Flushes WAL streams and closes active handles to support backups, exports, and shutdown flows. | ⚙️ Async | > All history/timeline methods share the same options contract below, including collection-level and vault-level calls where applicable. ### `options` (history/timeline) | Key | Type | Default | Notes | |---|---|---|---| | `includeArchived` | boolean | `true` | Include rotated `*.archived` files when present. | | `verifyChecksums` | boolean | `true` | Verify each entry’s SHA-256 checksum of `data`; skip failures. | | `includeSystem` | boolean | `true` | Merge related `system.log` entries for the same `collection`/`id` (e.g., DELETE, TX events). | | `actions` | string \| string[] | `undefined` | Allow-list of actions to include (e.g., `["WRITE","UPDATE"]`). Case-insensitive. | | `excludeActions` | string \| string[] | `undefined` | Block-list of actions to exclude (e.g., `["DELETE"]`). Case-insensitive. | | `since` | number \| string \| Date | `undefined` | Include entries at/after this time (ms epoch, ISO string, or `Date`). | | `until` | number \| string \| Date | `undefined` | Include entries at/before this time. | | `match` | (row) => boolean | `undefined` | Custom predicate that keeps/discards rows after other filters. | | `includeDecrypted` | boolean | *(defaults to `true` when a decryptor exists; otherwise `false`)* | When `true` and encryption enabled, decrypt rows and populate `dataDecrypted`. | | `audit` | boolean | `undefined` | Per-call audit switch; when `true`, writes an audit entry (see Auditing). | > ℹ️ History entries include `source`, `collection`, `action`, `id`, `timestamp`, `client`, `data`, `checksum`, plus `eventID`, `transactionID`, `idempotencyKey` when present. With `includeDecrypted` and a decryptor, output also includes `dataEncrypted` and `dataDecrypted`. ### Audit Examples ```js // Full history (archived segments + checksum verification) const history = vault.getRecordHistory("users/region1", "1760233994625", { includeArchived : true, verifyChecksums : true }, { actor : "admin-1" }, { token : "valid" }); // Windowed, updates only const updates = vault.getRecordHistory("users/region1", "1760233994625", { actions : "UPDATE", since : Date.now() - 60_000, until : Date.now() }); // Compact timeline, no deletions, explicit audit const timeline = vault.getRecordTimeline("users/region1", "1760233994625", { excludeActions : ["DELETE", "SOFT_DELETE"] }, { audit : true }, { token : "valid" }); // Collection-level history const collectionHistory = vault.getCollectionHistory("users/region1", { includeArchived : true, includeSystem : true }); // Vault-wide history with collection filter const vaultHistory = vault.getVaultHistory({ collections : ["users/*", "orders/*"], actions : ["CREATE", "UPDATE"] }); // Synchronous WAL flush before external backup/export await vault.syncHistory({}, { token : "valid" }); ``` --- ### 🧾 Auditing (History & Timeline Reads) History/timeline reads can trigger **auditing** to `audit.log`. Use this for compliance and access tracking. #### Control switches Auditing activates when **any** of the following holds: 1) `options.audit === true` 2) `actionInfo.audit === true` 3) `new StarVault(..., { auditHistory : true })` On audit, Star-Vault writes `logAudit` with: - `action` : `"HISTORY_READ"` or `"TIMELINE_READ"` - `details` : `{ kind, collection, id, count, filters }` - `filters` carries a sanitized view of options (booleans/timestamps, allow/block lists, and a flag that notes a custom `match` predicate). > Encryption enabled? Prefer encrypted audit logs for sensitive environments. #### Audit Example: global default ```js const vault = new StarVault("./data", "./logs", 4, "869MB", "1w", { auditHistory : true }); ``` #### Audit Example: per-call ```js vault.getRecordHistory("orders", "o-123", { audit : true }); ``` #### Audit Action names - `HISTORY_READ` — written after `getRecordHistory` completes - `TIMELINE_READ` — written after `getRecordTimeline` completes These actions cover record-level, collection-level, and vault-level history/timeline reads. --- #### ⚠️ Privacy Notes - Handling sensitive data? Prefer `includeDecrypted : false` unless the caller truly needs decrypted payloads. - Console snippets show full records for clarity; in production, prefer logging only **ids** and **checksums**. --- ## 🪶 Debug & Developer Logging Options Fine-grained switches help you trace StarVault behavior without noise overload. All flags default to `true`, and you can change them at construction via `options.debugOptions` or at runtime with `setDebugOptions()`. | **Flag** | **Default** | **Purpose** | |-----------|-------------|-------------| | `displayQueryRetrievalTimestamps` | `true` | Appends a timestamp to query retrieval logs for timing correlation. | | `displayQueryRetrievals` | `true` | Logs when queries complete, showing which collection and how many records. | | `displayEventEmissions` | `true` | Logs lifecycle events such as `"create"`, `"update"`, `"delete"`. | | `displayUnsetKeysLogs` | `true` | Logs details when fields are removed using dot-path unsets. | | `displayCollectionQueryErrors` | `true` | Warns when a collection fails to resolve. | | `displayRecordQueryErrors` | `true` | Warns when a specific record lookup fails. | | `displayListenerAdditions` | `true` | Logs listener registration through `listen()`. | | `displayEventEmissionMatches` | `true` | Logs path matching attempts for event emissions. | | `displayEventEmissionTriggers` | `true` | Logs when a matching listener fires. | | `displayCollectionMatchLogs` | `true` | Logs wildcard/path-based collection matching results. | | `displayPathMatches` | `true` | Logs per-segment comparison details during path matching. | ### ⚙️ Configure at construction ```js const vault = new StarVault( "./db", "./logs", 8, "512MB", "7d", { debugOptions : { displayRecordQueryErrors : false, displayEventEmissions : false } } ); ``` ### 🔧 Configure at runtime ```js // Merge partial updates (returns full resulting config) vault.setDebugOptions({ displayEventEmissions : true, displayEventEmissionTriggers : false }); // Read current flags const dbg = vault.getDebugOptions(); console.log(dbg); ``` --- ## ⚡ StarCache — Intelligent Caching Layer StarVault includes an adaptive in-memory caching system called **StarCache**, designed to minimize redundant reads, decryptions, and disk access across all operations. StarCache operates automatically in the background and adapts dynamically to workload patterns. --- ### ⚙️ Cache Management Methods | **Method** | **Description** | |-------------|-----------------| | `getCacheCapacities()` | Returns current cache capacity configuration. | | `setCacheCapacities({ primary, decoded })` | Updates cache capacities at runtime; omit a value to retain existing configuration. | | `getCacheStats()` | Retrieves hit, miss, and eviction statistics for the cache. | | `resetCacheStats()` | Resets all cache statistics counters to zero. | | `prewarmDecoded(collection, ids)` | Preloads and decrypts specific records in advance to remove first-access latency. | --- ### 💡 Example Usage ```js // Inspect cache capacity console.log(vault.getCacheCapacities()); // Adjust cache capacities dynamically vault.setCacheCapacities({ decoded : 5000, primary : 2000 }); // Reset cache statistics before benchmarking vault.resetCacheStats(); // Preload critical records for instant retrieval await vault.prewarmDecoded("users", ["user-001", "user-002"]); // Display cache performance statistics console.log(vault.getCacheStats()); ``` --- ## 🛰️ StarLogger High-throughput append-only logger with **ordered writes**, **atomic rotation**, and **cross-process safety** — powering Star-Vault’s record tracking and audit subsystem. ### Access & Configuration ```js const StarVault = require("@trap_stevo/star-vault"); const vault = new StarVault("./data", "./logs", 4, "869MB", "1w", { enableEncryption : true, masterKey : "starvault-demo", auditHistory : false }); // Configure the embedded logger through the vault instance vault.wal.setIdleCloseMs(30000); // Close pooled fds after N ms idle (0 = never) vault.wal.setLockMs(50); // Max time to try acquiring rotate lock (ms) vault.wal.setDebounceMs(0); // Debounce rotate checks (ms) vault.wal.setFsync(false); // fsync after each append (durability vs throughput) ``` StarLogger automatically provides: - **Ordered writes** — entries for each log file never interleave - **Automatic rotation** — log rotation on size threshold across processes - **Idle pooling** — file handles reuse for throughput and close after idle windows - **Retention management** — log cleanup guided by retention configuration - **Crash resilience** — optional fsync/fdatasync for higher durability --- ### 🔧 Core Logger Methods > For history-driven lookups (record/collection/vault), see > [**🧭 Record History & Timeline**](#-record-history--timeline) (single source of truth for those APIs). | Method | Description | Async/Sync | |--------|-------------|-------| | **new StarLogger(logPath, maxLogSizeBytes?, retention?, dirMode?, fileMode?)** | Create a logger under `logPath`. Normalize sizes/durations through `EngineUtilityManager`. | ✅ Sync | | **setDecryptor(fn)** | Install a decryptor `(namespace, payload) => decrypted`. History APIs consume it when exposing decrypted data. | ✅ Sync | | **setEncryptor(fn)** | Install an encryptor `(namespace, data) => { locked/iv/tag/... }`. System/audit write paths accept it automatically. | ✅ Sync | | **setIdleCloseMs(ms)** | Configure idle-FD close window. `0` disables idle closing. Return the active value. | ✅ Sync | | **setLockMs(ms)** | Configure rotation-lock patience. Return the active value. | ✅ Sync | | **setDebounceMs(ms)** | Configure debounce interval for rotation checks. Return the active value. | ✅ Sync | | **setFsync(enabled)** | Toggle `fsync/datasync` enforcement after each write. Return the active value. | ✅ Sync | | **logWrite(collection, record, clientInfo?, action?, metadata?)** | Append a JSONL WAL entry for a collection. Compute SHA-256 checksum. Chain writes for strict ordering. | ✅ Sync (fire-and-queue) | | **recoverWrites(collection)** | Load `.log`, parse JSONL, validate checksums, return only verified entries. | ✅ Sync | | **logSystemAction(action, collection, details, clientInfo?, metadata?)** | Append a system-level entry in `system.log`. Encrypt `details` when encryptor exists. | ✅ Sync (fire-and-queue) | | **logTxBegin(transactionID, clientInfo?)** | Emit `TX-BEGIN` into `system.log`. | ✅ Sync | | **logTxCommit(transactionID, clientInfo?)** | Emit `TX-COMMIT` into `system.log`. | ✅ Sync | | **logTxAbort(transactionID, clientInfo?)** | Emit `TX-ABORT` into `system.log`. | ✅ Sync | | **logAudit(action, details, clientInfo?, metadata?)** | Append an audit entry in `audit.log`. Encrypt details when available. | ✅ Sync (fire-and-queue) | | **clearLog(collection)** | Remove `.log` and close any pooled FD for that file. Leave archives intact. | ✅ Sync | | **syncLogs()** | Drain chains, flush all active file handles/FDs, close them. | ⚙️ Async (`Promise`) | | **shutdown()** | Drain chains and close all pooled FDs/handles during controlled teardown. | ⚙️ Async (`Promise`) | --- ### 📊 Query Metrics StarVault collects query-level execution metrics through **StarLogger** by default. #### `queryMetrics` Object | Key | Type | Description | Default | |-----|------|-------------|---------| | `enabled` | boolean | Toggle query metrics collection. | `true` | | `maxPending` | number | Maximum queued metric entries before triggering a flush. | `5000` | | `maxSources` | number | Maximum distinct query sources tracked concurrently. | `250` | | `sampleHitRate` | number | Sampling rate between `0` and `1`. | `1.0` | | `serviceName` | string | Optional service identifier attached to metric entries. | `null` | #### Example Configuration ```js const vault = new StarVault("./data", "./logs", 4, "869MB", "1w", { queryMetrics : { enabled : true, maxPending : 5000, maxSources : 250, sampleHitRate : 0.5, serviceName : "starvault-api" } }); ``` To disable metrics entirely: ```js const vault = new StarVault("./data", "./logs", 4, "869MB", "1w", { queryMetrics : false }); ``` > Metric entries flush automatically during normal operation, pressure thresholds, and shutdown signals. --- ### Write, System, and Audit Events ```js // Collection write log (JSONL rows with checksum) vault.wal.logWrite( "users", { id : "u-1", name : "Nova" }, { userID : "system" }, "CREATE", { transactionID : null, idempotencyKey : null, recordID : "u-1" } ); // System trail (optionally encrypted via setEncryptor) vault.wal.logSystemAction( "TX-COMMIT", "users", { id : "u-1" }, { userID : "svc-1" }, { transactionID : "tx-42", idempotencyKey : null, keyVersion : 1 } ); // Audit trail (reads / admin ops, etc.) vault.wal.logAudit( "READ_PROFILE", { id : "u-1" }, { userID : "admin-7" }, { idempotencyKey : null, transactionID : null, keyVersion : 1 } ); ``` --- ### Recovery, Query & Maintenance ```js // Validate JSONL lines by checksum and return good entries const recovered = vault.wal.recoverWrites("users"); // Hard-delete the *live* log file (archived files remain) vault.wal.clearLog("users"); // Full history (live + archived + optional system.log) const history = vault.wal.getRecordHistory("users", "u-1", { includeArchived : true, verifyChecksums : true, includeSystem : true, includeDecrypted : true, actions : ["CREATE","UPDATE","DELETE"], since : "2025-01-01", until : Date.now(), match : (row) => row.client?.userID !== "internal" }); // Timeline-friendly projection const timeline = vault.wal.getRecordTimeline("users", "u-1"); // Collection-level history const collectionHistory = vault.wal.getCollectionHistory("users/region1", { includeArchived : true }); // Vault-wide summarized timeline const vaultTimeline = vault.wal.getVaultTimeline({ actions : ["CREATE", "UPDATE"] }); // Durability sync (flush + handle sync + close) await vault.wal.syncLogs(); // Graceful shutdown (drains queues and closes resources) await vault.wal.shutdown(); ``` --- ### Rotation & Retention (built-in) - Rotation triggers when projected size passes `maxLogSizeBytes`. - Cross-process `.rotate.lock` maintains safe, serialized rotation steps. - FD closes before rename; a fresh FD opens immediately for new entries. - Retention removes files older than `retention`, skipping any file currently in active use. --- ## 📂 Starchive (Storage Layer) Starchive provides a **unified storage abstraction** for both local filesystem and remote S3-compatible backends. You can configure it for **local only**, **remote only**, or a **hybrid** (both). --- ### ⚙️ Starchive Configuration Modes | Mode | Description | Example | |----------|-------------|---------| | **Remote (S3)** | Store files in any S3-compatible backend (AWS, MinIO, Wasabi, etc.). | `starchive: { local: false, remote: { type: "s3", bucket: "mybucket", clientConfig: { region: "us-east-1" } }, defaultTarget: "remote" }` | | **Local (Filesystem)** | Store files locally on disk under a managed `.starchive` directory. | `starchive: { local: { root: path.join(__dirname, "data", ".starchive") }, remote: false, defaultTarget: "local" }` | | **Hybrid** | Support both local + S3; choose per call via `{ target: "local" }` or `{ target: "remote" }`. | `starchive: { local: { root: path.join(__dirname, "data", ".starchive") }, remote: { type: "s3", bucket: "mybucket", clientConfig: { region: "us-east-1" } }, defaultTarget: "remote" }` | --- ### ⚙️ Starchive Configurations | Key | Type | Description | Example / Default | |------------------|-----------|-----------------------------------------------------------------------------|---------------------------| | `collection` | `string` \| `"starchive.files"` | The collection name you desire holding Starchive file records. | `"starchive.files"` | | `local` | `object` \| `false` | Local adapter configuration or `false` to disable. | `{ root: "./data/.starchive" }` | | `remote` | `object` \| `false` | Remote adapter config (`type: "s3"`, `bucket`, `clientConfig`). | `{ type: "s3", bucket: "mybucket", clientConfig: { region: "us-east-1" } }` | | `defaultTarget` | `"local"` \| `"remote"` | Which backend to use if not explicitly specified. | `"local"` | | `remoteDefaultAlias` | `string` \| `null` | Default **remote alias** when using `target: "remote"` and no alias is supplied. | First configured alias | **Remote `clientConfig` fields** (S3-compatible): | Key | Type | Description | Example | |------------------|----------|------------------------------------------------|---------| | `region` | string | AWS region. | `"us-east-1"` | | `endpoint` | string | S3 endpoint override. | `"https://s3.us-east-1.amazonaws.com"` | | `accessKeyID` | string | Access key ID. | `"..."` | | `secretAccessKey`| string | Secret key. | `"..."` | | `forcePathStyle` | boolean | Use path-style URLs instead of virtual-hosted. | `true` | #### 🔐 Remote (S3) starchive: { local: false, remote: { type: "s3", bucket: "mybucket", clientConfig: { secretAccessKey: ".../...", accessKeyID: "...", endpoint: "https://s3.us-east-1.amazonaws.com", region: "us-east-...", forcePathStyle: true } }, defaultTarget: "remote" } #### Multi-Remote Config Options You can express `remote` as: 1) **Object map of aliases → config** starchive: { local: false, remote: { mediaCDN: { type: "s3", bucket: "media-prod", clientConfig: { region: "us-east-1" } }, logs: { type: "s3", bucket: "app-logs", clientConfig: { region: "us-west-2" } } }, defaultTarget: "remote", remoteDefaultAlias: "mediaCDN" // optional } 2) **Array of remotes** (alias inferred from `id`/`alias`/`bucket`) starchive: { local: false, remote: [ { id: "mediaCDN", type: "s3", bucket: "media-prod", clientConfig: { region: "us-east-1" } }, { alias: "logs", type: "s3", bucket: "app-logs", clientConfig: { region: "us-west-2" } } ], defaultTarget: "remote", remoteDefaultAlias: "mediaCDN" // optional } > ℹ️ Per call, override the backend with `{ target: "local" }` or `{ target: "remote", remoteAlias?: "alias", bucket?: "override-bucket" }`. #### 💻 Local (Filesystem) starchive: { local: { root: path.join(__dirname, "data", ".starchive") }, remote: false, defaultTarget: "local" } #### 🌐 Hybrid (Local + Remote) starchive: { local: { root: path.join(__dirname, "data", ".starchive") }, remote: { type: "s3", bucket: "mybucket", clientConfig: { region: "us-east-1" } }, defaultTarget: "remote" // default is S3; override per call with { target: "local" } } > ℹ️ Override the default per call with `{ target: "local" }` or `{ target: "remote" }`. --- ### 📦 Starchive Core Methods | Method | Description | Sync/Async | |---|---|---| | `put({ key, data, target?, remoteAlias?, bucket?, prefix?, absolute?, contentType?, filename?, tags?, encrypt?, extra? }, actionInfo?, clientAuth?)` | Upload a buffer/stream/string. Records metadata (incl. `{ provider, key, bucket, alias }` for S3). | ⚙️ Async | | `putFile({ key, path, target?, remoteAlias?, bucket?, prefix?, absolute?, contentType?, filename?, tags?, encrypt?, extra? }, actionInfo?, clientAuth?)` | Upload from a local file path. | ⚙️ Async | | `putMany(entries, { target?, remoteAlias?, bucket?, concurrency?, prefix?, absolute? }, actionInfo?, clientAuth?)` | Batch upload buffers/files. Returns `{ complete, incomplete, records, total }`. | ⚙️ Async | | `getStream(key, { target?, remoteAlias?, bucket?, prefix?, absolute? })` | Get an object as a readable stream. | ⚙️ Async | | `getStreamRange(key, start, end, { target?, remoteAlias?, bucket?, prefix?, absolute? })` | Stream a byte range of an object. | ⚙️ Async | | `head(key, { target?, remoteAlias?, bucket?, prefix?, absolute? })` | Get object headers/metadata. | ⚙️ Async | | `exists(key, { target?, remoteAlias?, bucket?, prefix?, absolute? })` | Check if an object exists. | ⚙️ Async | | `remove(key, { target?, remoteAlias?, bucket?, record?, reason?, prefix?, absolute?, actionInfo?, clientAuth? })` | Delete an object; optionally soft/hard-delete matching records. | ⚙️ Async | | `removeMany(keys, { target?, remoteAlias?, bucket?, record?, reason?, prefix?, absolute?, actionInfo?, clientAuth? })` | Delete multiple objects; optionally soft/hard-delete records. | ⚙️ Async | | `copy(srcKey, dstKey, { target?, remoteAlias?, bucket?, prefix?, absolute? })` | Copy an object. | ⚙️ Async | | `move(srcKey, dstKey, { target?, remoteAlias?, bucket?, prefix?, absolute? })` | Move (copy+delete) an object. | ⚙️ Async | | `list({ target?, remoteAlias?, bucket?, prefix?, maxKeys?, continuationToken?, startAfter?, timeoutMS? })` | List objects with pagination. | ⚙️ Async | | `listAll({ target?, remoteAlias?, bucket?, prefix?, pageSize?, timeoutMS? })` | List **all** objects under a prefix. | ⚙️ Async | | `walk({ target?, remoteAlias?, bucket?, prefix? })` | Async generator to recursively walk files/prefixes. | ⚙️ Async | --- ### ✍️ Starchive Convenience Methods | Method | Description | Sync/Async | |---|---|---| | `write(key, data, { target?, remoteAlias?, bucket?, prefix?, absolute?, contentType?, filename?, tags?, extra?, actionInfo?, clientAuth? })` | Wrapper over `put` for buffers/strings; infers filename. | ⚙️ Async | | `writeText(key, text, opts?)` | Write UTF-8 text (content type set automatically). | ⚙️ Async | | `writeJSON(key, obj, { pretty?, ...opts })` | Serialize and write JSON. | ⚙️ Async | | `read(key, opts?)` | Read entire object into a `Buffer`. | ⚙️ Async | | `readText(key, opts?)` | Read object as UTF-8 string. | ⚙️ Async | | `readJSON(key, opts?)` | Read and parse JSON (returns `null` on empty/invalid JSON). | ⚙️ Async | | `stat(key, opts?)` | Returns `{ size, lastModified, contentType }` derived from `head`. | ⚙️ Async | --- ### 🛰️ Remote Management Helpers | Method | Description | |---|---| | `listRemotes(opts?)` | Returns the configured remotes. Pass `{ asObject: true }` to get an object keyed by alias. | | `getRemoteDefaultAlias()` | Get the current default remote alias. | | `setRemoteDefaultAlias(alias)` | Set/switch the default remote alias at runtime. | --- ### 🔏 Starchive Signing & Presigning | Method | Description | Sync/Async | |---|---|---| | `sign.getDetails(fileID, { method = "GET", expiresInMs = 900000 })` | Create an internal HMAC signature payload `{ fileID, method, exp, sig }`. | ✅ Sync | | `sign.verify({ id, m, e, s })` | Verify payload against the current signing key and expiry. | ✅ Sync | | `presign.get(key, { target?, remoteAlias?, bucket?, ttlSeconds?, responseContentType?, responseContentDisposition?, timeoutMS?, prefix?, absolute? })` | Generate S3 presigned **GET** URL (remote only). | ⚙️ Async | | `presign.put(key, { target?, remoteAlias?, bucket?, ttlSeconds?, contentType?, cacheControl?, contentDisposition?, tagging?, timeoutMS?, prefix?, absolute? })` | Generate S3 presigned **PUT** URL (remote only). | ⚙️ Async | | `presign.getURL(key, opts?)` | Alias of `presign.get`. | ⚙️ Async | | `presign.putURL(key, opts?)` | Alias of `presign.put`. | ⚙️ Async | --- ### 🔏 Starchive Downloading | Method | Description | Sync/Async | |---|---|---| | `download.buffer(key, { target?, remoteAlias?, bucket?, timeoutMS? })` | Download file to `Buffer`. | ⚙️ Async | | `download.file(key, outPath, { target?, remoteAlias?, bucket?, timeoutMS? })` | Download file to disk. | ⚙️ Async | --- ### 📜 Starchive Records | Method | Description | Sync/Async | |---|---|---| | `records.getByID(id, actionInfo?)` | Fetch the stored file record by ID. | ✅ Sync | | `records.update(id, updates, actionInfo?, clientAuth?)` | Update metadata fields on a file record. | ⚙️ Async | | `records.deleteByID(id, { softDelete?, reason? }, actionInfo?, clientAuth?)` | Soft or hard delete a record by ID. | ⚙️ Async | | `records.deleteByQuery(queryFn, { softDelete?, reason? }, actionInfo?, clientAuth?)` | Delete/soft-delete many records matching a predicate. | ⚙️ Async | --- ### 📥 Import & Snapshot Methods Star-Vault exposes a family of **import helpers** for seeding, restoring, and migrating data into the vault from existing snapshots, JSON dumps, or external systems. These APIs: - Respect **auth** (`clientAuth`) and **actionInfo** (for auditing / history) - Integrate with **WAL**, **StarCache**, and the **indexer** - Emit **events** (`import`) and can trigger **auto-vacuum** --- ### 🌌 Import Methods Overview | Method | Description | Sync/Async | |--------|-------------|------------| | `importRecord(collection, record, options = {}, actionInfo = {}, clientAuth = null)` | Import a **single record** (with `id`, optional `timestamp`, `lastModified`, and `data`) into a collection. | Sync | | `importManyRecords(collection, records, options = {}, actionInfo = {}, clientAuth = null)` | Import **multiple records** into a collection and return a summary. | Sync | | `importCollection(collection, records, options = {}, actionInfo = {}, clientAuth = null)` | Import an **entire collection** from a list of records, optionally truncating first. | Sync | | `importStateSnapshot(snapshot, options = {}, actionInfo = {}, clientAuth = null)` | Import a **full vault snapshot** across many collections. | Sync | --- ### `importRecord(collection, record, options?, actionInfo?, clientAuth?)` Imports a **single record** into a collection. ```js const result = vault.importRecord("users", { id : "u-1", data : { name : "Nova", email : "nova@example.com" }, timestamp : 1730000000000, lastModified : 1730000000000 }, { overwrite : false }, { actor : "seed-script" }, { token : "admin-token" }); ``` #### Parameters - `collection : string` - `record : { id, data?, timestamp?, lastModified? }` - `options : { overwrite? : boolean }` - `actionInfo : object` - `clientAuth : any` #### Behavior - Validates record and ID. - If exists and `overwrite === false` → skip. - Normalizes timestamps. - Writes through storage layer + WAL. - Updates caches and indexer. - Emits `import` for `collection/id`. #### Returns ```js { id, imported, overwritten, reason?, record? } ``` --- ### `importManyRecords(collection, records, options?, actionInfo?, clientAuth?)` Bulk-import wrapper over `importRecord`. ```js const summary = vault.importManyRecords("users", [ { id : "u-1", data : { name : "Nova" } }, { id : "u-2", data : { name : "Orion" } } ], { overwrite : false }); ``` #### Returns ```js { collection, total, imported, skippedExisting, results : [...] } ``` --- ### `importCollection(collection, records, options?, actionInfo?, clientAuth?)` Imports a whole collection and optionally **truncates** first. ```js const result = vault.importCollection("users", [ { id : "u-1", data : { name : "Nova" } }, { id : "u-2", data : { name : "Orion" } } ], { overwrite : true, truncateFirst : true }); ``` #### Returns ```js { collection, total, imported, skippedExisting, truncated, results : [...] } ``` --- ### `importStateSnapshot(snapshot, options?, actionInfo?, clientAuth?)` Imports a **multi-collection** JSON snapshot. ```js const snapshot = { users : [ { id : "u-1", data : { name : "Nova" } } ], orders : [ { id : "o-1", data : { userID : "u-1", total : 99.99 } } ] }; const result = vault.importStateSnapshot(snapshot, { overwrite : true, truncateCollections : true }); ``` #### Options | Option | Type | Description | |--------|-------|-------------| | `overwrite` | boolean | Overwrite existing records | | `truncateCollections` | boolean | Truncate each collection before import | | `collections` | string[] \| null | Restrict import to these collections; default all | #### Returns ```js { overwrite, truncateCollections, collections : [ { collection, total, imported, skippedExisting, truncated, results : [...] } ] } ``` --- ### Example: Import from Backup File ```js const fs = require("node:fs"); const raw = fs.readFileSync("./backup/star-vault-snapshot.json", "utf8"); const snapshot = JSON.parse(raw); const result = vault.importStateSnapshot(snapshot, { overwrite : true, truncateCollections : true }, { actor : "restore-from-backup" }, { token : "root-admin-token" }); ``` --- ## 🎧 Event Listening Used for reactive, event-driven architecture patterns. | Method | Description | Sync/Async | |--------|-------------|------------| | `listen(event, nodePath, callback)` | Registers a callback for a specific event (`create`, `update`, `delete`) at a given path. | ✅ Sync | | `emit(event, nodePath, data)` | Triggers callbacks for listeners whose paths match the emitted path. | ✅ Sync | --- ## 🌌 Collection & Path Matching Utilities for dynamic collection operations and wildcard paths. | Method | Description | Sync/Async | |--------|-------------|------------| | `getMatchingCollections(basePath)` | Resolves wildcard-based paths like `logs/*/2025` to real collection names. | ✅ Sync | | `matchesPath(listenerPath, nodePath)` | Checks if an emitted node path matches a listener's wildcard path. | ✅ Sync | --- ## 🌌 StarVault Events StarVault emits real-time events that let you react to data and authentication lifecycle changes across your app. These include both **record-level events** and **auth lifecycle events**. You can listen to them at multiple levels of granularity: - 🔁 **Global listeners** (across all collections and records) - 📁 **Collection-level listeners** (any record in a collection) - 🧾 **Record-specific listeners** (a specific record path) --- ### 🔔 Emitted Record Events (StarVault Core) | Event | Description | Emitted Path | |----------|-----------------------------------------|--------------------------| | `create` | Fires on record creation. | `collectionName/recordID` | | `update` | An existing record updated | `collectionName/recordID` | | `delete` | A record got deleted (soft or hard) | `collectionName/recordID` | --- ### 🔐 Emitted Auth Lifecycle Events (StarAuth) | Event | Description | |---------------------|-----------------------------------------------------------| | `guest:upgraded` | A guest account successfully upgraded to a full user | | `guest:updated` | A guest account successfully updated | | `user:updated` | A user account successfully updated | | `session:resumed` | A user session resumed from a valid token | | `user:guestExpired` | A guest account expired due to inactivity | | `user:deleted` | A user account got marked for deletion | | `user:reactivated` | A previously deleted user got reactivated | --- ### 🔔 StarTransaction Lifecycle Events | Event | Description | Payload | |-------|-------------|----------| | `tx:commit` | Fires after transaction commit and data application. | `{ count, startedAt, endedAt }` | | `tx:abort` | Fires when transaction aborts manually or through error. | `{ reason }` | | `tx:summary` | Fires after a transaction successfully commits and finalizes all operations. Ideal for analytics, dashboards, or external monitoring. | `{ transactionID, ops, startedAt, endedAt }` | --- ### 🪐 Global Listener (All Changes Across Vault) ```js vault.listen("create", "*", (path, data) => { console.log("Created:", path, data); }); vault.listen("update", "*", (path, data) => { console.log("Updated:", path, data); }); vault.listen("delete", "*", (path, data) => { console.log("Deleted:", path, data); }); ``` --- ### 📂 Collection-Level Listener ```js vault.listen("create", "users/*", (path, data) => { console.log("User created:", path, data); }); vault.listen("update", "inventory/*", (path, data) => { console.log("Inventory updated:", path, data); }); ``` --- ### 🧍 Record-Specific Listener ```js vault.listen("update", "orders/12345", (path, data) => { console.log("Order updated:", data); }); vault.listen("delete", "accounts/user-abc", (path, data) => { console.log("User deleted:", data); }); ``` --- ### 🎯 Advanced Wildcard Usage ```js // Match nested logs by year vault.listen("update", "logs/*/2025/*", (path, data) => { console.log("Log updated:", path); }); ``` --- ### 🧠 Listening to StarAuth Lifecycle Events Use `.on()` to subscribe to StarAuth-specific events: ```js vault.on("guest:upgraded", (info) => { console.log("Guest upgraded:", info); }); vault.on("user:deleted", (info) => { console.log("User deleted:", info); }); vault.on("user:reactivated", (info) => { console.log("User reactivated:", info); }); ``` --- ### 🔔 Listening to StarTransaction Lifecycle Events ```js vault.listen("tx:commit", "*", (id, info) => { console.log("Committed txn:", id, info); }); vault.listen("tx:abort", "*", (id, info) => { console.log("Aborted txn:", id, info); }); vault.on("tx:summary", (transactionID, summary) => { console.log("Transaction Summary:", transactionID, summary); }); ``` --- These event hooks power real-time dashboards, activity tracking, notifications, and custom automation — all deeply integrated with StarVault's reactive system. --- ## 🧽 Vacuum & Auto-Maintenance System StarVault continuously tracks file growth, update frequency, and live-data ratios. Vacuum operations compact on-disk collections or shards to reclaim space, reduce fragmentation, and refresh metrics. --- ### ⚙️ Automatic Vacuum Policy The vacuum policy defines when StarVault triggers compaction automatically. You can inspect or adjust it using `getVacuumPolicy()` and `setVacuumPolicy()`. | **Property** | **Description** | **Default** | |---------------|-----------------|--------------| | `minUpdates` | Minimum number of updates per collection before eligibility. | `500` | | `liveRatioThreshold` | Threshold for live-to-total bytes ratio below which vacuuming starts. | `0.55` | | `minBytes` | Minimum file size (in bytes) before considering compaction. | `33554432` *(≈32 MB)* | | `minIntervalMs` | Minimum time interval (in ms) between consecutive vacuums. | `600000` *(10 minutes)* | | `doubleWrite` | When true, perform safe dual-file write-swap during compaction. | `true` | | `perRecordMaxUpdates` | Number of writes to a single record before triggering a targeted vacuum. | `2` | | `perRecordAction` | Action scope when per-record threshold exceeds limit (`"shard"` or `"collection"`). | `"shard"` | --- ### 🪄 Methods | **Method** | **Purpose** | |-------------|-------------| | `setVacuumPolicy(updates)` | Update one or more vacuum parameters dynamically. | | `getVacuumPolicy()` | Retrieve the current active vacuum policy. | | `vacuumCollection(collection, opts, actionInfo?, clientAuth?)` | Compact all shards in a given collection atomically. | | `vacuumShard(collection, shard, opts, actionInfo?, clientAuth?)` | Compact a specific shard for finer control. | | `vacuumAll(opts, actionInfo?)` | Compact all collections in the vault (global vacuum). | | `vacuumIfNeeded(collection, reason, actionInfo?, clientAuth?)` | Evaluate policy thresholds and perform vacuum automatically if required. | | `shouldVacuum(collection)` | Check whether a collection qualifies for compaction under current thresholds. | | `vacuumStats()` | Retrieve summarized live metrics for all collections (fileBytes, liveBytes, updates, lastVacuumAt). | --- ### 🪶 Example ```js // Adjust thresholds vault.setVacuumPolicy({ minUpdates : 1000, liveRatioThreshold : 0.40, perRecordMaxUpdates : 3 }); // Trigger manual vacuum for one collection const result = vault.vacuumCollection("users"); console.log(result); // View metrics and totals console.table(vault.vacuumStats()); ``` --- ### ⚡ Auto-Vacuum Triggers StarVault automatically calls `#maybeAutoVacuum()` whenever records update, delete, or restore. When thresholds exceed limits, it triggers background compaction depending on `perRecordAction`: - `"collection"` → rebuilds the entire collection - `"shard"` → compacts only the affected shard Each automatic run updates metrics in `.meta/metrics.json` and writes WAL entries (`VACUUM_BEGIN`, `VACUUM_SWAP`, `VACUUM_END`) for full traceability. --- ### 📊 Metrics Example (via `vacuumStats()`) ```json { "collections": { "users": { "fileBytes": 43219456, "liveBytes": 38453212, "updates": 913, "lastVacuumAt": 1730104001123 } }, "total": { "fileBytes": 43219456, "liveBytes": 38453212, "updates": 913 } } ``` --- ## ✨ Getting Started ### 📦 Installation ```bash npm install @trap_stevo/star-vault ``` ### Basic Usage ```js const StarVault = require("@trap_stevo/star-vault"); const vault = new StarVault( "./data", "./logs", 4, "869MB", "1w", { enableEncryption : true, masterKey : "supersecretkey", authHandler : (auth) => auth.token === "valid-token" } ); vault.create("users", { id : "001", name : "Nova" }, { source : "init" }, { token : "valid-token" }); ``` --- ## 📈 Querying ```js const results = vault.query("users") .where({ role : "admin" }) .sort({ name : 1 }) .select(["id", "name"]) .limit(10) .execute(); ``` --- ## 🎧 Listening to Changes ```js vault.listen("create", "*", (path, data) => { console.log("[CREATE - ANY] Path:", path, data); }); vault.listen("create", "users/*", (path, data) => { console.log("[CREATE - users] Path:", path, data); }); vault.listen("update", "users/*", (path, data) => { console.log("[UPDATE - users] Path:", path, data); }); // Listen for updates to a specific user record (e.g., user ID '001') vault.listen("update", "users/user-001", (path, data) => { console.log("[UPDATE - users/user-001] Path:", path, data); }); vault.listen("delete", "users/*", (path, data) => { console.log("[DELETE - users] Path:", path, data); }); vault.listen("delete", "logs/*/2025", (path, data) => { console.log("[DELETE - logs/*/2025] Path:", path, data); }); // Listen for a specific log record inside a structured path (e.g., logs/server1/2025/log-99) vault.listen("create", "logs/server1/2025/log-99", (path, data) => { console.log("[CREATE - logs/server1/2025/log-99] Path:", path, data); }); // Listen to everything happening in 'auth-sessions' vault.listen("create", "auth-sessions", (path, data) => { console.log("[CREATE - auth-sessions] Path:", path, data); }); vault.listen("update", "auth-sessions", (path, data) => { console.log("[UPDATE - auth-sessions] Path:", path, data); }); vault.listen("delete", "auth-sessions", (path, data) => { console.log("[DELETE - auth-sessions] Path:", path, data); }); ``` --- ## 🌐 Wildcard Collection Queries ```js vault.query("logs/*/2025") .recent("timestamp", "7d") .execute(); ``` --- ## 👥 Registering Users (Guest and Normal) ```js const StarVault = require("@trap_stevo/star-vault"); const vault = new StarVault( "./data", "./logs", 4, "869MB", "1w", { enableEncryption : true, masterKey : "supersecretkey", authHandler : (auth) => auth.token === "star-vault-demo", authOptions : { allowGuestSessions : true, generateGuestID : () => "guest-" + Math.floor(Math.random() * 1000000) } } ); // Register a full user const userResult = await vault.registerUser("nova@example.com", "strongpassword123", { source : "site" }, { token : "star-vault-demo" }); console.log("Registered user:", userResult); // Register a guest user const guestResult = await vault.registerGuestUser({ ip : "127.0.0.1" }, { referrer : "home" }, { source : "guest-form" }, { token : "star-vault-demo" }); console.log("Registered guest:", guestResult); ``` --- ## 📋 Listing Users and Sessions ```js const StarVault = require("@trap_stevo/star-vault"); const vault = new StarVault( "./data", "./logs", 4, "869MB", "1w", { enableEncryption : true, masterKey : "supersecretkey", authHandler : (auth) => auth.token === "star-vault-demo" } ); // List all users const users = await vault.listUsers({}, { limit : 25 }, { token : "star-vault-demo" }); console.log("All users:", users); // List all sessions (including inactive and deleted) const sessions = await vault.listAllSessions({ includeDeleted : "only" }, true, { token : "star-vault-demo" }); console.log("All sessions:", sessions); // List sessions for a specific user const userSessions = await vault.listSessions("user-001", { device : "mobile" }, false, { token : "star-vault-demo" }); console.log("Sessions for user-001:", userSessions); // List recent user activity (active and inactive) const activity = await vault.listUserActivity({ includeInactive : true, onlyMostRecent : true }, { token : "star-vault-demo" }); console.log("User activity:", activity); ``` --- ### 🔍 Quick Starchive Examples **1. Store & Retrieve a File** await vault.starchive.putFile({ key: "uploads/photo.png", path: "./photo.png", target: "remote" }); const exists = await vault.starchive.exists("uploads/photo.png"); console.log("Uploaded?", exists); **2. Write and Read JSON** await vault.starchive.writeJSON("configs/app.json", { theme: "dark" }); const cfg = await vault.starchive.readJSON("configs/app.json"); console.log("Loaded config:", cfg); **3. Batch Upload (Buffers + Files)** const results = await vault.starchive.putMany([ { key: "docs/a.txt", buffer: Buffer.from("Alpha") }, { key: "docs/b.txt", path: "./local-b.txt" } ], { target: "remote" }); console.log("Completed:", results.complete.length); **4. Presigned Upload (Client-Side)** const url = await vault.starchive.presign.put("uploads/user.txt", { contentType: "text/plain", ttlSeconds: 300 }); console.log("Upload directly to:", url); **5. Streaming Download** const stream = await vault.starchive.getStream("uploads/photo.png"); stream.pipe(fs.createWriteStream("./downloaded.png")); **6. Range Request (Video Streaming)** const part = await vault.starchive.getStreamRange("videos/movie.mp4", 0, 1024*1024); part.stream.pipe(fs.createWriteStream("./clip.mp4")); **7. Directory Walk** for await (const node of vault.starchive.walk({ prefix: "reports/2025/" })) { console.log(node.type, node.key); } **8. Metadata & Stat** const info = await vault.starchive.stat("uploads/photo.png"); console.log(info.size, info.lastModified, info.contentType); **9. Record Management** const rec = await vault.starchive.records.getByID("file-123"); await vault.starchive.records.updateFields("file-123", { tags: ["archive"] }); await vault.starchive.records.deleteByID("file-123", { softDelete: true }); **10. Copy & Move** await vault.starchive.copy("docs/a.txt", "docs/a-copy.txt"); await vault.starchive.move("docs/b.txt", "docs/b-archived.txt"); --- ## ✨ License [SCL-1.0-Universal](./SCL-1.0-Universal.md) --- ## 🚀 Transform Data into Action Star-Vault transcends traditional data systems—offering a full-fledged database engine and cosmic foundation that propels secure, reactive, and intelligent data-driven architectures.