# sflow
### Stream Flow for TypeScript/JavaScript 🚀
[](https://www.npmjs.com/package/sflow)
[](https://www.npmjs.com/package/sflow)
[](https://github.com/snomiao/sflow/blob/main/LICENSE)
[](https://www.typescriptlang.org/)
A powerful and highly-extensible library for processing and manipulating streams of data effortlessly based on WebStreams API.
**[📖 Documentation](https://deepwiki.com/snomiao/sflow)** • **[🎮 Online Demo](https://sflow-examples.vercel.app/)** • **[💬 Discussions](https://github.com/snomiao/sflow/discussions/2)**
---
## 📋 Table of Contents
- [Why sflow?](#-why-sflow)
- [Features](#-features)
- [Installation](#-installation)
- [Quick Start](#-quick-start)
- [API Overview](#-api-overview)
- [Initialization](#initialization)
- [Transformations](#transformations)
- [Chunking, Buffering, and Grouping](#chunking-buffering-and-grouping)
- [Advanced Utilities](#advanced-utilities)
- [Type-Safe Enhancements](#type-safe-enhancements)
- [Agent Skill](#-agent-skill-claude-code--codex--cursor)
- [Contributing](#-contributing)
- [References](#-references)
- [License](#-license)
---
## ✨ Why sflow?
sflow is **built for async-first pipelines** — every pipeline method accepts async functions, making it perfect for web applications that handle async I/O operations and need to manage concurrency efficiently. Inspired by functional programming paradigms, it provides a rich set of utilities for transforming streams: chunking, filtering, mapping, reducing, and many more. It's the perfect companion for developers who work extensively with streams and want to make their data processing pipelines more efficient and concise.
## 🎯 Features
- **⚡ Async-first design** — Every pipeline method accepts async functions, enabling seamless async I/O operations and concurrency management for web applications.
- **📦 Chunking and buffering** — Easily divide your stream into chunks based on different criteria such as count, intervals, custom conditions, etc.
- **🔄 Transformations** — Map, filter, reduce, and various other transformations to process your stream data.
- **🛠️ Stream utilities** — Merge, throttle, debounce, and more utilities for advanced stream controls.
- **🛡️ Error handling** — Prevent or handle errors during stream processing, ensuring robustness.
- **🔌 Integration-ready** — Seamlessly integrates with tools like `web-streams-extensions`, making it versatile for different streaming needs.
- **📘 TypeScript support** — Fully typed for a richer developer experience and better code quality.
## 📦 Installation
```bash
# AI coding agent skill (Claude Code / Codex / Cursor)
# install to project scope
bunx skills add snomiao/sflow -y -a
# install to global scope
bunx skills add snomiao/sflow -y -a -g
# npm
npm install sflow
# bun
bun add sflow
```
## 🚀 Quick Start
### Basic Usage
Here's a simple example of how to use sflow to process a stream:
```typescript
import { sflow } from "sflow";
async function run() {
let result = await sflow([1, 2, 3, 4])
.map((n) => n * 2)
.log() // this stage prints 2, 4, 6, 8
.filter((n) => n > 4)
.log() // this stage prints 6, 8
.reduce((a, b) => a + b, 0) // first emit 0+6=6, second emit 0+6+8=14
.log() // this stage prints 6, 14
.toArray();
console.log(result); // Outputs: [6, 14]
}
await run();
```
### Async Pipeline Example
sflow excels at handling async operations in pipelines - perfect for web applications:
```typescript
import { sflow } from "sflow";
// Example: Fetch user data and process concurrently
async function processUsers() {
const result = await sflow([1, 2, 3, 4, 5])
// Every method accepts async functions!
.map(async (userId) => {
const response = await fetch(`https://api.example.com/users/${userId}`);
return response.json();
})
.filter(async (user) => {
// Async filtering for complex checks
const isActive = await checkUserStatus(user.id);
return isActive;
})
.map(async (user) => {
// Transform with async operations
const profile = await enrichUserProfile(user);
return profile;
})
.toArray();
console.log(result); // Array of processed user profiles
}
async function checkUserStatus(id: number) {
// Simulate async check
return id % 2 === 0;
}
async function enrichUserProfile(user: any) {
// Simulate async enrichment
return { ...user, enriched: true };
}
```
## 📚 API Overview
### Initialization
Initialize a flow from various types of data sources:
```typescript
import { sflow } from "sflow";
// From an array
const flow1 = sflow([1, 2, 3, 4]);
// From a promise
const flow2 = sflow(Promise.resolve([1, 2, 3, 4]));
// From an async iterable
async function* asyncGenerator() {
yield 1;
yield 2;
yield 3;
}
const flow3 = sflow(asyncGenerator());
```
### Transformations
Transform your flow with various transformation methods. **All methods support async functions:**
```typescript
// Synchronous mapping
flow1.map((n) => n * 2);
// Async mapping - great for API calls, database queries, etc.
flow1.map(async (n) => {
const data = await fetchData(n);
return data.value * 2;
});
// Synchronous filtering
flow1.filter((n) => n % 2 === 0);
// Async filtering - perfect for complex validation
flow1.filter(async (n) => {
const isValid = await validateAsync(n);
return isValid;
});
// Async reducing
flow1.reduce(async (a, b) => {
const result = await computeAsync(a, b);
return result;
}, 0);
```
### Chunking, Buffering, and Grouping
sflow provides methods for chunking, buffering, and grouping data:
```typescript
// Chunking by count
flow1.chunk(2); // [[1, 2], [3, 4]]
// Buffering within a time interval
flow1.chunkByInterval(1000);
// Custom chunking
flow1.chunkBy((x) => Math.floor(x / 2));
```
### Advanced Utilities
sflow comes with a plethora of utilities to manipulate streams efficiently:
```typescript
// Throttling
flow1.throttle(100);
// Debouncing
flow1.debounce(200);
// Converting to array
flow1.toArray();
// Merging multiple streams
const mergedFlow = sflow([flow1, flow2]).merge();
// Use chunkIf to split tokens by line
await sflow("a,b,c\n\n1,2,3\nd,s,f".split(""))
.through(chunkIfs((e: string) => e.indexOf("\n") === -1))
.map((chars) => chars.join(""))
.toArray(); // ["a,b,c\n",'\n', "1,2,3\n", "d,s,f"]
```
### Type-Safe Enhancements
With TypeScript, sflow ensures your transformations are type-safe:
```typescript
import { sflow } from "sflow";
const typedFlow = sflow([{ a: 1, b: [1, 2, 3] }])
.unwind("b") // Use `unwind` for objects with nested arrays
.mapAddField("newField", (item) => item.a + item.b);
```
## 🤖 Agent Skill (Claude Code / Codex / Cursor)
sflow ships an **Agent Skill** so AI coding assistants can use it fluently in your project.
Install it with one command:
```sh
npx skills add https://github.com/snomiao/sflow --skill sflow
```
This copies the skill into your project's `.claude/skills/sflow/` (or equivalent) so Claude Code, Codex CLI, Cursor, and other compatible agents automatically understand how to write sflow pipelines — with the right API, patterns, and idioms.
The skill includes:
- `SKILL.md` — core API reference and patterns (loaded automatically when relevant)
- `examples.md` — 10 real-world scenarios: API fetching, CSV processing, real-time events, parallel processing, object transformation, stream merging, batch processing, text streams, rate limiting, and more
> Requires [skills CLI](https://github.com/vercel-labs/skills): `npm i -g skills` (or just use `npx`).
## 🤝 Contributing
Contributions to sflow are always welcome! We appreciate your help in making sflow better.
### How to Contribute
1. **🐛 Report bugs or suggest features** — [Open an issue](https://github.com/snomiao/sflow/issues) on GitHub
2. **💻 Submit pull requests** — Fork the repo and create your first PR from [GitHub.dev](https://github.dev/snomiao/sflow)
3. **💬 Join the community** — Share your thoughts and ask questions in [Discussions](https://github.com/snomiao/sflow/discussions/2)
## 📚 References
- [Infinite Streams with Elixir](https://gist.github.com/mgwidmann/5e0cb590f12e2ca239564d07d7c2a572)
- [web-streams-extensions on npm](https://www.npmjs.com/package/web-streams-extensions)
- [$unwind (aggregation) - MongoDB Manual](https://www.mongodb.com/docs/manual/reference/operator/aggregation/unwind/)
- [Introducing sflow on r/javascript](https://www.reddit.com/r/javascript/comments/1exv4we/introducing_sflow_a_new_era_of_web_stream/)
## 📄 License
sflow is released under the **MIT License**. See the [LICENSE](./LICENSE) file for more details.
---
**sflow aims to simplify stream processing and bring functional programming paradigms to modern JavaScript and TypeScript development.**
Happy streaming! 🚀