# react-multimodal
**Effortlessly Integrate Camera, Microphone, and AI-Powered Body/Hand Tracking into Your React Applications.**
`react-multimodal` is a comprehensive React library designed to simplify the integration of various media inputs and advanced AI-driven tracking capabilities into your web applications. It provides a set of easy-to-use React components and hooks, abstracting away the complexities of managing media streams, permissions, and real-time AI model processing (like MediaPipe for hand and body tracking).
## Why use `react-multimodal`?
- **Simplified Media Access:** Get up and running with camera and microphone feeds in minutes.
- **Advanced AI Features:** Seamlessly integrate cutting-edge hand and body tracking without deep AI/ML expertise.
- **Unified API:** Manage multiple media sources (video, audio, hands, body) through a consistent and declarative API.
- **React-Friendly:** Built with React developers in mind, leveraging hooks and context for a modern development experience.
- **Performance Conscious:** Designed to be efficient, especially for real-time AI processing tasks.
## Core Features
`react-multimodal` offers the following key components and hooks:
- 🎥 **`CameraProvider` & `useCamera`**: Access and manage camera video streams. Provides the raw `MediaStream` for direct use or rendering with helper components.
- 🎤 **`MicrophoneProvider` & `useMicrophone`**: Access and manage microphone audio streams. Provides the raw `MediaStream`.
- 🖐️ **`HandsProvider` & `useHands`**: Implements real-time hand tracking using MediaPipe. Provides detailed landmark data for detected hands.
- 🤸 **`BodyProvider` & `useBody`**: (Coming Soon/Conceptual) Intended for real-time body pose estimation.
- 🧩 **`MediaProvider` & `useMedia`**: The central, unified provider. Combines access to camera, microphone, hand tracking, and body tracking. This is the recommended way to use multiple modalities.
- Easily enable or disable specific media types (video, audio, hands, body).
- Manages underlying providers and their lifecycles.
- Provides a consolidated context with all active media data and control functions (`startMedia`, `stopMedia`).
Additionally, there are couple of reusable components in the examples:
- 🖼️ **`CameraView`**: A utility component to easily render a video stream (e.g., from `CameraProvider` or `MediaProvider`) onto a canvas, often used for overlaying tracking visualizations. (`/src/examples/common/src/CameraView.jsx`)
- 🎤 **`MicrophoneView`**: A utility component for a simple visualization of an audio stream (e.g., from `MicrophoneProvider` or `MediaProvider`) onto a canvas, often used for overlaying tracking visualizations. (`/src/examples/common/src/MicrophoneView.jsx`)
## Installation
```bash
npm install @kortexa-ai/react-multimodal
# or
yarn add @kortexa-ai/react-multimodal
```
You will also need to install peer dependencies if you plan to use features like hand tracking:
```bash
npm install @mediapipe/tasks-vision @mediapipe/hands
# or
yarn add @mediapipe/tasks-vision @mediapipe/hands
```
## Getting Started
Here's how you can quickly get started with `react-multimodal`:
### 1. Basic Setup with `MediaProvider`
Wrap your application or relevant component tree with `MediaProvider`.
```jsx
// App.js or your main component
import { MediaProvider } from "@kortexa-ai/react-multimodal";
import MyComponent from "./MyComponent";
function App() {
return (
);
}
export default App;
```
### 2. Accessing Media Streams and Data
Use the `useMedia` hook within a component wrapped by `MediaProvider`.
```jsx
// MyComponent.jsx
import React, { useEffect, useRef } from "react";
import { useMedia } from "@kortexa-ai/react-multimodal";
// Assuming CameraView is imported from your project or the library's examples
// import CameraView from './CameraView';
function MyComponent() {
const {
videoStream,
audioStream,
handsData, // Will be null or empty if handsProps is not provided
isMediaReady,
isStarting,
startMedia,
stopMedia,
currentVideoError,
currentAudioError,
currentHandsError,
} = useMedia();
useEffect(() => {
// Automatically start media when the component mounts
// Or trigger with a button click: startMedia();
if (!isMediaReady && !isStarting) {
startMedia();
}
return () => {
// Clean up when the component unmounts
stopMedia();
};
}, [startMedia, stopMedia, isMediaReady, isStarting]);
if (currentVideoError)
return
Video Error: {currentVideoError.message}
;
if (currentAudioError)
return
Audio Error: {currentAudioError.message}
;
if (currentHandsError)
return
Hands Error: {currentHandsError.message}
;
return (
Multimodal Demo
{isMediaReady && videoStream && (
Camera Feed
{/* For CameraView, you'd import and use it like: */}
{/* */}