# 🧠 Avatoon
[![npm package][npm-img]][npm-url]
[![Build Status][build-img]][build-url]
[![Issues][issues-img]][issues-url]
[![Code Coverage][codecov-img]][codecov-url]
**Avatoon** is a lightweight React Three Fiber component for rendering animated 3D avatars with real-time viseme-driven lip-sync. It supports lifelike head movements, morph target control, and optional goal-based gestures (e.g., flexing, sleeping), making it ideal for voice assistants, interactive characters, or storytelling apps.
### 🎮 [**Try the live demo →**](https://khaledalam.net/avatoon/)
[](https://stackblitz.com/github/khaledalam/avatoon/tree/main/example)
[demo video](https://youtu.be/9ODqu2EZvtw)
## ✨ Features
- 🎤 Real-time lip-sync using phoneme-viseme mapping
- 🧍 Subtle head motion animation while talking
- 👁️ Automatic eye-blinking (when the model exposes blink morph targets)
- 🎛️ Imperative `play()` / `stop()` control via a `ref`
- 🌐 GLTF model support via `useGLTF`
- ⚛️ Plug-and-play with React Three Fiber + Drei
- 🎯 Goal-based gestures like "Muscle" or "Sleep"
## 📦 Installation
```bash
npm install avatoon
```
### Requirements (peer dependencies)
Avatoon builds on React and the React Three Fiber ecosystem. Install these
alongside it if your project doesn't already have them:
```bash
npm install react react-dom three @react-three/fiber @react-three/drei
```
| Peer dependency | Supported version |
| --------------------- | ----------------- |
| `react` / `react-dom` | `>=18` |
| `three` | `>=0.153.0` |
| `@react-three/fiber` | `>=8.0.0` |
| `@react-three/drei` | `>=9.0.0` |
> Runs in any browser with WebGL support. Written in TypeScript — type
> definitions ship with the package.
## 🚀 Usage
### `Avatoon` — full avatar with audio-synced lip-sync
```jsx
import { Avatoon } from "avatoon";
const visemeJson = {
visemes: [
{ time: 0, viseme: "X" },
{ time: 1.3, viseme: "A" },
{ time: 1.367, viseme: "C" },
],
audio_base64: "", // base64-encoded WAV (optional)
};
export default function App() {
return (
console.log("Render Completed!")}
/>
);
}
```
### `LipSyncAvatoon` — standalone lip-sync (no audio required)
A self-contained component with a built-in Start/Stop button that animates
procedural mouth movement. Great for quick demos or "talking head" idle states.
```jsx
import { LipSyncAvatoon } from "avatoon";
export default function App() {
return (
);
}
```
### Imperative control (ref)
Drive playback from your own UI instead of the built-in button by passing a
`ref` — it exposes `play()`, `stop()`, and `toggle()`:
```jsx
import { useRef } from "react";
import { Avatoon, type AvatoonHandle } from "avatoon";
function App() {
const avatar = useRef(null);
return (
<>
[demo video](https://youtu.be/9ODqu2EZvtw)
## ✨ Features
- 🎤 Real-time lip-sync using phoneme-viseme mapping
- 🧍 Subtle head motion animation while talking
- 👁️ Automatic eye-blinking (when the model exposes blink morph targets)
- 🎛️ Imperative `play()` / `stop()` control via a `ref`
- 🌐 GLTF model support via `useGLTF`
- ⚛️ Plug-and-play with React Three Fiber + Drei
- 🎯 Goal-based gestures like "Muscle" or "Sleep"
## 📦 Installation
```bash
npm install avatoon
```
### Requirements (peer dependencies)
Avatoon builds on React and the React Three Fiber ecosystem. Install these
alongside it if your project doesn't already have them:
```bash
npm install react react-dom three @react-three/fiber @react-three/drei
```
| Peer dependency | Supported version |
| --------------------- | ----------------- |
| `react` / `react-dom` | `>=18` |
| `three` | `>=0.153.0` |
| `@react-three/fiber` | `>=8.0.0` |
| `@react-three/drei` | `>=9.0.0` |
> Runs in any browser with WebGL support. Written in TypeScript — type
> definitions ship with the package.
## 🚀 Usage
### `Avatoon` — full avatar with audio-synced lip-sync
```jsx
import { Avatoon } from "avatoon";
const visemeJson = {
visemes: [
{ time: 0, viseme: "X" },
{ time: 1.3, viseme: "A" },
{ time: 1.367, viseme: "C" },
],
audio_base64: "", // base64-encoded WAV (optional)
};
export default function App() {
return (
- T2 (Blendshape Face - Expressive)
- Separate eyeballs and mouth
- Supports morph targets / ARKit visemes
- Slightly less realistic but animatable
- 
## 📘 Types
```ts
interface VisemeData {
visemes: Array<{ time: number; viseme: string | null }>;
audio_base64?: string;
}
```
## 🎚️ Generating viseme data
`visemeJson` is what drives the mouth animation. Each entry pairs a **`time`**
(in seconds, from the start of the audio) with a single-letter **viseme code**.
Codes are mapped onto the model's morph targets; unknown or silent codes simply
rest the mouth.
| Code | Mouth shape / example sound |
| ---------- | ----------------------------- |
| `A` | open — "**a**" as in *apple* |
| `B` | lips together — *p, b, m* |
| `C` | *ch, sh, j* |
| `D` | *d, t, th* |
| `E` | "*eh*", "*ae*" |
| `F` | *f, v* |
| `G` | *g, k* |
| `I` | "*ee*" |
| `J` | *r, l, y* |
| `K` | "*oo*", "*u*" |
| `H` / `X` | silence / rest |
The standard **Oculus / Ready Player Me viseme names** are also accepted as codes
directly — `sil`, `PP`, `FF`, `TH`, `DD`, `kk`, `CH`, `SS`, `nn`, `RR`, `aa`,
`E`, `I`, `O`, `U` — which is what the converter helpers below emit.
Entries must be ordered by ascending `time`. Optionally provide the spoken audio
as a base64-encoded WAV via `audio_base64` to play it in sync (used when
`showPlayVoiceButton` is enabled).
```jsonc
{
"visemes": [
{ "time": 0.00, "viseme": "X" },
{ "time": 0.12, "viseme": "B" },
{ "time": 0.20, "viseme": "A" },
{ "time": 0.35, "viseme": "I" }
],
"audio_base64": "UklGR... (optional WAV)"
}
```
### From a speech engine
Rather than hand-authoring the timeline, generate it from a TTS/lip-sync engine.
Avatoon ships converters for the most common sources:
```js
import {
Avatoon,
fromAzureVisemes, // Azure Speech SDK viseme events
fromPollySpeechMarks, // AWS Polly speech marks
fromRhubarb, // Rhubarb Lip Sync JSON
} from "avatoon";
// Azure: collect events from the Speech SDK's `visemeReceived` callback
const visemeJson = fromAzureVisemes(azureEvents); // [{ visemeId, audioOffset }]
// AWS Polly: pass the `viseme` speech marks
const visemeJson = fromPollySpeechMarks(pollyMarks); // [{ time, value, type }]
// Rhubarb: pass its JSON output (or just the mouthCues array)
const visemeJson = fromRhubarb(rhubarbOutput); // { mouthCues: [{ start, end, value }] }
```
Each returns a `VisemeData` object ready to hand to `