320 lines
16 KiB
Markdown
320 lines
16 KiB
Markdown
# Technology Stack: v1.6 Voice Pipeline + Telegram Bridge
|
|
|
|
**Project:** Nexus v1.6 — additive to v1.5 stack (see prior STACK.md for hardware detection, smart-whisper, Puter.js, vectra, openid-client)
|
|
**Researched:** 2026-04-03
|
|
**Scope:** NEW libraries only for v1.6 — server-side voice pipeline integration, audio format conversion, browser VAD, Telegram bridge
|
|
**Confidence:** MEDIUM-HIGH (grammy HIGH via official docs; vad-react MEDIUM — React 19 peer dep confirmed fixed; ffmpeg-static MEDIUM — archived fluent-ffmpeg confirmed, spawn approach verified)
|
|
|
|
---
|
|
|
|
## Context: What v1.5 Already Installed
|
|
|
|
Do not re-add or re-research these — they are in `server/package.json` or `ui/package.json`:
|
|
|
|
| Package | Location | Purpose |
|
|
|---------|----------|---------|
|
|
| `smart-whisper ^0.8.1` | `server/` | Whisper.cpp Node bindings (recommended in v1.5 STACK.md) |
|
|
| `@mintplex-labs/piper-tts-web ^1.0.4` | `ui/` | Browser-side Piper WASM (already installed) |
|
|
| `systeminformation 5` | `server/` | Hardware detection |
|
|
| `multer ^2.0.2` | `server/` | Multipart upload (already handles audio blob uploads) |
|
|
| `express ^5.1.0` | `server/` | HTTP server |
|
|
|
|
The existing `VoiceRecordButton` already uses `MediaRecorder` + `POST /api/transcribe`. The existing `usePiperTts` hook already uses `@mintplex-labs/piper-tts-web` for browser-side TTS. The v1.6 work **extends** this — adding silence detection, server-side TTS, and Telegram relay.
|
|
|
|
---
|
|
|
|
## New Libraries by Feature Area
|
|
|
|
### 1. Browser VAD (Silence Detection + Auto-Send)
|
|
|
|
**Package:** `@ricky0123/vad-react`
|
|
**Version:** `^0.0.36`
|
|
**Where it lives:** `ui/` only — browser-side ONNX model running off the main thread
|
|
|
|
**Why:** The existing `VoiceRecordButton` requires the user to manually tap Stop. `@ricky0123/vad-react` uses Silero VAD (ONNX Runtime Web) to detect when the user stops speaking and fires `onSpeechEnd` automatically with the speech segment as a `Float32Array` at 16kHz. This eliminates the manual stop button and enables waveform-while-speaking UI via the `userSpeaking` state flag.
|
|
|
|
**React 19 compatibility:** Confirmed fixed in v0.0.36 (August 2025). The peer dependency constraint on React 18 was resolved. No `--legacy-peer-deps` needed.
|
|
|
|
**API surface:**
|
|
|
|
```typescript
|
|
import { useMicVAD } from "@ricky0123/vad-react";
|
|
|
|
const vad = useMicVAD({
|
|
startOnLoad: false, // user must explicitly start
|
|
positiveSpeechThreshold: 0.3, // sensitivity
|
|
minSpeechMs: 400, // ignore sub-400ms blips
|
|
redemptionMs: 1400, // 1.4s silence = end of utterance
|
|
onSpeechEnd: (audio: Float32Array) => {
|
|
// audio is 16kHz Float32Array — matches what Whisper expects
|
|
sendToTranscribeEndpoint(float32ToWav(audio));
|
|
},
|
|
});
|
|
|
|
// vad.userSpeaking — boolean for waveform animation
|
|
// vad.listening — boolean for mic state
|
|
// vad.start() / vad.pause()
|
|
```
|
|
|
|
**Key integration note:** `onSpeechEnd` delivers a `Float32Array` at 16000Hz — this maps directly to what `smart-whisper` expects on the server side, so no resampling is needed in the browser-to-server path.
|
|
|
|
**Confidence: MEDIUM** — Version verified via GitHub issues, React 19 fix confirmed. ONNX Runtime Web dependency means an extra ~5MB WASM download on first load.
|
|
|
|
---
|
|
|
|
### 2. Audio Format Conversion (Server-Side: WebM → WAV, WAV → OGG)
|
|
|
|
**Package:** `ffmpeg-static`
|
|
**Version:** `^5.2.0` (bundles FFmpeg 6.1.1 binaries for macOS arm64 + x64, Linux, Windows)
|
|
**Where it lives:** `server/` — provides the binary path; invoked via Node.js `child_process.spawn`
|
|
|
|
**Why `ffmpeg-static` over alternatives:**
|
|
- `fluent-ffmpeg` was archived on GitHub May 2025, no longer maintained — do NOT use as a new dependency
|
|
- `@ffmpeg-installer/ffmpeg` — last updated 2022, stale binary (FFmpeg 4.x)
|
|
- `ffmpeg-static` — actively maintained, ships FFmpeg 6.1.1, macOS arm64 confirmed, installed as an npm dependency (no system-level install needed)
|
|
- Direct `child_process.spawn("ffmpeg", [...])` with the binary path from `ffmpeg-static` is the recommended approach for 2025+
|
|
|
|
**Two conversions needed:**
|
|
|
|
**a) Incoming STT path: WebM/Opus → WAV 16kHz mono (for Whisper)**
|
|
|
|
```typescript
|
|
import ffmpegPath from "ffmpeg-static";
|
|
import { spawn } from "node:child_process";
|
|
|
|
function webmToWav16k(inputBuffer: Buffer): Promise<Buffer> {
|
|
return new Promise((resolve, reject) => {
|
|
const proc = spawn(ffmpegPath!, [
|
|
"-i", "pipe:0", // read from stdin
|
|
"-acodec", "pcm_s16le",
|
|
"-ac", "1", // mono
|
|
"-ar", "16000", // 16kHz
|
|
"-f", "wav",
|
|
"pipe:1", // write to stdout
|
|
]);
|
|
const out: Buffer[] = [];
|
|
proc.stdout.on("data", (c: Buffer) => out.push(c));
|
|
proc.stdout.on("end", () => resolve(Buffer.concat(out)));
|
|
proc.stderr.on("data", () => {}); // suppress ffmpeg banner
|
|
proc.on("error", reject);
|
|
proc.stdin.write(inputBuffer);
|
|
proc.stdin.end();
|
|
});
|
|
}
|
|
```
|
|
|
|
**b) Outgoing Telegram TTS path: WAV/PCM → OGG Opus (Telegram voice format)**
|
|
|
|
```typescript
|
|
function wavToOggOpus(inputBuffer: Buffer): Promise<Buffer> {
|
|
return new Promise((resolve, reject) => {
|
|
const proc = spawn(ffmpegPath!, [
|
|
"-i", "pipe:0",
|
|
"-c:a", "libopus",
|
|
"-b:a", "32k",
|
|
"-f", "ogg",
|
|
"pipe:1",
|
|
]);
|
|
// ... same pattern as above
|
|
});
|
|
}
|
|
```
|
|
|
|
**Confidence: MEDIUM** — `ffmpeg-static` macOS arm64 confirmed via GitHub README. Pipe-based approach is well-documented. fluent-ffmpeg archival confirmed May 2025.
|
|
|
|
---
|
|
|
|
### 3. Telegram Bridge
|
|
|
|
**Package:** `grammy`
|
|
**Version:** `^1.41.1` (latest, supports Bot API 9.6)
|
|
**Where it lives:** `server/` as an optional singleton service — only starts if `TELEGRAM_BOT_TOKEN` is set
|
|
|
|
**Why grammy over alternatives:**
|
|
- `grammy` has 1.4M weekly downloads vs `telegraf` at 900K — grammY is now the higher-adoption choice
|
|
- grammY is written in TypeScript-first (clean types, no DefinitelyTyped). Telegraf v4 migrated to TS but the type system is described as "too complex to understand" in grammY's own comparison docs
|
|
- `node-telegram-bot-api` is lower-level with no middleware, requires more boilerplate for this use case
|
|
- grammY's file handling API (`ctx.getFile()`) is the cleanest for the voice relay use case
|
|
|
|
**What the bridge needs to do (thin relay only — per PROJECT.md):**
|
|
|
|
```typescript
|
|
import { Bot, Context } from "grammy";
|
|
|
|
const bot = new Bot(process.env.TELEGRAM_BOT_TOKEN!);
|
|
|
|
// Relay text messages to Nexus chat API
|
|
bot.on("message:text", async (ctx) => {
|
|
const response = await relayToNexus(ctx.message.text, ctx.from.id);
|
|
await ctx.reply(response);
|
|
});
|
|
|
|
// Receive voice messages — download OGG, transcribe, relay
|
|
bot.on("message:voice", async (ctx) => {
|
|
const file = await ctx.getFile();
|
|
// file.download() returns Buffer (grammY handles temp URL expiry)
|
|
const oggBuffer = await downloadFile(file.file_path!, bot.token);
|
|
const transcript = await transcribeOgg(oggBuffer); // via smart-whisper
|
|
const response = await relayToNexus(transcript, ctx.from.id);
|
|
await ctx.reply(response);
|
|
});
|
|
|
|
// Run with long polling (no webhook needed for single-user local setup)
|
|
bot.start();
|
|
```
|
|
|
|
**Voice message format from Telegram:** Telegram sends voice messages as OGG/Opus, 32kbps, mono, 48kHz. To pass this to Whisper (which needs 16kHz WAV), convert with `ffmpeg-static` pipeline: `ogg→wav16k`.
|
|
|
|
**To send TTS back to Telegram:** Convert Piper WAV output → OGG Opus via `ffmpeg-static`, then use `ctx.replyWithVoice(new InputFile(oggBuffer, "voice.ogg"))`.
|
|
|
|
**Long polling vs webhook:** Long polling is correct for this deployment (Mac Mini, local network, no public HTTPS endpoint required). No reverse proxy or SSL cert needed.
|
|
|
|
**Confidence: HIGH** — grammy official docs verified at grammy.dev. File download pattern confirmed via grammY file handling guide. Bot API 9.6 support confirmed in homepage badge.
|
|
|
|
---
|
|
|
|
### 4. Server-Side Piper TTS (Audio Response Endpoint)
|
|
|
|
**No new library needed.** The v1.5 STACK.md already specified the `child_process.spawn` approach with the Piper binary.
|
|
|
|
**What v1.6 adds on top of v1.5:**
|
|
- A new Express route: `POST /api/voice/synthesize` that accepts `{ text, voice? }` and returns raw WAV audio (`Content-Type: audio/wav`)
|
|
- This endpoint is used by both the web chat playback (browser `<audio>` element) and the Telegram bridge (convert WAV → OGG for `sendVoice`)
|
|
- Voice mode flag: requests with `voiceMode: true` should receive a condensed plain-language response (no markdown, no code blocks) — this is a prompt instruction layer, not a library
|
|
|
|
**Response shape:**
|
|
|
|
```
|
|
POST /api/voice/synthesize
|
|
Body: { text: string, voice?: "en_US-lessac-medium" }
|
|
Response: audio/wav binary stream
|
|
```
|
|
|
|
**Confidence: HIGH** — this is an implementation pattern, not a new library.
|
|
|
|
---
|
|
|
|
### 5. Audio Playback (Web Chat)
|
|
|
|
**No new library needed.** The browser's native `<audio>` element handles WAV and OGG playback. The existing `TtsButton` uses `new Audio(url)` already. The v1.6 enhancement is:
|
|
|
|
- Upgrade from `new Audio(blob)` to a proper inline `<audio controls>` player with auto-play toggle stored in settings
|
|
- Use `URL.createObjectURL(blob)` for streaming playback of TTS responses
|
|
- Waveform visualization via `AnalyserNode` from the Web Audio API — no library needed
|
|
|
|
**Confidence: HIGH** — Web Audio API and `<audio>` are native browser APIs. No library required.
|
|
|
|
---
|
|
|
|
## Installation Summary
|
|
|
|
```bash
|
|
# ui/ — add VAD for silence detection + auto-send
|
|
pnpm --filter @paperclipai/ui add @ricky0123/vad-react
|
|
|
|
# server/ — add FFmpeg binary (for audio format conversion) and Telegram bot
|
|
pnpm --filter @paperclipai/server add ffmpeg-static grammy
|
|
|
|
# server/ — types (ffmpeg-static ships its own types; grammy is TS-native)
|
|
# No @types/* needed for grammy
|
|
# ffmpeg-static types are included in the package
|
|
```
|
|
|
|
---
|
|
|
|
## What NOT to Add
|
|
|
|
| Avoid | Why | Use Instead |
|
|
|-------|-----|-------------|
|
|
| `fluent-ffmpeg` | Archived May 2025, no longer maintained | Direct `child_process.spawn` with `ffmpeg-static` binary |
|
|
| `@ffmpeg-installer/ffmpeg` | Stale — last updated 2022, ships FFmpeg 4.x | `ffmpeg-static ^5.2.0` (ships FFmpeg 6.1.1, arm64 support) |
|
|
| `telegraf` | TypeScript type system "too complex to understand" per maintainers; lower weekly downloads than grammY | `grammy ^1.41.1` |
|
|
| `node-telegram-bot-api` | Low-level, requires callback polling setup, no middleware, more boilerplate | `grammy` |
|
|
| `@ricky0123/vad-node` | Node.js support was discontinued by the maintainer; wound down | `@ricky0123/vad-react` (browser-only, which is where recording lives) |
|
|
| `whisper.js` / `transformers.js` (browser WASM) | 200MB+ model download in browser; slow on first load; server-side Whisper via `smart-whisper` is already in place | `smart-whisper` on server (already in v1.5 stack) |
|
|
| `@mintplex-labs/piper-tts-web` for server TTS | Browser WASM only, no Node.js support | Piper binary via `child_process.spawn` (already specified in v1.5) |
|
|
| Wake word / real-time streaming audio | Out of scope per PROJECT.md | Future milestone |
|
|
|
|
---
|
|
|
|
## Alternatives Considered
|
|
|
|
| Recommended | Alternative | When to Use Alternative |
|
|
|-------------|-------------|-------------------------|
|
|
| `grammy ^1.41.1` | `telegraf ^4.x` | If you need a battle-tested library with larger plugin ecosystem and tolerate complex TypeScript types |
|
|
| `ffmpeg-static` + `spawn` | `@ffmpeg/ffmpeg` (WASM) | If running in a serverless/edge environment where native binaries are not available — not applicable here |
|
|
| `@ricky0123/vad-react` | Manual `AudioWorklet` energy threshold | If you need lower latency or don't want the 5MB ONNX WASM payload; simpler but less accurate silence detection |
|
|
| `@ricky0123/vad-react` | `MediaRecorder` with manual stop button (current impl) | The current v1.3 VoiceRecordButton works; VAD is strictly an UX upgrade |
|
|
|
|
---
|
|
|
|
## Version Compatibility
|
|
|
|
| Package | Compatible With | Notes |
|
|
|---------|-----------------|-------|
|
|
| `grammy ^1.41.1` | Node.js >=18, TypeScript >=5 | Ships its own types, no `@types/grammy` |
|
|
| `ffmpeg-static ^5.2.0` | Node.js >=14, macOS arm64 | Downloads correct binary at `npm install` time via `optionalDependencies` |
|
|
| `@ricky0123/vad-react ^0.0.36` | React 19, Vite 6 | React 19 peer dep fixed in August 2025; requires SharedArrayBuffer (COOP/COEP headers) for ONNX thread worker |
|
|
| `smart-whisper ^0.8.1` | Node.js >=18, macOS arm64 | From v1.5 — verify it's actually installed before v1.6 starts |
|
|
|
|
**Critical COOP/COEP note for `@ricky0123/vad-react`:** The Silero VAD model runs in an ONNX Runtime Web worker that requires `SharedArrayBuffer`. This means the server must send these headers on HTML responses:
|
|
|
|
```
|
|
Cross-Origin-Opener-Policy: same-origin
|
|
Cross-Origin-Embedder-Policy: require-corp
|
|
```
|
|
|
|
This is a one-line addition to the Express static file middleware. Without it, VAD silently fails in Chrome/Firefox. The existing PWA service worker may also need `Cross-Origin-Embedder-Policy: require-corp` to avoid breaking.
|
|
|
|
---
|
|
|
|
## Integration Architecture (v1.6 additions only)
|
|
|
|
```
|
|
Browser (UI) Server (Express)
|
|
───────────────────────────────── ───────────────────────────────────────
|
|
|
|
@ricky0123/vad-react POST /api/transcribe (existing)
|
|
└── useMicVAD ─────→ └── ffmpeg-static: webm→wav16k
|
|
└── onSpeechEnd(Float32Array) └── smart-whisper: wav→text
|
|
└── userSpeaking (waveform UI) └── returns { text: string }
|
|
|
|
React ChatInput (updated) POST /api/voice/synthesize (new)
|
|
└── voice mode toggle ─────→ └── Piper binary: text→wav
|
|
└── auto-send on speech end └── returns audio/wav stream
|
|
└── <audio> inline player ←──────────────┘
|
|
|
|
Telegram bridge (new, optional)
|
|
└── grammy long polling
|
|
└── message:text → relayToNexus()
|
|
└── message:voice →
|
|
ffmpeg: ogg→wav16k
|
|
smart-whisper → text
|
|
relayToNexus() → response
|
|
Piper → wav
|
|
ffmpeg: wav→ogg
|
|
ctx.replyWithVoice()
|
|
```
|
|
|
|
---
|
|
|
|
## Sources
|
|
|
|
- [grammY official docs](https://grammy.dev/) — TypeScript support, long polling, file handling confirmed
|
|
- [grammY GitHub](https://github.com/grammyjs/grammY) — Bot API 9.6 badge, v1.41.1 version
|
|
- [grammY file handling guide](https://grammy.dev/guide/files) — `ctx.getFile()`, download pattern
|
|
- [grammY comparison with Telegraf](https://grammy.dev/resources/comparison) — TypeScript type quality comparison
|
|
- [ffmpeg-static GitHub](https://github.com/eugeneware/ffmpeg-static) — macOS arm64 binary confirmed, FFmpeg 6.1.1
|
|
- [fluent-ffmpeg archival](https://github.com/fluent-ffmpeg/node-fluent-ffmpeg) — archived May 22 2025, confirmed
|
|
- [@ricky0123/vad-react npm](https://www.npmjs.com/package/@ricky0123/vad-react) — v0.0.36, last published 3 months ago
|
|
- [vad React 19 support issue #188](https://github.com/ricky0123/vad/issues/188) — fixed August 28 2025, confirmed
|
|
- [vad API docs](https://docs.vad.ricky0123.com/user-guide/api/) — `onSpeechEnd` Float32Array 16kHz confirmed
|
|
- [Telegram Bot API sendVoice](https://core.telegram.org/bots/api#sendvoice) — OGG Opus format requirement
|
|
- [nodejs-whisper GitHub](https://github.com/ChetanXpro/nodejs-whisper) — v0.2.9 comparison (rejected: subprocess-based, 10 months stale)
|
|
- [Piper TTS GitHub releases](https://github.com/rhasspy/piper/releases) — macOS aarch64 binary availability
|
|
|
|
---
|
|
|
|
*Stack research for: Nexus v1.6 Voice Pipeline + Telegram Bridge*
|
|
*Researched: 2026-04-03*
|
|
*Supersedes: v1.5 STACK.md entries for smart-whisper and Piper — those remain valid; this file adds the glue and new libraries*
|