diff --git a/.planning/phases/34-voice/34-RESEARCH.md b/.planning/phases/34-voice/34-RESEARCH.md new file mode 100644 index 00000000..2633bce1 --- /dev/null +++ b/.planning/phases/34-voice/34-RESEARCH.md @@ -0,0 +1,470 @@ +# Phase 34: Voice - Research + +**Researched:** 2026-04-01 +**Domain:** Browser STT (Whisper via smart-whisper), Browser TTS (Piper via @mintplex-labs/piper-tts-web WASM), Onboarding voice step +**Confidence:** MEDIUM + +--- + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions +None — all implementation choices are at Claude's discretion. + +### Claude's Discretion +All implementation choices are at Claude's discretion. + +### Deferred Ideas (OUT OF SCOPE) +None. + + +--- + + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|------------------| +| VOICE-01 | User gets Piper TTS speech output that works on CPU-only hardware | @mintplex-labs/piper-tts-web runs entirely in browser WASM via ONNX Runtime — no GPU needed | +| VOICE-02 | Piper TTS pre-warms on first use with visible download progress (no silent 15-30s hang) | `tts.download(voiceId, progressCallback)` API provides loaded/total bytes; render a progress bar before calling `predict()` | +| VOICE-03 | Voice features (Whisper STT + Piper TTS) offered during onboarding based on hardware capability | NexusOnboardingWizard currently has 5 steps; add a step 4 (voice) gated on `hardwareInfo.hardwareTier !== undefined`; all tiers can run voice since it is purely CPU-bound WASM | + + +--- + +## Summary + +Phase 34 adds two voice capabilities: speech-to-text (STT) via Whisper, and text-to-speech (TTS) via Piper, plus an onboarding step where users can opt into voice features. + +The STT side already has a server route (`POST /api/transcribe` in `chat-files.ts`) and a `VoiceRecordButton` component that calls it. The route is implemented correctly but has a critical gap: it is **exported from `routes/index.ts` but never registered in `app.ts`**, so `POST /api/transcribe` returns 404 at runtime. Fixing this registration is the primary STT task. + +For TTS, the project currently has zero Piper integration. The recommended approach is browser-side WASM via `@mintplex-labs/piper-tts-web` (v1.0.4, MIT). This library wraps the Piper ONNX models in WebAssembly so synthesis runs on-device without a server round-trip, satisfying VOICE-01 (CPU-only hardware). The key UX concern (VOICE-02) is a 10-50 MB model download that blocks first synthesis — the library provides a `download()` method with a progress callback that must be wired to a visible UI element before calling `predict()`. + +The onboarding voice step (VOICE-03) should be inserted into `NexusOnboardingWizard.tsx` as step 4 (shifting the existing "root directory" step to 5 and "summary" to 6). The step should probe mic permission availability and detect whether the browser supports `MediaRecorder` to inform the user, then offer a "yes, enable voice" / "skip" choice. Since all hardware tiers can run browser-WASM TTS, the gate is not tier-based — it is browser-capability-based. + +**Primary recommendation:** Register `chatFileRoutes` in `app.ts` to fix STT; add `@mintplex-labs/piper-tts-web` for browser-side TTS with a progress-bar pre-warm flow; add a voice opt-in step in `NexusOnboardingWizard`. + +--- + +## Standard Stack + +### Core + +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| `@mintplex-labs/piper-tts-web` | 1.0.4 | Browser-side Piper TTS via WASM/ONNX | Browser-only, no server infra, CPU-safe, actively maintained fork used in AnythingLLM | +| `smart-whisper` | 0.8.1 | Native Node.js Whisper.cpp binding for STT | Auto-downloads models, Metal on Apple Silicon, used as drop-in replacement for CLI approach | + +### Supporting + +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| `node-wav` | 0.0.2 | Decode WAV buffer to Float32Array for smart-whisper | Required: smart-whisper only accepts 16kHz Float32Array PCM, not raw webm | + +**Note on audio conversion:** The browser sends `audio/webm;codecs=opus`. smart-whisper requires 16kHz mono Float32Array PCM. `ffmpeg` is not present on this machine. The existing `/transcribe` route writes a temp `.webm` file and calls the `whisper` or `whisper-cpp` CLI — this works when those CLIs are installed. If upgrading to `smart-whisper`, a conversion step is required. The server-side `ffmpeg` is not available, so either: (a) require `ffmpeg` as an install-time dep via `fluent-ffmpeg` + system `ffmpeg`, or (b) keep the CLI-fallback pattern in the existing route and just **fix the route registration** rather than rewriting the transcription logic. Option (b) is lower risk. + +### Alternatives Considered + +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| `@mintplex-labs/piper-tts-web` (browser) | `piper` Python CLI via server route | CLI requires Python + model install; adds server complexity; VOICE-F01 deferred to future | +| Fix route registration | Rewrite transcription with `smart-whisper` | smart-whisper requires PCM conversion (no ffmpeg on host); high risk; the existing CLI fallback is simpler | + +**Installation (UI):** +```bash +pnpm --filter @paperclipai/ui add @mintplex-labs/piper-tts-web +``` + +**No new server deps needed** for the minimal fix (just registering existing route). If upgrading to smart-whisper in a future phase: +```bash +pnpm --filter @paperclipai/server add smart-whisper node-wav +``` + +**Version verification (confirmed against npm registry 2026-04-01):** +- `@mintplex-labs/piper-tts-web`: 1.0.4 (latest) +- `smart-whisper`: 0.8.1 (latest) +- `node-wav`: 0.0.2 (latest) + +--- + +## Architecture Patterns + +### Recommended Project Structure +``` +server/src/ +├── app.ts # ADD: chatFileRoutes registration (1-line fix) +├── routes/ +│ └── chat-files.ts # Existing /transcribe route — no changes needed +│ └── voice.ts # Optional: extract a dedicated voice route if /synthesize added + +ui/src/ +├── components/ +│ ├── VoiceRecordButton.tsx # Existing — no changes needed once server route is fixed +│ ├── TtsButton.tsx # NEW: speaker icon button that calls piper-tts-web predict() +│ └── onboarding/ +│ └── VoiceStep.tsx # NEW: opt-in step for voice features +├── hooks/ +│ └── usePiperTts.ts # NEW: singleton TtsSession, download(), predict(), status +├── NexusOnboardingWizard.tsx # MODIFY: insert step 4 (voice), shift steps 4→5, 5→6 +``` + +### Pattern 1: Route Registration Fix (STT) + +**What:** `chatFileRoutes` is defined and exported but never registered in `app.ts`. Add one import and one `api.use()` call. + +**When to use:** This is the only required change for STT to function. + +**Example:** +```typescript +// server/src/app.ts — add after line ~31 (other imports) +import { chatFileRoutes } from "./routes/chat-files.js"; + +// ...inside createApp, after api.use(assistantHandoffRoutes(db)): +api.use(chatFileRoutes(db, opts.storageService)); +``` + +The `chatFileRoutes` function signature: `chatFileRoutes(db: Db, storage: StorageService)`. +In `app.ts`, `opts.storageService` is the storage argument. + +### Pattern 2: Piper TTS Hook (Browser-Side WASM) + +**What:** A React hook wrapping `@mintplex-labs/piper-tts-web` that manages model download state and synthesis. The model download is the pre-warm step that prevents the silent 15-30s hang on first synthesis. + +**When to use:** Any component that needs to read assistant responses aloud. + +**Example:** +```typescript +// ui/src/hooks/usePiperTts.ts +import { tts } from "@mintplex-labs/piper-tts-web"; + +const DEFAULT_VOICE = "en_US-hfc_female-medium"; + +export function usePiperTts() { + const [status, setStatus] = useState<"idle" | "downloading" | "ready" | "speaking">("idle"); + const [progress, setProgress] = useState(0); // 0–100 + + async function prewarm() { + setStatus("downloading"); + const stored = await tts.stored(); + if (!stored.includes(DEFAULT_VOICE)) { + await tts.download(DEFAULT_VOICE, (p) => { + setProgress(Math.round((p.loaded / p.total) * 100)); + }); + } + setStatus("ready"); + } + + async function speak(text: string) { + if (status !== "ready") return; + setStatus("speaking"); + const wav = await tts.predict({ text, voiceId: DEFAULT_VOICE }); + const audio = new Audio(wav); + audio.onended = () => setStatus("ready"); + audio.play(); + } + + return { status, progress, prewarm, speak }; +} +``` + +**Key points:** +- `tts.predict()` returns a Blob URL (WAV format). Use `new Audio(blobUrl).play()` — simplest approach, no Web Audio API needed. +- `tts.stored()` checks IndexedDB cache; download is skipped if model already present. +- The library is browser-only. Do not import in server code. + +### Pattern 3: Onboarding Voice Step + +**What:** Add a step 4 in `NexusOnboardingWizard.tsx` that shows STT+TTS capability, checks mic permission, and lets users opt in. Because piper-tts-web is CPU-safe WASM, the gate is browser capability (`navigator.mediaDevices`), not hardware tier. + +**When to use:** VOICE-03 requirement — offer voice during onboarding. + +**Step numbering shift:** +- Current: 1=hardware, 2=mode, 3=provider, 4=rootDir, 5=summary +- New: 1=hardware, 2=mode, 3=provider, **4=voice**, 5=rootDir, 6=summary +- Update `Step` type from `1 | 2 | 3 | 4 | 5` to `1 | 2 | 3 | 4 | 5 | 6` +- Update "Step X of 5" label to "Step X of 5" (keep label at 5 since summary is a bonus; or "Step X of 6") +- Update all `setStep()` calls to use new numbers + +**Voice opt-in state to track:** +```typescript +const [voiceEnabled, setVoiceEnabled] = useState(false); +``` +Store in `nexus-settings.json` via a new field (e.g., `voiceEnabled: boolean`) if persistence across sessions is desired. Or store in localStorage if the no-DB-schema constraint applies (it does — no schema changes, use file-backed JSON). + +**Example VoiceStep component structure:** +```tsx +// ui/src/components/onboarding/VoiceStep.tsx +export function VoiceStep({ onEnable, onSkip }: VoiceStepProps) { + const [micAvailable, setMicAvailable] = useState(null); + + useEffect(() => { + // Non-blocking probe: does browser support mic? + navigator.mediaDevices?.enumerateDevices() + .then(devices => setMicAvailable(devices.some(d => d.kind === "audioinput"))) + .catch(() => setMicAvailable(false)); + }, []); + + return ( + <> +

Voice features

+

Speak to your assistant (Whisper STT) and hear responses read aloud (Piper TTS). Runs entirely on your device.

+ {micAvailable === false && ( +

No microphone detected — STT unavailable, but TTS still works.

+ )} + + + + ); +} +``` + +### Anti-Patterns to Avoid + +- **Importing piper-tts-web in Node.js:** The library explicitly does not support Node.js. It must only be imported in browser code (UI package). Vite will not include it in the server bundle. +- **Calling `tts.predict()` before downloading the model:** Results in a 15-30s silent hang. Always call `tts.download()` first (or check `tts.stored()`), show progress, then call `predict()`. +- **Registering `/transcribe` before auth middleware:** The existing `/transcribe` route calls `assertBoard(req)` — it must sit inside the `api` sub-router (after `boardMutationGuard`), not before it. The `chatFileRoutes` call belongs at line ~161 of `app.ts` alongside other `api.use()` calls. +- **Using `new Audio()` with a raw Buffer:** `tts.predict()` returns a Blob URL string — pass it directly to `new Audio(url)`, not `new Audio(Buffer)`. + +--- + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| TTS synthesis in browser | Custom ONNX loader + Piper WASM integration | `@mintplex-labs/piper-tts-web` | Already bundles ort-wasm, phenomizer, model management — 492KB package handles all of it | +| Model download progress | Manual fetch with XHR progress | `tts.download(voiceId, progressCb)` | Built-in progress callback, automatic IndexedDB caching | +| PCM audio decoding for Whisper | Custom webm→PCM decoder | Keep CLI fallback in existing `/transcribe` route | No ffmpeg on host; smart-whisper requires PCM — adding conversion is out-of-scope for this phase | +| Mic permission detection | Custom navigator probe | `navigator.mediaDevices.enumerateDevices()` | Native browser API, no library needed | + +**Key insight:** The browser handles TTS completely — no server-side Piper install needed for VOICE-01/02. Server-side Piper (VOICE-F01) is explicitly deferred. + +--- + +## Common Pitfalls + +### Pitfall 1: `/transcribe` Returns 404 + +**What goes wrong:** VoiceRecordButton sends audio to `/api/transcribe`, gets a 404, swallows the error silently. Voice input appears broken with no feedback to the user. + +**Why it happens:** `chatFileRoutes` is exported in `routes/index.ts` but not imported or registered in `app.ts`. The route exists in code but is never mounted. + +**How to avoid:** Add `import { chatFileRoutes } from "./routes/chat-files.js"` and `api.use(chatFileRoutes(db, opts.storageService))` in `app.ts`. + +**Warning signs:** `GET /api/transcribe` returns 404; no logs from the route handler; VoiceRecordButton spinner appears and disappears with no text inserted. + +### Pitfall 2: Piper TTS Silent Hang on First Use + +**What goes wrong:** User clicks "speak" button. Nothing happens for 15-30 seconds, then audio plays. User thinks it's broken. + +**Why it happens:** `tts.predict()` internally downloads the ONNX model (~10-50MB) on first call with no progress feedback. + +**How to avoid:** Call `tts.download(voiceId, progressCb)` explicitly before the first `predict()`. Show a progress bar or spinner with percentage. The `prewarm()` pattern in the hook above is the canonical fix for VOICE-02. + +**Warning signs:** First TTS invocation is slow (10-30s), subsequent calls are fast. Model in browser DevTools IndexedDB after first successful call. + +### Pitfall 3: `chatFileRoutes` Argument Mismatch + +**What goes wrong:** Passing incorrect arguments to `chatFileRoutes(db, storage)` — e.g., passing the wrong storage interface type. + +**Why it happens:** `app.ts` uses `opts.storageService` which is a `StorageService`. The function signature is `chatFileRoutes(db: Db, storage: StorageService)`. + +**How to avoid:** Verify the StorageService import path and type. In `app.ts`, `opts.storageService` is already typed as `StorageService` and is used by other routes (e.g., `assetRoutes(db, opts.storageService)`). Mirror that pattern exactly. + +### Pitfall 4: Onboarding Step Counter Mismatch + +**What goes wrong:** Adding step 4 (voice) but forgetting to update Back/Continue `setStep()` calls in steps 5 and 6, causing step transitions to skip or loop. + +**Why it happens:** `NexusOnboardingWizard.tsx` has hard-coded step numbers throughout (`setStep(4)`, `setStep(5)`, etc.) and a `Step` type union (`1 | 2 | 3 | 4 | 5`). + +**How to avoid:** When inserting step 4, do a full audit of all `setStep(N)` calls and the `Step` type. The `type Step = 1 | 2 | 3 | 4 | 5` must become `1 | 2 | 3 | 4 | 5 | 6`. All old `setStep(4)` → `setStep(5)`, `setStep(5)` → `setStep(6)`. + +### Pitfall 5: piper-tts-web In a Web Worker Context + +**What goes wrong:** Importing `@mintplex-labs/piper-tts-web` fails in a Web Worker because of missing `window` or `document` globals. + +**Why it happens:** The library expects browser globals. It also mentions supporting Web Worker patterns but requires careful WASM path configuration. + +**How to avoid:** Use the library from a regular React component/hook (main thread). Do not import in server-side code, Node.js workers, or Vitest Node environment tests. Mark test files importing it with `@vitest-environment jsdom` if needed, or mock the module in tests. + +--- + +## Code Examples + +### Fix: Register chatFileRoutes in app.ts + +```typescript +// Source: server/src/app.ts (existing pattern — mirror assetRoutes) + +// Near top of file with other route imports: +import { chatFileRoutes } from "./routes/chat-files.js"; + +// Inside createApp(), after api.use(assistantHandoffRoutes(db)): +api.use(chatFileRoutes(db, opts.storageService)); +``` + +Confirmed pattern from `app.ts` line 147: +```typescript +api.use(assetRoutes(db, opts.storageService)); +``` + +### TTS: Minimal predict() call + +```typescript +// Source: @mintplex-labs/piper-tts-web README + +import { tts } from "@mintplex-labs/piper-tts-web"; + +// Download model with progress (pre-warm): +await tts.download("en_US-hfc_female-medium", (progress) => { + const pct = Math.round((progress.loaded / progress.total) * 100); + console.log(`Downloading voice model: ${pct}%`); +}); + +// Synthesize: +const wav = await tts.predict({ + text: "Hello, I am your assistant.", + voiceId: "en_US-hfc_female-medium", +}); + +// Play: +const audio = new Audio(wav); +audio.play(); +``` + +### TTS: Check if already downloaded (skip re-download) + +```typescript +// Source: @mintplex-labs/piper-tts-web README + +const stored = await tts.stored(); // string[] of cached voiceIds +if (!stored.includes("en_US-hfc_female-medium")) { + await tts.download("en_US-hfc_female-medium", progressCb); +} +``` + +### Mic availability probe (no library required) + +```typescript +// Source: MDN Web API (browser standard) + +async function hasMicrophone(): Promise { + try { + const devices = await navigator.mediaDevices.enumerateDevices(); + return devices.some((d) => d.kind === "audioinput"); + } catch { + return false; + } +} +``` + +--- + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| whisper CLI (openai-whisper Python) | smart-whisper Node.js native binding | 2023-2024 | No Python runtime needed; better perf | +| Piper CLI binary | @mintplex-labs/piper-tts-web WASM | 2024 | Runs in browser, no server setup | +| Server-rendered TTS audio | Client-side WASM synthesis | 2024 | Eliminates network round-trip; offline-safe | + +**Deprecated/outdated:** +- `whisper-cpp` CLI: still works but requires system-level install; the existing `/transcribe` route already has this fallback — adequate for now +- `rhasspy/piper` repository: archived Oct 2025, development moved to `OHF-Voice/piper1-gpl`; the `@mintplex-labs/piper-tts-web` npm package uses the original archived models (MIT) and still works + +--- + +## Open Questions + +1. **nexus-settings voice persistence** + - What we know: `nexus-settings.json` currently only stores `{ mode }`. The `nexusSettingsSchema` is a Zod schema. + - What's unclear: Should `voiceEnabled: boolean` be added to the schema? The constraint says "no DB schema changes" but this is a file-backed JSON, not a DB table. + - Recommendation: Add `voiceEnabled: z.boolean().default(false)` to `nexusSettingsSchema`. This is a file field, not a DB migration. The planner should confirm this is acceptable under the "no DB schema changes" constraint. + +2. **smart-whisper Apple Silicon unverified claim (from STATE.md blockers)** + - What we know: STATE.md notes "smart-whisper Apple Silicon acceleration claim unverified on Mac Mini M4 — fall back to `tiny.en` if `base.en` acceleration not confirmed on device." + - What's unclear: Whether Metal acceleration actually works for `base.en` on M4. + - Recommendation: The current `/transcribe` route uses CLI fallback anyway. Since this phase is NOT rewriting STT with smart-whisper (just fixing route registration), this blocker does not apply to Phase 34. + +3. **VoiceRecordButton in PersonalAssistant** + - What we know: `ChatPanel` sets `enableVoiceInput={true}`. `PersonalAssistant.tsx` does not use `ChatInput` and has its own send form that does NOT include a `VoiceRecordButton`. Voice input only works in the project-mode `ChatPanel`, not in the personal assistant chat. + - What's unclear: Whether VOICE-01/02/03 require voice in personal assistant chat specifically. + - Recommendation: Planner should add `VoiceRecordButton` to `PersonalAssistant.tsx`'s input area as part of this phase, since personal assistant is the primary chat surface for v1.5. + +--- + +## Environment Availability + +| Dependency | Required By | Available | Version | Fallback | +|------------|------------|-----------|---------|----------| +| Node.js | Server runtime | Yes | v20.20.2 | — | +| piper CLI | VOICE-01 (server-side) | No | — | Browser WASM via piper-tts-web (preferred) | +| whisper CLI | /transcribe route | No | — | Route returns 503 with user-visible error | +| whisper-cpp CLI | /transcribe route | No | — | Falls through to openai-whisper, then 503 | +| ffmpeg | WebM→PCM conversion | No | — | Keep CLI-fallback STT; no smart-whisper upgrade this phase | +| Browser MediaRecorder | VoiceRecordButton | N/A (browser) | — | Degrades gracefully (mic unavailable state) | + +**Missing dependencies with no fallback:** +- None that block this phase — the `/transcribe` route already handles missing Whisper CLIs gracefully with a 503 + descriptive error. Piper TTS runs entirely in browser WASM, no server dep. + +**Missing dependencies with fallback:** +- `whisper` / `whisper-cpp`: Not installed. Route returns `{ error: "Whisper not available. Install whisper-cpp or openai-whisper for voice input." }` with 503. This is existing behavior. STT will silently fail until user installs Whisper, which is acceptable given the 503 message guides them. + +--- + +## Validation Architecture + +### Test Framework + +| Property | Value | +|----------|-------| +| Framework | Vitest 3.0.5 | +| Config file | `server/vitest.config.ts` | +| Quick run command | `npx vitest run server/src/__tests__/34-voice-routes.test.ts` | +| Full suite command | `npx vitest run` (from `/opt/nexus`) | + +### Phase Requirements → Test Map + +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| VOICE-01 | `chatFileRoutes` is mounted in `app.ts` — GET/POST routes are reachable | unit (route) | `npx vitest run server/src/__tests__/34-voice-routes.test.ts` | No — Wave 0 | +| VOICE-02 | `usePiperTts` hook exposes `prewarm()`, `status`, `progress` | unit (hook) | `npx vitest run ui/src/hooks/usePiperTts.test.ts` | No — Wave 0 | +| VOICE-03 | `NexusOnboardingWizard` renders voice step at step 4 | unit (component) | Manual / `npx vitest run ui/src/components/NexusOnboardingWizard.test.ts` | No — Wave 0 | + +### Sampling Rate +- **Per task commit:** `npx vitest run server/src/__tests__/34-voice-routes.test.ts` +- **Per wave merge:** `npx vitest run` +- **Phase gate:** Full suite green before `/gsd:verify-work` + +### Wave 0 Gaps +- [ ] `server/src/__tests__/34-voice-routes.test.ts` — covers VOICE-01 (route registration, 503 when no whisper CLI) +- [ ] `ui/src/hooks/usePiperTts.test.ts` — covers VOICE-02 hook state machine (mock piper-tts-web) +- [ ] `ui/src/components/onboarding/VoiceStep.test.tsx` — covers VOICE-03 step rendering + +--- + +## Sources + +### Primary (HIGH confidence) +- Codebase inspection — `server/src/routes/chat-files.ts`, `server/src/app.ts`, `ui/src/components/VoiceRecordButton.tsx`, `ui/src/components/NexusOnboardingWizard.tsx` +- npm registry — `@mintplex-labs/piper-tts-web@1.0.4`, `smart-whisper@0.8.1` (verified 2026-04-01) + +### Secondary (MEDIUM confidence) +- [Mintplex-Labs/piper-tts-web README](https://github.com/Mintplex-Labs/piper-tts-web/blob/main/README.md) — `tts.download()`, `tts.predict()`, `tts.stored()` API +- [JacobLinCool/smart-whisper GitHub](https://github.com/JacobLinCool/smart-whisper) — Whisper class, PCM Float32Array requirement, Metal on Apple Silicon +- [smart-whisper documentation](https://jacoblincool.github.io/smart-whisper/) — transcribe API, model manager + +### Tertiary (LOW confidence) +- WebSearch results for Piper TTS Node.js integration — browser-only WASM pattern confirmed by multiple sources + +--- + +## Metadata + +**Confidence breakdown:** +- Standard stack: MEDIUM — npm package versions verified; API verified via README; no local test environment to run the library +- Architecture: HIGH — based on direct codebase inspection (route registration gap confirmed, wizard step structure confirmed) +- Pitfalls: HIGH — route registration gap is a confirmed code-level fact, not speculation + +**Research date:** 2026-04-01 +**Valid until:** 2026-05-01 (stable libraries; piper-tts-web and smart-whisper are low-churn)