105 lines
5.3 KiB
Markdown
105 lines
5.3 KiB
Markdown
---
|
|
phase: 39-voice-polish
|
|
plan: "01"
|
|
subsystem: voice
|
|
tags: [tts, streaming, sse, multi-lang, sentence-buffering]
|
|
dependency_graph:
|
|
requires: [36-voice-pipeline, 37-ui-voice]
|
|
provides: [sentence-streaming-sse, multi-lang-synthesis, streaming-playback]
|
|
affects: [ChatVoicePlayer, voice-pipeline, voice-routes]
|
|
tech_stack:
|
|
added: []
|
|
patterns: [AsyncGenerator, SSE/EventSource, ReadableStream, base64-audio]
|
|
key_files:
|
|
created:
|
|
- server/src/__tests__/39-sentence-streaming.test.ts
|
|
modified:
|
|
- server/src/services/voice-pipeline.ts
|
|
- server/src/routes/voice.ts
|
|
- ui/src/components/ChatVoicePlayer.tsx
|
|
key_decisions:
|
|
- "splitSentences protects only title abbreviations (Dr., Mr., etc.) - acronyms like D.C. can still end sentences"
|
|
- "synthesizeSentenceStream uses AsyncGenerator for lazy per-sentence audio production"
|
|
- "ChatVoicePlayer uses fetch POST + ReadableStream to parse SSE manually (EventSource only supports GET)"
|
|
- "Object URL cleanup on unmount and on new text arrival prevents blob memory leaks"
|
|
metrics:
|
|
duration: "~5 minutes"
|
|
completed_date: "2026-04-04"
|
|
tasks_completed: 2
|
|
files_modified: 4
|
|
---
|
|
|
|
# Phase 39 Plan 01: Sentence-Buffered TTS Streaming + Multi-Language Synthesis Summary
|
|
|
|
Sentence-buffered SSE TTS streaming with `synthesizeSentenceStream` AsyncGenerator + `synthesizeMultiLang` Promise.all parallel synthesis + ChatVoicePlayer progressive playback with sentence queue and progress indicator.
|
|
|
|
## Tasks Completed
|
|
|
|
| Task | Name | Commit | Files |
|
|
|------|------|--------|-------|
|
|
| 1 (TDD RED) | Failing tests for sentence streaming | 2efe6f30 | server/src/__tests__/39-sentence-streaming.test.ts |
|
|
| 1 (TDD GREEN) | Sentence streaming + multi-lang in pipeline + routes | 5c888c1a | server/src/services/voice-pipeline.ts, server/src/routes/voice.ts |
|
|
| 2 | ChatVoicePlayer sentence-buffered streaming playback | c4b05399 | ui/src/components/ChatVoicePlayer.tsx |
|
|
|
|
## What Was Built
|
|
|
|
### voice-pipeline.ts additions
|
|
|
|
- `splitSentences(text)` — exported function, protects title abbreviations (Dr., Mr., Mrs., etc.) from false sentence splits using placeholder technique; acronyms like D.C. at sentence end still trigger splits
|
|
- `synthesizeSentenceStream(text, voiceId?)` — `AsyncGenerator<{ index, total, audio }>` that calls piper per sentence and yields audio buffers immediately as each completes
|
|
- `synthesizeMultiLang(text, voiceIds[])` — runs `synthesize()` for each voiceId in parallel via `Promise.all`, returns `Map<voiceId, Buffer>`
|
|
- Internal `synthesizeSentence()` helper extracted to avoid code duplication between the three synthesis methods
|
|
- Existing `synthesize()` updated to use `splitSentences()` internally (backward compatible)
|
|
|
|
### voice.ts additions
|
|
|
|
- `POST /api/synthesize/stream` — SSE endpoint with `Content-Type: text/event-stream`, iterates `synthesizeSentenceStream`, writes `data: { index, total, audio: base64 }\n\n` per sentence, finishes with `data: { done: true }\n\n`
|
|
- `POST /api/synthesize/multi-lang` — validates voiceIds array (1-5 entries), calls `synthesizeMultiLang`, returns `{ results: [{ voiceId, audio: base64 }] }`
|
|
- Existing `POST /api/synthesize` unchanged
|
|
|
|
### ChatVoicePlayer.tsx
|
|
|
|
- Added `streaming` prop (default `true`)
|
|
- When streaming=true: uses `fetch POST + response.body.getReader()` to parse SSE lines as they arrive (not EventSource — POST required)
|
|
- First chunk received → decode base64, create Blob URL, set audio src, call `.play()` immediately
|
|
- Subsequent chunks → pushed to `audioQueue` ref
|
|
- `onEnded` handler pops next URL from queue and plays it
|
|
- Progress indicator: "Sentence N of M" text + dot progress bar (filled/unfilled per sentence)
|
|
- Fallback: stream error or `streaming=false` → falls back to existing full-fetch via `POST /api/synthesize`
|
|
- All Blob URLs cleaned up on unmount or new `text` prop
|
|
|
|
## Test Results
|
|
|
|
```
|
|
Tests 8 passed (8)
|
|
- splitSentences: splits basic sentences ✓
|
|
- splitSentences: abbreviation-aware (Dr., D.C.) ✓
|
|
- splitSentences: single sentence ✓
|
|
- splitSentences: filters empty strings ✓
|
|
- synthesizeSentenceStream: yields correct chunk count + metadata ✓
|
|
- synthesizeSentenceStream: single sentence ✓
|
|
- synthesizeMultiLang: returns Map with all voices ✓
|
|
- synthesizeMultiLang: calls piper in parallel ✓
|
|
```
|
|
|
|
## Deviations from Plan
|
|
|
|
### Auto-fixed Issues
|
|
|
|
None.
|
|
|
|
### Design Notes
|
|
|
|
**abbreviation handling:** The plan mentioned "Dr. Smith went to D.C. He liked it." → two sentences. The key insight was that title abbreviations (Dr., Mr., etc.) need protection but two-letter acronyms like D.C. at sentence position can still be split. The implementation uses a space-as-placeholder technique: title abbreviations have their trailing space replaced with `\x00`, then split on `(?<=[.!?])\s+`, then restore.
|
|
|
|
## Known Stubs
|
|
|
|
None — all endpoints are wired to real piper TTS synthesis.
|
|
|
|
## Self-Check: PASSED
|
|
|
|
- server/src/services/voice-pipeline.ts exists with splitSentences + synthesizeSentenceStream + synthesizeMultiLang ✓
|
|
- server/src/routes/voice.ts has synthesize/stream + synthesize/multi-lang + text/event-stream ✓
|
|
- server/src/__tests__/39-sentence-streaming.test.ts exists and all 8 tests pass ✓
|
|
- ui/src/components/ChatVoicePlayer.tsx has synthesize/stream + ReadableStream + audioQueue + sentence progress ✓
|
|
- Commits: 2efe6f30, 5c888c1a, c4b05399 ✓
|