3690a47fd5
Two plans in wave 1 (parallel): - 39-01: Sentence-buffered TTS streaming + multi-language synthesis (VPIPE-07, VPIPE-08) - 39-02: Onboarding voice hardware capability probe (ONBRD-01, ONBRD-02) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
11 KiB
11 KiB
| phase | plan | type | wave | depends_on | files_modified | autonomous | requirements | must_haves | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 39-voice-polish | 01 | execute | 1 |
|
true |
|
|
Purpose: Voice responses begin playing before full synthesis completes (under 1s to first audio), and users can synthesize the same response in multiple languages without a second agent call. Output: Streaming synthesize endpoint, multi-language endpoint, updated ChatVoicePlayer with progressive playback.
<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/ROADMAP.md @.planning/REQUIREMENTS.md @.planning/phases/39-voice-polish/39-CONTEXT.md@server/src/services/voice-pipeline.ts @server/src/routes/voice.ts @ui/src/components/ChatVoicePlayer.tsx
From server/src/services/voice-pipeline.ts:
// synthesize already does sentence splitting internally:
// const sentences = text.split(/(?<=[.!?])\s+/).filter((s) => s.length > 0);
// Currently concatenates all sentence buffers before returning.
// Need to yield each sentence buffer as it completes.
export function voicePipelineService(): {
transcribe(buffer: Buffer, format: "webm" | "ogg" | "wav"): Promise<{ text: string; language?: string }>;
synthesize(text: string, voiceId?: string): Promise<Buffer>;
formatForVoice(text: string): string;
transcodeToWav16k(inputBuffer: Buffer, inputFormat: string): Promise<Buffer>;
}
From server/src/routes/voice.ts:
// POST /api/synthesize — takes { text, voiceId }, returns audio/wav buffer
// POST /api/transcribe — takes multipart audio, returns { text, language? }
From ui/src/components/ChatVoicePlayer.tsx:
interface ChatVoicePlayerProps {
text: string;
autoPlay?: boolean;
}
// Currently fetches full audio blob from POST /api/synthesize, then plays
2. Add `async *synthesizeSentenceStream(text: string, voiceId?: string): AsyncGenerator<{ index: number; total: number; audio: Buffer }>` method:
- Call splitSentences(text) to get sentences array
- For each sentence, call piper (same as current synthesize logic), yield { index, total: sentences.length, audio: audioBuffer } immediately
- This gives the consumer each sentence's audio as soon as it is ready
3. Add `async synthesizeMultiLang(text: string, voiceIds: string[]): Promise<Map<string, Buffer>>` method:
- For each voiceId, call existing synthesize(text, voiceId) in parallel via Promise.all
- Return Map<voiceId, Buffer>
4. Update the return signature of voicePipelineService() to include the new methods.
5. In voice.ts, add streaming endpoint:
`POST /api/synthesize/stream` — accepts { text: string, voiceId?: string }
- assertBoard(req)
- Set headers: Content-Type: text/event-stream, Cache-Control: no-cache, Connection: keep-alive
- Iterate synthesizeSentenceStream, for each chunk: write SSE `data: { "index": N, "total": M, "audio": "<base64>" }\n\n`
- On completion: write `data: { "done": true }\n\n` then res.end()
6. In voice.ts, add multi-language endpoint:
`POST /api/synthesize/multi-lang` — accepts { text: string, voiceIds: string[] }
- assertBoard(req)
- Validate voiceIds is array with 1-5 entries
- Call synthesizeMultiLang, return JSON: { results: [{ voiceId, audio: base64 }] }
7. Write tests in 39-sentence-streaming.test.ts:
- Test splitSentences with basic and edge cases
- Test synthesizeSentenceStream yields correct number of chunks (mock piper execFile)
- Test synthesizeMultiLang returns correct number of entries (mock piper)
2. Streaming playback logic:
- Maintain a queue of audio Buffers (base64-decoded from SSE data)
- On first chunk received: decode base64 to ArrayBuffer, create Blob with audio/wav type, create object URL, set as audio src, begin playback immediately — this satisfies the "under 1 second" requirement
- On subsequent chunks: queue them. When current audio `onEnded`, pop next from queue, set as new src, play
- Show progress: "Playing 1/3..." in the UI
3. Sentence progress indicator:
- Display "Sentence N of M" text when streaming is active
- Show a small progress bar or dot indicator below the play button
4. Fallback: when streaming=false or if SSE connection fails, fall back to existing full-fetch behavior (current implementation)
5. Clean up: revoke all object URLs on unmount or when new text arrives
6. Keep the existing play/pause controls working for both modes
<success_criteria>
- VPIPE-07: First sentence plays while subsequent sentences still synthesizing (sentence-buffered SSE streaming)
- VPIPE-08: Single text can be synthesized in multiple languages via /api/synthesize/multi-lang
- Backward compatible: existing /api/synthesize POST unchanged
- All tests green </success_criteria>