docs(39): create phase plan for voice polish
Two plans in wave 1 (parallel): - 39-01: Sentence-buffered TTS streaming + multi-language synthesis (VPIPE-07, VPIPE-08) - 39-02: Onboarding voice hardware capability probe (ONBRD-01, ONBRD-02) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Nexus Dev
parent
b54130d991
commit
3690a47fd5
3 changed files with 499 additions and 2 deletions
|
|
@ -165,7 +165,11 @@ Plans:
|
||||||
2. A user can request the same agent response as audio in both English and Danish; both OGG files are generated and available for playback without a second agent call
|
2. A user can request the same agent response as audio in both English and Danish; both OGG files are generated and available for playback without a second agent call
|
||||||
3. On a fresh install, the onboarding hardware probe reports whether Whisper STT and Piper TTS are runnable on the detected hardware tier
|
3. On a fresh install, the onboarding hardware probe reports whether Whisper STT and Piper TTS are runnable on the detected hardware tier
|
||||||
4. The onboarding voice step activates (showing enable/skip options) only when the hardware probe confirms sufficient capability; on hardware below threshold it shows a capability note and skips to the next step
|
4. The onboarding voice step activates (showing enable/skip options) only when the hardware probe confirms sufficient capability; on hardware below threshold it shows a capability note and skips to the next step
|
||||||
**Plans**: TBD
|
**Plans**: 2 plans
|
||||||
|
|
||||||
|
Plans:
|
||||||
|
- [ ] 39-01-PLAN.md — Sentence-buffered TTS streaming + multi-language synthesis
|
||||||
|
- [ ] 39-02-PLAN.md — Onboarding voice hardware capability probe
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -224,4 +228,4 @@ All 23 v1.6 requirements are mapped to exactly one phase. No orphans.
|
||||||
| 36. Voice Pipeline Foundation | v1.6 | 2/3 | Complete | 2026-04-04 |
|
| 36. Voice Pipeline Foundation | v1.6 | 2/3 | Complete | 2026-04-04 |
|
||||||
| 37. Web Chat Voice UI | v1.6 | 3/4 | Complete | 2026-04-04 |
|
| 37. Web Chat Voice UI | v1.6 | 3/4 | Complete | 2026-04-04 |
|
||||||
| 38. Telegram Bridge | v1.6 | 3/3 | Complete | 2026-04-04 |
|
| 38. Telegram Bridge | v1.6 | 3/3 | Complete | 2026-04-04 |
|
||||||
| 39. Voice Polish | v1.6 | 0/TBD | Not started | - |
|
| 39. Voice Polish | v1.6 | 0/2 | Not started | - |
|
||||||
|
|
|
||||||
238
.planning/phases/39-voice-polish/39-01-PLAN.md
Normal file
238
.planning/phases/39-voice-polish/39-01-PLAN.md
Normal file
|
|
@ -0,0 +1,238 @@
|
||||||
|
---
|
||||||
|
phase: 39-voice-polish
|
||||||
|
plan: 01
|
||||||
|
type: execute
|
||||||
|
wave: 1
|
||||||
|
depends_on: []
|
||||||
|
files_modified:
|
||||||
|
- server/src/services/voice-pipeline.ts
|
||||||
|
- server/src/routes/voice.ts
|
||||||
|
- ui/src/components/ChatVoicePlayer.tsx
|
||||||
|
- server/src/__tests__/39-sentence-streaming.test.ts
|
||||||
|
autonomous: true
|
||||||
|
requirements: [VPIPE-07, VPIPE-08]
|
||||||
|
|
||||||
|
must_haves:
|
||||||
|
truths:
|
||||||
|
- "First sentence audio begins playing before full response finishes synthesizing"
|
||||||
|
- "User can request same text synthesized in multiple languages simultaneously"
|
||||||
|
- "Existing single-language synthesize endpoint still works unchanged"
|
||||||
|
artifacts:
|
||||||
|
- path: "server/src/services/voice-pipeline.ts"
|
||||||
|
provides: "synthesizeSentenceStream generator + synthesizeMultiLang method"
|
||||||
|
contains: "synthesizeSentenceStream"
|
||||||
|
- path: "server/src/routes/voice.ts"
|
||||||
|
provides: "GET /api/synthesize/stream SSE endpoint + POST /api/synthesize/multi-lang"
|
||||||
|
contains: "synthesize/stream"
|
||||||
|
- path: "ui/src/components/ChatVoicePlayer.tsx"
|
||||||
|
provides: "Streaming audio playback via sentence-buffered fetch"
|
||||||
|
contains: "EventSource\\|ReadableStream\\|sentence"
|
||||||
|
- path: "server/src/__tests__/39-sentence-streaming.test.ts"
|
||||||
|
provides: "Tests for sentence splitting, multi-lang, and streaming"
|
||||||
|
contains: "describe.*sentence"
|
||||||
|
key_links:
|
||||||
|
- from: "ui/src/components/ChatVoicePlayer.tsx"
|
||||||
|
to: "/api/synthesize/stream"
|
||||||
|
via: "EventSource or fetch with ReadableStream"
|
||||||
|
pattern: "synthesize/stream"
|
||||||
|
- from: "server/src/routes/voice.ts"
|
||||||
|
to: "server/src/services/voice-pipeline.ts"
|
||||||
|
via: "synthesizeSentenceStream generator"
|
||||||
|
pattern: "synthesizeSentenceStream"
|
||||||
|
---
|
||||||
|
|
||||||
|
<objective>
|
||||||
|
Sentence-buffered TTS streaming and multi-language synthesis.
|
||||||
|
|
||||||
|
Purpose: Voice responses begin playing before full synthesis completes (under 1s to first audio), and users can synthesize the same response in multiple languages without a second agent call.
|
||||||
|
Output: Streaming synthesize endpoint, multi-language endpoint, updated ChatVoicePlayer with progressive playback.
|
||||||
|
</objective>
|
||||||
|
|
||||||
|
<execution_context>
|
||||||
|
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||||
|
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||||
|
</execution_context>
|
||||||
|
|
||||||
|
<context>
|
||||||
|
@.planning/ROADMAP.md
|
||||||
|
@.planning/REQUIREMENTS.md
|
||||||
|
@.planning/phases/39-voice-polish/39-CONTEXT.md
|
||||||
|
|
||||||
|
@server/src/services/voice-pipeline.ts
|
||||||
|
@server/src/routes/voice.ts
|
||||||
|
@ui/src/components/ChatVoicePlayer.tsx
|
||||||
|
|
||||||
|
<interfaces>
|
||||||
|
<!-- Key types and contracts from existing codebase -->
|
||||||
|
|
||||||
|
From server/src/services/voice-pipeline.ts:
|
||||||
|
```typescript
|
||||||
|
// synthesize already does sentence splitting internally:
|
||||||
|
// const sentences = text.split(/(?<=[.!?])\s+/).filter((s) => s.length > 0);
|
||||||
|
// Currently concatenates all sentence buffers before returning.
|
||||||
|
// Need to yield each sentence buffer as it completes.
|
||||||
|
|
||||||
|
export function voicePipelineService(): {
|
||||||
|
transcribe(buffer: Buffer, format: "webm" | "ogg" | "wav"): Promise<{ text: string; language?: string }>;
|
||||||
|
synthesize(text: string, voiceId?: string): Promise<Buffer>;
|
||||||
|
formatForVoice(text: string): string;
|
||||||
|
transcodeToWav16k(inputBuffer: Buffer, inputFormat: string): Promise<Buffer>;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
From server/src/routes/voice.ts:
|
||||||
|
```typescript
|
||||||
|
// POST /api/synthesize — takes { text, voiceId }, returns audio/wav buffer
|
||||||
|
// POST /api/transcribe — takes multipart audio, returns { text, language? }
|
||||||
|
```
|
||||||
|
|
||||||
|
From ui/src/components/ChatVoicePlayer.tsx:
|
||||||
|
```typescript
|
||||||
|
interface ChatVoicePlayerProps {
|
||||||
|
text: string;
|
||||||
|
autoPlay?: boolean;
|
||||||
|
}
|
||||||
|
// Currently fetches full audio blob from POST /api/synthesize, then plays
|
||||||
|
```
|
||||||
|
</interfaces>
|
||||||
|
</context>
|
||||||
|
|
||||||
|
<tasks>
|
||||||
|
|
||||||
|
<task type="auto" tdd="true">
|
||||||
|
<name>Task 1: Sentence-buffered synthesis + multi-language TTS in voice pipeline and routes</name>
|
||||||
|
<files>server/src/services/voice-pipeline.ts, server/src/routes/voice.ts, server/src/__tests__/39-sentence-streaming.test.ts</files>
|
||||||
|
<read_first>
|
||||||
|
- server/src/services/voice-pipeline.ts (full file — understand synthesize internals)
|
||||||
|
- server/src/routes/voice.ts (full file — understand route patterns)
|
||||||
|
- server/src/routes/authz.ts (assertBoard pattern)
|
||||||
|
</read_first>
|
||||||
|
<behavior>
|
||||||
|
- Test: splitSentences("Hello world. How are you? I am fine.") returns ["Hello world.", "How are you?", "I am fine."]
|
||||||
|
- Test: splitSentences("Dr. Smith went to D.C. He liked it.") returns ["Dr. Smith went to D.C.", "He liked it."] (abbreviation-aware)
|
||||||
|
- Test: synthesizeSentenceStream yields Buffer chunks one per sentence
|
||||||
|
- Test: synthesizeMultiLang({ text, languages: ["en_US-lessac-medium", "da_DK-talesyntese-medium"] }) returns Map with two Buffer entries
|
||||||
|
</behavior>
|
||||||
|
<action>
|
||||||
|
1. In voice-pipeline.ts, extract sentence splitting into an exported `splitSentences(text: string): string[]` function. Use regex: split on /(?<=[.!?])\s+/ (same as current), filter empty. Keep existing synthesize() working by calling splitSentences internally.
|
||||||
|
|
||||||
|
2. Add `async *synthesizeSentenceStream(text: string, voiceId?: string): AsyncGenerator<{ index: number; total: number; audio: Buffer }>` method:
|
||||||
|
- Call splitSentences(text) to get sentences array
|
||||||
|
- For each sentence, call piper (same as current synthesize logic), yield { index, total: sentences.length, audio: audioBuffer } immediately
|
||||||
|
- This gives the consumer each sentence's audio as soon as it is ready
|
||||||
|
|
||||||
|
3. Add `async synthesizeMultiLang(text: string, voiceIds: string[]): Promise<Map<string, Buffer>>` method:
|
||||||
|
- For each voiceId, call existing synthesize(text, voiceId) in parallel via Promise.all
|
||||||
|
- Return Map<voiceId, Buffer>
|
||||||
|
|
||||||
|
4. Update the return signature of voicePipelineService() to include the new methods.
|
||||||
|
|
||||||
|
5. In voice.ts, add streaming endpoint:
|
||||||
|
`POST /api/synthesize/stream` — accepts { text: string, voiceId?: string }
|
||||||
|
- assertBoard(req)
|
||||||
|
- Set headers: Content-Type: text/event-stream, Cache-Control: no-cache, Connection: keep-alive
|
||||||
|
- Iterate synthesizeSentenceStream, for each chunk: write SSE `data: { "index": N, "total": M, "audio": "<base64>" }\n\n`
|
||||||
|
- On completion: write `data: { "done": true }\n\n` then res.end()
|
||||||
|
|
||||||
|
6. In voice.ts, add multi-language endpoint:
|
||||||
|
`POST /api/synthesize/multi-lang` — accepts { text: string, voiceIds: string[] }
|
||||||
|
- assertBoard(req)
|
||||||
|
- Validate voiceIds is array with 1-5 entries
|
||||||
|
- Call synthesizeMultiLang, return JSON: { results: [{ voiceId, audio: base64 }] }
|
||||||
|
|
||||||
|
7. Write tests in 39-sentence-streaming.test.ts:
|
||||||
|
- Test splitSentences with basic and edge cases
|
||||||
|
- Test synthesizeSentenceStream yields correct number of chunks (mock piper execFile)
|
||||||
|
- Test synthesizeMultiLang returns correct number of entries (mock piper)
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd /opt/nexus && npx vitest run server/src/__tests__/39-sentence-streaming.test.ts --reporter=verbose 2>&1 | tail -30</automated>
|
||||||
|
</verify>
|
||||||
|
<acceptance_criteria>
|
||||||
|
- grep -q "splitSentences" server/src/services/voice-pipeline.ts
|
||||||
|
- grep -q "synthesizeSentenceStream" server/src/services/voice-pipeline.ts
|
||||||
|
- grep -q "synthesizeMultiLang" server/src/services/voice-pipeline.ts
|
||||||
|
- grep -q "synthesize/stream" server/src/routes/voice.ts
|
||||||
|
- grep -q "synthesize/multi-lang" server/src/routes/voice.ts
|
||||||
|
- grep -q "text/event-stream" server/src/routes/voice.ts
|
||||||
|
- test -f server/src/__tests__/39-sentence-streaming.test.ts
|
||||||
|
</acceptance_criteria>
|
||||||
|
<done>
|
||||||
|
- splitSentences exported and tested
|
||||||
|
- synthesizeSentenceStream yields per-sentence audio chunks via AsyncGenerator
|
||||||
|
- synthesizeMultiLang synthesizes same text in N languages in parallel
|
||||||
|
- POST /api/synthesize/stream sends SSE with base64 audio per sentence
|
||||||
|
- POST /api/synthesize/multi-lang returns array of { voiceId, audio } pairs
|
||||||
|
- Existing POST /api/synthesize unchanged (backward compatible)
|
||||||
|
- All tests pass
|
||||||
|
</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 2: ChatVoicePlayer sentence-buffered streaming playback</name>
|
||||||
|
<files>ui/src/components/ChatVoicePlayer.tsx</files>
|
||||||
|
<read_first>
|
||||||
|
- ui/src/components/ChatVoicePlayer.tsx (full file — current playback implementation)
|
||||||
|
- ui/src/components/ChatMessage.tsx (how ChatVoicePlayer is used)
|
||||||
|
</read_first>
|
||||||
|
<action>
|
||||||
|
1. Refactor ChatVoicePlayer to support streaming playback mode:
|
||||||
|
- Add a `streaming` prop (default true) to ChatVoicePlayerProps
|
||||||
|
- When streaming=true, use EventSource to connect to POST /api/synthesize/stream (use fetch with ReadableStream since EventSource only supports GET — instead use fetch POST then parse SSE text manually from response body stream)
|
||||||
|
- Actually: use fetch with { method: "POST", body, headers } and read response.body as ReadableStream, parsing SSE lines manually
|
||||||
|
|
||||||
|
2. Streaming playback logic:
|
||||||
|
- Maintain a queue of audio Buffers (base64-decoded from SSE data)
|
||||||
|
- On first chunk received: decode base64 to ArrayBuffer, create Blob with audio/wav type, create object URL, set as audio src, begin playback immediately — this satisfies the "under 1 second" requirement
|
||||||
|
- On subsequent chunks: queue them. When current audio `onEnded`, pop next from queue, set as new src, play
|
||||||
|
- Show progress: "Playing 1/3..." in the UI
|
||||||
|
|
||||||
|
3. Sentence progress indicator:
|
||||||
|
- Display "Sentence N of M" text when streaming is active
|
||||||
|
- Show a small progress bar or dot indicator below the play button
|
||||||
|
|
||||||
|
4. Fallback: when streaming=false or if SSE connection fails, fall back to existing full-fetch behavior (current implementation)
|
||||||
|
|
||||||
|
5. Clean up: revoke all object URLs on unmount or when new text arrives
|
||||||
|
|
||||||
|
6. Keep the existing play/pause controls working for both modes
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd /opt/nexus && npx tsc --noEmit --project ui/tsconfig.json 2>&1 | tail -20</automated>
|
||||||
|
</verify>
|
||||||
|
<acceptance_criteria>
|
||||||
|
- grep -q "synthesize/stream" ui/src/components/ChatVoicePlayer.tsx
|
||||||
|
- grep -q "ReadableStream\|getReader\|TextDecoder" ui/src/components/ChatVoicePlayer.tsx
|
||||||
|
- grep -q "queue\|Queue\|audioQueue" ui/src/components/ChatVoicePlayer.tsx
|
||||||
|
- grep -q "Sentence.*of\|sentence.*progress\|Playing.*of" ui/src/components/ChatVoicePlayer.tsx
|
||||||
|
</acceptance_criteria>
|
||||||
|
<done>
|
||||||
|
- ChatVoicePlayer connects to /api/synthesize/stream via fetch POST + ReadableStream
|
||||||
|
- First sentence audio begins playing as soon as first SSE chunk arrives
|
||||||
|
- Subsequent sentences auto-play in sequence from queue
|
||||||
|
- Progress indicator shows current sentence position
|
||||||
|
- Falls back to full-fetch on stream error
|
||||||
|
- TypeScript compiles without errors
|
||||||
|
</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
</tasks>
|
||||||
|
|
||||||
|
<verification>
|
||||||
|
1. TypeScript compiles: `npx tsc --noEmit` in both server and ui
|
||||||
|
2. Tests pass: `npx vitest run server/src/__tests__/39-sentence-streaming.test.ts`
|
||||||
|
3. Existing synthesize endpoint still works: grep confirms original POST /api/synthesize route unchanged
|
||||||
|
4. SSE endpoint exists: grep confirms text/event-stream header in voice.ts
|
||||||
|
5. Multi-lang endpoint exists: grep confirms synthesize/multi-lang in voice.ts
|
||||||
|
</verification>
|
||||||
|
|
||||||
|
<success_criteria>
|
||||||
|
- VPIPE-07: First sentence plays while subsequent sentences still synthesizing (sentence-buffered SSE streaming)
|
||||||
|
- VPIPE-08: Single text can be synthesized in multiple languages via /api/synthesize/multi-lang
|
||||||
|
- Backward compatible: existing /api/synthesize POST unchanged
|
||||||
|
- All tests green
|
||||||
|
</success_criteria>
|
||||||
|
|
||||||
|
<output>
|
||||||
|
After completion, create `.planning/phases/39-voice-polish/39-01-SUMMARY.md`
|
||||||
|
</output>
|
||||||
255
.planning/phases/39-voice-polish/39-02-PLAN.md
Normal file
255
.planning/phases/39-voice-polish/39-02-PLAN.md
Normal file
|
|
@ -0,0 +1,255 @@
|
||||||
|
---
|
||||||
|
phase: 39-voice-polish
|
||||||
|
plan: 02
|
||||||
|
type: execute
|
||||||
|
wave: 1
|
||||||
|
depends_on: []
|
||||||
|
files_modified:
|
||||||
|
- server/src/services/hardware.ts
|
||||||
|
- server/src/routes/hardware.ts
|
||||||
|
- ui/src/components/onboarding/VoiceStep.tsx
|
||||||
|
- ui/src/components/NexusOnboardingWizard.tsx
|
||||||
|
- ui/src/hooks/useHardwareInfo.ts
|
||||||
|
- server/src/__tests__/39-voice-hardware-probe.test.ts
|
||||||
|
autonomous: true
|
||||||
|
requirements: [ONBRD-01, ONBRD-02]
|
||||||
|
|
||||||
|
must_haves:
|
||||||
|
truths:
|
||||||
|
- "Onboarding hardware probe reports whether Whisper STT is runnable on detected hardware"
|
||||||
|
- "Onboarding hardware probe reports whether Piper TTS is runnable on detected hardware"
|
||||||
|
- "VoiceStep shows enable/skip when hardware is sufficient"
|
||||||
|
- "VoiceStep shows capability note and auto-skips or shows skip-only when hardware is insufficient"
|
||||||
|
artifacts:
|
||||||
|
- path: "server/src/services/hardware.ts"
|
||||||
|
provides: "voiceCapability probe in HardwareInfo"
|
||||||
|
contains: "whisperAvailable"
|
||||||
|
- path: "server/src/routes/hardware.ts"
|
||||||
|
provides: "voice capability data in /system/providers response"
|
||||||
|
contains: "voiceCapability"
|
||||||
|
- path: "ui/src/components/onboarding/VoiceStep.tsx"
|
||||||
|
provides: "Hardware-aware voice step with conditional enable/skip"
|
||||||
|
contains: "whisperAvailable\\|piperAvailable\\|voiceCapability"
|
||||||
|
- path: "server/src/__tests__/39-voice-hardware-probe.test.ts"
|
||||||
|
provides: "Tests for voice capability detection"
|
||||||
|
contains: "describe.*voice.*capability"
|
||||||
|
key_links:
|
||||||
|
- from: "ui/src/components/onboarding/VoiceStep.tsx"
|
||||||
|
to: "ui/src/hooks/useHardwareInfo.ts"
|
||||||
|
via: "voiceCapability prop from hardware info"
|
||||||
|
pattern: "voiceCapability"
|
||||||
|
- from: "ui/src/components/NexusOnboardingWizard.tsx"
|
||||||
|
to: "ui/src/components/onboarding/VoiceStep.tsx"
|
||||||
|
via: "passes hardware voiceCapability as prop"
|
||||||
|
pattern: "voiceCapability"
|
||||||
|
---
|
||||||
|
|
||||||
|
<objective>
|
||||||
|
Onboarding voice hardware detection — probe for Whisper STT and Piper TTS capability during onboarding and gate the voice enable step accordingly.
|
||||||
|
|
||||||
|
Purpose: New installs detect whether the machine can run STT/TTS before offering voice features, preventing users from enabling voice on incapable hardware.
|
||||||
|
Output: Extended hardware probe with voice capability, updated VoiceStep with hardware-aware UI.
|
||||||
|
</objective>
|
||||||
|
|
||||||
|
<execution_context>
|
||||||
|
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||||
|
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||||
|
</execution_context>
|
||||||
|
|
||||||
|
<context>
|
||||||
|
@.planning/ROADMAP.md
|
||||||
|
@.planning/REQUIREMENTS.md
|
||||||
|
@.planning/phases/39-voice-polish/39-CONTEXT.md
|
||||||
|
|
||||||
|
@server/src/services/hardware.ts
|
||||||
|
@server/src/routes/hardware.ts
|
||||||
|
@ui/src/components/onboarding/VoiceStep.tsx
|
||||||
|
@ui/src/components/NexusOnboardingWizard.tsx
|
||||||
|
@ui/src/hooks/useHardwareInfo.ts
|
||||||
|
|
||||||
|
<interfaces>
|
||||||
|
From server/src/services/hardware.ts:
|
||||||
|
```typescript
|
||||||
|
export type HardwareTier = "gpu" | "apple_silicon" | "cpu_only";
|
||||||
|
|
||||||
|
export interface HardwareInfo {
|
||||||
|
totalGb: number;
|
||||||
|
freeGb: number;
|
||||||
|
usableGb: number;
|
||||||
|
platform: NodeJS.Platform;
|
||||||
|
gpuName: string | null;
|
||||||
|
gpuVramGb: number | null;
|
||||||
|
unifiedMemory: boolean;
|
||||||
|
hardwareTier: HardwareTier;
|
||||||
|
cpuModel: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function hardwareService(): { detect(): Promise<HardwareInfo> }
|
||||||
|
```
|
||||||
|
|
||||||
|
From ui/src/components/onboarding/VoiceStep.tsx:
|
||||||
|
```typescript
|
||||||
|
interface VoiceStepProps {
|
||||||
|
onEnable: () => void;
|
||||||
|
onSkip: () => void;
|
||||||
|
}
|
||||||
|
// Currently only checks for microphone via navigator.mediaDevices
|
||||||
|
// Does NOT check if Whisper/Piper binaries are available on server
|
||||||
|
```
|
||||||
|
|
||||||
|
From ui/src/hooks/useHardwareInfo.ts:
|
||||||
|
```typescript
|
||||||
|
// Hook that fetches GET /system/providers and returns HardwareInfo
|
||||||
|
// Used by NexusOnboardingWizard.tsx
|
||||||
|
```
|
||||||
|
</interfaces>
|
||||||
|
</context>
|
||||||
|
|
||||||
|
<tasks>
|
||||||
|
|
||||||
|
<task type="auto" tdd="true">
|
||||||
|
<name>Task 1: Voice capability probe in hardware service and route</name>
|
||||||
|
<files>server/src/services/hardware.ts, server/src/routes/hardware.ts, server/src/__tests__/39-voice-hardware-probe.test.ts</files>
|
||||||
|
<read_first>
|
||||||
|
- server/src/services/hardware.ts (full file — understand detect() and HardwareInfo)
|
||||||
|
- server/src/routes/hardware.ts (full file — understand route patterns)
|
||||||
|
- server/src/services/voice-pipeline.ts (lines 76-125 — understand whisper/piper detection patterns)
|
||||||
|
</read_first>
|
||||||
|
<behavior>
|
||||||
|
- Test: detectVoiceCapability() returns { whisperAvailable: true, piperAvailable: true } when both binaries resolve via execFile --version
|
||||||
|
- Test: detectVoiceCapability() returns { whisperAvailable: false, piperAvailable: false } when both binaries throw ENOENT
|
||||||
|
- Test: detectVoiceCapability() returns { whisperAvailable: true, piperAvailable: false } when only whisper is found
|
||||||
|
- Test: Hardware tier "cpu_only" with < 4GB RAM sets voiceTierSufficient to false
|
||||||
|
- Test: Hardware tier "apple_silicon" with >= 8GB RAM sets voiceTierSufficient to true
|
||||||
|
</behavior>
|
||||||
|
<action>
|
||||||
|
1. In hardware.ts, add a `VoiceCapability` interface:
|
||||||
|
```typescript
|
||||||
|
export interface VoiceCapability {
|
||||||
|
whisperAvailable: boolean;
|
||||||
|
piperAvailable: boolean;
|
||||||
|
voiceTierSufficient: boolean; // true if hardware tier >= apple_silicon OR (cpu_only with >= 4GB free)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Extend HardwareInfo interface with `voiceCapability: VoiceCapability`.
|
||||||
|
|
||||||
|
3. Add `async detectVoiceCapability(): Promise<VoiceCapability>` to hardwareService:
|
||||||
|
- Probe whisper-cpp: try `execFile("whisper-cpp", ["--help"])` with 2s timeout. If resolves → whisperAvailable=true. If ENOENT → try `execFile("whisper", ["--help"])` as fallback. Both fail → false.
|
||||||
|
- Probe piper: try `execFile("piper", ["--help"])` with 2s timeout. If resolves → piperAvailable=true. Catch → false.
|
||||||
|
- voiceTierSufficient: true if hardwareTier is "apple_silicon" or "gpu", OR if "cpu_only" with freeGb >= 4
|
||||||
|
- Use execFile from node:child_process with promisify pattern (or the existing execFileAsync if extracted)
|
||||||
|
|
||||||
|
4. Call detectVoiceCapability() inside detect() AFTER the existing hardware detection, add result to HardwareInfo. Use a separate 3s timeout to avoid slowing down hardware detection if voice probes hang.
|
||||||
|
|
||||||
|
5. In hardware.ts route: no changes needed — it already returns the full HardwareInfo object from detect(), so voiceCapability will be included automatically.
|
||||||
|
|
||||||
|
6. Write tests in 39-voice-hardware-probe.test.ts:
|
||||||
|
- Mock execFile (child_process) to test whisper/piper detection
|
||||||
|
- Test voiceTierSufficient logic for each hardware tier
|
||||||
|
- Test that detectVoiceCapability timeout does not exceed 3s
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd /opt/nexus && npx vitest run server/src/__tests__/39-voice-hardware-probe.test.ts --reporter=verbose 2>&1 | tail -30</automated>
|
||||||
|
</verify>
|
||||||
|
<acceptance_criteria>
|
||||||
|
- grep -q "VoiceCapability" server/src/services/hardware.ts
|
||||||
|
- grep -q "whisperAvailable" server/src/services/hardware.ts
|
||||||
|
- grep -q "piperAvailable" server/src/services/hardware.ts
|
||||||
|
- grep -q "voiceTierSufficient" server/src/services/hardware.ts
|
||||||
|
- grep -q "voiceCapability" server/src/services/hardware.ts
|
||||||
|
- test -f server/src/__tests__/39-voice-hardware-probe.test.ts
|
||||||
|
</acceptance_criteria>
|
||||||
|
<done>
|
||||||
|
- HardwareInfo includes voiceCapability with whisperAvailable, piperAvailable, voiceTierSufficient
|
||||||
|
- Binary detection probes whisper-cpp/whisper and piper with 2s timeout each
|
||||||
|
- voiceTierSufficient is true for apple_silicon/gpu, or cpu_only with >= 4GB free RAM
|
||||||
|
- GET /system/providers response now includes voiceCapability object
|
||||||
|
- All tests pass
|
||||||
|
</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 2: VoiceStep hardware-aware UI with conditional enable/skip</name>
|
||||||
|
<files>ui/src/components/onboarding/VoiceStep.tsx, ui/src/components/NexusOnboardingWizard.tsx, ui/src/hooks/useHardwareInfo.ts</files>
|
||||||
|
<read_first>
|
||||||
|
- ui/src/components/onboarding/VoiceStep.tsx (full file)
|
||||||
|
- ui/src/components/NexusOnboardingWizard.tsx (full file — understand step 4 voice wiring)
|
||||||
|
- ui/src/hooks/useHardwareInfo.ts (full file — understand HardwareInfo type on client)
|
||||||
|
</read_first>
|
||||||
|
<action>
|
||||||
|
1. Update useHardwareInfo.ts: ensure the TypeScript type for hardware info includes the new voiceCapability field. Add:
|
||||||
|
```typescript
|
||||||
|
interface VoiceCapability {
|
||||||
|
whisperAvailable: boolean;
|
||||||
|
piperAvailable: boolean;
|
||||||
|
voiceTierSufficient: boolean;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Add `voiceCapability?: VoiceCapability` to the HardwareInfo type used in the hook. The "?" makes it backward-compatible if the server hasn't been updated yet.
|
||||||
|
|
||||||
|
2. Update VoiceStep props to accept voice capability:
|
||||||
|
```typescript
|
||||||
|
interface VoiceStepProps {
|
||||||
|
onEnable: () => void;
|
||||||
|
onSkip: () => void;
|
||||||
|
voiceCapability?: {
|
||||||
|
whisperAvailable: boolean;
|
||||||
|
piperAvailable: boolean;
|
||||||
|
voiceTierSufficient: boolean;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Update VoiceStep rendering logic:
|
||||||
|
- If voiceCapability is undefined (loading/missing): show current behavior (mic check only)
|
||||||
|
- If voiceCapability.voiceTierSufficient === false: show capability note ("Your hardware may not support voice features. Voice requires at least 4GB free RAM."), show Skip button only (no Enable), do NOT auto-skip — let user read the note
|
||||||
|
- If voiceCapability.whisperAvailable && voiceCapability.piperAvailable: show green checkmark next to STT and TTS labels ("Whisper detected", "Piper detected"), show Enable + Skip buttons
|
||||||
|
- If whisperAvailable but NOT piperAvailable: show checkmark for STT, warning for TTS ("Piper not found — install piper for voice output"), still allow Enable (voice input will work, output won't)
|
||||||
|
- If neither available but tier is sufficient: show note "Install whisper-cpp and piper for voice features", show Skip button, dim the Enable button but keep it clickable (user may install later)
|
||||||
|
|
||||||
|
4. In NexusOnboardingWizard.tsx, pass voiceCapability to VoiceStep:
|
||||||
|
- hardwareInfo already comes from useHardwareInfo hook
|
||||||
|
- Pass `voiceCapability={hardwareInfo?.voiceCapability}` to VoiceStep in step 4
|
||||||
|
|
||||||
|
5. Keep existing microphone detection in VoiceStep — it checks client-side mic availability which is complementary to server-side binary detection.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd /opt/nexus && npx tsc --noEmit --project ui/tsconfig.json 2>&1 | tail -20</automated>
|
||||||
|
</verify>
|
||||||
|
<acceptance_criteria>
|
||||||
|
- grep -q "voiceCapability" ui/src/components/onboarding/VoiceStep.tsx
|
||||||
|
- grep -q "whisperAvailable" ui/src/components/onboarding/VoiceStep.tsx
|
||||||
|
- grep -q "piperAvailable" ui/src/components/onboarding/VoiceStep.tsx
|
||||||
|
- grep -q "voiceTierSufficient" ui/src/components/onboarding/VoiceStep.tsx
|
||||||
|
- grep -q "voiceCapability" ui/src/components/NexusOnboardingWizard.tsx
|
||||||
|
- grep -q "VoiceCapability" ui/src/hooks/useHardwareInfo.ts
|
||||||
|
</acceptance_criteria>
|
||||||
|
<done>
|
||||||
|
- VoiceStep accepts voiceCapability prop and renders conditionally based on hardware detection
|
||||||
|
- Sufficient hardware + binaries present: shows enable/skip with green checkmarks
|
||||||
|
- Insufficient hardware: shows capability note and skip-only
|
||||||
|
- Missing binaries on sufficient hardware: shows install note with dimmed enable
|
||||||
|
- NexusOnboardingWizard passes voiceCapability from hardware probe to VoiceStep
|
||||||
|
- TypeScript compiles without errors
|
||||||
|
</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
</tasks>
|
||||||
|
|
||||||
|
<verification>
|
||||||
|
1. TypeScript compiles: `npx tsc --noEmit` for both server and ui
|
||||||
|
2. Tests pass: `npx vitest run server/src/__tests__/39-voice-hardware-probe.test.ts`
|
||||||
|
3. VoiceStep renders hardware-aware UI: grep confirms voiceCapability, whisperAvailable, piperAvailable in VoiceStep.tsx
|
||||||
|
4. Wizard wiring: grep confirms voiceCapability prop passed in NexusOnboardingWizard.tsx
|
||||||
|
</verification>
|
||||||
|
|
||||||
|
<success_criteria>
|
||||||
|
- ONBRD-01: Onboarding hardware probe reports Whisper STT and Piper TTS capability (whisperAvailable, piperAvailable, voiceTierSufficient in HardwareInfo)
|
||||||
|
- ONBRD-02: VoiceStep activates enable/skip when hardware is capable, shows capability note when below threshold
|
||||||
|
- Backward compatible: existing hardware endpoint still works, VoiceStep degrades gracefully if voiceCapability is undefined
|
||||||
|
</success_criteria>
|
||||||
|
|
||||||
|
<output>
|
||||||
|
After completion, create `.planning/phases/39-voice-polish/39-02-SUMMARY.md`
|
||||||
|
</output>
|
||||||
Loading…
Add table
Reference in a new issue