13 KiB
| phase | plan | type | wave | depends_on | files_modified | autonomous | requirements | must_haves | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 37-web-chat-voice-ui | 01 | execute | 1 |
|
true |
|
|
Purpose: Phase 36 Tasks 2-3 (nexus-settings voiceMode schema, voice HTTP routes, voiceMode wiring in chat.ts) are not present on this branch. This plan cherry-picks or re-implements those deliverables, adds COOP/COEP headers for SharedArrayBuffer, installs @ricky0123/vad-react, copies VAD ONNX assets to ui/public/, and configures Vite dev server headers.
Output: Working server endpoints (transcribe, synthesize, nexus-settings), COOP/COEP isolation, VAD assets ready in ui/public/
<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/phases/37-web-chat-voice-ui/37-RESEARCH.mdFrom server/src/services/voice-pipeline.ts (ALREADY on this branch):
// voicePipelineService() exposes transcribe(buffer, format) and synthesize(text, voiceId?)
export function voicePipelineService(): { transcribe, synthesize, formatForVoice, transcodeToWav16k }
From server/src/app.ts (parent branch — route mounting pattern):
// Routes are mounted on an `api` Router via api.use(...)
// Pattern: import { xyzRoutes } from "./routes/xyz.js"; then api.use(xyzRoutes());
import { chatRoutes } from "./routes/chat.js";
api.use(chatRoutes(db, storageService, config));
From packages/shared/src/types/chat.ts (parent branch):
export interface ChatMessage {
id: string;
conversationId: string;
role: "user" | "assistant" | "system";
content: string;
messageType?: string | null;
// ... other fields
}
From packages/shared/src/validators/chat.ts (parent branch):
export const createMessageSchema = z.object({
content: z.string().min(1),
role: z.enum(["user", "assistant", "system"]).default("user"),
agentId: z.string().uuid().optional(),
// voiceMode NOT present on parent branch — must add
});
d0d7a23a (nexus-settings voiceMode schema extension)
- b964c0e4 (voiceMode in createMessageSchema + ChatMessage interface)
- 11508547 (voice HTTP routes)
- fd372eaf (voiceMode wiring in chat.ts + route mounting)
Try cherry-picking these 4 commits in order:
git cherry-pick d0d7a23a b964c0e4 11508547 fd372eaf
If cherry-pick conflicts, re-implement manually:
-
server/src/services/nexus-settings.ts — Add VOICE_MODES and VoiceMode type:
export const VOICE_MODES = ["text", "voice_input", "full_voice"] as const; export type VoiceMode = (typeof VOICE_MODES)[number];Add
voiceMode: z.enum(VOICE_MODES).default("text")to nexusSettingsSchema. AddtelegramToken: z.string().optional(),piperBinaryPath: z.string().optional(),whisperBinaryPath: z.string().optional(). -
server/src/routes/nexus-settings.ts — Create new file:
- GET /nexus/settings — returns nexusSettingsService().get()
- PATCH /nexus/settings — calls nexusSettingsService().set(req.body), returns updated
- Both routes call assertBoard(req) first
- Import Router from express, assertBoard from ./authz.js, nexusSettingsService from ../services/nexus-settings.js
-
server/src/routes/voice.ts — Create new file:
- POST /transcribe — accepts multipart audio upload via multer memoryStorage, calls voicePipelineService().transcribe(buffer, format), returns { text }
- POST /synthesize — accepts JSON { text, voiceId? }, calls voicePipelineService().synthesize(text, voiceId), returns audio/wav buffer
- Both routes call assertBoard(req)
- Import multer, Router, assertBoard, voicePipelineService, MAX_ATTACHMENT_BYTES
-
packages/shared/src/types/chat.ts — Add
voiceMode?: string | null;to ChatMessage interface if not present. -
packages/shared/src/validators/chat.ts — Add
voiceMode: z.enum(["text", "voice_input", "full_voice"]).optional()to createMessageSchema. -
server/src/routes/chat.ts — In the stream POST handler, destructure
voiceModefrom req.body alongside content and agentId. When voiceMode is "full_voice", call voicePipelineService().formatForVoice(aiContent) to produce SPOKEN/DETAILED format. Set messageType on stored message: "voice_full" if voiceMode==="full_voice", "voice_input" if voiceMode==="voice_input", else null. -
server/src/app.ts — Import and mount voiceRoutes and nexusSettingsRoutes:
import { nexusSettingsRoutes } from "./routes/nexus-settings.js"; import { voiceRoutes } from "./routes/voice.js"; // In the api router setup: api.use(nexusSettingsRoutes()); api.use(voiceRoutes()); -
COOP/COEP headers — In server/src/app.ts, add middleware BEFORE static file serving and vite dev middleware:
app.use((_req, res, next) => { res.setHeader("Cross-Origin-Opener-Policy", "same-origin"); res.setHeader("Cross-Origin-Embedder-Policy", "require-corp"); next(); });Place this before any
app.use(express.static(...))or vite middleware attachment. cd /opt/nexus/.claude/worktrees/agent-a009558f && grep -q "voiceRoutes" server/src/app.ts && grep -q "nexusSettingsRoutes" server/src/app.ts && grep -q "Cross-Origin-Opener-Policy" server/src/app.ts && grep -q "voiceMode" server/src/routes/chat.ts && grep -q "voice_full" server/src/routes/chat.ts && test -f server/src/routes/voice.ts && test -f server/src/routes/nexus-settings.ts && echo "PASS" || echo "FAIL" <acceptance_criteria>- grep "voiceRoutes" server/src/app.ts returns match
- grep "nexusSettingsRoutes" server/src/app.ts returns match
- grep "Cross-Origin-Opener-Policy" server/src/app.ts returns "same-origin"
- grep "Cross-Origin-Embedder-Policy" server/src/app.ts returns "require-corp"
- grep "voiceMode" server/src/routes/chat.ts returns match
- grep "voice_full" server/src/routes/chat.ts returns match
- server/src/routes/voice.ts exists with POST /transcribe and POST /synthesize
- server/src/routes/nexus-settings.ts exists with GET and PATCH /nexus/settings
- grep "VOICE_MODES" server/src/services/nexus-settings.ts returns match </acceptance_criteria> Phase 36 server deliverables present on branch. COOP/COEP headers added. Voice routes mounted. Chat stream accepts voiceMode.
-
Copy VAD assets from node_modules to ui/public/ for same-origin serving (avoids COEP blocking CDN):
cp node_modules/@ricky0123/vad-web/dist/vad.worklet.bundle.min.js ui/public/ cp node_modules/@ricky0123/vad-web/dist/silero_vad_legacy.onnx ui/public/ cp node_modules/@ricky0123/vad-web/dist/silero_vad_v5.onnx ui/public/If vad-web is in ui/node_modules/@ricky0123/vad-web/dist/, use that path instead. Verify all three files exist after copy.
-
Add a "copy-vad-assets" script to ui/package.json:
"copy-vad-assets": "cp node_modules/@ricky0123/vad-web/dist/vad.worklet.bundle.min.js public/ && cp node_modules/@ricky0123/vad-web/dist/silero_vad_legacy.onnx public/ && cp node_modules/@ricky0123/vad-web/dist/silero_vad_v5.onnx public/" -
Update ui/vite.config.ts — add COOP/COEP headers to dev server config:
server: { port: 5173, headers: { "Cross-Origin-Opener-Policy": "same-origin", "Cross-Origin-Embedder-Policy": "require-corp", }, proxy: { ... }, // keep existing proxy config },This ensures SharedArrayBuffer works in Vite dev mode too. cd /opt/nexus/.claude/worktrees/agent-a009558f && test -f ui/public/vad.worklet.bundle.min.js && test -f ui/public/silero_vad_legacy.onnx && test -f ui/public/silero_vad_v5.onnx && grep -q "vad-react" ui/package.json && grep -q "Cross-Origin-Opener-Policy" ui/vite.config.ts && echo "PASS" || echo "FAIL" <acceptance_criteria>
- ui/public/vad.worklet.bundle.min.js exists (non-zero size)
- ui/public/silero_vad_legacy.onnx exists (non-zero size)
- ui/public/silero_vad_v5.onnx exists (non-zero size)
- grep "vad-react" ui/package.json returns match
- grep "Cross-Origin-Opener-Policy" ui/vite.config.ts returns "same-origin"
- grep "Cross-Origin-Embedder-Policy" ui/vite.config.ts returns "require-corp"
- grep "copy-vad-assets" ui/package.json returns match </acceptance_criteria> VAD library installed. ONNX model files and worklet bundle served from ui/public/. Vite dev server sends COOP/COEP headers. SharedArrayBuffer available in dev.
<success_criteria> All Phase 36 server deliverables present. COOP/COEP headers set on both Express and Vite dev server. VAD assets served from same-origin. Foundation ready for frontend voice components. </success_criteria>
After completion, create `.planning/phases/37-web-chat-voice-ui/37-01-SUMMARY.md`