297 lines
13 KiB
Markdown
297 lines
13 KiB
Markdown
---
|
|
phase: 37-web-chat-voice-ui
|
|
plan: 01
|
|
type: execute
|
|
wave: 1
|
|
depends_on: []
|
|
files_modified:
|
|
- server/src/services/nexus-settings.ts
|
|
- server/src/routes/nexus-settings.ts
|
|
- server/src/routes/voice.ts
|
|
- server/src/routes/chat.ts
|
|
- server/src/app.ts
|
|
- packages/shared/src/types/chat.ts
|
|
- packages/shared/src/validators/chat.ts
|
|
- ui/vite.config.ts
|
|
- ui/package.json
|
|
- ui/public/vad.worklet.bundle.min.js
|
|
- ui/public/silero_vad_legacy.onnx
|
|
- ui/public/silero_vad_v5.onnx
|
|
autonomous: true
|
|
requirements:
|
|
- WCHAT-01
|
|
- WCHAT-02
|
|
- WCHAT-04
|
|
|
|
must_haves:
|
|
truths:
|
|
- "POST /api/transcribe accepts audio upload and returns { text }"
|
|
- "POST /api/synthesize accepts { text } and returns audio/wav"
|
|
- "GET /api/nexus/settings returns voiceMode field"
|
|
- "PATCH /api/nexus/settings accepts voiceMode update"
|
|
- "Chat stream endpoint accepts voiceMode in request body"
|
|
- "SharedArrayBuffer is available in browser (COOP/COEP headers set)"
|
|
- "VAD ONNX model files are served from /vad.worklet.bundle.min.js, /silero_vad_legacy.onnx, /silero_vad_v5.onnx"
|
|
artifacts:
|
|
- path: "server/src/routes/voice.ts"
|
|
provides: "POST /api/transcribe and POST /api/synthesize"
|
|
exports: ["voiceRoutes"]
|
|
- path: "server/src/routes/nexus-settings.ts"
|
|
provides: "GET/PATCH /api/nexus/settings"
|
|
exports: ["nexusSettingsRoutes"]
|
|
- path: "server/src/services/nexus-settings.ts"
|
|
provides: "nexusSettingsService with voiceMode field"
|
|
exports: ["nexusSettingsService", "VoiceMode", "VOICE_MODES"]
|
|
- path: "ui/public/vad.worklet.bundle.min.js"
|
|
provides: "VAD AudioWorklet bundle"
|
|
- path: "ui/public/silero_vad_legacy.onnx"
|
|
provides: "Silero VAD legacy ONNX model"
|
|
key_links:
|
|
- from: "server/src/app.ts"
|
|
to: "server/src/routes/voice.ts"
|
|
via: "api.use(voiceRoutes())"
|
|
pattern: "voiceRoutes"
|
|
- from: "server/src/app.ts"
|
|
to: "server/src/routes/nexus-settings.ts"
|
|
via: "api.use(nexusSettingsRoutes())"
|
|
pattern: "nexusSettingsRoutes"
|
|
- from: "server/src/routes/chat.ts"
|
|
to: "voiceMode parameter"
|
|
via: "req.body.voiceMode in stream handler"
|
|
pattern: "voiceMode.*voice_input|voice_full"
|
|
---
|
|
|
|
<objective>
|
|
Establish all server-side prerequisites and browser infrastructure for voice I/O.
|
|
|
|
Purpose: Phase 36 Tasks 2-3 (nexus-settings voiceMode schema, voice HTTP routes, voiceMode wiring in chat.ts) are not present on this branch. This plan cherry-picks or re-implements those deliverables, adds COOP/COEP headers for SharedArrayBuffer, installs @ricky0123/vad-react, copies VAD ONNX assets to ui/public/, and configures Vite dev server headers.
|
|
|
|
Output: Working server endpoints (transcribe, synthesize, nexus-settings), COOP/COEP isolation, VAD assets ready in ui/public/
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
|
@$HOME/.claude/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@.planning/phases/37-web-chat-voice-ui/37-RESEARCH.md
|
|
|
|
<interfaces>
|
|
<!-- Phase 36 branch deliverables that must be present before Phase 37 UI work -->
|
|
|
|
From server/src/services/voice-pipeline.ts (ALREADY on this branch):
|
|
```typescript
|
|
// voicePipelineService() exposes transcribe(buffer, format) and synthesize(text, voiceId?)
|
|
export function voicePipelineService(): { transcribe, synthesize, formatForVoice, transcodeToWav16k }
|
|
```
|
|
|
|
From server/src/app.ts (parent branch — route mounting pattern):
|
|
```typescript
|
|
// Routes are mounted on an `api` Router via api.use(...)
|
|
// Pattern: import { xyzRoutes } from "./routes/xyz.js"; then api.use(xyzRoutes());
|
|
import { chatRoutes } from "./routes/chat.js";
|
|
api.use(chatRoutes(db, storageService, config));
|
|
```
|
|
|
|
From packages/shared/src/types/chat.ts (parent branch):
|
|
```typescript
|
|
export interface ChatMessage {
|
|
id: string;
|
|
conversationId: string;
|
|
role: "user" | "assistant" | "system";
|
|
content: string;
|
|
messageType?: string | null;
|
|
// ... other fields
|
|
}
|
|
```
|
|
|
|
From packages/shared/src/validators/chat.ts (parent branch):
|
|
```typescript
|
|
export const createMessageSchema = z.object({
|
|
content: z.string().min(1),
|
|
role: z.enum(["user", "assistant", "system"]).default("user"),
|
|
agentId: z.string().uuid().optional(),
|
|
// voiceMode NOT present on parent branch — must add
|
|
});
|
|
```
|
|
</interfaces>
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto">
|
|
<name>Task 1: Cherry-pick Phase 36 server deliverables and add COOP/COEP headers</name>
|
|
<files>
|
|
server/src/services/nexus-settings.ts,
|
|
server/src/routes/nexus-settings.ts,
|
|
server/src/routes/voice.ts,
|
|
server/src/routes/chat.ts,
|
|
server/src/app.ts,
|
|
packages/shared/src/types/chat.ts,
|
|
packages/shared/src/validators/chat.ts
|
|
</files>
|
|
<read_first>
|
|
server/src/services/nexus-settings.ts,
|
|
server/src/services/voice-pipeline.ts,
|
|
server/src/app.ts,
|
|
server/src/routes/chat.ts,
|
|
packages/shared/src/types/chat.ts,
|
|
packages/shared/src/validators/chat.ts
|
|
</read_first>
|
|
<action>
|
|
Cherry-pick or re-implement Phase 36 Tasks 2-3 deliverables. The commits on gsd/phase-36-voice-pipeline-foundation are:
|
|
- d0d7a23a (nexus-settings voiceMode schema extension)
|
|
- b964c0e4 (voiceMode in createMessageSchema + ChatMessage interface)
|
|
- 11508547 (voice HTTP routes)
|
|
- fd372eaf (voiceMode wiring in chat.ts + route mounting)
|
|
|
|
Try cherry-picking these 4 commits in order:
|
|
```bash
|
|
git cherry-pick d0d7a23a b964c0e4 11508547 fd372eaf
|
|
```
|
|
|
|
If cherry-pick conflicts, re-implement manually:
|
|
|
|
1. **server/src/services/nexus-settings.ts** — Add VOICE_MODES and VoiceMode type:
|
|
```typescript
|
|
export const VOICE_MODES = ["text", "voice_input", "full_voice"] as const;
|
|
export type VoiceMode = (typeof VOICE_MODES)[number];
|
|
```
|
|
Add `voiceMode: z.enum(VOICE_MODES).default("text")` to nexusSettingsSchema.
|
|
Add `telegramToken: z.string().optional()`, `piperBinaryPath: z.string().optional()`, `whisperBinaryPath: z.string().optional()`.
|
|
|
|
2. **server/src/routes/nexus-settings.ts** — Create new file:
|
|
- GET /nexus/settings — returns nexusSettingsService().get()
|
|
- PATCH /nexus/settings — calls nexusSettingsService().set(req.body), returns updated
|
|
- Both routes call assertBoard(req) first
|
|
- Import Router from express, assertBoard from ./authz.js, nexusSettingsService from ../services/nexus-settings.js
|
|
|
|
3. **server/src/routes/voice.ts** — Create new file:
|
|
- POST /transcribe — accepts multipart audio upload via multer memoryStorage, calls voicePipelineService().transcribe(buffer, format), returns { text }
|
|
- POST /synthesize — accepts JSON { text, voiceId? }, calls voicePipelineService().synthesize(text, voiceId), returns audio/wav buffer
|
|
- Both routes call assertBoard(req)
|
|
- Import multer, Router, assertBoard, voicePipelineService, MAX_ATTACHMENT_BYTES
|
|
|
|
4. **packages/shared/src/types/chat.ts** — Add `voiceMode?: string | null;` to ChatMessage interface if not present.
|
|
|
|
5. **packages/shared/src/validators/chat.ts** — Add `voiceMode: z.enum(["text", "voice_input", "full_voice"]).optional()` to createMessageSchema.
|
|
|
|
6. **server/src/routes/chat.ts** — In the stream POST handler, destructure `voiceMode` from req.body alongside content and agentId. When voiceMode is "full_voice", call voicePipelineService().formatForVoice(aiContent) to produce SPOKEN/DETAILED format. Set messageType on stored message: "voice_full" if voiceMode==="full_voice", "voice_input" if voiceMode==="voice_input", else null.
|
|
|
|
7. **server/src/app.ts** — Import and mount voiceRoutes and nexusSettingsRoutes:
|
|
```typescript
|
|
import { nexusSettingsRoutes } from "./routes/nexus-settings.js";
|
|
import { voiceRoutes } from "./routes/voice.js";
|
|
// In the api router setup:
|
|
api.use(nexusSettingsRoutes());
|
|
api.use(voiceRoutes());
|
|
```
|
|
|
|
8. **COOP/COEP headers** — In server/src/app.ts, add middleware BEFORE static file serving and vite dev middleware:
|
|
```typescript
|
|
app.use((_req, res, next) => {
|
|
res.setHeader("Cross-Origin-Opener-Policy", "same-origin");
|
|
res.setHeader("Cross-Origin-Embedder-Policy", "require-corp");
|
|
next();
|
|
});
|
|
```
|
|
Place this before any `app.use(express.static(...))` or vite middleware attachment.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /opt/nexus/.claude/worktrees/agent-a009558f && grep -q "voiceRoutes" server/src/app.ts && grep -q "nexusSettingsRoutes" server/src/app.ts && grep -q "Cross-Origin-Opener-Policy" server/src/app.ts && grep -q "voiceMode" server/src/routes/chat.ts && grep -q "voice_full" server/src/routes/chat.ts && test -f server/src/routes/voice.ts && test -f server/src/routes/nexus-settings.ts && echo "PASS" || echo "FAIL"</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- grep "voiceRoutes" server/src/app.ts returns match
|
|
- grep "nexusSettingsRoutes" server/src/app.ts returns match
|
|
- grep "Cross-Origin-Opener-Policy" server/src/app.ts returns "same-origin"
|
|
- grep "Cross-Origin-Embedder-Policy" server/src/app.ts returns "require-corp"
|
|
- grep "voiceMode" server/src/routes/chat.ts returns match
|
|
- grep "voice_full" server/src/routes/chat.ts returns match
|
|
- server/src/routes/voice.ts exists with POST /transcribe and POST /synthesize
|
|
- server/src/routes/nexus-settings.ts exists with GET and PATCH /nexus/settings
|
|
- grep "VOICE_MODES" server/src/services/nexus-settings.ts returns match
|
|
</acceptance_criteria>
|
|
<done>Phase 36 server deliverables present on branch. COOP/COEP headers added. Voice routes mounted. Chat stream accepts voiceMode.</done>
|
|
</task>
|
|
|
|
<task type="auto">
|
|
<name>Task 2: Install VAD library, copy ONNX assets, configure Vite COOP/COEP headers</name>
|
|
<files>
|
|
ui/package.json,
|
|
ui/public/vad.worklet.bundle.min.js,
|
|
ui/public/silero_vad_legacy.onnx,
|
|
ui/public/silero_vad_v5.onnx,
|
|
ui/vite.config.ts
|
|
</files>
|
|
<read_first>
|
|
ui/package.json,
|
|
ui/vite.config.ts
|
|
</read_first>
|
|
<action>
|
|
1. Install @ricky0123/vad-react in the ui package:
|
|
```bash
|
|
pnpm add @ricky0123/vad-react --filter @paperclipai/ui
|
|
```
|
|
|
|
2. Copy VAD assets from node_modules to ui/public/ for same-origin serving (avoids COEP blocking CDN):
|
|
```bash
|
|
cp node_modules/@ricky0123/vad-web/dist/vad.worklet.bundle.min.js ui/public/
|
|
cp node_modules/@ricky0123/vad-web/dist/silero_vad_legacy.onnx ui/public/
|
|
cp node_modules/@ricky0123/vad-web/dist/silero_vad_v5.onnx ui/public/
|
|
```
|
|
If vad-web is in ui/node_modules/@ricky0123/vad-web/dist/, use that path instead.
|
|
Verify all three files exist after copy.
|
|
|
|
3. Add a "copy-vad-assets" script to ui/package.json:
|
|
```json
|
|
"copy-vad-assets": "cp node_modules/@ricky0123/vad-web/dist/vad.worklet.bundle.min.js public/ && cp node_modules/@ricky0123/vad-web/dist/silero_vad_legacy.onnx public/ && cp node_modules/@ricky0123/vad-web/dist/silero_vad_v5.onnx public/"
|
|
```
|
|
|
|
4. Update ui/vite.config.ts — add COOP/COEP headers to dev server config:
|
|
```typescript
|
|
server: {
|
|
port: 5173,
|
|
headers: {
|
|
"Cross-Origin-Opener-Policy": "same-origin",
|
|
"Cross-Origin-Embedder-Policy": "require-corp",
|
|
},
|
|
proxy: { ... }, // keep existing proxy config
|
|
},
|
|
```
|
|
This ensures SharedArrayBuffer works in Vite dev mode too.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /opt/nexus/.claude/worktrees/agent-a009558f && test -f ui/public/vad.worklet.bundle.min.js && test -f ui/public/silero_vad_legacy.onnx && test -f ui/public/silero_vad_v5.onnx && grep -q "vad-react" ui/package.json && grep -q "Cross-Origin-Opener-Policy" ui/vite.config.ts && echo "PASS" || echo "FAIL"</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- ui/public/vad.worklet.bundle.min.js exists (non-zero size)
|
|
- ui/public/silero_vad_legacy.onnx exists (non-zero size)
|
|
- ui/public/silero_vad_v5.onnx exists (non-zero size)
|
|
- grep "vad-react" ui/package.json returns match
|
|
- grep "Cross-Origin-Opener-Policy" ui/vite.config.ts returns "same-origin"
|
|
- grep "Cross-Origin-Embedder-Policy" ui/vite.config.ts returns "require-corp"
|
|
- grep "copy-vad-assets" ui/package.json returns match
|
|
</acceptance_criteria>
|
|
<done>VAD library installed. ONNX model files and worklet bundle served from ui/public/. Vite dev server sends COOP/COEP headers. SharedArrayBuffer available in dev.</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<verification>
|
|
- server/src/routes/voice.ts exists with transcribe and synthesize endpoints
|
|
- server/src/routes/nexus-settings.ts exists with GET/PATCH
|
|
- server/src/app.ts mounts both route sets and has COOP/COEP middleware
|
|
- server/src/routes/chat.ts handles voiceMode in stream handler
|
|
- ui/public/ has all 3 VAD asset files
|
|
- ui/vite.config.ts has COOP/COEP headers
|
|
- @ricky0123/vad-react in ui/package.json dependencies
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
All Phase 36 server deliverables present. COOP/COEP headers set on both Express and Vite dev server. VAD assets served from same-origin. Foundation ready for frontend voice components.
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/37-web-chat-voice-ui/37-01-SUMMARY.md`
|
|
</output>
|