544 lines
34 KiB
Markdown
544 lines
34 KiB
Markdown
# Architecture Research
|
||
|
||
**Domain:** Smart Onboarding + Personal AI Assistant (v1.5) — integration with existing Nexus/Paperclip monorepo
|
||
**Researched:** 2026-04-02
|
||
**Confidence:** HIGH — based on direct codebase inspection + verified current documentation
|
||
|
||
---
|
||
|
||
## System Overview
|
||
|
||
The v1.5 features layer on top of the existing monorepo without touching DB schema, API routes, or TypeScript identifiers. Every new service, component, and data flow hooks into the existing extension points: the adapter registry, the secrets service, the instance settings JSONB columns, the chat SSE pipeline, and the onboarding wizard overlay.
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────────────┐
|
||
│ UI Layer (React/Vite) │
|
||
│ │
|
||
│ ┌─────────────────────────────────────────┐ ┌────────────────────────┐ │
|
||
│ │ NexusOnboardingWizard (MODIFIED) │ │ PersonalAssistantPage │ │
|
||
│ │ ┌──────────────┐ ┌──────────────────┐ │ │ (NEW — lazy loaded) │ │
|
||
│ │ │ ModeSelector │ │ HardwareSummary │ │ │ ┌──────────────────┐ │ │
|
||
│ │ │ (NEW) │ │ (NEW) │ │ │ │ AssistantChatHub │ │ │
|
||
│ │ └──────────────┘ └──────────────────┘ │ │ │ (MODIFIED │ │ │
|
||
│ │ ┌──────────────┐ ┌──────────────────┐ │ │ │ ChatPanel) │ │ │
|
||
│ │ │ProviderSetup │ │ VoiceSetupStep │ │ │ └──────────────────┘ │ │
|
||
│ │ │ (NEW) │ │ (NEW) │ │ └────────────────────────┘ │
|
||
│ │ └──────────────┘ └──────────────────┘ │ │
|
||
│ └─────────────────────────────────────────┘ │
|
||
│ │
|
||
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
||
│ │ Existing Extension Points │ │
|
||
│ │ ChatPanel • ChatInput • useStreamingChat • ChatAgentSelector │ │
|
||
│ └───────────────────────────────────────────────────────────────────┘ │
|
||
└──────────────────────────────────────────────────────────────────────────┘
|
||
↕ REST + SSE
|
||
┌──────────────────────────────────────────────────────────────────────────┐
|
||
│ Server Layer (Express) │
|
||
│ │
|
||
│ NEW routes mounted in app.ts: │
|
||
│ ┌────────────────────┐ ┌───────────────────┐ ┌──────────────────────┐ │
|
||
│ │ /api/hardware │ │ /api/puter-proxy │ │ /api/voice │ │
|
||
│ │ (hardware detect) │ │ (Puter.js relay) │ │ (Whisper + Piper) │ │
|
||
│ └────────────────────┘ └───────────────────┘ └──────────────────────┘ │
|
||
│ ┌────────────────────┐ ┌───────────────────┐ │
|
||
│ │ /api/memory │ │ Existing routes: │ │
|
||
│ │ (assistant memory) │ │ /ollama • /chat │ │
|
||
│ └────────────────────┘ │ /secrets • /llms │ │
|
||
│ └───────────────────┘ │
|
||
│ │
|
||
│ NEW services (named-export pattern, no classes): │
|
||
│ hardwareService • puterProxyService • voiceService • memoryService │
|
||
└──────────────────────────────────────────────────────────────────────────┘
|
||
↕ Drizzle ORM
|
||
┌──────────────────────────────────────────────────────────────────────────┐
|
||
│ Data Layer (PostgreSQL) │
|
||
│ │
|
||
│ NO new tables — all v1.5 state lives in existing extension columns: │
|
||
│ ┌────────────────────────────────────────────────────────────────────┐ │
|
||
│ │ instance_settings.general JSONB (onboarding mode, voice config) │ │
|
||
│ │ company_secrets table (OAuth tokens, Puter token) │ │
|
||
│ │ chat_conversations table (no change — re-used as-is) │ │
|
||
│ └────────────────────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
│ NEW file-based storage (server data dir, no migration needed): │
|
||
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
||
│ │ data/memory/<companyId>.json (assistant memory store) │ │
|
||
│ │ data/whisper-models/ (downloaded .bin files) │ │
|
||
│ │ data/piper-voices/ (downloaded .onnx voice files) │ │
|
||
│ └─────────────────────────────────────────────────────────────────────┘ │
|
||
└──────────────────────────────────────────────────────────────────────────┘
|
||
↕ npx
|
||
┌──────────────────────────────────────────────────────────────────────────┐
|
||
│ CLI Layer (Commander.js) │
|
||
│ │
|
||
│ NEW standalone package: packages/buildthis/ │
|
||
│ ┌──────────────────────────────────────────────────────────────────────┐│
|
||
│ │ npx buildthis → detects if Nexus running → opens browser ││
|
||
│ │ OR runs nexus onboard wizard → starts server ││
|
||
│ └──────────────────────────────────────────────────────────────────────┘│
|
||
└──────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Component Responsibilities
|
||
|
||
| Component | Responsibility | New or Modified | Where |
|
||
|-----------|----------------|-----------------|-------|
|
||
| `NexusOnboardingWizard` | Multi-step onboarding: mode, hardware, provider, voice, summary | MODIFIED (replace single-step) | `ui/src/components/NexusOnboardingWizard.tsx` |
|
||
| `ModeSelector` | Card picker: Personal AI / Project Builder / Both | NEW | `ui/src/components/onboarding/ModeSelector.tsx` |
|
||
| `HardwareSummaryStep` | Displays detected GPU/RAM/Unified Memory result | NEW | `ui/src/components/onboarding/HardwareSummaryStep.tsx` |
|
||
| `ProviderTierStep` | Puter.js auth button, OAuth tier, API key entry | NEW | `ui/src/components/onboarding/ProviderTierStep.tsx` |
|
||
| `VoiceSetupStep` | Whisper model picker + Piper voice picker | NEW | `ui/src/components/onboarding/VoiceSetupStep.tsx` |
|
||
| `OnboardingSummaryStep` | Final summary before launch | NEW | `ui/src/components/onboarding/OnboardingSummaryStep.tsx` |
|
||
| `PersonalAssistantPage` | Full-screen chat experience for assistant mode | NEW | `ui/src/pages/PersonalAssistant.tsx` |
|
||
| `AssistantMemoryBar` | Shows memory slots / recall indicator in chat | NEW | `ui/src/components/AssistantMemoryBar.tsx` |
|
||
| `hardwareService` | Reads `os.totalmem()`, runs `system_profiler` on macOS for GPU info | NEW | `server/src/services/hardware.ts` |
|
||
| `puterProxyService` | Wraps Puter.js Node.js client; relays AI calls through SSE | NEW | `server/src/services/puter-proxy.ts` |
|
||
| `voiceService` | Manages Whisper (via `whisper-node`) + Piper (via `@mintplex-labs/piper-tts-web` server-side) | NEW | `server/src/services/voice.ts` |
|
||
| `memoryService` | CRUD on file-based JSON memory store; injects context into system prompt | NEW | `server/src/services/memory.ts` |
|
||
| `hardwareRoutes` | `GET /api/hardware/info` | NEW | `server/src/routes/hardware.ts` |
|
||
| `puterProxyRoutes` | `POST /api/puter-proxy/chat` (SSE), `POST /api/puter-proxy/auth` | NEW | `server/src/routes/puter-proxy.ts` |
|
||
| `voiceRoutes` | `POST /api/voice/transcribe`, `POST /api/voice/speak`, `GET /api/voice/status` | NEW | `server/src/routes/voice.ts` |
|
||
| `memoryRoutes` | `GET/POST/DELETE /api/companies/:id/memory` | NEW | `server/src/routes/memory.ts` |
|
||
| `buildthis` package | `npx buildthis` entry point — detect/launch Nexus | NEW | `packages/buildthis/` |
|
||
|
||
---
|
||
|
||
## Recommended Project Structure
|
||
|
||
```
|
||
packages/
|
||
├── buildthis/ # NEW — npx buildthis entry point
|
||
│ ├── src/
|
||
│ │ └── index.ts # bin entry: detect running Nexus, open browser or run onboard
|
||
│ └── package.json # name: "buildthis", bin: { buildthis: "./dist/index.js" }
|
||
|
||
server/src/
|
||
├── services/
|
||
│ ├── hardware.ts # NEW — detect GPU/RAM/Apple Silicon
|
||
│ ├── puter-proxy.ts # NEW — Puter.js Node.js client wrapper
|
||
│ ├── voice.ts # NEW — Whisper + Piper lifecycle
|
||
│ └── memory.ts # NEW — file-based JSON assistant memory
|
||
├── routes/
|
||
│ ├── hardware.ts # NEW — GET /api/hardware/info
|
||
│ ├── puter-proxy.ts # NEW — POST /api/puter-proxy/chat (SSE)
|
||
│ ├── voice.ts # NEW — POST /api/voice/transcribe, /speak
|
||
│ └── memory.ts # NEW — GET/POST/DELETE /companies/:id/memory
|
||
└── app.ts # MODIFIED — mount 4 new route sets
|
||
|
||
ui/src/
|
||
├── components/
|
||
│ ├── NexusOnboardingWizard.tsx # MODIFIED — multi-step replaces single-step
|
||
│ ├── AssistantMemoryBar.tsx # NEW
|
||
│ └── onboarding/ # NEW directory — onboarding step components
|
||
│ ├── ModeSelector.tsx
|
||
│ ├── HardwareSummaryStep.tsx
|
||
│ ├── ProviderTierStep.tsx
|
||
│ ├── VoiceSetupStep.tsx
|
||
│ └── OnboardingSummaryStep.tsx
|
||
├── pages/
|
||
│ └── PersonalAssistant.tsx # NEW — full-screen assistant page
|
||
├── hooks/
|
||
│ ├── useHardwareInfo.ts # NEW — query /api/hardware/info
|
||
│ ├── usePuterChat.ts # NEW — SSE streaming from puter-proxy
|
||
│ ├── useVoiceInput.ts # NEW — Whisper transcription hook
|
||
│ ├── useVoiceSpeech.ts # NEW — Piper TTS hook
|
||
│ └── useAssistantMemory.ts # NEW — memory CRUD hook
|
||
└── api/
|
||
├── hardware.ts # NEW — typed fetch wrappers
|
||
├── puter-proxy.ts # NEW
|
||
├── voice.ts # NEW
|
||
└── memory.ts # NEW
|
||
```
|
||
|
||
### Structure Rationale
|
||
|
||
- **`packages/buildthis/`:** Standalone package with its own `package.json` and `bin` field — publishable to npm as `buildthis` independently. Does not depend on the monorepo server package at runtime; it only detects a running Nexus instance via HTTP or launches the CLI onboard flow.
|
||
- **`server/src/services/` additions:** All follow the existing named-export pattern (`export function hardwareService() { return { ... } }`). No classes. Dependencies injected as parameters. Drizzle `db` is only accepted if the service actually queries the DB.
|
||
- **`ui/src/components/onboarding/`:** Sub-directory isolates the 5 new step components from the main components directory. `NexusOnboardingWizard.tsx` imports them. This limits the upstream-conflict surface to the single wizard file.
|
||
- **`ui/src/pages/PersonalAssistant.tsx`:** New route registered in App.tsx routing (the only modification needed in the routing layer). The page re-uses `ChatPanel` with an `assistantMode` prop.
|
||
|
||
---
|
||
|
||
## Architectural Patterns
|
||
|
||
### Pattern 1: Hardware Detection via Server-Side Shell Probe
|
||
|
||
**What:** `hardwareService` runs on the Express server where it has access to `os.totalmem()` and can shell out to `system_profiler SPDisplaysDataType` on macOS to get GPU details. Apple Silicon unified memory is detected by checking the `cpu_brand_string` for "Apple M". Results are cached in memory (5-minute TTL) so the onboarding wizard can poll cheaply.
|
||
|
||
**When to use:** Any time the onboarding wizard needs to display hardware capabilities to make model recommendations.
|
||
|
||
**Trade-offs:** Server-side only — the UI cannot do this itself in the browser. The route is scoped to `assertBoard` (existing auth middleware), so it's protected. Apple Silicon reports unified memory as both RAM and VRAM; the service returns `{ unifiedMemory: true, totalBytes }` instead of separate fields.
|
||
|
||
**Example:**
|
||
```typescript
|
||
// server/src/services/hardware.ts
|
||
export function hardwareService() {
|
||
let cache: HardwareInfo | null = null;
|
||
let cacheExpiry = 0;
|
||
|
||
return {
|
||
async detect(): Promise<HardwareInfo> {
|
||
if (cache && Date.now() < cacheExpiry) return cache;
|
||
const totalBytes = os.totalmem();
|
||
const gpuInfo = await probeGpu(); // shells system_profiler on macOS, /proc/driver/nvidia on Linux
|
||
cache = { totalBytes, gpu: gpuInfo, platform: process.platform };
|
||
cacheExpiry = Date.now() + 5 * 60 * 1000;
|
||
return cache;
|
||
}
|
||
};
|
||
}
|
||
```
|
||
|
||
### Pattern 2: Puter.js as a Server-Side Adapter (not browser-direct)
|
||
|
||
**What:** Puter.js supports Node.js via `@heyputer/puter.js` with `init(authToken)`. The server acts as a proxy: it holds the Puter auth token (stored in `company_secrets` via the existing `secretService`), forwards chat requests to `puter.ai.chat({ stream: true })`, and pipes the async iterable back to the browser as SSE — exactly the same format the existing `useStreamingChat` hook already consumes.
|
||
|
||
**Why not browser-direct:** The existing chat architecture is server-mediated (all agent messages go through Express SSE). Bypassing this would require forking the streaming infrastructure. Using the server as proxy re-uses `useStreamingChat` unchanged and keeps the Puter token off the client.
|
||
|
||
**When to use:** During onboarding when user selects "Puter.js cloud" tier and authenticates. The Puter auth flow opens a browser popup (`puter.auth.signIn()` must be user-initiated from the UI), receives a token, then POSTs it to `/api/puter-proxy/auth` for server storage.
|
||
|
||
**Trade-offs:** One extra round-trip compared to browser-direct, but avoids token exposure and re-uses the existing SSE pipeline. Puter.js Node.js usage requires `@heyputer/puter.js` as a server dependency (not currently in the monorepo).
|
||
|
||
**Example (server-side relay):**
|
||
```typescript
|
||
// server/src/routes/puter-proxy.ts
|
||
router.post("/api/puter-proxy/chat", async (req, res) => {
|
||
const token = await svc.getStoredToken(companyId);
|
||
const puter = init(token);
|
||
res.setHeader("Content-Type", "text/event-stream");
|
||
const stream = await puter.ai.chat(req.body.messages, { stream: true });
|
||
for await (const chunk of stream) {
|
||
res.write(`data: ${JSON.stringify(chunk)}\n\n`);
|
||
}
|
||
res.end();
|
||
});
|
||
```
|
||
|
||
### Pattern 3: Whisper on Server, Piper in Browser (Hybrid Voice)
|
||
|
||
**What:** Voice input (speech-to-text) runs server-side via `whisper-node` (Node.js bindings for whisper.cpp). The UI records audio via `MediaRecorder`, POSTs a blob to `POST /api/voice/transcribe`, and gets back a transcript string. Voice output (text-to-speech) uses `@mintplex-labs/piper-tts-web` which runs client-side via WebAssembly — no server round-trip needed for TTS.
|
||
|
||
**Why this split:** whisper.cpp requires native binaries that work on CPU-only hardware, which the server controls. Piper TTS web runs via WASM in the browser and has no native dependency — this keeps TTS latency low (no network round-trip) and works even if the server is slow.
|
||
|
||
**When to use:** When user selects "voice mode" in onboarding (VoiceSetupStep). Whisper runs only if the user chooses a local Whisper model (downloaded to `data/whisper-models/`); as a fallback, the browser's native `webkitSpeechRecognition` / `SpeechRecognition` API is used.
|
||
|
||
**Trade-offs:** Whisper download adds 75MB–1.5GB to first-run setup. For CPU-only hardware, whisper-tiny.en (75MB) transcribes in ~2s for a 10s clip on M4 — acceptable. Piper WASM download is ~20MB (models ~30-100MB each).
|
||
|
||
**Example (voice input hook):**
|
||
```typescript
|
||
// ui/src/hooks/useVoiceInput.ts
|
||
export function useVoiceInput() {
|
||
// Records with MediaRecorder → blob → POST /api/voice/transcribe
|
||
// Falls back to window.SpeechRecognition if whisper not configured
|
||
}
|
||
```
|
||
|
||
### Pattern 4: Persistent Memory via File-Backed JSON (No New DB Table)
|
||
|
||
**What:** The assistant memory store is a per-workspace JSON file at `data/memory/<companyId>.json`. Each memory entry has `{ id, content, createdAt, tags }`. The `memoryService` reads this file on startup (lazy-loaded per companyId), keeps it in-process, and writes on mutation. Memory injection works by prepending a formatted memory block to the system prompt at chat-send time in the existing chat service.
|
||
|
||
**Why not PostgreSQL:** Adding a new table violates the "no DB schema changes" constraint for upstream rebase safety. File-backed JSON with an in-process cache is fast for a single-user setup (sub-millisecond reads) and requires no migration.
|
||
|
||
**When to use:** Personal AI Assistant mode only. Project Builder mode does not use the memory service.
|
||
|
||
**Trade-offs:** Not transactional. For a single-user local deployment, this is acceptable. File writes are atomic via write-then-rename pattern. Memory search is linear scan (no vector embeddings in v1.5 — semantic search is a future enhancement).
|
||
|
||
**Example:**
|
||
```typescript
|
||
// server/src/services/memory.ts
|
||
export function memoryService() {
|
||
const cache = new Map<string, MemoryStore>();
|
||
|
||
return {
|
||
async inject(companyId: string, systemPrompt: string): Promise<string> {
|
||
const store = await load(companyId);
|
||
if (store.entries.length === 0) return systemPrompt;
|
||
const block = store.entries.map(e => `- ${e.content}`).join("\n");
|
||
return `${systemPrompt}\n\n## What I remember about you:\n${block}`;
|
||
}
|
||
};
|
||
}
|
||
```
|
||
|
||
### Pattern 5: Onboarding State via instance_settings.general JSONB
|
||
|
||
**What:** All onboarding configuration (selected mode, voice config, active provider tier) is stored in the existing `instance_settings.general` JSONB column. The `instanceSettingsService` already handles arbitrary JSONB keys. Nexus adds its config under a `nexus` namespace key to avoid upstream key collisions.
|
||
|
||
**When to use:** Reading/writing onboarding mode, voice model selection, and provider tier configuration. No new table, no migration.
|
||
|
||
**Example:**
|
||
```typescript
|
||
// instance_settings.general.nexus = {
|
||
// mode: "personal_ai" | "project_builder" | "both",
|
||
// voiceModel: "whisper-tiny.en" | "whisper-base" | null,
|
||
// piperVoice: "en_US-amy-medium" | null,
|
||
// providerTier: "local" | "puter" | "oauth_gemini" | "api_key",
|
||
// }
|
||
```
|
||
|
||
### Pattern 6: OAuth Token Storage via Existing Secrets Service
|
||
|
||
**What:** OAuth tokens (Google Gemini, OpenAI) and the Puter.js auth token are stored via the existing `secretService` using the `local_encrypted` provider. The onboarding wizard calls `POST /api/companies/:id/secrets` with a well-known name (e.g., `nexus_puter_token`, `nexus_gemini_token`). Adapters read these at spawn time.
|
||
|
||
**When to use:** Any time an OAuth flow completes and a token needs persistence.
|
||
|
||
**Trade-offs:** Secrets are per-company (workspace), not per-instance. This is fine for single-user setup. The existing secrets UI lets users view/rotate tokens manually.
|
||
|
||
---
|
||
|
||
## Data Flow
|
||
|
||
### Onboarding Wizard Data Flow
|
||
|
||
```
|
||
User opens Nexus (no workspace yet)
|
||
↓
|
||
NexusOnboardingWizard renders
|
||
↓
|
||
Step 1: ModeSelector → user picks "Personal AI" / "Project Builder" / "Both"
|
||
↓
|
||
Step 2: HardwareSummaryStep
|
||
→ GET /api/hardware/info (new route)
|
||
→ hardwareService.detect() → os.totalmem() + system_profiler
|
||
→ returns { totalGb, gpuName, unifiedMemory, platform }
|
||
→ wizard shows model tier recommendations
|
||
↓
|
||
Step 3: ProviderTierStep
|
||
→ Local: already detected via existing Hermes probe
|
||
→ Puter.js: user clicks "Connect" → puter.auth.signIn() popup
|
||
→ UI POSTs token to POST /api/puter-proxy/auth
|
||
→ server stores in secretService("nexus_puter_token")
|
||
→ OAuth (Gemini/OpenAI): OAuth PKCE flow in a popup window
|
||
→ callback captured by temp local server or redirect
|
||
→ token stored via secretService
|
||
→ API Key: direct input → stored via secretService
|
||
↓
|
||
Step 4: VoiceSetupStep (optional, skippable)
|
||
→ GET /api/voice/status → check if whisper binary present
|
||
→ User picks model → POST /api/voice/download (async download + SSE progress)
|
||
→ User picks Piper voice → stored in instance_settings.general.nexus
|
||
↓
|
||
Step 5: OnboardingSummaryStep
|
||
→ Creates workspace + agents (existing companiesApi + agentsApi flow)
|
||
→ Writes nexus config to instance_settings.general.nexus
|
||
→ Navigates to PersonalAssistant page OR Dashboard based on mode
|
||
```
|
||
|
||
### Personal AI Assistant Chat Data Flow
|
||
|
||
```
|
||
User types message in AssistantChatHub
|
||
↓
|
||
ChatInput → useStreamingChat.startStream(conversationId, message)
|
||
↓
|
||
POST /api/companies/:id/chat/conversations/:convId/messages
|
||
↓ (existing chat route, no change)
|
||
chat route detects "personal_assistant" agent type
|
||
↓
|
||
memoryService.inject(companyId, systemPrompt) ← NEW injection point
|
||
↓
|
||
Route selects provider based on instance_settings.general.nexus.providerTier:
|
||
- "local" → existing Hermes adapter (no change)
|
||
- "puter" → puterProxyService.chat() → Puter.js Node client → SSE relay
|
||
- "oauth_*" → respective provider API with stored OAuth token → SSE relay
|
||
↓
|
||
SSE events stream to UI via existing /api/chat/stream endpoint pattern
|
||
↓
|
||
useStreamingChat receives chunks → ChatMessageList renders them
|
||
```
|
||
|
||
### Voice Input Data Flow
|
||
|
||
```
|
||
User presses mic button in ChatInput (MODIFIED)
|
||
↓
|
||
useVoiceInput starts MediaRecorder → records WebM/Opus blob
|
||
↓
|
||
User releases mic → blob POSTed to POST /api/voice/transcribe
|
||
↓
|
||
voiceService.transcribe(audioBuffer)
|
||
→ whisper-node.transcribe(path) → returns text
|
||
↓
|
||
Text injected into ChatInput.value
|
||
↓
|
||
User reviews → sends normally
|
||
```
|
||
|
||
### npx buildthis Data Flow
|
||
|
||
```
|
||
Developer runs: npx buildthis
|
||
↓
|
||
buildthis/src/index.ts checks for running Nexus:
|
||
GET http://localhost:4000/api/health → 200?
|
||
YES → open browser to http://localhost:4000
|
||
NO → run nexus onboard wizard (delegates to paperclipai onboard)
|
||
OR detect Docker → suggest docker-compose up
|
||
```
|
||
|
||
---
|
||
|
||
## Integration Points: New vs Modified
|
||
|
||
### Server Routes — app.ts (MODIFIED)
|
||
|
||
One file to add 4 route mounts. Minimal conflict surface with upstream.
|
||
|
||
```typescript
|
||
// In server/src/app.ts — add after ollamaRoutes():
|
||
app.use(hardwareRoutes());
|
||
app.use(voiceRoutes());
|
||
app.use(memoryRoutes(db));
|
||
app.use(puterProxyRoutes(db));
|
||
```
|
||
|
||
### Chat Route — MODIFIED for memory injection
|
||
|
||
The existing chat service (`server/src/services/chat.ts`) needs one injection point: when building the system prompt for a conversation, call `memoryService.inject()`. This is scoped to conversations where the agent has `adapterConfig.assistantMode === true`.
|
||
|
||
**Risk:** This touches an upstream file. The injection is a 3-line addition inside the message-send handler. Low conflict probability — upstream rarely modifies this section.
|
||
|
||
### NexusOnboardingWizard.tsx — REPLACED
|
||
|
||
The current single-step wizard becomes a multi-step wizard. Since this file is already a Nexus replacement (not an upstream file), there is zero conflict risk — it will never exist in upstream.
|
||
|
||
### App.tsx routing — MODIFIED (one new route)
|
||
|
||
Add the `PersonalAssistant` page as a new lazy-loaded route. Minimal upstream conflict (routing section rarely changes).
|
||
|
||
### ChatInput.tsx — MODIFIED (voice button)
|
||
|
||
Add a microphone button that triggers `useVoiceInput`. This is an upstream file — the modification is additive (new button, no existing logic changed). Conflict risk: LOW, as upstream rarely modifies ChatInput.
|
||
|
||
---
|
||
|
||
## Anti-Patterns
|
||
|
||
### Anti-Pattern 1: Browser-Direct Puter.js
|
||
|
||
**What people do:** Import `@heyputer/puter.js` in the React frontend and call `puter.ai.chat()` directly from the browser.
|
||
|
||
**Why it's wrong:** Exposes the Puter auth token in browser storage/network. Bypasses the existing SSE pipeline, requiring a second streaming implementation. Breaks the memory injection pattern (no server-side hook). Cannot use the existing `useStreamingChat` hook.
|
||
|
||
**Do this instead:** Use the server proxy pattern (Pattern 2). The UI sends messages to `/api/puter-proxy/chat` exactly like any other chat endpoint.
|
||
|
||
### Anti-Pattern 2: New PostgreSQL Tables for Memory
|
||
|
||
**What people do:** Create a `assistant_memories` migration with a proper relational schema.
|
||
|
||
**Why it's wrong:** Violates the hard constraint: no DB migrations, no schema changes, to keep upstream rebase clean. A migration file created in Nexus will conflict every time upstream adds a migration.
|
||
|
||
**Do this instead:** File-backed JSON in the server's data directory (Pattern 4). The single-user M4 Mini deployment will never hit performance limits with this approach.
|
||
|
||
### Anti-Pattern 3: Multi-Step Wizard as Modified OnboardingWizard.tsx
|
||
|
||
**What people do:** Modify the upstream `OnboardingWizard.tsx` directly to add v1.5 steps.
|
||
|
||
**Why it's wrong:** The upstream wizard is actively maintained (120+ upstream commits since fork). Touching it creates guaranteed rebase conflicts.
|
||
|
||
**Do this instead:** Continue the existing pattern — `NexusOnboardingWizard.tsx` is already the Nexus replacement via Vite alias. All v1.5 changes go there. Upstream file untouched.
|
||
|
||
### Anti-Pattern 4: OAuth in the Browser via Redirect
|
||
|
||
**What people do:** Redirect the main app window to the OAuth provider and handle the callback via `window.location`.
|
||
|
||
**Why it's wrong:** Loses React state mid-flow. Hard to handle callback URL in a local server that may not have a publicly routable HTTPS endpoint.
|
||
|
||
**Do this instead:** Use a popup window for OAuth (`window.open`). The popup handles the full OAuth redirect. On callback, the popup calls `window.opener.postMessage` with the token, closes itself, and the main window receives it. For Puter.js specifically, `puter.auth.signIn()` handles the popup internally.
|
||
|
||
---
|
||
|
||
## Scaling Considerations
|
||
|
||
This is a single-user local deployment on an M4 Mini. Scaling is not a concern for v1.5. The architecture is designed for correctness and upstream merge-ability, not horizontal scale.
|
||
|
||
| Concern | Single User (M4 Mini) |
|
||
|---------|----------------------|
|
||
| Hardware detection | os.totalmem() + sync shell probe, cached 5min — negligible |
|
||
| Puter.js relay | One connection at a time, no pooling needed |
|
||
| Whisper transcription | ~2s for 10s clip on M4, sequential queue sufficient |
|
||
| Memory store | File JSON, <10ms read, no contention |
|
||
| Voice TTS | WASM in browser, zero server load |
|
||
|
||
---
|
||
|
||
## Build Order (Dependency Graph)
|
||
|
||
The build order matters because later phases consume services built in earlier ones.
|
||
|
||
```
|
||
Phase 1: Hardware Detection
|
||
→ hardwareService (server)
|
||
→ GET /api/hardware/info (route)
|
||
→ useHardwareInfo hook (UI)
|
||
→ HardwareSummaryStep component (UI)
|
||
No dependencies on other new phases.
|
||
|
||
Phase 2: Provider Tiers (depends on Phase 1 for display)
|
||
→ puterProxyService (server) — Puter.js Node client
|
||
→ secretService integration for token storage (uses EXISTING service)
|
||
→ POST /api/puter-proxy/auth (route)
|
||
→ ProviderTierStep component (UI)
|
||
→ OAuth popup flow (UI)
|
||
|
||
Phase 3: Multi-Step Onboarding Wizard (depends on Phases 1+2)
|
||
→ ModeSelector, OnboardingSummaryStep components (UI)
|
||
→ Refactor NexusOnboardingWizard.tsx into multi-step
|
||
→ instance_settings.general.nexus config write
|
||
|
||
Phase 4: Persistent Memory + Assistant Mode (depends on Phase 3)
|
||
→ memoryService (server)
|
||
→ Memory injection in chat route (MODIFIED — highest risk step)
|
||
→ GET/POST/DELETE /api/companies/:id/memory (routes)
|
||
→ PersonalAssistantPage (UI)
|
||
→ useAssistantMemory hook (UI)
|
||
|
||
Phase 5: Voice (depends on Phase 3, independent of Phase 4)
|
||
→ voiceService (server) — whisper-node + piper setup
|
||
→ POST /api/voice/transcribe, /speak, /status (routes)
|
||
→ VoiceSetupStep in onboarding (UI)
|
||
→ useVoiceInput, useVoiceSpeech hooks (UI)
|
||
→ ChatInput microphone button (MODIFIED — upstream file, low risk)
|
||
|
||
Phase 6: npx buildthis (independent of all above)
|
||
→ packages/buildthis/ new package
|
||
→ package.json bin field setup
|
||
→ npm publish configuration
|
||
```
|
||
|
||
**Recommended sequence:** 1 → 2 → 3 → 4 → 5 → 6. Phase 4 (memory injection into chat route) is the highest-risk upstream-file modification and should come after onboarding is validated.
|
||
|
||
---
|
||
|
||
## Integration Points: External Services
|
||
|
||
| Service | Integration Pattern | Auth Storage | Notes |
|
||
|---------|---------------------|--------------|-------|
|
||
| Puter.js | Server-side Node.js client proxy, SSE relay | `company_secrets` table | Token obtained via browser popup on first connect |
|
||
| Google Gemini OAuth | PKCE popup flow, access token + refresh token | `company_secrets` table | Policy risk: using Gemini CLI OAuth with third-party apps may trigger abuse detection — use only if user has an active Gemini subscription |
|
||
| OpenAI OAuth | PKCE flow via auth.openai.com | `company_secrets` table | Only for free tier / ChatGPT Plus users |
|
||
| Whisper (whisper-node) | Native binary, spawned by voiceService | N/A — local binary | Download on first use, cached in data/whisper-models/ |
|
||
| Piper TTS | @mintplex-labs/piper-tts-web WASM, runs in browser | N/A — client-side | Model files downloaded to browser cache |
|
||
| Ollama | Existing integration (v1.4) — no changes | N/A | ollama.ts service and /ollama routes unchanged |
|
||
|
||
---
|
||
|
||
## Sources
|
||
|
||
- Codebase inspection: `/opt/nexus/server/src/`, `/opt/nexus/ui/src/`, `/opt/nexus/packages/`
|
||
- Puter.js Node.js support: https://docs.puter.com/supported-platforms/
|
||
- Puter.js chat streaming API: https://docs.puter.com/AI/chat/
|
||
- Puter.js auth flow: https://developer.puter.com/blog/browser-based-auth-puter-js-node/
|
||
- whisper-node npm package: https://www.npmjs.com/package/whisper-node
|
||
- Piper TTS WASM: https://www.npmjs.com/package/@mintplex-labs/piper-tts-web
|
||
- @xenova/transformers Node.js audio guide: https://huggingface.co/docs/transformers.js/main/en/guides/node-audio-processing
|
||
- Google Gemini OAuth: https://ai.google.dev/gemini-api/docs/oauth
|
||
- Google Gemini OAuth policy risk: https://github.com/google-gemini/gemini-cli/issues/21866
|
||
- Vectra local vector DB (future memory enhancement): https://github.com/Stevenic/vectra
|
||
- Apple Silicon unified memory: https://eclecticlight.co/2022/03/01/making-sense-of-m1-memory-use/
|
||
|
||
---
|
||
*Architecture research for: Nexus v1.5 Smart Onboarding + Personal AI Assistant*
|
||
*Researched: 2026-04-02*
|