Nexus Dev c53e419c66 docs: complete project research

2026-04-04 03:55:49 +00:00

34 KiB

Raw Blame History

Architecture Research

Domain: Smart Onboarding + Personal AI Assistant (v1.5) — integration with existing Nexus/Paperclip monorepo Researched: 2026-04-02 Confidence: HIGH — based on direct codebase inspection + verified current documentation

System Overview

The v1.5 features layer on top of the existing monorepo without touching DB schema, API routes, or TypeScript identifiers. Every new service, component, and data flow hooks into the existing extension points: the adapter registry, the secrets service, the instance settings JSONB columns, the chat SSE pipeline, and the onboarding wizard overlay.

┌──────────────────────────────────────────────────────────────────────────┐
│                              UI Layer (React/Vite)                        │
│                                                                           │
│  ┌─────────────────────────────────────────┐  ┌────────────────────────┐ │
│  │   NexusOnboardingWizard (MODIFIED)      │  │  PersonalAssistantPage  │ │
│  │   ┌──────────────┐ ┌──────────────────┐ │  │  (NEW — lazy loaded)   │ │
│  │   │ ModeSelector │ │ HardwareSummary  │ │  │  ┌──────────────────┐  │ │
│  │   │ (NEW)        │ │ (NEW)            │ │  │  │ AssistantChatHub │  │ │
│  │   └──────────────┘ └──────────────────┘ │  │  │ (MODIFIED        │  │ │
│  │   ┌──────────────┐ ┌──────────────────┐ │  │  │  ChatPanel)      │  │ │
│  │   │ProviderSetup │ │ VoiceSetupStep   │ │  │  └──────────────────┘  │ │
│  │   │ (NEW)        │ │ (NEW)            │ │  └────────────────────────┘ │
│  │   └──────────────┘ └──────────────────┘ │                            │
│  └─────────────────────────────────────────┘                            │
│                                                                           │
│  ┌───────────────────────────────────────────────────────────────────┐   │
│  │                 Existing Extension Points                          │   │
│  │  ChatPanel • ChatInput • useStreamingChat • ChatAgentSelector     │   │
│  └───────────────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────────┘
                               ↕ REST + SSE
┌──────────────────────────────────────────────────────────────────────────┐
│                        Server Layer (Express)                             │
│                                                                           │
│  NEW routes mounted in app.ts:                                            │
│  ┌────────────────────┐  ┌───────────────────┐  ┌──────────────────────┐ │
│  │ /api/hardware      │  │ /api/puter-proxy   │  │ /api/voice           │ │
│  │ (hardware detect)  │  │ (Puter.js relay)   │  │ (Whisper + Piper)    │ │
│  └────────────────────┘  └───────────────────┘  └──────────────────────┘ │
│  ┌────────────────────┐  ┌───────────────────┐                           │
│  │ /api/memory        │  │ Existing routes:   │                           │
│  │ (assistant memory) │  │ /ollama • /chat    │                           │
│  └────────────────────┘  │ /secrets • /llms   │                           │
│                           └───────────────────┘                          │
│                                                                           │
│  NEW services (named-export pattern, no classes):                        │
│  hardwareService • puterProxyService • voiceService • memoryService      │
└──────────────────────────────────────────────────────────────────────────┘
                               ↕ Drizzle ORM
┌──────────────────────────────────────────────────────────────────────────┐
│                      Data Layer (PostgreSQL)                              │
│                                                                           │
│  NO new tables — all v1.5 state lives in existing extension columns:     │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │ instance_settings.general  JSONB  (onboarding mode, voice config)  │  │
│  │ company_secrets             table  (OAuth tokens, Puter token)     │  │
│  │ chat_conversations          table  (no change — re-used as-is)     │  │
│  └────────────────────────────────────────────────────────────────────┘  │
│                                                                           │
│  NEW file-based storage (server data dir, no migration needed):          │
│  ┌─────────────────────────────────────────────────────────────────────┐ │
│  │ data/memory/<companyId>.json   (assistant memory store)             │ │
│  │ data/whisper-models/           (downloaded .bin files)              │ │
│  │ data/piper-voices/             (downloaded .onnx voice files)       │ │
│  └─────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
                               ↕ npx
┌──────────────────────────────────────────────────────────────────────────┐
│                      CLI Layer (Commander.js)                             │
│                                                                           │
│  NEW standalone package: packages/buildthis/                              │
│  ┌──────────────────────────────────────────────────────────────────────┐│
│  │  npx buildthis  →  detects if Nexus running  →  opens browser        ││
│  │                  OR  runs nexus onboard wizard  →  starts server      ││
│  └──────────────────────────────────────────────────────────────────────┘│
└──────────────────────────────────────────────────────────────────────────┘

Component Responsibilities

Component	Responsibility	New or Modified	Where
`NexusOnboardingWizard`	Multi-step onboarding: mode, hardware, provider, voice, summary	MODIFIED (replace single-step)	`ui/src/components/NexusOnboardingWizard.tsx`
`ModeSelector`	Card picker: Personal AI / Project Builder / Both	NEW	`ui/src/components/onboarding/ModeSelector.tsx`
`HardwareSummaryStep`	Displays detected GPU/RAM/Unified Memory result	NEW	`ui/src/components/onboarding/HardwareSummaryStep.tsx`
`ProviderTierStep`	Puter.js auth button, OAuth tier, API key entry	NEW	`ui/src/components/onboarding/ProviderTierStep.tsx`
`VoiceSetupStep`	Whisper model picker + Piper voice picker	NEW	`ui/src/components/onboarding/VoiceSetupStep.tsx`
`OnboardingSummaryStep`	Final summary before launch	NEW	`ui/src/components/onboarding/OnboardingSummaryStep.tsx`
`PersonalAssistantPage`	Full-screen chat experience for assistant mode	NEW	`ui/src/pages/PersonalAssistant.tsx`
`AssistantMemoryBar`	Shows memory slots / recall indicator in chat	NEW	`ui/src/components/AssistantMemoryBar.tsx`
`hardwareService`	Reads `os.totalmem()`, runs `system_profiler` on macOS for GPU info	NEW	`server/src/services/hardware.ts`
`puterProxyService`	Wraps Puter.js Node.js client; relays AI calls through SSE	NEW	`server/src/services/puter-proxy.ts`
`voiceService`	Manages Whisper (via `whisper-node`) + Piper (via `@mintplex-labs/piper-tts-web` server-side)	NEW	`server/src/services/voice.ts`
`memoryService`	CRUD on file-based JSON memory store; injects context into system prompt	NEW	`server/src/services/memory.ts`
`hardwareRoutes`	`GET /api/hardware/info`	NEW	`server/src/routes/hardware.ts`
`puterProxyRoutes`	`POST /api/puter-proxy/chat` (SSE), `POST /api/puter-proxy/auth`	NEW	`server/src/routes/puter-proxy.ts`
`voiceRoutes`	`POST /api/voice/transcribe`, `POST /api/voice/speak`, `GET /api/voice/status`	NEW	`server/src/routes/voice.ts`
`memoryRoutes`	`GET/POST/DELETE /api/companies/:id/memory`	NEW	`server/src/routes/memory.ts`
`buildthis` package	`npx buildthis` entry point — detect/launch Nexus	NEW	`packages/buildthis/`

Recommended Project Structure

packages/
├── buildthis/                  # NEW — npx buildthis entry point
│   ├── src/
│   │   └── index.ts            # bin entry: detect running Nexus, open browser or run onboard
│   └── package.json            # name: "buildthis", bin: { buildthis: "./dist/index.js" }

server/src/
├── services/
│   ├── hardware.ts             # NEW — detect GPU/RAM/Apple Silicon
│   ├── puter-proxy.ts          # NEW — Puter.js Node.js client wrapper
│   ├── voice.ts                # NEW — Whisper + Piper lifecycle
│   └── memory.ts               # NEW — file-based JSON assistant memory
├── routes/
│   ├── hardware.ts             # NEW — GET /api/hardware/info
│   ├── puter-proxy.ts          # NEW — POST /api/puter-proxy/chat (SSE)
│   ├── voice.ts                # NEW — POST /api/voice/transcribe, /speak
│   └── memory.ts               # NEW — GET/POST/DELETE /companies/:id/memory
└── app.ts                      # MODIFIED — mount 4 new route sets

ui/src/
├── components/
│   ├── NexusOnboardingWizard.tsx    # MODIFIED — multi-step replaces single-step
│   ├── AssistantMemoryBar.tsx       # NEW
│   └── onboarding/                  # NEW directory — onboarding step components
│       ├── ModeSelector.tsx
│       ├── HardwareSummaryStep.tsx
│       ├── ProviderTierStep.tsx
│       ├── VoiceSetupStep.tsx
│       └── OnboardingSummaryStep.tsx
├── pages/
│   └── PersonalAssistant.tsx        # NEW — full-screen assistant page
├── hooks/
│   ├── useHardwareInfo.ts           # NEW — query /api/hardware/info
│   ├── usePuterChat.ts              # NEW — SSE streaming from puter-proxy
│   ├── useVoiceInput.ts             # NEW — Whisper transcription hook
│   ├── useVoiceSpeech.ts            # NEW — Piper TTS hook
│   └── useAssistantMemory.ts        # NEW — memory CRUD hook
└── api/
    ├── hardware.ts                  # NEW — typed fetch wrappers
    ├── puter-proxy.ts               # NEW
    ├── voice.ts                     # NEW
    └── memory.ts                    # NEW

Structure Rationale

packages/buildthis/: Standalone package with its own package.json and bin field — publishable to npm as buildthis independently. Does not depend on the monorepo server package at runtime; it only detects a running Nexus instance via HTTP or launches the CLI onboard flow.
server/src/services/ additions: All follow the existing named-export pattern (export function hardwareService() { return { ... } }). No classes. Dependencies injected as parameters. Drizzle db is only accepted if the service actually queries the DB.
ui/src/components/onboarding/: Sub-directory isolates the 5 new step components from the main components directory. NexusOnboardingWizard.tsx imports them. This limits the upstream-conflict surface to the single wizard file.
ui/src/pages/PersonalAssistant.tsx: New route registered in App.tsx routing (the only modification needed in the routing layer). The page re-uses ChatPanel with an assistantMode prop.

Architectural Patterns

Pattern 1: Hardware Detection via Server-Side Shell Probe

What: hardwareService runs on the Express server where it has access to os.totalmem() and can shell out to system_profiler SPDisplaysDataType on macOS to get GPU details. Apple Silicon unified memory is detected by checking the cpu_brand_string for "Apple M". Results are cached in memory (5-minute TTL) so the onboarding wizard can poll cheaply.

When to use: Any time the onboarding wizard needs to display hardware capabilities to make model recommendations.

Trade-offs: Server-side only — the UI cannot do this itself in the browser. The route is scoped to assertBoard (existing auth middleware), so it's protected. Apple Silicon reports unified memory as both RAM and VRAM; the service returns { unifiedMemory: true, totalBytes } instead of separate fields.

Example:

// server/src/services/hardware.ts
export function hardwareService() {
  let cache: HardwareInfo | null = null;
  let cacheExpiry = 0;

  return {
    async detect(): Promise<HardwareInfo> {
      if (cache && Date.now() < cacheExpiry) return cache;
      const totalBytes = os.totalmem();
      const gpuInfo = await probeGpu(); // shells system_profiler on macOS, /proc/driver/nvidia on Linux
      cache = { totalBytes, gpu: gpuInfo, platform: process.platform };
      cacheExpiry = Date.now() + 5 * 60 * 1000;
      return cache;
    }
  };
}

Pattern 2: Puter.js as a Server-Side Adapter (not browser-direct)

What: Puter.js supports Node.js via @heyputer/puter.js with init(authToken). The server acts as a proxy: it holds the Puter auth token (stored in company_secrets via the existing secretService), forwards chat requests to puter.ai.chat({ stream: true }), and pipes the async iterable back to the browser as SSE — exactly the same format the existing useStreamingChat hook already consumes.

Why not browser-direct: The existing chat architecture is server-mediated (all agent messages go through Express SSE). Bypassing this would require forking the streaming infrastructure. Using the server as proxy re-uses useStreamingChat unchanged and keeps the Puter token off the client.

When to use: During onboarding when user selects "Puter.js cloud" tier and authenticates. The Puter auth flow opens a browser popup (puter.auth.signIn() must be user-initiated from the UI), receives a token, then POSTs it to /api/puter-proxy/auth for server storage.

Trade-offs: One extra round-trip compared to browser-direct, but avoids token exposure and re-uses the existing SSE pipeline. Puter.js Node.js usage requires @heyputer/puter.js as a server dependency (not currently in the monorepo).

Example (server-side relay):

// server/src/routes/puter-proxy.ts
router.post("/api/puter-proxy/chat", async (req, res) => {
  const token = await svc.getStoredToken(companyId);
  const puter = init(token);
  res.setHeader("Content-Type", "text/event-stream");
  const stream = await puter.ai.chat(req.body.messages, { stream: true });
  for await (const chunk of stream) {
    res.write(`data: ${JSON.stringify(chunk)}\n\n`);
  }
  res.end();
});

Pattern 3: Whisper on Server, Piper in Browser (Hybrid Voice)

What: Voice input (speech-to-text) runs server-side via whisper-node (Node.js bindings for whisper.cpp). The UI records audio via MediaRecorder, POSTs a blob to POST /api/voice/transcribe, and gets back a transcript string. Voice output (text-to-speech) uses @mintplex-labs/piper-tts-web which runs client-side via WebAssembly — no server round-trip needed for TTS.

Why this split: whisper.cpp requires native binaries that work on CPU-only hardware, which the server controls. Piper TTS web runs via WASM in the browser and has no native dependency — this keeps TTS latency low (no network round-trip) and works even if the server is slow.

When to use: When user selects "voice mode" in onboarding (VoiceSetupStep). Whisper runs only if the user chooses a local Whisper model (downloaded to data/whisper-models/); as a fallback, the browser's native webkitSpeechRecognition / SpeechRecognition API is used.

Trade-offs: Whisper download adds 75MB–1.5GB to first-run setup. For CPU-only hardware, whisper-tiny.en (75MB) transcribes in ~2s for a 10s clip on M4 — acceptable. Piper WASM download is ~20MB (models ~30-100MB each).

Example (voice input hook):

// ui/src/hooks/useVoiceInput.ts
export function useVoiceInput() {
  // Records with MediaRecorder → blob → POST /api/voice/transcribe
  // Falls back to window.SpeechRecognition if whisper not configured
}

Pattern 4: Persistent Memory via File-Backed JSON (No New DB Table)

What: The assistant memory store is a per-workspace JSON file at data/memory/<companyId>.json. Each memory entry has { id, content, createdAt, tags }. The memoryService reads this file on startup (lazy-loaded per companyId), keeps it in-process, and writes on mutation. Memory injection works by prepending a formatted memory block to the system prompt at chat-send time in the existing chat service.

Why not PostgreSQL: Adding a new table violates the "no DB schema changes" constraint for upstream rebase safety. File-backed JSON with an in-process cache is fast for a single-user setup (sub-millisecond reads) and requires no migration.

When to use: Personal AI Assistant mode only. Project Builder mode does not use the memory service.

Trade-offs: Not transactional. For a single-user local deployment, this is acceptable. File writes are atomic via write-then-rename pattern. Memory search is linear scan (no vector embeddings in v1.5 — semantic search is a future enhancement).

Example:

// server/src/services/memory.ts
export function memoryService() {
  const cache = new Map<string, MemoryStore>();

  return {
    async inject(companyId: string, systemPrompt: string): Promise<string> {
      const store = await load(companyId);
      if (store.entries.length === 0) return systemPrompt;
      const block = store.entries.map(e => `- ${e.content}`).join("\n");
      return `${systemPrompt}\n\n## What I remember about you:\n${block}`;
    }
  };
}

Pattern 5: Onboarding State via instance_settings.general JSONB

What: All onboarding configuration (selected mode, voice config, active provider tier) is stored in the existing instance_settings.general JSONB column. The instanceSettingsService already handles arbitrary JSONB keys. Nexus adds its config under a nexus namespace key to avoid upstream key collisions.

When to use: Reading/writing onboarding mode, voice model selection, and provider tier configuration. No new table, no migration.

Example:

// instance_settings.general.nexus = {
//   mode: "personal_ai" | "project_builder" | "both",
//   voiceModel: "whisper-tiny.en" | "whisper-base" | null,
//   piperVoice: "en_US-amy-medium" | null,
//   providerTier: "local" | "puter" | "oauth_gemini" | "api_key",
// }

Pattern 6: OAuth Token Storage via Existing Secrets Service

What: OAuth tokens (Google Gemini, OpenAI) and the Puter.js auth token are stored via the existing secretService using the local_encrypted provider. The onboarding wizard calls POST /api/companies/:id/secrets with a well-known name (e.g., nexus_puter_token, nexus_gemini_token). Adapters read these at spawn time.

When to use: Any time an OAuth flow completes and a token needs persistence.

Trade-offs: Secrets are per-company (workspace), not per-instance. This is fine for single-user setup. The existing secrets UI lets users view/rotate tokens manually.

Data Flow

Onboarding Wizard Data Flow

User opens Nexus (no workspace yet)
    ↓
NexusOnboardingWizard renders
    ↓
Step 1: ModeSelector → user picks "Personal AI" / "Project Builder" / "Both"
    ↓
Step 2: HardwareSummaryStep
    → GET /api/hardware/info (new route)
    → hardwareService.detect() → os.totalmem() + system_profiler
    → returns { totalGb, gpuName, unifiedMemory, platform }
    → wizard shows model tier recommendations
    ↓
Step 3: ProviderTierStep
    → Local: already detected via existing Hermes probe
    → Puter.js: user clicks "Connect" → puter.auth.signIn() popup
                → UI POSTs token to POST /api/puter-proxy/auth
                → server stores in secretService("nexus_puter_token")
    → OAuth (Gemini/OpenAI): OAuth PKCE flow in a popup window
                → callback captured by temp local server or redirect
                → token stored via secretService
    → API Key: direct input → stored via secretService
    ↓
Step 4: VoiceSetupStep (optional, skippable)
    → GET /api/voice/status → check if whisper binary present
    → User picks model → POST /api/voice/download (async download + SSE progress)
    → User picks Piper voice → stored in instance_settings.general.nexus
    ↓
Step 5: OnboardingSummaryStep
    → Creates workspace + agents (existing companiesApi + agentsApi flow)
    → Writes nexus config to instance_settings.general.nexus
    → Navigates to PersonalAssistant page OR Dashboard based on mode

Personal AI Assistant Chat Data Flow

User types message in AssistantChatHub
    ↓
ChatInput → useStreamingChat.startStream(conversationId, message)
    ↓
POST /api/companies/:id/chat/conversations/:convId/messages
    ↓ (existing chat route, no change)
chat route detects "personal_assistant" agent type
    ↓
memoryService.inject(companyId, systemPrompt)  ← NEW injection point
    ↓
Route selects provider based on instance_settings.general.nexus.providerTier:
  - "local"     → existing Hermes adapter (no change)
  - "puter"     → puterProxyService.chat() → Puter.js Node client → SSE relay
  - "oauth_*"   → respective provider API with stored OAuth token → SSE relay
    ↓
SSE events stream to UI via existing /api/chat/stream endpoint pattern
    ↓
useStreamingChat receives chunks → ChatMessageList renders them

Voice Input Data Flow

User presses mic button in ChatInput (MODIFIED)
    ↓
useVoiceInput starts MediaRecorder → records WebM/Opus blob
    ↓
User releases mic → blob POSTed to POST /api/voice/transcribe
    ↓
voiceService.transcribe(audioBuffer) 
    → whisper-node.transcribe(path) → returns text
    ↓
Text injected into ChatInput.value
    ↓
User reviews → sends normally

npx buildthis Data Flow

Developer runs: npx buildthis
    ↓
buildthis/src/index.ts checks for running Nexus:
    GET http://localhost:4000/api/health → 200?
        YES → open browser to http://localhost:4000
        NO  → run nexus onboard wizard (delegates to paperclipai onboard)
              OR detect Docker → suggest docker-compose up

Integration Points: New vs Modified

Server Routes — app.ts (MODIFIED)

One file to add 4 route mounts. Minimal conflict surface with upstream.

// In server/src/app.ts — add after ollamaRoutes():
app.use(hardwareRoutes());
app.use(voiceRoutes());
app.use(memoryRoutes(db));
app.use(puterProxyRoutes(db));

Chat Route — MODIFIED for memory injection

The existing chat service (server/src/services/chat.ts) needs one injection point: when building the system prompt for a conversation, call memoryService.inject(). This is scoped to conversations where the agent has adapterConfig.assistantMode === true.

Risk: This touches an upstream file. The injection is a 3-line addition inside the message-send handler. Low conflict probability — upstream rarely modifies this section.

NexusOnboardingWizard.tsx — REPLACED

The current single-step wizard becomes a multi-step wizard. Since this file is already a Nexus replacement (not an upstream file), there is zero conflict risk — it will never exist in upstream.

App.tsx routing — MODIFIED (one new route)

Add the PersonalAssistant page as a new lazy-loaded route. Minimal upstream conflict (routing section rarely changes).

ChatInput.tsx — MODIFIED (voice button)

Add a microphone button that triggers useVoiceInput. This is an upstream file — the modification is additive (new button, no existing logic changed). Conflict risk: LOW, as upstream rarely modifies ChatInput.

Anti-Patterns

Anti-Pattern 1: Browser-Direct Puter.js

What people do: Import @heyputer/puter.js in the React frontend and call puter.ai.chat() directly from the browser.

Why it's wrong: Exposes the Puter auth token in browser storage/network. Bypasses the existing SSE pipeline, requiring a second streaming implementation. Breaks the memory injection pattern (no server-side hook). Cannot use the existing useStreamingChat hook.

Do this instead: Use the server proxy pattern (Pattern 2). The UI sends messages to /api/puter-proxy/chat exactly like any other chat endpoint.

Anti-Pattern 2: New PostgreSQL Tables for Memory

What people do: Create a assistant_memories migration with a proper relational schema.

Why it's wrong: Violates the hard constraint: no DB migrations, no schema changes, to keep upstream rebase clean. A migration file created in Nexus will conflict every time upstream adds a migration.

Do this instead: File-backed JSON in the server's data directory (Pattern 4). The single-user M4 Mini deployment will never hit performance limits with this approach.

Anti-Pattern 3: Multi-Step Wizard as Modified OnboardingWizard.tsx

What people do: Modify the upstream OnboardingWizard.tsx directly to add v1.5 steps.

Why it's wrong: The upstream wizard is actively maintained (120+ upstream commits since fork). Touching it creates guaranteed rebase conflicts.

Do this instead: Continue the existing pattern — NexusOnboardingWizard.tsx is already the Nexus replacement via Vite alias. All v1.5 changes go there. Upstream file untouched.

Anti-Pattern 4: OAuth in the Browser via Redirect

What people do: Redirect the main app window to the OAuth provider and handle the callback via window.location.

Why it's wrong: Loses React state mid-flow. Hard to handle callback URL in a local server that may not have a publicly routable HTTPS endpoint.

Do this instead: Use a popup window for OAuth (window.open). The popup handles the full OAuth redirect. On callback, the popup calls window.opener.postMessage with the token, closes itself, and the main window receives it. For Puter.js specifically, puter.auth.signIn() handles the popup internally.

Scaling Considerations

This is a single-user local deployment on an M4 Mini. Scaling is not a concern for v1.5. The architecture is designed for correctness and upstream merge-ability, not horizontal scale.

Concern	Single User (M4 Mini)
Hardware detection	os.totalmem() + sync shell probe, cached 5min — negligible
Puter.js relay	One connection at a time, no pooling needed
Whisper transcription	~2s for 10s clip on M4, sequential queue sufficient
Memory store	File JSON, <10ms read, no contention
Voice TTS	WASM in browser, zero server load

Build Order (Dependency Graph)

The build order matters because later phases consume services built in earlier ones.

Phase 1: Hardware Detection
  → hardwareService (server)
  → GET /api/hardware/info (route)
  → useHardwareInfo hook (UI)
  → HardwareSummaryStep component (UI)
  No dependencies on other new phases.

Phase 2: Provider Tiers (depends on Phase 1 for display)
  → puterProxyService (server) — Puter.js Node client
  → secretService integration for token storage (uses EXISTING service)
  → POST /api/puter-proxy/auth (route)
  → ProviderTierStep component (UI)
  → OAuth popup flow (UI)

Phase 3: Multi-Step Onboarding Wizard (depends on Phases 1+2)
  → ModeSelector, OnboardingSummaryStep components (UI)
  → Refactor NexusOnboardingWizard.tsx into multi-step
  → instance_settings.general.nexus config write

Phase 4: Persistent Memory + Assistant Mode (depends on Phase 3)
  → memoryService (server)
  → Memory injection in chat route (MODIFIED — highest risk step)
  → GET/POST/DELETE /api/companies/:id/memory (routes)
  → PersonalAssistantPage (UI)
  → useAssistantMemory hook (UI)

Phase 5: Voice (depends on Phase 3, independent of Phase 4)
  → voiceService (server) — whisper-node + piper setup
  → POST /api/voice/transcribe, /speak, /status (routes)
  → VoiceSetupStep in onboarding (UI)
  → useVoiceInput, useVoiceSpeech hooks (UI)
  → ChatInput microphone button (MODIFIED — upstream file, low risk)

Phase 6: npx buildthis (independent of all above)
  → packages/buildthis/ new package
  → package.json bin field setup
  → npm publish configuration

Recommended sequence: 1 → 2 → 3 → 4 → 5 → 6. Phase 4 (memory injection into chat route) is the highest-risk upstream-file modification and should come after onboarding is validated.

Integration Points: External Services

Service	Integration Pattern	Auth Storage	Notes
Puter.js	Server-side Node.js client proxy, SSE relay	`company_secrets` table	Token obtained via browser popup on first connect
Google Gemini OAuth	PKCE popup flow, access token + refresh token	`company_secrets` table	Policy risk: using Gemini CLI OAuth with third-party apps may trigger abuse detection — use only if user has an active Gemini subscription
OpenAI OAuth	PKCE flow via auth.openai.com	`company_secrets` table	Only for free tier / ChatGPT Plus users
Whisper (whisper-node)	Native binary, spawned by voiceService	N/A — local binary	Download on first use, cached in data/whisper-models/
Piper TTS	@mintplex-labs/piper-tts-web WASM, runs in browser	N/A — client-side	Model files downloaded to browser cache
Ollama	Existing integration (v1.4) — no changes	N/A	ollama.ts service and /ollama routes unchanged

Sources

Codebase inspection: /opt/nexus/server/src/, /opt/nexus/ui/src/, /opt/nexus/packages/
Puter.js Node.js support: https://docs.puter.com/supported-platforms/
Puter.js chat streaming API: https://docs.puter.com/AI/chat/
Puter.js auth flow: https://developer.puter.com/blog/browser-based-auth-puter-js-node/
whisper-node npm package: https://www.npmjs.com/package/whisper-node
Piper TTS WASM: https://www.npmjs.com/package/@mintplex-labs/piper-tts-web
@xenova/transformers Node.js audio guide: https://huggingface.co/docs/transformers.js/main/en/guides/node-audio-processing
Google Gemini OAuth: https://ai.google.dev/gemini-api/docs/oauth
Google Gemini OAuth policy risk: https://github.com/google-gemini/gemini-cli/issues/21866
Vectra local vector DB (future memory enhancement): https://github.com/Stevenic/vectra
Apple Silicon unified memory: https://eclecticlight.co/2022/03/01/making-sense-of-m1-memory-use/

Architecture research for: Nexus v1.5 Smart Onboarding + Personal AI Assistant Researched: 2026-04-02

34 KiB Raw Blame History Unescape Escape

Architecture Research

System Overview

Component Responsibilities

Recommended Project Structure

Structure Rationale

Architectural Patterns

Pattern 1: Hardware Detection via Server-Side Shell Probe

Pattern 2: Puter.js as a Server-Side Adapter (not browser-direct)

Pattern 3: Whisper on Server, Piper in Browser (Hybrid Voice)

Pattern 4: Persistent Memory via File-Backed JSON (No New DB Table)

Pattern 5: Onboarding State via instance_settings.general JSONB

Pattern 6: OAuth Token Storage via Existing Secrets Service

Data Flow

Onboarding Wizard Data Flow

Personal AI Assistant Chat Data Flow

Voice Input Data Flow

npx buildthis Data Flow

Integration Points: New vs Modified

Server Routes — app.ts (MODIFIED)

Chat Route — MODIFIED for memory injection

NexusOnboardingWizard.tsx — REPLACED

App.tsx routing — MODIFIED (one new route)

ChatInput.tsx — MODIFIED (voice button)

Anti-Patterns

Anti-Pattern 1: Browser-Direct Puter.js

Anti-Pattern 2: New PostgreSQL Tables for Memory

Anti-Pattern 3: Multi-Step Wizard as Modified OnboardingWizard.tsx

Anti-Pattern 4: OAuth in the Browser via Redirect

Scaling Considerations

Build Order (Dependency Graph)

Integration Points: External Services

Sources

34 KiB

Raw Blame History