docs(33): research phase persistent memory domain

2026-04-03 21:44:10 +00:00 · 2026-04-03 21:44:10 +00:00 · 2efed8797e
commit 2efed8797e
parent 9ff01aeda9
1 changed files with 420 additions and 0 deletions
--- a/.planning/phases/33-persistent-memory/33-RESEARCH.md
+++ b/.planning/phases/33-persistent-memory/33-RESEARCH.md
@ -0,0 +1,420 @@
+# Phase 33: Persistent Memory + Personal Assistant Mode — Research
+
+**Researched:** 2026-04-01
+**Domain:** File-backed memory service, prompt injection, assistant handoff to PM agent
+**Confidence:** HIGH
+
+---
+
+<user_constraints>
+## User Constraints (from CONTEXT.md)
+
+### Locked Decisions
+All implementation choices are at Claude's discretion — discuss phase was skipped per user setting. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions.
+
+### Claude's Discretion
+All implementation choices.
+
+### Deferred Ideas (OUT OF SCOPE)
+None — discuss phase skipped.
+</user_constraints>
+
+---
+
+<phase_requirements>
+## Phase Requirements
+
+| ID | Description | Research Support |
+|----|-------------|------------------|
+| ASST-01 | User has persistent memory across chat sessions (summary-based, injected into system prompts) | File-backed memory service at `data/assistant-memory/<companyId>.json`; inject via new `/conversations/:id/assistant-stream` endpoint or by extending existing `/conversations/:id/stream` |
+| ASST-02 | Memory content sanitized at write time to prevent prompt injection | `server/src/redaction.ts` has `SECRET_PAYLOAD_KEY_RE` + `JWT_VALUE_RE`; extend with a plain-text credential blocklist pattern applied at `memorySummary` write time |
+| ASST-03 | User can hand off an assistant conversation to a PM agent with one click, transferring context | Existing `POST /conversations/:id/handoff` route creates an issue from a spec. Phase 33 needs a parallel route — or mode on the existing route — that creates a PM agent conversation with a system message containing a conversation summary |
+| ASST-04 | Assistant and Project Builder modes work standalone or together | `nexusSettingsService` persists `mode` as `"personal_ai" | "project_builder" | "both"` to `data/nexus-settings.json`. Phase 33 adds a `PersonalAssistantPage` gated on `mode !== "project_builder"` |
+</phase_requirements>
+
+---
+
+## Summary
+
+Phase 33 adds three capabilities to Nexus: (1) a file-backed memory layer that accumulates facts across sessions and injects them as a system prompt prefix before every assistant response, (2) credential scrubbing that prevents API keys and tokens from being stored in memory, and (3) a one-click "hand off to PM" flow that creates a new conversation pre-seeded with a summary of the assistant exchange.
+
+No new DB tables are permitted (milestone constraint). All state lives either in `data/assistant-memory/<companyId>.json` (following the `nexus-settings.json` file-backed pattern) or in existing JSONB columns. The `mode` setting already persists correctly in `data/nexus-settings.json` via `nexusSettingsService` and is accessible via `GET /nexus/settings`. The UI only needs to read this mode to gate the `PersonalAssistantPage` route.
+
+The streaming endpoint at `POST /conversations/:id/stream` currently responds with the echo service. Phase 33 must replace this with a real AI call (most likely through `puterProxyService`) and add memory injection into the message array sent to the model. Memory is accumulated after each completed assistant turn by calling a new `assistantMemoryService.append()` which sanitizes the assistant text before persisting.
+
+**Primary recommendation:** Build a `assistantMemoryService` modelled on `nexusSettingsService` (file-backed JSON, no DB changes), extend the `/conversations/:id/stream` endpoint to prepend memory as a system message, and add a `/conversations/:id/assistant-handoff` route that creates a PM-linked conversation with a seeded system message.
+
+---
+
+## Standard Stack
+
+### Core
+| Library | Version | Purpose | Why Standard |
+|---------|---------|---------|--------------|
+| `node:fs` (sync) | Node built-in | Read/write `assistant-memory/<companyId>.json` | Same pattern as `nexus-settings.json` — no extra deps |
+| `zod` | already in codebase | Schema validation for memory JSON on read | All service layer uses zod |
+| `drizzle-orm` | already in codebase | Querying `chatMessages` for summary extraction | Already used in `chatService` |
+| Express Router | already in codebase | New memory CRUD routes | All server routes use Express |
+
+### Supporting
+| Library | Version | Purpose | When to Use |
+|---------|---------|---------|-------------|
+| `puterProxyService` | internal | Real AI completions for memory summarization | Use for the summarize-conversation step; same service already used for chat |
+
+### Alternatives Considered
+| Instead of | Could Use | Tradeoff |
+|------------|-----------|----------|
+| File-backed JSON memory | `instance_settings.general` JSONB | JSONB is per-instance, not per-company; file-backed allows per-company isolation and matches REQUIREMENTS.md out-of-scope note |
+| File-backed JSON memory | Vector DB (Mem0, Chroma) | Explicitly out of scope in REQUIREMENTS.md — "Vector database for memory: Summary-based approach sufficient; no infra overhead" |
+| File-backed JSON memory | `chat_conversations` extra column | Would require a DB schema migration — prohibited by milestone constraint |
+
+**Installation:** No new packages needed.
+
+---
+
+## Architecture Patterns
+
+### Recommended Project Structure
+```
+server/src/services/
+├── assistant-memory.ts     # new — read/write data/assistant-memory/<companyId>.json
+server/src/routes/
+├── assistant-memory.ts     # new — GET/PATCH memory endpoints
+ui/src/
+├── pages/PersonalAssistantPage.tsx  # new — gated on mode !== "project_builder"
+├── api/assistantMemory.ts           # new — API client for memory endpoints
+```
+
+### Pattern 1: File-Backed Service (follows nexusSettingsService)
+
+**What:** Read a JSON file from `resolvePaperclipInstanceRoot()/data/assistant-memory/<companyId>.json`, validate with zod, write back after updates.
+
+**When to use:** All memory reads and writes.
+
+**Example:**
+```typescript
+// Source: server/src/services/nexus-settings.ts (existing pattern)
+import fs from "node:fs";
+import path from "node:path";
+import { z } from "zod";
+import { resolvePaperclipInstanceRoot } from "../home-paths.js";
+
+const assistantMemorySchema = z.object({
+  facts: z.array(z.string()).default([]),
+  updatedAt: z.string().datetime().optional(),
+});
+
+type AssistantMemory = z.infer<typeof assistantMemorySchema>;
+
+function resolveMemoryPath(companyId: string): string {
+  return path.resolve(
+    resolvePaperclipInstanceRoot(),
+    "data",
+    "assistant-memory",
+    `${companyId}.json`,
+  );
+}
+```
+
+### Pattern 2: Memory Injection via System Message Prefix
+
+**What:** Before streaming a response, prepend a system message containing the memory facts to the messages array passed to the model.
+
+**When to use:** Every `/conversations/:id/stream` call when the conversation's company is in `personal_ai` or `both` mode.
+
+**Example:**
+```typescript
+// Inject pattern — server/src/routes/chat.ts stream endpoint extension
+const memory = await assistantMemoryService.get(companyId);
+const systemPrefix = memory.facts.length > 0
+  ? `[Memory from previous sessions]\n${memory.facts.map(f => `- ${f}`).join("\n")}\n\n`
+  : "";
+const messagesWithMemory = systemPrefix
+  ? [{ role: "system", content: systemPrefix }, ...conversationMessages]
+  : conversationMessages;
+```
+
+### Pattern 3: Write-Time Sanitization
+
+**What:** Before appending any fact to memory, run it through a credential-scrubbing function.
+
+**When to use:** Every call to `assistantMemoryService.append()`.
+
+**Example:**
+```typescript
+// Source: server/src/redaction.ts (existing pattern extended for plain text)
+const CREDENTIAL_INLINE_RE = /\b(sk-[A-Za-z0-9]{20,}|ghp_[A-Za-z0-9]{36}|AIza[0-9A-Za-z_-]{35}|[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{6,}\.[A-Za-z0-9_-]{20,})/g;
+const SENSITIVE_KEY_VALUE_RE = /(?:api[_-]?key|token|secret|password|bearer|auth)\s*[:=]\s*\S+/gi;
+
+function sanitizeMemoryFact(raw: string): string {
+  return raw
+    .replace(CREDENTIAL_INLINE_RE, "[REDACTED]")
+    .replace(SENSITIVE_KEY_VALUE_RE, "[REDACTED]");
+}
+```
+
+### Pattern 4: PM Agent Handoff with Conversation Summary
+
+**What:** `POST /conversations/:id/assistant-handoff` — creates a new conversation pre-seeded with a system message summarising the current exchange, then navigates the user to that new conversation.
+
+**When to use:** When user clicks "Turn this into a project" in PersonalAssistantPage.
+
+The existing `POST /conversations/:id/handoff` creates an issue from a structured spec. The new assistant handoff is different — it creates a conversation context transfer, not necessarily an issue. It should:
+1. Fetch the last N messages from the current conversation.
+2. Produce a brief text summary (simple first-pass: concatenate user messages up to a token budget; optional: use AI summarisation via puterProxyService if token available).
+3. Create a new conversation (or find the PM agent's default conversation) and insert a `messageType: "handoff_context"` system message.
+4. Return `{ targetConversationId }` so the client can navigate.
+
+### Anti-Patterns to Avoid
+- **Injecting memory at retrieval time (not write time):** ASST-02 requires sanitization at write time. Sanitizing only on read means raw credentials are stored to disk.
+- **Storing the full conversation as memory:** Memory should be a summary — not raw message content. Full messages can include credentials, PII, or prompt-injection payloads.
+- **Putting memory in `instance_settings.general`:** That field is per-instance not per-company. Multiple companies (workspaces) need separate memory namespaces.
+- **Reading memory synchronously in the hot path of the SSE stream without caching:** The file read is fast (< 1ms for small JSON) but should be done once before writing SSE headers, not inside the token loop.
+
+---
+
+## Don't Hand-Roll
+
+| Problem | Don't Build | Use Instead | Why |
+|---------|-------------|-------------|-----|
+| Credential scrubbing regex | Custom patterns from scratch | Extend `server/src/redaction.ts` with plain-text patterns | Existing patterns cover `api_key`, `access_token`, `jwt`, etc. — reusing reduces divergence |
+| Secrets storage | New encrypted store | `secretService` already handles encrypted storage — memory does not store secrets, only sanitized facts | Credential data must never reach memory layer at all |
+| AI summarisation service | New inference client | `puterProxyService` for optional summarisation — same endpoint, same cost tracking | Avoids a second AI client implementation |
+
+**Key insight:** The memory layer is deliberately simple — a list of plain-text facts. The complexity is in the sanitization gate, not the storage.
+
+---
+
+## Runtime State Inventory
+
+> SKIPPED — this is a greenfield phase, not a rename/refactor/migration.
+
+---
+
+## Common Pitfalls
+
+### Pitfall 1: Mode Check Location
+**What goes wrong:** UI renders PersonalAssistantPage for all users regardless of mode, or the stream endpoint injects memory for project-builder-only users.
+**Why it happens:** `nexusSettings.mode` is read from a file, not from DB — easy to forget to check it server-side.
+**How to avoid:** The stream endpoint must call `nexusSettingsService().get()` and skip memory injection when `mode === "project_builder"`.
+**Warning signs:** Memory facts appearing in project builder conversations.
+
+### Pitfall 2: companyId Scoping in Memory File Path
+**What goes wrong:** All companies share a single memory file.
+**Why it happens:** Forgetting `companyId` in the path, e.g. `data/assistant-memory.json` instead of `data/assistant-memory/<companyId>.json`.
+**How to avoid:** The path resolver function must include `companyId` as a directory segment (as shown in Pattern 1 above).
+**Warning signs:** Facts from workspace A appearing in workspace B.
+
+### Pitfall 3: SSE Stream Already Flushed When Memory Read Fails
+**What goes wrong:** A memory read error causes an unhandled exception after SSE headers are flushed, leaving the client connection open but broken.
+**Why it happens:** The existing stream endpoint flushes headers at line 101 (`res.flushHeaders()`) before any async logic.
+**How to avoid:** Read memory before calling `res.flushHeaders()`. If memory read fails, fall back gracefully (empty memory, log warning) — never throw after flush.
+**Warning signs:** Client sees `:ok` event but then nothing further.
+
+### Pitfall 4: Memory Growing Without Bound
+**What goes wrong:** `facts` array accumulates thousands of entries; the injected system prompt exceeds the model's context window.
+**Why it happens:** No fact eviction or cap.
+**How to avoid:** Cap at 50 facts (FIFO — drop oldest when limit is reached). Cap the injected system prefix at 2000 characters max.
+**Warning signs:** Streaming responses truncated; model errors about context length.
+
+### Pitfall 5: Handoff Creates Duplicate PM Conversations
+**What goes wrong:** Every "Turn this into a project" click creates a new PM conversation, leading to many orphaned conversations.
+**Why it happens:** The handoff endpoint creates a new conversation on every call.
+**How to avoid:** Use the PM agent's existing most-recent-conversation or create a fresh one per handoff (acceptable for v1.5); document this as a known v1.5 limitation. The route should return `targetConversationId` so the UI can detect if a new one was created.
+**Warning signs:** Conversation list growing rapidly.
+
+### Pitfall 6: Chat Route Injection Point
+**What goes wrong:** Memory injection is added to the wrong handler — the echo `streamEcho` helper instead of the real AI call.
+**Why it happens:** `POST /conversations/:id/stream` currently calls `svc.streamEcho()` which is a stub. Phase 33 replaces this with a real AI call. The memory injection must be added here, not in the stub.
+**How to avoid:** During plan-phase, inspect the exact injection point in `server/src/routes/chat.ts` lines 107-137 and confirm the replacement strategy. (State.md blocker: "Chat route injection point needs codebase inspection — confirm correct hook location in `server/src/services/chat.ts` during plan-phase".)
+**Warning signs:** Memory injected but responses are still the echo stub.
+
+---
+
+## Code Examples
+
+Verified patterns from existing codebase:
+
+### File-Backed Service Read/Write
+```typescript
+// Source: server/src/services/nexus-settings.ts
+async function get(): Promise<NexusSettings> {
+  const filePath = resolveNexusSettingsPath();
+  try {
+    const raw = fs.readFileSync(filePath, "utf-8");
+    const parsed = nexusSettingsSchema.safeParse(JSON.parse(raw));
+    if (parsed.success) return parsed.data;
+    return { mode: "both" };
+  } catch {
+    return { mode: "both" };
+  }
+}
+
+async function set(patch: Partial<NexusSettings>): Promise<NexusSettings> {
+  const current = await get();
+  const merged = { ...current, ...patch };
+  const validated = nexusSettingsSchema.parse(merged);
+  const filePath = resolveNexusSettingsPath();
+  fs.mkdirSync(path.dirname(filePath), { recursive: true });
+  fs.writeFileSync(filePath, JSON.stringify(validated, null, 2), "utf-8");
+  return validated;
+}
+```
+
+### Existing Redaction Patterns (extend for plain-text memory)
+```typescript
+// Source: server/src/redaction.ts
+const SECRET_PAYLOAD_KEY_RE =
+  /(api[-_]?key|access[-_]?token|auth(?:_?token)?|authorization|bearer|secret|passwd|password|credential|jwt|private[-_]?key|cookie|connectionstring)/i;
+const JWT_VALUE_RE = /^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+(?:\.[A-Za-z0-9_-]+)?$/;
+```
+
+### Existing Handoff Route Pattern
+```typescript
+// Source: server/src/routes/chat.ts lines 163-203
+router.post("/conversations/:id/handoff", async (req, res) => {
+  assertBoard(req);
+  const data = handoffSchema.parse(req.body);
+  const conversation = await svc.getConversation(req.params.id!);
+  const companyId = conversation.companyId;
+  // 1. Insert handoff system message
+  // 2. Create issue from spec
+  // 3. Insert task_created system message
+  res.json({ handoffMessageId: handoffMsg.id, issues: [issue] });
+});
+```
+
+### Existing SSE Streaming Pattern
+```typescript
+// Source: server/src/routes/chat.ts lines 88-137
+res.setHeader("Content-Type", "text/event-stream");
+res.setHeader("Cache-Control", "no-cache");
+res.setHeader("Connection", "keep-alive");
+res.setHeader("X-Accel-Buffering", "no");
+res.flushHeaders();
+res.write(":ok\n\n");
+// ... async generator loop
+res.write(`data: ${JSON.stringify({ token })}\n\n`);
+res.write(`data: ${JSON.stringify({ done: true, messageId, content })}\n\n`);
+```
+
+### PuterProxyService Chat Stream
+```typescript
+// Source: server/src/services/puter-proxy.ts
+async function* chatStream(
+  companyId: string,
+  agentId: string | null | undefined,
+  messages: unknown[],
+  model: string | undefined,
+  signal: AbortSignal | undefined,
+): AsyncGenerator<string>
+// Called with messages array in OpenAI format: [{ role: "system"|"user"|"assistant", content: string }]
+```
+
+---
+
+## State of the Art
+
+| Old Approach | Current Approach | When Changed | Impact |
+|--------------|------------------|--------------|--------|
+| Vector DB for memory | Summary-based file-backed facts | Phase 33 (new) | No infra overhead; no pgvector needed |
+| Echo stub in stream endpoint | Real AI via puterProxyService | Phase 33 (new) | Memory injection only makes sense once real AI is wired |
+
+**Deprecated/outdated:**
+- `streamEcho()` in `chatService`: stub used for testing — Phase 33 replaces it with real AI calls in the stream route (or supplements it with a parallel assistant-stream endpoint).
+
+---
+
+## Open Questions
+
+1. **Does the stream endpoint replace the echo stub or add a parallel endpoint?**
+   - What we know: `/conversations/:id/stream` currently calls `svc.streamEcho()`. The puter proxy has its own endpoint at `/puter-proxy/chat`.
+   - What's unclear: Whether to extend the existing stream endpoint to delegate to puterProxyService (if the conversation is in a personal-assistant company), or create a dedicated `/conversations/:id/assistant-stream`.
+   - Recommendation: Extend the existing endpoint — the UI already calls it via `chatApi.postMessageAndStream()`. The endpoint should check `nexusSettings.mode` and route to puterProxy if a token is available, else fall back to echo stub. This avoids a UI-side change.
+
+2. **What constitutes a "memory fact" — when is it written?**
+   - What we know: ASST-01 says "summary-based". Facts could be extracted from (a) every assistant turn, (b) end of conversation, or (c) on demand.
+   - What's unclear: v1.5 success criterion 1 says "A fact stated in one chat session… is referenced correctly by the assistant in a new session." This implies facts are persisted at the end of a session or after each user-confirmed turn.
+   - Recommendation: Append after each assistant response turn — simpler than session-end detection, and aligns with success criterion 1 (fact from one session visible in next). The `onDone` callback in `chatApi.postMessageAndStream` is the natural trigger; or do it server-side in the stream endpoint after saving the final message.
+
+3. **How does the personal assistant handoff interact with the existing brainstormer handoff?**
+   - What we know: The existing handoff (Phase 23) creates a project issue from a structured spec (`{ what, why, constraints, success }`). The assistant handoff (ASST-03) transfers conversation context to a PM agent.
+   - What's unclear: Whether these are the same button or different flows.
+   - Recommendation: Implement a separate `POST /conversations/:id/assistant-handoff` endpoint. It does not create an issue — it creates a new conversation with a context summary as a seeded system message. The UI button is "Turn this into a project" (distinct from the brainstormer's "Send to PM" which targets a structured spec).
+
+---
+
+## Environment Availability
+
+> Phase is code/config-only. No new external services required.
+> `puterProxyService` is already integrated (Phase 31). File I/O uses Node.js built-ins.
+
+| Dependency | Required By | Available | Version | Fallback |
+|------------|------------|-----------|---------|----------|
+| Node.js `fs` module | Memory file storage | ✓ | built-in | — |
+| `puterProxyService` | AI responses + optional summarisation | ✓ | internal | Fall back to echo stub if no token |
+| `nexusSettingsService` | Mode check (ASST-04) | ✓ | internal | — |
+| `secretService` | Token resolution in puterProxy | ✓ | internal | — |
+
+---
+
+## Validation Architecture
+
+### Test Framework
+| Property | Value |
+|----------|-------|
+| Framework | Vitest 3.x |
+| Config file | `server/vitest.config.ts` |
+| Quick run command | `pnpm --filter @paperclipai/server vitest run --reporter=verbose src/__tests__/33-*.test.ts` |
+| Full suite command | `pnpm test:run` |
+
+### Phase Requirements → Test Map
+| Req ID | Behavior | Test Type | Automated Command | File Exists? |
+|--------|----------|-----------|-------------------|-------------|
+| ASST-01 | Memory persists across sessions — new session includes previously stored fact in system prompt | unit | `pnpm --filter @paperclipai/server vitest run src/__tests__/33-assistant-memory.test.ts` | ❌ Wave 0 |
+| ASST-02 | API key pasted into chat is NOT stored in memory file | unit | `pnpm --filter @paperclipai/server vitest run src/__tests__/33-memory-sanitization.test.ts` | ❌ Wave 0 |
+| ASST-03 | Assistant handoff creates target conversation with context summary system message | unit | `pnpm --filter @paperclipai/server vitest run src/__tests__/33-assistant-handoff.test.ts` | ❌ Wave 0 |
+| ASST-04 | PersonalAssistantPage visible when mode is `personal_ai` or `both`; hidden when `project_builder` | unit | `pnpm --filter @paperclipai/ui vitest run src/components/PersonalAssistantPage.test.tsx` | ❌ Wave 0 |
+
+### Sampling Rate
+- **Per task commit:** `pnpm --filter @paperclipai/server vitest run src/__tests__/33-*.test.ts`
+- **Per wave merge:** `pnpm test:run`
+- **Phase gate:** Full suite green before `/gsd:verify-work`
+
+### Wave 0 Gaps
+- [ ] `server/src/__tests__/33-assistant-memory.test.ts` — covers ASST-01
+- [ ] `server/src/__tests__/33-memory-sanitization.test.ts` — covers ASST-02
+- [ ] `server/src/__tests__/33-assistant-handoff.test.ts` — covers ASST-03
+- [ ] `ui/src/components/PersonalAssistantPage.test.tsx` — covers ASST-04
+
+---
+
+## Sources
+
+### Primary (HIGH confidence)
+- `server/src/services/nexus-settings.ts` — file-backed JSON service pattern (confirmed by reading source)
+- `server/src/services/puter-proxy.ts` — AI streaming service (confirmed by reading source)
+- `server/src/routes/chat.ts` — SSE streaming pattern and handoff route (confirmed by reading source)
+- `server/src/redaction.ts` — credential scrubbing patterns (confirmed by reading source)
+- `packages/db/src/schema/chat_conversations.ts` — conversation schema (confirmed by reading source)
+- `packages/db/src/schema/chat_messages.ts` — message schema with `messageType` column (confirmed by reading source)
+- `.planning/REQUIREMENTS.md` — "DB schema changes: Out of Scope" (confirmed by reading source)
+
+### Secondary (MEDIUM confidence)
+- `.planning/ROADMAP.md` Phase 33 success criteria — defines exact behavior for ASST-01/ASST-02/ASST-03/ASST-04
+- `.planning/STATE.md` Accumulated Context decisions — confirms "No DB schema changes", "Memory sanitization blocklist applied at write time"
+
+### Tertiary (LOW confidence)
+- None — all findings verified from source files.
+
+---
+
+## Metadata
+
+**Confidence breakdown:**
+- Standard stack: HIGH — verified by reading existing service files
+- Architecture: HIGH — follows established patterns in codebase (nexusSettingsService, redaction.ts, handoff route)
+- Pitfalls: HIGH — derived from reading the actual stream endpoint code and constraint docs
+
+**Research date:** 2026-04-01
+**Valid until:** 2026-05-01 (stable codebase; these patterns won't change without major refactors)