diff --git a/.planning/phases/24-search-history-branching/24-RESEARCH.md b/.planning/phases/24-search-history-branching/24-RESEARCH.md new file mode 100644 index 00000000..a8e597ea --- /dev/null +++ b/.planning/phases/24-search-history-branching/24-RESEARCH.md @@ -0,0 +1,668 @@ +# Phase 24: Search, History & Branching - Research + +**Researched:** 2026-04-01 +**Domain:** PostgreSQL full-text search, conversation branching data model, bookmark storage, export formatting, React command palette UI +**Confidence:** HIGH + +## Summary + +Phase 24 adds four capabilities to the existing chat system: full-text search across all messages (CHAT-07, PERF-04), bookmarking any message (CHAT-13), conversation branching from any point (CHAT-14), and conversation export (HIST-04). The project uses PostgreSQL 17 with Drizzle ORM, Express 5, React 19, and TanStack Query — all patterns established in Phases 21–23. + +The biggest technical decision is how to implement full-text search performantly. The codebase already uses `ilike` for conversation title search and for issue search, but the requirement is sub-500ms search across 10,000+ messages. PostgreSQL's native `tsvector`/`to_tsvector` with a GIN index is the correct approach for this scale — `ILIKE '%term%'` requires a sequential table scan and will not meet PERF-04 at volume. The codebase has no existing `tsvector` columns, but the migration infrastructure fully supports adding one. + +Conversation branching is architecturally the most complex item. The existing data model is flat (each message belongs to one conversation, each conversation is linear). Branching requires either (a) a new branch relationship that forks a conversation from a message point, creating a new conversation with a reference to the branch-point message, or (b) a tree structure inside a conversation. Option (a) — creating a new child conversation — aligns cleanly with existing patterns: a `parentConversationId` foreign key plus a `branchFromMessageId` on `chatConversations` lets the UI show both branches without changing message storage logic. + +Bookmarks and export are straightforward additions. Bookmarks require a new `chat_message_bookmarks` table and a corresponding service method and route. Export is a server-side endpoint that queries all messages for a conversation and serialises them as Markdown or JSON. + +**Primary recommendation:** Use PostgreSQL `tsvector` + GIN index for message search (add via migration); model branching as child conversations with `parentConversationId` + `branchFromMessageId` foreign keys; store bookmarks in a dedicated join table; export from a dedicated server route that streams a file download. + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions + +None — discuss phase was skipped per workflow.skip_discuss. + +### Claude's Discretion + +All implementation choices are at Claude's discretion. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions. + +### Deferred Ideas (OUT OF SCOPE) + +None — discuss phase skipped. Refer to ROADMAP phase description and success criteria. + + + +## Phase Requirements + +The ROADMAP (authoritative) lists these requirements for Phase 24: + +| ID | Description | Research Support | +|----|-------------|------------------| +| CHAT-07 | Full-text search across all conversations | PostgreSQL tsvector + GIN index; new `/companies/:id/messages/search` route | +| CHAT-13 | Message reactions / bookmarks: mark important messages for later reference | New `chat_message_bookmarks` table; toggle route; bookmark list UI in sidebar | +| CHAT-14 | Conversation branching: editing a mid-conversation message creates a branch; both branches are preserved | `parentConversationId` + `branchFromMessageId` columns on `chatConversations`; branch create route; branch selector UI in ChatPanel | +| HIST-04 | Conversation export: download as Markdown or JSON | Server-side export route returning file download; client-side trigger | +| PERF-04 | Full-text search returns results in under 500ms across 10,000+ messages | GIN index on `tsvector` column satisfies this at 10k+ scale; ILIKE does not | + +Note: The additional_context block referenced HIST-07 through HIST-12, which do not exist in REQUIREMENTS.md. The canonical requirement list is from ROADMAP.md, confirmed above. + + +--- + +## Standard Stack + +### Core +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| drizzle-orm | ^0.38.4 | DB queries, schema definition, migrations | Project ORM — all DB access uses Drizzle | +| drizzle-kit | ^0.31.9 | Migration generation | Project migration tool | +| postgres | ^3.4.5 | PostgreSQL driver | Project driver | +| zod | ^3.24.2 | Request validation schemas | Project validator | +| express | 5.1.0 | Routes and middleware | Project HTTP framework | +| @tanstack/react-query | 5.x | Server state, data fetching | Project state layer | +| cmdk | ^1.1.1 | Command palette primitives | Already installed; `CommandDialog` in `ui/src/components/ui/command.tsx` | +| lucide-react | ^0.574.0 | Icons | Project icon library | + +### Supporting +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| sql (drizzle-orm) | ^0.38.4 | Raw SQL fragments in Drizzle queries | Needed for `to_tsvector`, `plainto_tsquery`, GIN index DDL | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| PostgreSQL tsvector | ILIKE `%term%` | ILIKE is simpler but requires full table scan — fails PERF-04 at 10k+ messages | +| PostgreSQL tsvector | Dedicated search engine (Meilisearch, Typesense) | External service adds operational complexity; Postgres FTS is sufficient at this scale and is already available | +| Child-conversation branching | In-conversation tree (parentMessageId) | Tree structure requires rewriting the entire message list rendering; child conversations reuse all existing conversation display logic | + +**Installation:** No new packages required. All libraries are already in the project. + +--- + +## Architecture Patterns + +### Recommended Project Structure + +New files follow the established pattern from Phase 21–23: + +``` +packages/db/src/schema/ +├── chat_conversations.ts # ADD: parentConversationId, branchFromMessageId columns +├── chat_messages.ts # ADD: tsvector column (generated, persisted) +├── chat_message_bookmarks.ts # NEW: bookmark join table +└── index.ts # EXPORT: chat_message_bookmarks + +packages/db/src/migrations/ +├── 0050_.sql # ADD parentConversationId + branchFromMessageId to chat_conversations +├── 0051_.sql # ADD tsvector column + GIN index to chat_messages +└── 0052_.sql # CREATE chat_message_bookmarks table + +packages/shared/src/types/ +└── chat.ts # ADD: ChatMessageSearchResult, ChatBookmark types + +packages/shared/src/validators/ +└── chat.ts # ADD: searchMessagesSchema + +server/src/services/ +└── chat.ts # ADD: searchMessages, toggleBookmark, getBookmarks, + # branchConversation, exportConversation + +server/src/routes/ +└── chat.ts # ADD: search route, bookmark routes, branch route, export route + +server/src/__tests__/ +├── chat-service.test.ts # ADD: tests for new service methods +└── chat-routes.test.ts # ADD: tests for new routes + +ui/src/api/ +└── chat.ts # ADD: searchMessages, toggleBookmark, getBookmarks, + # branchConversation, exportConversation + +ui/src/hooks/ +└── useChatSearch.ts # NEW: React Query hook for message search +└── useChatBookmarks.ts # NEW: hook for bookmark list + +ui/src/components/ +├── ChatSearchDialog.tsx # NEW: full-text message search overlay +├── ChatMessageBookmark.tsx # NEW: bookmark toggle button on messages +├── ChatBookmarkList.tsx # NEW: filterable bookmark list panel +├── ChatBranchSelector.tsx # NEW: branch picker shown when conversation has branches +└── ChatConversationItem.tsx # MODIFY: show branch indicator when branchFromMessageId set +``` + +### Pattern 1: PostgreSQL Full-Text Search with tsvector + +**What:** Store a pre-computed `tsvector` in `chat_messages` using a generated stored column. Index it with GIN. Query with `plainto_tsquery`. + +**When to use:** Any search across `content` column at scale. + +**How it works in PostgreSQL 17:** + +```sql +-- Migration: add generated tsvector column + GIN index +ALTER TABLE "chat_messages" + ADD COLUMN "content_search" tsvector + GENERATED ALWAYS AS (to_tsvector('english', content)) STORED; + +CREATE INDEX "chat_messages_content_search_idx" + ON "chat_messages" USING GIN ("content_search"); +``` + +**Query pattern in Drizzle (using `sql` tag):** + +```typescript +// Source: drizzle-orm docs — sql template literal for raw expressions +import { sql, and, eq, desc } from "drizzle-orm"; + +async searchMessages(companyId: string, query: string, opts: { limit?: number }) { + const limit = Math.min(opts.limit ?? 20, 50); + const tsQuery = sql`plainto_tsquery('english', ${query})`; + + const rows = await db + .select({ + messageId: chatMessages.id, + conversationId: chatMessages.conversationId, + content: chatMessages.content, + role: chatMessages.role, + createdAt: chatMessages.createdAt, + conversationTitle: chatConversations.title, + rank: sql`ts_rank(${chatMessages.contentSearch}, ${tsQuery})`, + }) + .from(chatMessages) + .innerJoin(chatConversations, eq(chatMessages.conversationId, chatConversations.id)) + .where( + and( + eq(chatConversations.companyId, companyId), + sql`${chatMessages.contentSearch} @@ ${tsQuery}`, + ), + ) + .orderBy(desc(sql`ts_rank(${chatMessages.contentSearch}, ${tsQuery})`)) + .limit(limit); + + return rows; +} +``` + +**Drizzle schema for the generated column:** + +Drizzle ORM does not have a first-class API for `GENERATED ALWAYS AS ... STORED` columns as of 0.38.x. The column must be added via raw SQL migration (not `drizzle-kit generate`) and declared as a `customType` or plain `sql` column in the schema for query use only. The safest approach: + +1. Add the column via a hand-written migration SQL file. +2. Declare it in the Drizzle schema as a non-insertable column using `customType` or simply reference it via `sql` in queries without adding it to the Drizzle schema object (since it's never written by application code). + +The cleanest pattern: declare a virtual query alias: + +```typescript +// In chat_messages.ts schema — do NOT add contentSearch to insert types +// Reference it only in raw sql() expressions when querying +// The column exists in Postgres but Drizzle need not know its type for insertion +``` + +This avoids fighting Drizzle's type system for generated columns. + +### Pattern 2: Conversation Branching via Child Conversations + +**What:** When a user branches from message M in conversation C, create a new conversation C' that: +- Has `parentConversationId = C.id` +- Has `branchFromMessageId = M.id` +- Copies all messages from C up to and including M into C' (or references them) + +**Recommended approach — copy messages up to branch point:** + +Copy is simpler than reference because the existing `listMessages` and streaming logic requires messages to live in the same conversation. A reference approach would require `JOIN` on every message list query. Copying is O(n) at branch time and results in independent conversations — users can edit messages in branches without affecting each other. + +```typescript +async branchConversation(parentConversationId: string, branchFromMessageId: string, companyId: string) { + // 1. Get messages up to and including branch point, ordered chronologically + const branchMsg = await db + .select({ createdAt: chatMessages.createdAt }) + .from(chatMessages) + .where(eq(chatMessages.id, branchFromMessageId)); + if (!branchMsg[0]) throw notFound("Branch message not found"); + + const messagesUpToBranch = await db + .select() + .from(chatMessages) + .where( + and( + eq(chatMessages.conversationId, parentConversationId), + lte(chatMessages.createdAt, branchMsg[0].createdAt), + ), + ) + .orderBy(asc(chatMessages.createdAt)); + + // 2. Create new conversation with branch metadata + const [newConv] = await db + .insert(chatConversations) + .values({ + companyId, + title: null, + parentConversationId, + branchFromMessageId, + }) + .returning(); + + // 3. Copy messages into new conversation + if (messagesUpToBranch.length > 0) { + await db.insert(chatMessages).values( + messagesUpToBranch.map(({ id: _id, conversationId: _cid, ...rest }) => ({ + ...rest, + conversationId: newConv!.id, + })), + ); + } + + return newConv!; +} +``` + +**Schema additions to `chat_conversations`:** + +```typescript +parentConversationId: uuid("parent_conversation_id") + .references(() => chatConversations.id, { onDelete: "set null" }), +branchFromMessageId: uuid("branch_from_message_id"), +// No FK on branchFromMessageId — message may have been deleted +``` + +**Listing branches:** Add `listBranches(conversationId)` service method that queries `WHERE parentConversationId = ?`. + +### Pattern 3: Bookmarks as a Join Table + +**What:** `chat_message_bookmarks` with `(userId, messageId)` or `(conversationId, messageId)`. Since the project uses board-level auth (not per-user), scope bookmarks to `companyId`. + +**Schema:** + +```typescript +export const chatMessageBookmarks = pgTable( + "chat_message_bookmarks", + { + id: uuid("id").primaryKey().defaultRandom(), + companyId: uuid("company_id").notNull().references(() => companies.id), + messageId: uuid("message_id").notNull().references(() => chatMessages.id, { onDelete: "cascade" }), + conversationId: uuid("conversation_id").notNull().references(() => chatConversations.id, { onDelete: "cascade" }), + createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), + }, + (table) => ({ + companyMessageIdx: index("chat_bookmarks_company_message_idx").on(table.companyId, table.messageId), + companyConvIdx: index("chat_bookmarks_company_conv_idx").on(table.companyId, table.conversationId), + }), +); +``` + +**Toggle pattern:** Upsert-or-delete: if bookmark exists for `(companyId, messageId)`, delete it; otherwise insert. Return `{ bookmarked: boolean }`. + +### Pattern 4: Export as Server Route + +**What:** `GET /api/conversations/:id/export?format=markdown|json` returns a file download. + +**Markdown format:** + +``` +# {conversation.title} +Exported: {date} + +--- + +**{agentName}** ({timestamp}) +{message.content} + +--- + +**You** ({timestamp}) +{message.content} +``` + +**JSON format:** Return the full `ChatMessageListResponse`-shaped object with all messages and conversation metadata. + +**Route:** + +```typescript +router.get("/conversations/:id/export", async (req, res) => { + assertBoard(req); + const format = req.query.format === "json" ? "json" : "markdown"; + const { content, filename } = await svc.exportConversation(req.params.id!, format); + res.setHeader("Content-Disposition", `attachment; filename="${filename}"`); + res.setHeader("Content-Type", format === "json" ? "application/json" : "text/markdown"); + res.send(content); +}); +``` + +**Client-side trigger:** Use `window.location.href = url` or create a temporary `` element with `download` attribute pointing to the API URL. + +### Pattern 5: Chat Search Dialog — Cmd+K Routing + +**Problem:** The existing `Cmd+K` handler opens `CommandPalette` (general app search). The ROADMAP success criterion says "Cmd+K opens a search overlay" for chat message search. These two handlers conflict. + +**Resolution options:** + +A. Add a "Search messages" item to the existing `CommandPalette` that opens a separate `ChatSearchDialog`. +B. When chat panel is open, `Cmd+K` opens `ChatSearchDialog` instead of `CommandPalette`. + +**Recommendation: Option A.** The existing `CommandPalette` already intercepts `Cmd+K` globally. Adding a "Search chat messages" command item with a keyboard shortcut hint (e.g., `Cmd+Shift+F`) avoids handler conflicts and aligns with how CommandPalette is used for navigation. The `ChatSearchDialog` is a separate `CommandDialog` that can also be opened from the chat panel header via a search icon button. + +**ChatSearchDialog uses existing `CommandDialog` + `CommandList` primitives from `ui/src/components/ui/command.tsx`.** Results are fetched via `useChatSearch` hook using TanStack Query (debounced, enabled when query length >= 2). + +### Anti-Patterns to Avoid + +- **ILIKE for message content search:** `ILIKE '%term%'` on `chat_messages.content` requires a full table scan. At 10,000+ messages this will not meet 500ms. Always use the GIN-indexed `tsvector` column. +- **In-place message tree for branching:** Adding `parentMessageId` to `chat_messages` requires rewriting all message list queries and the virtualised list rendering. Use child conversations instead. +- **Storing tsvector as a regular column:** The `tsvector` column must be a generated stored column that auto-updates when `content` changes. If you add it as a regular column, you must remember to update it on every `editMessage` call — a maintenance burden. +- **Using `drizzle-kit generate` for generated columns:** Drizzle Kit 0.31.x has incomplete support for PostgreSQL generated columns. Write the migration SQL by hand and keep the Drizzle schema declaration minimal (no `generatedAlwaysAs` — just reference via `sql` in queries). +- **Exporting all messages in memory for large conversations:** The export endpoint queries all messages without pagination. For very large conversations this could be slow. For Phase 24 this is acceptable (no limit specified); note it in code as a future streaming candidate. + +--- + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| Full-text tokenisation | Custom tokeniser, regex word split | PostgreSQL `to_tsvector('english', ...)` | Handles stemming, stop words, multilingual edge cases | +| Search ranking | Custom score function | `ts_rank()` / `ts_rank_cd()` | Built-in ranking weighted by proximity and frequency | +| Command palette UI | Custom modal + keyboard handler | `cmdk` via existing `CommandDialog` | Already installed, accessible, handles keyboard nav | +| Markdown serialization for export | Custom Markdown renderer | Plain string template — messages are already stored in Markdown | No library needed; just format with headings and separators | +| Bookmark toggle atomicity | Application-level check-then-insert | Upsert / INSERT ... ON CONFLICT DO NOTHING + DELETE | Race-condition safe | + +--- + +## Common Pitfalls + +### Pitfall 1: tsvector Not Updated on Message Edit +**What goes wrong:** `chat_messages.content_search` is a `GENERATED ALWAYS AS ... STORED` column in Postgres. Since it is a generated column, Postgres updates it automatically on `UPDATE`. This works correctly. No manual sync needed. +**Why it happens:** Developers familiar with application-managed FTS (triggers, background jobs) assume they must update the search column manually. +**How to avoid:** Use `GENERATED ALWAYS AS ... STORED` — Postgres handles updates transparently. +**Warning signs:** Tests that edit a message and then search for the new content fail to find it. + +### Pitfall 2: Branch Isolation — Messages Must Be Copied, Not Shared +**What goes wrong:** If branch implementation uses a shared message table with a many-to-many join, `listMessages` must always filter by conversation. A shared model requires changing every downstream call. Streaming, edit, retry, and truncate all use `conversationId` as the primary scope. +**Why it happens:** Seems more storage-efficient to reference shared messages. +**How to avoid:** Copy messages on branch creation. Conversations stay independent. Total storage overhead is modest (messages are text, not binary). +**Warning signs:** `truncateMessagesAfter` called in a branched conversation deletes messages that belong to the parent. + +### Pitfall 3: Cmd+K Conflict with Existing CommandPalette +**What goes wrong:** Two `document.addEventListener('keydown', ...)` handlers both match `Cmd+K`. Order is nondeterministic; one or both open. +**Why it happens:** The existing `useKeyboardShortcuts` hook handles `Cmd+K` globally and calls `onSearch()` which opens `CommandPalette`. If `ChatSearchDialog` adds its own `Cmd+K` handler, both fire. +**How to avoid:** Route all `Cmd+K` traffic through the existing `onSearch` hook callback. Add a "Search chat" item inside `CommandPalette`, or pass a separate shortcut (Cmd+Shift+F) to a chat-specific search trigger. Do not register a second `Cmd+K` handler. +**Warning signs:** `CommandPalette` and `ChatSearchDialog` both open simultaneously. + +### Pitfall 4: Drizzle Schema Drift for Generated Columns +**What goes wrong:** `drizzle-kit generate` is run after adding hand-written migrations for the `tsvector` column. Drizzle Kit sees the schema out of sync and generates a migration that drops/re-adds columns. +**Why it happens:** The Drizzle schema (`chat_messages.ts`) does not declare `contentSearch` as a column, so Drizzle Kit has no record of it. +**How to avoid:** Do not add `contentSearch` to the Drizzle schema TypeScript file (or add it read-only with a `customType`). After adding the migration manually, snapshot the migration metadata to prevent Drizzle Kit from trying to reverse it. Alternatively, add the column to the schema as a `sql` custom type marked as not insertable — this aligns the snapshot without breaking type safety. +**Warning signs:** Running `pnpm db:generate` produces a migration that drops `content_search`. + +### Pitfall 5: Export Route Missing Agent Names +**What goes wrong:** The export Markdown includes `agentId` UUIDs instead of human-readable agent names, because `chat_messages` only stores `agentId`, not the agent name. +**Why it happens:** Agent identity is resolved in the UI by joining `agentId` against the agents list. The export service needs to do the same join. +**How to avoid:** The `exportConversation` service method should join `chatMessages` with `agents` to resolve names, or accept an optional map from the caller. +**Warning signs:** Exported Markdown shows UUIDs like `(00000000-0000-0000-0000-000000000001)` as the speaker identity. + +### Pitfall 6: Search Overlay Shows Stale Results After Message Edit +**What goes wrong:** After editing a message, a search for the old content still returns the message; searching for the new content does not find it. +**Why it happens:** TanStack Query caches the search result. The query key includes the search term but not the message `updatedAt`. +**How to avoid:** Invalidate `["chat", "search"]` queries whenever a message is edited: add `queryClient.invalidateQueries({ queryKey: ["chat", "search"] })` to the `handleEdit` callback in `ChatPanel`. + +--- + +## Code Examples + +### Adding tsvector Generated Column (Migration SQL) + +```sql +-- 0051_add_message_search_vector.sql +ALTER TABLE "chat_messages" + ADD COLUMN "content_search" tsvector + GENERATED ALWAYS AS (to_tsvector('english', "content")) STORED; + +CREATE INDEX "chat_messages_content_search_idx" + ON "chat_messages" USING GIN ("content_search"); +``` + +### Drizzle Search Query Pattern + +```typescript +// server/src/services/chat.ts — searchMessages +import { sql, and, eq, isNull, desc } from "drizzle-orm"; + +async searchMessages(companyId: string, query: string, opts: { limit?: number }) { + const limit = Math.min(opts.limit ?? 20, 50); + const tsQuery = query.trim(); + if (!tsQuery) return { items: [] }; + + const rows = await db + .select({ + messageId: chatMessages.id, + conversationId: chatMessages.conversationId, + conversationTitle: chatConversations.title, + content: chatMessages.content, + role: chatMessages.role, + agentId: chatMessages.agentId, + createdAt: chatMessages.createdAt, + rank: sql`ts_rank("chat_messages"."content_search", plainto_tsquery('english', ${tsQuery}))`, + }) + .from(chatMessages) + .innerJoin(chatConversations, and( + eq(chatMessages.conversationId, chatConversations.id), + eq(chatConversations.companyId, companyId), + isNull(chatConversations.deletedAt), + )) + .where( + sql`"chat_messages"."content_search" @@ plainto_tsquery('english', ${tsQuery})`, + ) + .orderBy(desc(sql`ts_rank("chat_messages"."content_search", plainto_tsquery('english', ${tsQuery}))`)) + .limit(limit); + + return { items: rows }; +} +``` + +### useChatSearch Hook Pattern + +```typescript +// ui/src/hooks/useChatSearch.ts +import { useQuery } from "@tanstack/react-query"; +import { chatApi } from "../api/chat"; + +export function useChatSearch(companyId: string | null, query: string) { + return useQuery({ + queryKey: ["chat", "search", companyId, query], + queryFn: () => chatApi.searchMessages(companyId!, query), + enabled: !!companyId && query.trim().length >= 2, + placeholderData: (prev) => prev, + }); +} +``` + +### ChatMessage Bookmark Toggle + +```typescript +// Toggling in ChatMessageActions — add onBookmark prop and Bookmark icon + +``` + +### Branch Conversation Service Skeleton + +```typescript +async branchConversation( + parentConversationId: string, + branchFromMessageId: string, + companyId: string, +) { + const [branchMsg] = await db + .select({ createdAt: chatMessages.createdAt }) + .from(chatMessages) + .where(eq(chatMessages.id, branchFromMessageId)); + if (!branchMsg) throw notFound("Branch message not found"); + + const messagesToCopy = await db + .select() + .from(chatMessages) + .where(and( + eq(chatMessages.conversationId, parentConversationId), + lte(chatMessages.createdAt, branchMsg.createdAt), + )) + .orderBy(asc(chatMessages.createdAt)); + + const [newConv] = await db + .insert(chatConversations) + .values({ + companyId, + title: null, + parentConversationId, + branchFromMessageId, + }) + .returning(); + + if (messagesToCopy.length > 0) { + await db.insert(chatMessages).values( + messagesToCopy.map(({ id: _id, conversationId: _cid, ...rest }) => ({ + ...rest, + conversationId: newConv!.id, + })), + ); + } + + return newConv!; +} +``` + +--- + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| ILIKE for content search | tsvector + GIN (PostgreSQL FTS) | PostgreSQL 8.3 (2008) | Order-of-magnitude faster at scale | +| Manual tsvector maintenance via triggers | GENERATED ALWAYS AS ... STORED | PostgreSQL 12 (2019) | No triggers, auto-updated on UPDATE | +| cmdk v0.x (uncontrolled) | cmdk v1.x (controlled, `shouldFilter` prop) | cmdk 1.0 (2024) | Must set `shouldFilter={false}` when using server-side search to prevent cmdk's own client-side filter from re-filtering server results | + +**Deprecated / outdated:** + +- `plainto_tsquery` vs `websearch_to_tsquery`: `websearch_to_tsquery` (Postgres 11+) handles quoted phrases and `-exclusions` like a web search engine. For a basic first implementation, `plainto_tsquery` is simpler and correct. Upgrade to `websearch_to_tsquery` later if users need phrase search. + +--- + +## Open Questions + +1. **Branch UI placement** + - What we know: `ChatConversationItem` shows conversation in the left column of ChatPanel. Branches would be child conversations visible in the same list with a visual indent or branch icon. + - What's unclear: Whether branches should be nested under the parent in the conversation list or shown as peers with a parent link. + - Recommendation: Show branches as indented items under the parent in `ChatConversationList`. Add a `parentConversationId` to `ChatConversationListItem` type and group by parent in the UI. Keep the server list endpoint flat (client-side grouping). + +2. **Bookmark scope: company vs. per-user** + - What we know: The project uses board-level (company-scoped) auth. There is no per-user identity surfaced in the chat service — `assertBoard` validates company access but does not expose `userId` to service methods in the current chat service signature. + - What's unclear: Whether bookmarks should be shared across all users in a workspace or per-user. + - Recommendation: Scope bookmarks to `companyId` for simplicity (shared bookmarks across the workspace). This matches how conversations are scoped. Per-user bookmarks can be added later when the user model is more prominent. + +3. **Search result navigation** + - What we know: Search results include `conversationId` and `messageId`. Clicking a result should navigate to that conversation and scroll to the message. + - What's unclear: The existing `ChatPanel` has no mechanism to scroll to a specific message by ID. + - Recommendation: Add a `scrollToMessageId` state to `ChatPanelContext`. When set, `ChatMessageList` uses the virtualiser's `scrollToIndex` method to jump to the message. Reset after scrolling. This follows the `nexus:focus-chat-search` custom event pattern already used for Cmd+K focus. + +--- + +## Environment Availability + +Step 2.6: No new external dependencies identified. PostgreSQL 17 is already the project's database. All npm packages are already installed. + +--- + +## Validation Architecture + +### Test Framework +| Property | Value | +|----------|-------| +| Framework | Vitest 3.2.4 | +| Config file | `server/vitest.config.ts` (node env), `ui/vitest.config.ts` (node env) | +| Quick run command | `pnpm --filter @paperclipai/server test run -- chat-service` | +| Full suite command | `pnpm test:run` | + +### Phase Requirements → Test Map + +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| CHAT-07 | `searchMessages` returns ranked results for matching term | unit | `pnpm --filter @paperclipai/server test run -- chat-service` | ✅ (new describe block in existing file) | +| CHAT-07 | `GET /companies/:id/messages/search?q=term` returns 200 with results | integration | `pnpm --filter @paperclipai/server test run -- chat-routes` | ✅ (new describe block in existing file) | +| CHAT-13 | `toggleBookmark` inserts or removes bookmark row | unit | `pnpm --filter @paperclipai/server test run -- chat-service` | ✅ | +| CHAT-13 | `GET /companies/:id/bookmarks` returns bookmarked messages | integration | `pnpm --filter @paperclipai/server test run -- chat-routes` | ✅ | +| CHAT-14 | `branchConversation` creates new conversation with copied messages | unit | `pnpm --filter @paperclipai/server test run -- chat-service` | ✅ | +| CHAT-14 | `POST /conversations/:id/branch` returns 201 with new conversation | integration | `pnpm --filter @paperclipai/server test run -- chat-routes` | ✅ | +| HIST-04 | `exportConversation` returns correct Markdown structure | unit | `pnpm --filter @paperclipai/server test run -- chat-service` | ✅ | +| HIST-04 | `GET /conversations/:id/export?format=markdown` returns file download headers | integration | `pnpm --filter @paperclipai/server test run -- chat-routes` | ✅ | +| PERF-04 | GIN index query plan uses index scan (not seq scan) | manual/DB | `EXPLAIN ANALYZE ...` in Postgres | ❌ Wave 0 — manual verification | + +### Sampling Rate +- **Per task commit:** `pnpm --filter @paperclipai/server test run -- chat-service` +- **Per wave merge:** `pnpm test:run` +- **Phase gate:** Full suite green before `/gsd:verify-work` + +### Wave 0 Gaps +- [ ] New `describe("searchMessages")` block in `server/src/__tests__/chat-service.test.ts` — covers CHAT-07 +- [ ] New `describe("toggleBookmark / getBookmarks")` block in `server/src/__tests__/chat-service.test.ts` — covers CHAT-13 +- [ ] New `describe("branchConversation")` block in `server/src/__tests__/chat-service.test.ts` — covers CHAT-14 +- [ ] New `describe("exportConversation")` block in `server/src/__tests__/chat-service.test.ts` — covers HIST-04 +- [ ] New route-level describe blocks in `server/src/__tests__/chat-routes.test.ts` — covers all four route groups + +--- + +## Project Constraints (from CLAUDE.md) + +CLAUDE.md does not exist at `/opt/nexus/CLAUDE.md`. No additional project-level directives to document. + +**Codebase conventions observed from prior phases:** + +- Use `object-syntax (table) => ({})` for Drizzle index callbacks (not arrow-returning-object shorthand). +- Use `it.todo()` (not `it.skip()`) for Wave 0 test scaffolding. +- Use `@/lib/router` Link abstraction for navigation, not `react-router-dom` directly. +- Use `useToast()/pushToast()` for error toasts — not `sonner`. +- DB schema files are individual per table; exported from `packages/db/src/schema/index.ts`. +- Migrations are hand-numbered (`0050_`, `0051_`, ...) and journaled in `meta/_journal.json`. +- Shared types in `packages/shared/src/types/chat.ts`; validators in `packages/shared/src/validators/chat.ts`; both re-exported from `packages/shared/src/index.ts`. +- Custom window events (e.g. `nexus:focus-chat-search`) are the project pattern for decoupled cross-component communication. +- The `assertBoard(req)` + `assertCompanyAccess(req, companyId)` guard pattern is required on all chat routes. +- Service functions are factory functions `chatService(db)` returning a plain object — not classes. + +--- + +## Sources + +### Primary (HIGH confidence) +- Direct codebase inspection — all schema, service, route, and UI files read from `/opt/nexus/` +- `packages/db/src/schema/chat_conversations.ts` — confirmed existing columns +- `packages/db/src/schema/chat_messages.ts` — confirmed no existing tsvector +- `server/src/services/chat.ts` — confirmed ilike-only current search +- `server/src/routes/chat.ts` — confirmed route patterns +- `ui/src/components/CommandPalette.tsx` — confirmed cmdk usage and Cmd+K binding +- `ui/src/hooks/useKeyboardShortcuts.ts` — confirmed Cmd+K conflict point +- `.planning/ROADMAP.md` Phase 24 — canonical requirement list (CHAT-07, CHAT-13, CHAT-14, HIST-04, PERF-04) +- `.planning/codebase/STACK.md` — confirmed PostgreSQL 17, Drizzle 0.38.x, cmdk 1.1.1 + +### Secondary (MEDIUM confidence) +- PostgreSQL 12 documentation — GENERATED ALWAYS AS STORED columns +- PostgreSQL FTS documentation — tsvector, GIN indexes, plainto_tsquery, ts_rank + +### Tertiary (LOW confidence, flag for validation) +- Drizzle ORM 0.38.x generated column support — my training data indicates incomplete support; verified indirectly by absence of `generatedAlwaysAs` usage in the codebase + +--- + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH — all libraries confirmed present in codebase +- Architecture: HIGH — patterns derived directly from existing Phases 21–23 code +- Search implementation: HIGH — PostgreSQL FTS is well-established; tsvector generated column confirmed supported in PG17 +- Drizzle generated column handling: MEDIUM — Drizzle Kit limitation confirmed by absence of existing usage; hand-written migration approach is the safe path +- Pitfalls: HIGH — derived from direct code inspection of conflict points + +**Research date:** 2026-04-01 +**Valid until:** 2026-05-01 (stable stack; no fast-moving dependencies)