docs(24): research phase domain
This commit is contained in:
Nexus Dev
parent
a2ea035aa1
commit
502ad9c63e
1 changed files with 668 additions and 0 deletions
668
.planning/phases/24-search-history-branching/24-RESEARCH.md
Normal file
668
.planning/phases/24-search-history-branching/24-RESEARCH.md
Normal file
|
|
@ -0,0 +1,668 @@
|
|||
# Phase 24: Search, History & Branching - Research
|
||||
|
||||
**Researched:** 2026-04-01
|
||||
**Domain:** PostgreSQL full-text search, conversation branching data model, bookmark storage, export formatting, React command palette UI
|
||||
**Confidence:** HIGH
|
||||
|
||||
## Summary
|
||||
|
||||
Phase 24 adds four capabilities to the existing chat system: full-text search across all messages (CHAT-07, PERF-04), bookmarking any message (CHAT-13), conversation branching from any point (CHAT-14), and conversation export (HIST-04). The project uses PostgreSQL 17 with Drizzle ORM, Express 5, React 19, and TanStack Query — all patterns established in Phases 21–23.
|
||||
|
||||
The biggest technical decision is how to implement full-text search performantly. The codebase already uses `ilike` for conversation title search and for issue search, but the requirement is sub-500ms search across 10,000+ messages. PostgreSQL's native `tsvector`/`to_tsvector` with a GIN index is the correct approach for this scale — `ILIKE '%term%'` requires a sequential table scan and will not meet PERF-04 at volume. The codebase has no existing `tsvector` columns, but the migration infrastructure fully supports adding one.
|
||||
|
||||
Conversation branching is architecturally the most complex item. The existing data model is flat (each message belongs to one conversation, each conversation is linear). Branching requires either (a) a new branch relationship that forks a conversation from a message point, creating a new conversation with a reference to the branch-point message, or (b) a tree structure inside a conversation. Option (a) — creating a new child conversation — aligns cleanly with existing patterns: a `parentConversationId` foreign key plus a `branchFromMessageId` on `chatConversations` lets the UI show both branches without changing message storage logic.
|
||||
|
||||
Bookmarks and export are straightforward additions. Bookmarks require a new `chat_message_bookmarks` table and a corresponding service method and route. Export is a server-side endpoint that queries all messages for a conversation and serialises them as Markdown or JSON.
|
||||
|
||||
**Primary recommendation:** Use PostgreSQL `tsvector` + GIN index for message search (add via migration); model branching as child conversations with `parentConversationId` + `branchFromMessageId` foreign keys; store bookmarks in a dedicated join table; export from a dedicated server route that streams a file download.
|
||||
|
||||
<user_constraints>
|
||||
## User Constraints (from CONTEXT.md)
|
||||
|
||||
### Locked Decisions
|
||||
|
||||
None — discuss phase was skipped per workflow.skip_discuss.
|
||||
|
||||
### Claude's Discretion
|
||||
|
||||
All implementation choices are at Claude's discretion. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions.
|
||||
|
||||
### Deferred Ideas (OUT OF SCOPE)
|
||||
|
||||
None — discuss phase skipped. Refer to ROADMAP phase description and success criteria.
|
||||
</user_constraints>
|
||||
|
||||
<phase_requirements>
|
||||
## Phase Requirements
|
||||
|
||||
The ROADMAP (authoritative) lists these requirements for Phase 24:
|
||||
|
||||
| ID | Description | Research Support |
|
||||
|----|-------------|------------------|
|
||||
| CHAT-07 | Full-text search across all conversations | PostgreSQL tsvector + GIN index; new `/companies/:id/messages/search` route |
|
||||
| CHAT-13 | Message reactions / bookmarks: mark important messages for later reference | New `chat_message_bookmarks` table; toggle route; bookmark list UI in sidebar |
|
||||
| CHAT-14 | Conversation branching: editing a mid-conversation message creates a branch; both branches are preserved | `parentConversationId` + `branchFromMessageId` columns on `chatConversations`; branch create route; branch selector UI in ChatPanel |
|
||||
| HIST-04 | Conversation export: download as Markdown or JSON | Server-side export route returning file download; client-side trigger |
|
||||
| PERF-04 | Full-text search returns results in under 500ms across 10,000+ messages | GIN index on `tsvector` column satisfies this at 10k+ scale; ILIKE does not |
|
||||
|
||||
Note: The additional_context block referenced HIST-07 through HIST-12, which do not exist in REQUIREMENTS.md. The canonical requirement list is from ROADMAP.md, confirmed above.
|
||||
</phase_requirements>
|
||||
|
||||
---
|
||||
|
||||
## Standard Stack
|
||||
|
||||
### Core
|
||||
| Library | Version | Purpose | Why Standard |
|
||||
|---------|---------|---------|--------------|
|
||||
| drizzle-orm | ^0.38.4 | DB queries, schema definition, migrations | Project ORM — all DB access uses Drizzle |
|
||||
| drizzle-kit | ^0.31.9 | Migration generation | Project migration tool |
|
||||
| postgres | ^3.4.5 | PostgreSQL driver | Project driver |
|
||||
| zod | ^3.24.2 | Request validation schemas | Project validator |
|
||||
| express | 5.1.0 | Routes and middleware | Project HTTP framework |
|
||||
| @tanstack/react-query | 5.x | Server state, data fetching | Project state layer |
|
||||
| cmdk | ^1.1.1 | Command palette primitives | Already installed; `CommandDialog` in `ui/src/components/ui/command.tsx` |
|
||||
| lucide-react | ^0.574.0 | Icons | Project icon library |
|
||||
|
||||
### Supporting
|
||||
| Library | Version | Purpose | When to Use |
|
||||
|---------|---------|---------|-------------|
|
||||
| sql (drizzle-orm) | ^0.38.4 | Raw SQL fragments in Drizzle queries | Needed for `to_tsvector`, `plainto_tsquery`, GIN index DDL |
|
||||
|
||||
### Alternatives Considered
|
||||
| Instead of | Could Use | Tradeoff |
|
||||
|------------|-----------|----------|
|
||||
| PostgreSQL tsvector | ILIKE `%term%` | ILIKE is simpler but requires full table scan — fails PERF-04 at 10k+ messages |
|
||||
| PostgreSQL tsvector | Dedicated search engine (Meilisearch, Typesense) | External service adds operational complexity; Postgres FTS is sufficient at this scale and is already available |
|
||||
| Child-conversation branching | In-conversation tree (parentMessageId) | Tree structure requires rewriting the entire message list rendering; child conversations reuse all existing conversation display logic |
|
||||
|
||||
**Installation:** No new packages required. All libraries are already in the project.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Patterns
|
||||
|
||||
### Recommended Project Structure
|
||||
|
||||
New files follow the established pattern from Phase 21–23:
|
||||
|
||||
```
|
||||
packages/db/src/schema/
|
||||
├── chat_conversations.ts # ADD: parentConversationId, branchFromMessageId columns
|
||||
├── chat_messages.ts # ADD: tsvector column (generated, persisted)
|
||||
├── chat_message_bookmarks.ts # NEW: bookmark join table
|
||||
└── index.ts # EXPORT: chat_message_bookmarks
|
||||
|
||||
packages/db/src/migrations/
|
||||
├── 0050_<slug>.sql # ADD parentConversationId + branchFromMessageId to chat_conversations
|
||||
├── 0051_<slug>.sql # ADD tsvector column + GIN index to chat_messages
|
||||
└── 0052_<slug>.sql # CREATE chat_message_bookmarks table
|
||||
|
||||
packages/shared/src/types/
|
||||
└── chat.ts # ADD: ChatMessageSearchResult, ChatBookmark types
|
||||
|
||||
packages/shared/src/validators/
|
||||
└── chat.ts # ADD: searchMessagesSchema
|
||||
|
||||
server/src/services/
|
||||
└── chat.ts # ADD: searchMessages, toggleBookmark, getBookmarks,
|
||||
# branchConversation, exportConversation
|
||||
|
||||
server/src/routes/
|
||||
└── chat.ts # ADD: search route, bookmark routes, branch route, export route
|
||||
|
||||
server/src/__tests__/
|
||||
├── chat-service.test.ts # ADD: tests for new service methods
|
||||
└── chat-routes.test.ts # ADD: tests for new routes
|
||||
|
||||
ui/src/api/
|
||||
└── chat.ts # ADD: searchMessages, toggleBookmark, getBookmarks,
|
||||
# branchConversation, exportConversation
|
||||
|
||||
ui/src/hooks/
|
||||
└── useChatSearch.ts # NEW: React Query hook for message search
|
||||
└── useChatBookmarks.ts # NEW: hook for bookmark list
|
||||
|
||||
ui/src/components/
|
||||
├── ChatSearchDialog.tsx # NEW: full-text message search overlay
|
||||
├── ChatMessageBookmark.tsx # NEW: bookmark toggle button on messages
|
||||
├── ChatBookmarkList.tsx # NEW: filterable bookmark list panel
|
||||
├── ChatBranchSelector.tsx # NEW: branch picker shown when conversation has branches
|
||||
└── ChatConversationItem.tsx # MODIFY: show branch indicator when branchFromMessageId set
|
||||
```
|
||||
|
||||
### Pattern 1: PostgreSQL Full-Text Search with tsvector
|
||||
|
||||
**What:** Store a pre-computed `tsvector` in `chat_messages` using a generated stored column. Index it with GIN. Query with `plainto_tsquery`.
|
||||
|
||||
**When to use:** Any search across `content` column at scale.
|
||||
|
||||
**How it works in PostgreSQL 17:**
|
||||
|
||||
```sql
|
||||
-- Migration: add generated tsvector column + GIN index
|
||||
ALTER TABLE "chat_messages"
|
||||
ADD COLUMN "content_search" tsvector
|
||||
GENERATED ALWAYS AS (to_tsvector('english', content)) STORED;
|
||||
|
||||
CREATE INDEX "chat_messages_content_search_idx"
|
||||
ON "chat_messages" USING GIN ("content_search");
|
||||
```
|
||||
|
||||
**Query pattern in Drizzle (using `sql` tag):**
|
||||
|
||||
```typescript
|
||||
// Source: drizzle-orm docs — sql template literal for raw expressions
|
||||
import { sql, and, eq, desc } from "drizzle-orm";
|
||||
|
||||
async searchMessages(companyId: string, query: string, opts: { limit?: number }) {
|
||||
const limit = Math.min(opts.limit ?? 20, 50);
|
||||
const tsQuery = sql`plainto_tsquery('english', ${query})`;
|
||||
|
||||
const rows = await db
|
||||
.select({
|
||||
messageId: chatMessages.id,
|
||||
conversationId: chatMessages.conversationId,
|
||||
content: chatMessages.content,
|
||||
role: chatMessages.role,
|
||||
createdAt: chatMessages.createdAt,
|
||||
conversationTitle: chatConversations.title,
|
||||
rank: sql<number>`ts_rank(${chatMessages.contentSearch}, ${tsQuery})`,
|
||||
})
|
||||
.from(chatMessages)
|
||||
.innerJoin(chatConversations, eq(chatMessages.conversationId, chatConversations.id))
|
||||
.where(
|
||||
and(
|
||||
eq(chatConversations.companyId, companyId),
|
||||
sql`${chatMessages.contentSearch} @@ ${tsQuery}`,
|
||||
),
|
||||
)
|
||||
.orderBy(desc(sql`ts_rank(${chatMessages.contentSearch}, ${tsQuery})`))
|
||||
.limit(limit);
|
||||
|
||||
return rows;
|
||||
}
|
||||
```
|
||||
|
||||
**Drizzle schema for the generated column:**
|
||||
|
||||
Drizzle ORM does not have a first-class API for `GENERATED ALWAYS AS ... STORED` columns as of 0.38.x. The column must be added via raw SQL migration (not `drizzle-kit generate`) and declared as a `customType` or plain `sql` column in the schema for query use only. The safest approach:
|
||||
|
||||
1. Add the column via a hand-written migration SQL file.
|
||||
2. Declare it in the Drizzle schema as a non-insertable column using `customType` or simply reference it via `sql` in queries without adding it to the Drizzle schema object (since it's never written by application code).
|
||||
|
||||
The cleanest pattern: declare a virtual query alias:
|
||||
|
||||
```typescript
|
||||
// In chat_messages.ts schema — do NOT add contentSearch to insert types
|
||||
// Reference it only in raw sql() expressions when querying
|
||||
// The column exists in Postgres but Drizzle need not know its type for insertion
|
||||
```
|
||||
|
||||
This avoids fighting Drizzle's type system for generated columns.
|
||||
|
||||
### Pattern 2: Conversation Branching via Child Conversations
|
||||
|
||||
**What:** When a user branches from message M in conversation C, create a new conversation C' that:
|
||||
- Has `parentConversationId = C.id`
|
||||
- Has `branchFromMessageId = M.id`
|
||||
- Copies all messages from C up to and including M into C' (or references them)
|
||||
|
||||
**Recommended approach — copy messages up to branch point:**
|
||||
|
||||
Copy is simpler than reference because the existing `listMessages` and streaming logic requires messages to live in the same conversation. A reference approach would require `JOIN` on every message list query. Copying is O(n) at branch time and results in independent conversations — users can edit messages in branches without affecting each other.
|
||||
|
||||
```typescript
|
||||
async branchConversation(parentConversationId: string, branchFromMessageId: string, companyId: string) {
|
||||
// 1. Get messages up to and including branch point, ordered chronologically
|
||||
const branchMsg = await db
|
||||
.select({ createdAt: chatMessages.createdAt })
|
||||
.from(chatMessages)
|
||||
.where(eq(chatMessages.id, branchFromMessageId));
|
||||
if (!branchMsg[0]) throw notFound("Branch message not found");
|
||||
|
||||
const messagesUpToBranch = await db
|
||||
.select()
|
||||
.from(chatMessages)
|
||||
.where(
|
||||
and(
|
||||
eq(chatMessages.conversationId, parentConversationId),
|
||||
lte(chatMessages.createdAt, branchMsg[0].createdAt),
|
||||
),
|
||||
)
|
||||
.orderBy(asc(chatMessages.createdAt));
|
||||
|
||||
// 2. Create new conversation with branch metadata
|
||||
const [newConv] = await db
|
||||
.insert(chatConversations)
|
||||
.values({
|
||||
companyId,
|
||||
title: null,
|
||||
parentConversationId,
|
||||
branchFromMessageId,
|
||||
})
|
||||
.returning();
|
||||
|
||||
// 3. Copy messages into new conversation
|
||||
if (messagesUpToBranch.length > 0) {
|
||||
await db.insert(chatMessages).values(
|
||||
messagesUpToBranch.map(({ id: _id, conversationId: _cid, ...rest }) => ({
|
||||
...rest,
|
||||
conversationId: newConv!.id,
|
||||
})),
|
||||
);
|
||||
}
|
||||
|
||||
return newConv!;
|
||||
}
|
||||
```
|
||||
|
||||
**Schema additions to `chat_conversations`:**
|
||||
|
||||
```typescript
|
||||
parentConversationId: uuid("parent_conversation_id")
|
||||
.references(() => chatConversations.id, { onDelete: "set null" }),
|
||||
branchFromMessageId: uuid("branch_from_message_id"),
|
||||
// No FK on branchFromMessageId — message may have been deleted
|
||||
```
|
||||
|
||||
**Listing branches:** Add `listBranches(conversationId)` service method that queries `WHERE parentConversationId = ?`.
|
||||
|
||||
### Pattern 3: Bookmarks as a Join Table
|
||||
|
||||
**What:** `chat_message_bookmarks` with `(userId, messageId)` or `(conversationId, messageId)`. Since the project uses board-level auth (not per-user), scope bookmarks to `companyId`.
|
||||
|
||||
**Schema:**
|
||||
|
||||
```typescript
|
||||
export const chatMessageBookmarks = pgTable(
|
||||
"chat_message_bookmarks",
|
||||
{
|
||||
id: uuid("id").primaryKey().defaultRandom(),
|
||||
companyId: uuid("company_id").notNull().references(() => companies.id),
|
||||
messageId: uuid("message_id").notNull().references(() => chatMessages.id, { onDelete: "cascade" }),
|
||||
conversationId: uuid("conversation_id").notNull().references(() => chatConversations.id, { onDelete: "cascade" }),
|
||||
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
},
|
||||
(table) => ({
|
||||
companyMessageIdx: index("chat_bookmarks_company_message_idx").on(table.companyId, table.messageId),
|
||||
companyConvIdx: index("chat_bookmarks_company_conv_idx").on(table.companyId, table.conversationId),
|
||||
}),
|
||||
);
|
||||
```
|
||||
|
||||
**Toggle pattern:** Upsert-or-delete: if bookmark exists for `(companyId, messageId)`, delete it; otherwise insert. Return `{ bookmarked: boolean }`.
|
||||
|
||||
### Pattern 4: Export as Server Route
|
||||
|
||||
**What:** `GET /api/conversations/:id/export?format=markdown|json` returns a file download.
|
||||
|
||||
**Markdown format:**
|
||||
|
||||
```
|
||||
# {conversation.title}
|
||||
Exported: {date}
|
||||
|
||||
---
|
||||
|
||||
**{agentName}** ({timestamp})
|
||||
{message.content}
|
||||
|
||||
---
|
||||
|
||||
**You** ({timestamp})
|
||||
{message.content}
|
||||
```
|
||||
|
||||
**JSON format:** Return the full `ChatMessageListResponse`-shaped object with all messages and conversation metadata.
|
||||
|
||||
**Route:**
|
||||
|
||||
```typescript
|
||||
router.get("/conversations/:id/export", async (req, res) => {
|
||||
assertBoard(req);
|
||||
const format = req.query.format === "json" ? "json" : "markdown";
|
||||
const { content, filename } = await svc.exportConversation(req.params.id!, format);
|
||||
res.setHeader("Content-Disposition", `attachment; filename="${filename}"`);
|
||||
res.setHeader("Content-Type", format === "json" ? "application/json" : "text/markdown");
|
||||
res.send(content);
|
||||
});
|
||||
```
|
||||
|
||||
**Client-side trigger:** Use `window.location.href = url` or create a temporary `<a>` element with `download` attribute pointing to the API URL.
|
||||
|
||||
### Pattern 5: Chat Search Dialog — Cmd+K Routing
|
||||
|
||||
**Problem:** The existing `Cmd+K` handler opens `CommandPalette` (general app search). The ROADMAP success criterion says "Cmd+K opens a search overlay" for chat message search. These two handlers conflict.
|
||||
|
||||
**Resolution options:**
|
||||
|
||||
A. Add a "Search messages" item to the existing `CommandPalette` that opens a separate `ChatSearchDialog`.
|
||||
B. When chat panel is open, `Cmd+K` opens `ChatSearchDialog` instead of `CommandPalette`.
|
||||
|
||||
**Recommendation: Option A.** The existing `CommandPalette` already intercepts `Cmd+K` globally. Adding a "Search chat messages" command item with a keyboard shortcut hint (e.g., `Cmd+Shift+F`) avoids handler conflicts and aligns with how CommandPalette is used for navigation. The `ChatSearchDialog` is a separate `CommandDialog` that can also be opened from the chat panel header via a search icon button.
|
||||
|
||||
**ChatSearchDialog uses existing `CommandDialog` + `CommandList` primitives from `ui/src/components/ui/command.tsx`.** Results are fetched via `useChatSearch` hook using TanStack Query (debounced, enabled when query length >= 2).
|
||||
|
||||
### Anti-Patterns to Avoid
|
||||
|
||||
- **ILIKE for message content search:** `ILIKE '%term%'` on `chat_messages.content` requires a full table scan. At 10,000+ messages this will not meet 500ms. Always use the GIN-indexed `tsvector` column.
|
||||
- **In-place message tree for branching:** Adding `parentMessageId` to `chat_messages` requires rewriting all message list queries and the virtualised list rendering. Use child conversations instead.
|
||||
- **Storing tsvector as a regular column:** The `tsvector` column must be a generated stored column that auto-updates when `content` changes. If you add it as a regular column, you must remember to update it on every `editMessage` call — a maintenance burden.
|
||||
- **Using `drizzle-kit generate` for generated columns:** Drizzle Kit 0.31.x has incomplete support for PostgreSQL generated columns. Write the migration SQL by hand and keep the Drizzle schema declaration minimal (no `generatedAlwaysAs` — just reference via `sql` in queries).
|
||||
- **Exporting all messages in memory for large conversations:** The export endpoint queries all messages without pagination. For very large conversations this could be slow. For Phase 24 this is acceptable (no limit specified); note it in code as a future streaming candidate.
|
||||
|
||||
---
|
||||
|
||||
## Don't Hand-Roll
|
||||
|
||||
| Problem | Don't Build | Use Instead | Why |
|
||||
|---------|-------------|-------------|-----|
|
||||
| Full-text tokenisation | Custom tokeniser, regex word split | PostgreSQL `to_tsvector('english', ...)` | Handles stemming, stop words, multilingual edge cases |
|
||||
| Search ranking | Custom score function | `ts_rank()` / `ts_rank_cd()` | Built-in ranking weighted by proximity and frequency |
|
||||
| Command palette UI | Custom modal + keyboard handler | `cmdk` via existing `CommandDialog` | Already installed, accessible, handles keyboard nav |
|
||||
| Markdown serialization for export | Custom Markdown renderer | Plain string template — messages are already stored in Markdown | No library needed; just format with headings and separators |
|
||||
| Bookmark toggle atomicity | Application-level check-then-insert | Upsert / INSERT ... ON CONFLICT DO NOTHING + DELETE | Race-condition safe |
|
||||
|
||||
---
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Pitfall 1: tsvector Not Updated on Message Edit
|
||||
**What goes wrong:** `chat_messages.content_search` is a `GENERATED ALWAYS AS ... STORED` column in Postgres. Since it is a generated column, Postgres updates it automatically on `UPDATE`. This works correctly. No manual sync needed.
|
||||
**Why it happens:** Developers familiar with application-managed FTS (triggers, background jobs) assume they must update the search column manually.
|
||||
**How to avoid:** Use `GENERATED ALWAYS AS ... STORED` — Postgres handles updates transparently.
|
||||
**Warning signs:** Tests that edit a message and then search for the new content fail to find it.
|
||||
|
||||
### Pitfall 2: Branch Isolation — Messages Must Be Copied, Not Shared
|
||||
**What goes wrong:** If branch implementation uses a shared message table with a many-to-many join, `listMessages` must always filter by conversation. A shared model requires changing every downstream call. Streaming, edit, retry, and truncate all use `conversationId` as the primary scope.
|
||||
**Why it happens:** Seems more storage-efficient to reference shared messages.
|
||||
**How to avoid:** Copy messages on branch creation. Conversations stay independent. Total storage overhead is modest (messages are text, not binary).
|
||||
**Warning signs:** `truncateMessagesAfter` called in a branched conversation deletes messages that belong to the parent.
|
||||
|
||||
### Pitfall 3: Cmd+K Conflict with Existing CommandPalette
|
||||
**What goes wrong:** Two `document.addEventListener('keydown', ...)` handlers both match `Cmd+K`. Order is nondeterministic; one or both open.
|
||||
**Why it happens:** The existing `useKeyboardShortcuts` hook handles `Cmd+K` globally and calls `onSearch()` which opens `CommandPalette`. If `ChatSearchDialog` adds its own `Cmd+K` handler, both fire.
|
||||
**How to avoid:** Route all `Cmd+K` traffic through the existing `onSearch` hook callback. Add a "Search chat" item inside `CommandPalette`, or pass a separate shortcut (Cmd+Shift+F) to a chat-specific search trigger. Do not register a second `Cmd+K` handler.
|
||||
**Warning signs:** `CommandPalette` and `ChatSearchDialog` both open simultaneously.
|
||||
|
||||
### Pitfall 4: Drizzle Schema Drift for Generated Columns
|
||||
**What goes wrong:** `drizzle-kit generate` is run after adding hand-written migrations for the `tsvector` column. Drizzle Kit sees the schema out of sync and generates a migration that drops/re-adds columns.
|
||||
**Why it happens:** The Drizzle schema (`chat_messages.ts`) does not declare `contentSearch` as a column, so Drizzle Kit has no record of it.
|
||||
**How to avoid:** Do not add `contentSearch` to the Drizzle schema TypeScript file (or add it read-only with a `customType`). After adding the migration manually, snapshot the migration metadata to prevent Drizzle Kit from trying to reverse it. Alternatively, add the column to the schema as a `sql` custom type marked as not insertable — this aligns the snapshot without breaking type safety.
|
||||
**Warning signs:** Running `pnpm db:generate` produces a migration that drops `content_search`.
|
||||
|
||||
### Pitfall 5: Export Route Missing Agent Names
|
||||
**What goes wrong:** The export Markdown includes `agentId` UUIDs instead of human-readable agent names, because `chat_messages` only stores `agentId`, not the agent name.
|
||||
**Why it happens:** Agent identity is resolved in the UI by joining `agentId` against the agents list. The export service needs to do the same join.
|
||||
**How to avoid:** The `exportConversation` service method should join `chatMessages` with `agents` to resolve names, or accept an optional map from the caller.
|
||||
**Warning signs:** Exported Markdown shows UUIDs like `(00000000-0000-0000-0000-000000000001)` as the speaker identity.
|
||||
|
||||
### Pitfall 6: Search Overlay Shows Stale Results After Message Edit
|
||||
**What goes wrong:** After editing a message, a search for the old content still returns the message; searching for the new content does not find it.
|
||||
**Why it happens:** TanStack Query caches the search result. The query key includes the search term but not the message `updatedAt`.
|
||||
**How to avoid:** Invalidate `["chat", "search"]` queries whenever a message is edited: add `queryClient.invalidateQueries({ queryKey: ["chat", "search"] })` to the `handleEdit` callback in `ChatPanel`.
|
||||
|
||||
---
|
||||
|
||||
## Code Examples
|
||||
|
||||
### Adding tsvector Generated Column (Migration SQL)
|
||||
|
||||
```sql
|
||||
-- 0051_add_message_search_vector.sql
|
||||
ALTER TABLE "chat_messages"
|
||||
ADD COLUMN "content_search" tsvector
|
||||
GENERATED ALWAYS AS (to_tsvector('english', "content")) STORED;
|
||||
|
||||
CREATE INDEX "chat_messages_content_search_idx"
|
||||
ON "chat_messages" USING GIN ("content_search");
|
||||
```
|
||||
|
||||
### Drizzle Search Query Pattern
|
||||
|
||||
```typescript
|
||||
// server/src/services/chat.ts — searchMessages
|
||||
import { sql, and, eq, isNull, desc } from "drizzle-orm";
|
||||
|
||||
async searchMessages(companyId: string, query: string, opts: { limit?: number }) {
|
||||
const limit = Math.min(opts.limit ?? 20, 50);
|
||||
const tsQuery = query.trim();
|
||||
if (!tsQuery) return { items: [] };
|
||||
|
||||
const rows = await db
|
||||
.select({
|
||||
messageId: chatMessages.id,
|
||||
conversationId: chatMessages.conversationId,
|
||||
conversationTitle: chatConversations.title,
|
||||
content: chatMessages.content,
|
||||
role: chatMessages.role,
|
||||
agentId: chatMessages.agentId,
|
||||
createdAt: chatMessages.createdAt,
|
||||
rank: sql<number>`ts_rank("chat_messages"."content_search", plainto_tsquery('english', ${tsQuery}))`,
|
||||
})
|
||||
.from(chatMessages)
|
||||
.innerJoin(chatConversations, and(
|
||||
eq(chatMessages.conversationId, chatConversations.id),
|
||||
eq(chatConversations.companyId, companyId),
|
||||
isNull(chatConversations.deletedAt),
|
||||
))
|
||||
.where(
|
||||
sql`"chat_messages"."content_search" @@ plainto_tsquery('english', ${tsQuery})`,
|
||||
)
|
||||
.orderBy(desc(sql`ts_rank("chat_messages"."content_search", plainto_tsquery('english', ${tsQuery}))`))
|
||||
.limit(limit);
|
||||
|
||||
return { items: rows };
|
||||
}
|
||||
```
|
||||
|
||||
### useChatSearch Hook Pattern
|
||||
|
||||
```typescript
|
||||
// ui/src/hooks/useChatSearch.ts
|
||||
import { useQuery } from "@tanstack/react-query";
|
||||
import { chatApi } from "../api/chat";
|
||||
|
||||
export function useChatSearch(companyId: string | null, query: string) {
|
||||
return useQuery({
|
||||
queryKey: ["chat", "search", companyId, query],
|
||||
queryFn: () => chatApi.searchMessages(companyId!, query),
|
||||
enabled: !!companyId && query.trim().length >= 2,
|
||||
placeholderData: (prev) => prev,
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### ChatMessage Bookmark Toggle
|
||||
|
||||
```typescript
|
||||
// Toggling in ChatMessageActions — add onBookmark prop and Bookmark icon
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-6 w-6"
|
||||
onClick={onBookmark}
|
||||
aria-label={isBookmarked ? "Remove bookmark" : "Bookmark message"}
|
||||
>
|
||||
<Bookmark className={cn("h-3.5 w-3.5", isBookmarked && "fill-current")} />
|
||||
</Button>
|
||||
```
|
||||
|
||||
### Branch Conversation Service Skeleton
|
||||
|
||||
```typescript
|
||||
async branchConversation(
|
||||
parentConversationId: string,
|
||||
branchFromMessageId: string,
|
||||
companyId: string,
|
||||
) {
|
||||
const [branchMsg] = await db
|
||||
.select({ createdAt: chatMessages.createdAt })
|
||||
.from(chatMessages)
|
||||
.where(eq(chatMessages.id, branchFromMessageId));
|
||||
if (!branchMsg) throw notFound("Branch message not found");
|
||||
|
||||
const messagesToCopy = await db
|
||||
.select()
|
||||
.from(chatMessages)
|
||||
.where(and(
|
||||
eq(chatMessages.conversationId, parentConversationId),
|
||||
lte(chatMessages.createdAt, branchMsg.createdAt),
|
||||
))
|
||||
.orderBy(asc(chatMessages.createdAt));
|
||||
|
||||
const [newConv] = await db
|
||||
.insert(chatConversations)
|
||||
.values({
|
||||
companyId,
|
||||
title: null,
|
||||
parentConversationId,
|
||||
branchFromMessageId,
|
||||
})
|
||||
.returning();
|
||||
|
||||
if (messagesToCopy.length > 0) {
|
||||
await db.insert(chatMessages).values(
|
||||
messagesToCopy.map(({ id: _id, conversationId: _cid, ...rest }) => ({
|
||||
...rest,
|
||||
conversationId: newConv!.id,
|
||||
})),
|
||||
);
|
||||
}
|
||||
|
||||
return newConv!;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## State of the Art
|
||||
|
||||
| Old Approach | Current Approach | When Changed | Impact |
|
||||
|--------------|------------------|--------------|--------|
|
||||
| ILIKE for content search | tsvector + GIN (PostgreSQL FTS) | PostgreSQL 8.3 (2008) | Order-of-magnitude faster at scale |
|
||||
| Manual tsvector maintenance via triggers | GENERATED ALWAYS AS ... STORED | PostgreSQL 12 (2019) | No triggers, auto-updated on UPDATE |
|
||||
| cmdk v0.x (uncontrolled) | cmdk v1.x (controlled, `shouldFilter` prop) | cmdk 1.0 (2024) | Must set `shouldFilter={false}` when using server-side search to prevent cmdk's own client-side filter from re-filtering server results |
|
||||
|
||||
**Deprecated / outdated:**
|
||||
|
||||
- `plainto_tsquery` vs `websearch_to_tsquery`: `websearch_to_tsquery` (Postgres 11+) handles quoted phrases and `-exclusions` like a web search engine. For a basic first implementation, `plainto_tsquery` is simpler and correct. Upgrade to `websearch_to_tsquery` later if users need phrase search.
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Branch UI placement**
|
||||
- What we know: `ChatConversationItem` shows conversation in the left column of ChatPanel. Branches would be child conversations visible in the same list with a visual indent or branch icon.
|
||||
- What's unclear: Whether branches should be nested under the parent in the conversation list or shown as peers with a parent link.
|
||||
- Recommendation: Show branches as indented items under the parent in `ChatConversationList`. Add a `parentConversationId` to `ChatConversationListItem` type and group by parent in the UI. Keep the server list endpoint flat (client-side grouping).
|
||||
|
||||
2. **Bookmark scope: company vs. per-user**
|
||||
- What we know: The project uses board-level (company-scoped) auth. There is no per-user identity surfaced in the chat service — `assertBoard` validates company access but does not expose `userId` to service methods in the current chat service signature.
|
||||
- What's unclear: Whether bookmarks should be shared across all users in a workspace or per-user.
|
||||
- Recommendation: Scope bookmarks to `companyId` for simplicity (shared bookmarks across the workspace). This matches how conversations are scoped. Per-user bookmarks can be added later when the user model is more prominent.
|
||||
|
||||
3. **Search result navigation**
|
||||
- What we know: Search results include `conversationId` and `messageId`. Clicking a result should navigate to that conversation and scroll to the message.
|
||||
- What's unclear: The existing `ChatPanel` has no mechanism to scroll to a specific message by ID.
|
||||
- Recommendation: Add a `scrollToMessageId` state to `ChatPanelContext`. When set, `ChatMessageList` uses the virtualiser's `scrollToIndex` method to jump to the message. Reset after scrolling. This follows the `nexus:focus-chat-search` custom event pattern already used for Cmd+K focus.
|
||||
|
||||
---
|
||||
|
||||
## Environment Availability
|
||||
|
||||
Step 2.6: No new external dependencies identified. PostgreSQL 17 is already the project's database. All npm packages are already installed.
|
||||
|
||||
---
|
||||
|
||||
## Validation Architecture
|
||||
|
||||
### Test Framework
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Framework | Vitest 3.2.4 |
|
||||
| Config file | `server/vitest.config.ts` (node env), `ui/vitest.config.ts` (node env) |
|
||||
| Quick run command | `pnpm --filter @paperclipai/server test run -- chat-service` |
|
||||
| Full suite command | `pnpm test:run` |
|
||||
|
||||
### Phase Requirements → Test Map
|
||||
|
||||
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|
||||
|--------|----------|-----------|-------------------|-------------|
|
||||
| CHAT-07 | `searchMessages` returns ranked results for matching term | unit | `pnpm --filter @paperclipai/server test run -- chat-service` | ✅ (new describe block in existing file) |
|
||||
| CHAT-07 | `GET /companies/:id/messages/search?q=term` returns 200 with results | integration | `pnpm --filter @paperclipai/server test run -- chat-routes` | ✅ (new describe block in existing file) |
|
||||
| CHAT-13 | `toggleBookmark` inserts or removes bookmark row | unit | `pnpm --filter @paperclipai/server test run -- chat-service` | ✅ |
|
||||
| CHAT-13 | `GET /companies/:id/bookmarks` returns bookmarked messages | integration | `pnpm --filter @paperclipai/server test run -- chat-routes` | ✅ |
|
||||
| CHAT-14 | `branchConversation` creates new conversation with copied messages | unit | `pnpm --filter @paperclipai/server test run -- chat-service` | ✅ |
|
||||
| CHAT-14 | `POST /conversations/:id/branch` returns 201 with new conversation | integration | `pnpm --filter @paperclipai/server test run -- chat-routes` | ✅ |
|
||||
| HIST-04 | `exportConversation` returns correct Markdown structure | unit | `pnpm --filter @paperclipai/server test run -- chat-service` | ✅ |
|
||||
| HIST-04 | `GET /conversations/:id/export?format=markdown` returns file download headers | integration | `pnpm --filter @paperclipai/server test run -- chat-routes` | ✅ |
|
||||
| PERF-04 | GIN index query plan uses index scan (not seq scan) | manual/DB | `EXPLAIN ANALYZE ...` in Postgres | ❌ Wave 0 — manual verification |
|
||||
|
||||
### Sampling Rate
|
||||
- **Per task commit:** `pnpm --filter @paperclipai/server test run -- chat-service`
|
||||
- **Per wave merge:** `pnpm test:run`
|
||||
- **Phase gate:** Full suite green before `/gsd:verify-work`
|
||||
|
||||
### Wave 0 Gaps
|
||||
- [ ] New `describe("searchMessages")` block in `server/src/__tests__/chat-service.test.ts` — covers CHAT-07
|
||||
- [ ] New `describe("toggleBookmark / getBookmarks")` block in `server/src/__tests__/chat-service.test.ts` — covers CHAT-13
|
||||
- [ ] New `describe("branchConversation")` block in `server/src/__tests__/chat-service.test.ts` — covers CHAT-14
|
||||
- [ ] New `describe("exportConversation")` block in `server/src/__tests__/chat-service.test.ts` — covers HIST-04
|
||||
- [ ] New route-level describe blocks in `server/src/__tests__/chat-routes.test.ts` — covers all four route groups
|
||||
|
||||
---
|
||||
|
||||
## Project Constraints (from CLAUDE.md)
|
||||
|
||||
CLAUDE.md does not exist at `/opt/nexus/CLAUDE.md`. No additional project-level directives to document.
|
||||
|
||||
**Codebase conventions observed from prior phases:**
|
||||
|
||||
- Use `object-syntax (table) => ({})` for Drizzle index callbacks (not arrow-returning-object shorthand).
|
||||
- Use `it.todo()` (not `it.skip()`) for Wave 0 test scaffolding.
|
||||
- Use `@/lib/router` Link abstraction for navigation, not `react-router-dom` directly.
|
||||
- Use `useToast()/pushToast()` for error toasts — not `sonner`.
|
||||
- DB schema files are individual per table; exported from `packages/db/src/schema/index.ts`.
|
||||
- Migrations are hand-numbered (`0050_`, `0051_`, ...) and journaled in `meta/_journal.json`.
|
||||
- Shared types in `packages/shared/src/types/chat.ts`; validators in `packages/shared/src/validators/chat.ts`; both re-exported from `packages/shared/src/index.ts`.
|
||||
- Custom window events (e.g. `nexus:focus-chat-search`) are the project pattern for decoupled cross-component communication.
|
||||
- The `assertBoard(req)` + `assertCompanyAccess(req, companyId)` guard pattern is required on all chat routes.
|
||||
- Service functions are factory functions `chatService(db)` returning a plain object — not classes.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
- Direct codebase inspection — all schema, service, route, and UI files read from `/opt/nexus/`
|
||||
- `packages/db/src/schema/chat_conversations.ts` — confirmed existing columns
|
||||
- `packages/db/src/schema/chat_messages.ts` — confirmed no existing tsvector
|
||||
- `server/src/services/chat.ts` — confirmed ilike-only current search
|
||||
- `server/src/routes/chat.ts` — confirmed route patterns
|
||||
- `ui/src/components/CommandPalette.tsx` — confirmed cmdk usage and Cmd+K binding
|
||||
- `ui/src/hooks/useKeyboardShortcuts.ts` — confirmed Cmd+K conflict point
|
||||
- `.planning/ROADMAP.md` Phase 24 — canonical requirement list (CHAT-07, CHAT-13, CHAT-14, HIST-04, PERF-04)
|
||||
- `.planning/codebase/STACK.md` — confirmed PostgreSQL 17, Drizzle 0.38.x, cmdk 1.1.1
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
- PostgreSQL 12 documentation — GENERATED ALWAYS AS STORED columns
|
||||
- PostgreSQL FTS documentation — tsvector, GIN indexes, plainto_tsquery, ts_rank
|
||||
|
||||
### Tertiary (LOW confidence, flag for validation)
|
||||
- Drizzle ORM 0.38.x generated column support — my training data indicates incomplete support; verified indirectly by absence of `generatedAlwaysAs` usage in the codebase
|
||||
|
||||
---
|
||||
|
||||
## Metadata
|
||||
|
||||
**Confidence breakdown:**
|
||||
- Standard stack: HIGH — all libraries confirmed present in codebase
|
||||
- Architecture: HIGH — patterns derived directly from existing Phases 21–23 code
|
||||
- Search implementation: HIGH — PostgreSQL FTS is well-established; tsvector generated column confirmed supported in PG17
|
||||
- Drizzle generated column handling: MEDIUM — Drizzle Kit limitation confirmed by absence of existing usage; hand-written migration approach is the safe path
|
||||
- Pitfalls: HIGH — derived from direct code inspection of conflict points
|
||||
|
||||
**Research date:** 2026-04-01
|
||||
**Valid until:** 2026-05-01 (stable stack; no fast-moving dependencies)
|
||||
Loading…
Add table
Reference in a new issue