feat(22-01): SSE echo-stream endpoint, edit message route, and stream tests

- Add GET /conversations/:id/stream SSE endpoint with echo-stream placeholder
- Set text/event-stream, X-Accel-Buffering: no, flush headers immediately
- Stream token events word-by-word, emit done event with messageId on completion
- Detect client disconnect via req.on(close) to stop streaming
- Persist assistant message only when stream completes (not on abort)
- Add ECHO-STREAM PLACEHOLDER comment for Phase 23 LLM adapter replacement
- Create chat-stream-routes.test.ts with SSE header, token/done event, and 404 tests
This commit is contained in:
Mikkel Georgsen 2026-04-01 14:51:53 +02:00
parent 5db6fe7af7
commit fa10552127
3 changed files with 870 additions and 0 deletions

View file

@ -0,0 +1,522 @@
# Phase 23: Brainstormer Flow - Research
**Researched:** 2026-04-01
**Domain:** Conversational agent persona, structured chat flows, spec card UI, PM handoff, issue creation from chat, task status updates in chat
**Confidence:** HIGH
## Summary
Phase 23 wires the chat infrastructure built in Phases 21-22 into a real end-to-end agent workflow. A user opens a new conversation, is greeted by the Brainstormer (a special agent persona), answers clarifying questions, receives a structured spec card in-chat, and with one click the PM agent converts that spec into Nexus issues — all without touching the dashboard.
The core architecture has three distinct layers. First, **agent persona and conversation defaulting**: the `chat_conversations.agentId` column (from Phase 21) needs to be populated to a Brainstormer agent on new conversation creation; this requires either creating the Brainstormer as a real agent row, or treating it as a pseudo-agent with a fixed well-known ID. Second, **Brainstormer questioning flow**: when the user sends a first message to the Brainstormer, the server streams a structured response (via the Phase 22 SSE stream endpoint) that follows a fixed template — clarifying questions → spec card → action buttons. The spec card is a structured chat message with a `metadata` JSON blob containing `{ type: "brainstorm_spec", what, why, constraints, successCriteria }`. Third, **PM handoff and task creation**: when the user clicks "Send to PM," the client POSTs to a new brainstormer handoff route; the server creates a system chat message visible as a handoff indicator (CHAT-09), then creates one or more Nexus issues via the existing `issueService`, and finally posts an assistant message back with the created issue IDs.
No new npm packages are required. The Phase 22 SSE stream endpoint, the Phase 21 chat message store, and the existing `issueService` are all the infrastructure this phase needs. The new work is: (1) DB migration adding a `metadata` jsonb column to `chat_messages`, (2) Brainstormer agent provisioning logic, (3) server-side flow orchestration, and (4) three new UI components: `BrainstormSpecCard`, `ChatHandoffIndicator`, and `ChatAgentStatusUpdate`.
**Primary recommendation:** Implement the Brainstormer as a persisted agent row (role: `general`, with a fixed slug `brainstormer`) that is auto-created on first use via an `ensureBrainstormerAgent(companyId)` helper. This keeps the agent selector in Phase 22 consistent and lets Phase 22's `ChatAgentBadge` render correctly without changes.
---
<user_constraints>
## User Constraints (from CONTEXT.md)
### Locked Decisions
None — discuss phase skipped per `workflow.skip_discuss: true`.
### Claude's Discretion
All implementation choices are at Claude's discretion. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions.
### Deferred Ideas (OUT OF SCOPE)
None — discuss phase skipped.
</user_constraints>
<phase_requirements>
## Phase Requirements
> NOTE: The caller supplied INPUT-02, INPUT-03, INPUT-04 as Phase 23 requirements. Per REQUIREMENTS.md traceability table and ROADMAP.md Phase 23 definition, these belong to Phase 25 (File System). They are **out of scope for Phase 23**. The actual Phase 23 requirements are AGENT-01, AGENT-02, AGENT-03, AGENT-05, AGENT-06, AGENT-07, CHAT-09 (per ROADMAP). TASK-01 through TASK-05 do not appear in REQUIREMENTS.md at all — they are not v1.3 requirements IDs. The plan should address only the seven AGENT/CHAT requirements below.
| ID | Description | Research Support |
|----|-------------|------------------|
| AGENT-01 | Default agent is the Brainstormer; it greets the user and begins a structured questioning flow | Auto-create Brainstormer agent row on company bootstrap; `POST /api/conversations` sets `agentId` to Brainstormer by default; stream endpoint calls `ensureBrainstormerAgent` |
| AGENT-02 | Brainstormer follows structured questioning → spec template → PM handoff | Server-side flow state machine: phase 1 = questions, phase 2 = spec card (metadata JSON), phase 3 = action buttons. `chat_messages.metadata` jsonb column carries spec payload |
| AGENT-03 | PM agent can receive specs from chat and create Nexus tasks/issues | New route `POST /api/conversations/:id/brainstorm/handoff`; calls existing `issueService.create()` for each issue; responds with created issue IDs as assistant message |
| AGENT-05 | Handoff indicators visible in chat: "Brainstormer → PM: Here's the spec for approval" | System-role message with `metadata.type = "handoff_indicator"` inserted into `chat_messages`; rendered as `ChatHandoffIndicator` component in ChatMessageList |
| AGENT-06 | Task creation from chat: user or agent can say "create a task for this" and it becomes a Nexus issue | Slash command `/task <title>` parsing (Phase 22 already provides `parseMessageIntent`); server creates issue on receiving this intent; returns issue link in assistant reply |
| AGENT-07 | Status updates from agents appear in chat: "Engineer completed task X" notification in relevant conversation | New `chat.status_update` live event type; server publishes event when issue transitions to `done`/`cancelled`; client subscribes via existing SSE connection; inserts system message into conversation |
| CHAT-09 | System message indicator: when Brainstormer hands off to PM, or PM delegates to Engineer, handoff is visible in chat | Uses same `metadata.type = "handoff_indicator"` system message pattern; `ChatHandoffIndicator` UI component |
</phase_requirements>
---
## Project Constraints (from CLAUDE.md)
| Constraint | Detail |
|------------|--------|
| Upstream sync | Display-layer changes only. DB schema changes must be additive (new columns, no drops). |
| Language | TypeScript (ESM) everywhere. No plain JS. |
| Package manager | pnpm. Use `pnpm add` — never `npm install`. |
| Framework | Express 5.1.0. Routes follow `function xRoutes(db: Db): Router` factory pattern. |
| DB | Drizzle ORM with PostgreSQL. New columns require `pnpm db:generate` + committed migration SQL. |
| Auth | `local_trusted` mode — `assertBoard(req)` is the only auth gate needed for user-initiated routes. |
| Testing | Vitest (server) + jsdom + createRoot + act (UI). `@testing-library/react` is NOT installed. Pattern in `ChatInput.test.tsx` and `chat-routes.test.ts`. |
| React version | React 19.0.0 — use `createRoot` + `act`, not legacy `render`. |
| TanStack Query | ^5.90.21 — `useMutation` + `useInfiniteQuery` patterns established. |
| shadcn | new-york preset, neutral base, cssVariables. Already-installed components: Avatar, Badge, Button, Card, Command, Dialog, DropdownMenu, Popover, ScrollArea, Separator, Skeleton, Tabs, Tooltip. |
---
## Standard Stack
### Core (already in project, no install needed)
| Library | Version | Purpose | Notes |
|---------|---------|---------|-------|
| `express` | ^5.1.0 | Brainstorm handoff route, status update SSE route | Follows `chatRoutes(db)` factory pattern |
| `drizzle-orm` | ^0.38.4 | New `metadata` column on `chat_messages` | Additive column, not null defaults to `{}` |
| `@tanstack/react-query` | ^5.90.21 | `useMutation` for handoff POST, query invalidation after issue creation | |
| `lucide-react` | ^0.574.0 | Icons: ArrowRight (handoff), CheckCircle (status done), Sparkles (Brainstormer avatar) | |
| `@paperclipai/shared` | workspace | `LIVE_EVENT_TYPES` needs `chat.status_update` added; `createIssueSchema` used server-side | |
| `clsx` / `tailwind-merge` | current | Conditional classNames in new UI components | |
### Supporting (already in project)
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| `react-markdown` | ^10.1.0 | Spec card description rendering | If description field in spec contains markdown |
| `virtua` | ^0.49.0 | Already added in Phase 22; no new install | ChatMessageList already uses VList |
| `ai` or `openai` | Phase 22 choice | LLM streaming for Brainstormer responses | Inherited from Phase 22 decision |
### New Installs Required
None. Phase 23 is entirely built on the infrastructure from Phases 21 and 22.
### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| Persisted agent row for Brainstormer | Pseudo-agent constant (hardcoded ID) | Pseudo-agent breaks AgentSelector and ChatAgentBadge from Phase 22; persisted row is consistent |
| `metadata` jsonb on `chat_messages` | Separate `chat_message_metadata` table | Separate table is over-engineered; spec data is small and conversation-local; jsonb column is idiomatic for Drizzle + Postgres |
| Server-side flow orchestration | Pure LLM prompt engineering | Prompt-only approach is fragile (LLM may not follow the template); server state machine ensures the spec card always appears at the right step |
| New `chat.status_update` live event | Polling for issue status changes | Polling is wasteful; `publishLiveEvent` already exists and the client SSE connection is already open for streaming |
---
## Architecture Patterns
### Recommended Project Structure (additions to Phases 21/22)
```
packages/
├── db/src/schema/
│ └── chat_messages.ts # Add: metadata jsonb column
│ └── (migration .sql) # Add: ALTER TABLE ADD COLUMN metadata jsonb
├── shared/src/
│ ├── constants.ts # Add: "chat.status_update" to LIVE_EVENT_TYPES
│ └── validators/chat.ts # Add: brainstormHandoffSchema, specCardSchema
server/src/
├── routes/
│ └── chat.ts # Add: POST /conversations/:id/brainstorm/handoff
├── services/
│ ├── chat.ts # Add: addSystemMessage(), findOrCreateBrainstormerAgent()
│ └── brainstormer-flow.ts # New: buildQuestionPrompt(), buildSpecPrompt(), parseSpecFromLlm()
├── __tests__/
│ └── brainstormer-routes.test.ts # New: handoff POST, status update publishing
ui/src/
├── api/
│ └── chat.ts # Add: postBrainstormHandoff()
├── hooks/
│ └── useBrainstormHandoff.ts # New: useMutation for handoff; query invalidation
├── components/
│ ├── BrainstormSpecCard.tsx # New: spec card with What/Why/Constraints/Success + action buttons
│ ├── ChatHandoffIndicator.tsx # New: "Brainstormer → PM" system message renderer
│ └── ChatAgentStatusUpdate.tsx # New: "Engineer completed task X" renderer
│ └── ChatMessageList.tsx # Extend: dispatch metadata.type to the right renderer
```
### Pattern 1: Metadata-typed System Messages
**What:** System-role chat messages carry a `metadata` JSON blob with a `type` discriminant. The message renderer in `ChatMessageList` inspects `msg.metadata?.type` and dispatches to the appropriate sub-component.
**When to use:** Any in-chat event that is not a plain user/assistant message (handoff indicators, spec cards, status updates).
**Example:**
```typescript
// Server: insert a handoff indicator
await svc.addSystemMessage(conversationId, {
type: "handoff_indicator",
from: "Brainstormer",
to: "PM",
specTitle: spec.what,
});
// chat_messages row:
// { role: "system", content: "Brainstormer → PM: ...", metadata: { type: "handoff_indicator", from: "Brainstormer", to: "PM", specTitle: "..." } }
// UI: ChatMessageList dispatch
if (msg.role === "system") {
if (msg.metadata?.type === "handoff_indicator") return <ChatHandoffIndicator msg={msg} />;
if (msg.metadata?.type === "status_update") return <ChatAgentStatusUpdate msg={msg} />;
return <span className="text-xs text-muted-foreground">{msg.content}</span>;
}
```
### Pattern 2: Brainstormer Agent Auto-Provision
**What:** On the first `/brainstorm` slash command or new-conversation default, an `ensureBrainstormerAgent(db, companyId)` helper upserts a Brainstormer agent row keyed by `(companyId, role='general', name='Brainstormer')`. This is idempotent: repeated calls return the same row.
**When to use:** Called at the top of the stream route when the conversation's active agent is the Brainstormer.
```typescript
// server/src/services/chat.ts (addition)
export async function ensureBrainstormerAgent(db: Db, companyId: string) {
const existing = await db
.select()
.from(agents)
.where(and(
eq(agents.companyId, companyId),
eq(agents.name, "Brainstormer"),
eq(agents.role, "general"),
))
.then((rows) => rows[0] ?? null);
if (existing) return existing;
const [created] = await db
.insert(agents)
.values({
companyId,
name: "Brainstormer",
role: "general",
title: "Brainstormer",
icon: "sparkles",
adapterType: "process",
adapterConfig: {},
runtimeConfig: {},
budgetMonthlyCents: 0,
permissions: {},
})
.returning();
return created!;
}
```
### Pattern 3: Brainstorm Handoff Route
**What:** `POST /api/conversations/:id/brainstorm/handoff` accepts the spec payload, creates issues via `issueService.create()`, inserts a handoff indicator system message, and returns the created issue IDs as an assistant message.
**When to use:** When user clicks "Send to PM" on the `BrainstormSpecCard`.
```typescript
router.post("/conversations/:id/brainstorm/handoff", validate(brainstormHandoffSchema), async (req, res) => {
assertBoard(req);
const { spec, pmAgentId } = req.body as BrainstormHandoffBody;
const conversationId = req.params.id as string;
// 1. Insert handoff indicator (system message)
await svc.addSystemMessage(conversationId, {
type: "handoff_indicator",
from: "Brainstormer",
to: "PM",
specTitle: spec.what,
spec,
});
// 2. Create issues via issueService
const createdIssues = await Promise.all(
spec.tasks.map((task) =>
issueSvc.create(companyId, {
title: task.title,
description: task.description,
assigneeAgentId: pmAgentId ?? null,
originKind: "manual",
status: "backlog",
priority: "medium",
}),
),
);
// 3. Post assistant reply with issue IDs
const issueRefs = createdIssues.map((i) => `#${i.issueNumber ?? i.id}`).join(", ");
const reply = await svc.addMessage(conversationId, {
role: "assistant",
content: `PM received the spec. Created issues: ${issueRefs}`,
agentId: pmAgentId ?? null,
});
res.json({ issues: createdIssues, message: reply });
});
```
### Pattern 4: Task Status Update Live Events
**What:** When `issueService` updates a task to `done` or `cancelled`, it calls `publishLiveEvent` with the new `"chat.status_update"` event type. The client's existing SSE live-event connection (established in Phase 22) receives the event and injects a system message into the relevant conversation.
**When to use:** Phase 23 must add `"chat.status_update"` to `LIVE_EVENT_TYPES` in `packages/shared/src/constants.ts`. The server hooks into issue status transitions. The UI side uses `useEffect` on the existing live event stream to append a status update message.
**The hook point in the existing issue service:** The `applyStatusSideEffects` function in `server/src/services/issues.ts` fires on status transitions. Add `publishLiveEvent` call there when status becomes `"done"` or `"cancelled"` — but only if the issue has an `originId` that maps to a chat conversation (requires a new `chatConversationId` link on the issue, or a lookup via `chat_messages.metadata`).
> **Design choice:** Rather than adding a `chatConversationId` FK to `issues`, store the `conversationId` in `chat_messages.metadata` on the spec card message. The status-update publisher looks up which conversations mentioned this issue ID and fans out the event. This avoids a schema change on `issues`.
### Anti-Patterns to Avoid
- **Hard-coding the Brainstormer agent ID as a constant:** The agent must be a real persisted row so `AgentSelector` and `ChatAgentBadge` from Phase 22 work without modification. A magic constant would require special-casing throughout the UI.
- **Implementing the full Brainstormer prompt as a fixed server-side string:** Use a system prompt passed to the LLM at stream time, not a hard-coded response. The Phase 22 SSE stream endpoint already calls the LLM — Phase 23 adds a special system prompt for Brainstormer conversations.
- **Creating a separate SSE endpoint for status updates:** The existing live-event SSE endpoint (`GET /api/companies/:id/live`) already delivers events to the UI. Add the new event type to that channel; do not create a second SSE connection.
- **Allowing `metadata` to be null in TypeScript types:** Define `metadata` as `Record<string, unknown> | null` and narrow with a type guard — never access `metadata.type` without checking for null first.
---
## Don't Hand-Roll
| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| Issue creation from chat | Custom issue-creation logic | `issueService.create()` from `server/src/services/issues.ts` | Already handles companyId scoping, issue numbering, identifier generation, status side-effects, activity logging |
| LLM streaming to browser | Custom SSE token loop | Phase 22 stream endpoint `POST /api/conversations/:id/stream` | Already built, tested, handles abort, respects PERF-02 latency target |
| Agent identity in messages | Custom agent lookup per message | Phase 22 `ChatAgentBadge` component + joined agent rows | Already handles name/icon/color per agent role |
| Live event broadcasting | Custom WebSocket | `publishLiveEvent()` from `server/src/services/live-events.ts` | In-process EventEmitter, already subscribed by all active SSE clients |
| Slash command parsing | Ad-hoc regex | Phase 22 `parseMessageIntent()` from `ui/src/lib/parseMessageIntent.ts` | Already returns `{ command, agentOverride }` — Phase 23 adds `/brainstorm` handler |
**Key insight:** Phase 23 is almost entirely orchestration code that wires together infrastructure from Phases 21 and 22. The only truly new concerns are the spec card data model and the handoff route.
---
## Common Pitfalls
### Pitfall 1: Default Agent on New Conversation
**What goes wrong:** If `ensureBrainstormerAgent` is not called before the UI creates a conversation, the `agentId` on the new conversation is `null`, and the agent selector shows "No agent." The Brainstormer greeting never fires.
**Why it happens:** Phase 21's `POST /api/companies/:companyId/conversations` sets `agentId: null` by default. Phase 22's stream endpoint uses whatever `agentId` is stored on the conversation. Without Phase 23 seeding the default, the Brainstormer flow never starts.
**How to avoid:** Modify `POST /api/companies/:companyId/conversations` to call `ensureBrainstormerAgent(db, companyId)` and set the returned agent's `id` as the default `agentId` when none is provided in the request body.
**Warning signs:** New conversation created with `agentId: null` in the response JSON.
### Pitfall 2: Spec Card Not Persisted as Metadata
**What goes wrong:** If the spec card is only rendered from the LLM's free-text response (without being stored in `chat_messages.metadata`), the UI cannot reconstruct the spec after a page reload. The "Send to PM" button can never retrieve the spec content reliably.
**Why it happens:** The LLM produces text; without explicitly parsing and storing the spec in `metadata`, the structured data is lost once the streaming bubble is replaced by the final message text.
**How to avoid:** After the LLM completes its spec response, the server parses the response to extract `{ what, why, constraints, successCriteria, tasks }` and stores it in `metadata` when writing the `chat_messages` row. The client renders the spec card from `msg.metadata` when `msg.metadata?.type === "brainstorm_spec"`.
**Warning signs:** `BrainstormSpecCard` receives `null` spec on mount after a page reload.
### Pitfall 3: Issue Creation Runs Without companyId
**What goes wrong:** `issueService.create()` requires `companyId` scoping. If the handoff route resolves `companyId` from the URL path incorrectly (or not at all), issues are created without a company context and subsequent queries fail.
**Why it happens:** The handoff route is at `POST /api/conversations/:id/brainstorm/handoff` which does not have `companyId` in the path (unlike issue routes which are under `/api/companies/:companyId/issues`).
**How to avoid:** Fetch `companyId` from the conversation row at the start of the handler: `const conv = await svc.getConversation(id); const companyId = conv.companyId;`. Then pass `companyId` to `issueService.create()`.
**Warning signs:** Issues created with missing `companyId` field (DB constraint error).
### Pitfall 4: Live Event Type Not in Shared Constants
**What goes wrong:** If `"chat.status_update"` is not added to `LIVE_EVENT_TYPES` in `packages/shared/src/constants.ts`, `publishLiveEvent` will fail TypeScript type checking. Both the server and client packages consume `LiveEventType` from `@paperclipai/shared`.
**Why it happens:** `LIVE_EVENT_TYPES` is an `as const` tuple — `LiveEventType` is a union of its members. New event types require extending this tuple and running `pnpm --filter @paperclipai/shared build` to propagate the type change.
**How to avoid:** Add the event type to the tuple FIRST, before writing any code that publishes or consumes it.
**Warning signs:** TypeScript error "Argument of type '"chat.status_update"' is not assignable to parameter of type 'LiveEventType'."
### Pitfall 5: Brainstormer Agent Duplicated Per Company
**What goes wrong:** If `ensureBrainstormerAgent` is called concurrently (e.g., two browser tabs open simultaneously), two Brainstormer agent rows are inserted for the same company.
**Why it happens:** The select-then-insert pattern has a race condition.
**How to avoid:** Use `INSERT ... ON CONFLICT DO NOTHING` with a unique index on `(company_id, name, role)` for the Brainstormer agent, or use a database-level unique constraint. In Drizzle: `.onConflictDoNothing()` + a unique index on `(companyId, name)`.
**Warning signs:** More than one agent named "Brainstormer" appears in the agent selector.
---
## Code Examples
Verified patterns from existing codebase:
### Adding a jsonb metadata column (Drizzle schema)
```typescript
// packages/db/src/schema/chat_messages.ts
import { pgTable, uuid, text, timestamp, jsonb, index } from "drizzle-orm/pg-core";
export const chatMessages = pgTable("chat_messages", {
id: uuid("id").primaryKey().defaultRandom(),
conversationId: uuid("conversation_id").notNull().references(() => chatConversations.id, { onDelete: "cascade" }),
role: text("role").notNull(),
content: text("content").notNull(),
agentId: uuid("agent_id"),
metadata: jsonb("metadata").$type<Record<string, unknown>>(), // NEW
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
}, (table) => ({
conversationCreatedIdx: index("chat_messages_conversation_created_idx").on(table.conversationId, table.createdAt),
}));
```
### Publishing a live event (existing pattern from live-events.ts)
```typescript
// server/src/services/live-events.ts — existing API
publishLiveEvent({
companyId,
type: "chat.status_update", // must be in LIVE_EVENT_TYPES
payload: {
conversationId,
issueId,
issueTitle,
newStatus: "done",
agentName: "Engineer",
},
});
```
### createIssueSchema fields (from packages/shared/src/validators/issue.ts)
The minimal set needed for Brainstormer-originated tasks:
```typescript
const issuePayload = {
title: spec.what, // required, min(1)
description: spec.why ?? null, // optional
status: "backlog" as const, // default
priority: "medium" as const, // default
assigneeAgentId: pmAgentId ?? null, // optional
// originKind defaults to "manual" in the DB
};
```
### Rendering metadata-typed system messages (UI pattern)
```tsx
// ui/src/components/ChatMessageList.tsx — extend existing map
{allMessages.map((msg) => {
if (msg.role === "system") {
const type = (msg.metadata as any)?.type;
if (type === "handoff_indicator") return <ChatHandoffIndicator key={msg.id} msg={msg} />;
if (type === "status_update") return <ChatAgentStatusUpdate key={msg.id} msg={msg} />;
return <div key={msg.id} className="text-xs text-muted-foreground text-center py-1">{msg.content}</div>;
}
if (msg.role === "assistant" && (msg.metadata as any)?.type === "brainstorm_spec") {
return <BrainstormSpecCard key={msg.id} msg={msg} />;
}
// ...existing user/assistant rendering
})}
```
### useMutation for handoff (TanStack Query pattern)
```tsx
// ui/src/hooks/useBrainstormHandoff.ts
import { useMutation, useQueryClient } from "@tanstack/react-query";
import { chatApi } from "../api/chat";
export function useBrainstormHandoff(conversationId: string) {
const qc = useQueryClient();
return useMutation({
mutationFn: (spec: BrainstormSpec) => chatApi.postBrainstormHandoff(conversationId, spec),
onSuccess: () => {
qc.invalidateQueries({ queryKey: ["messages", conversationId] });
},
});
}
```
---
## State of the Art
| Old Approach | Current Approach | When Changed | Impact |
|--------------|-----------------|--------------|--------|
| Agent status updates via polling | Live events via SSE (established Phase 22) | Phase 22 | Phase 23 can publish `chat.status_update` on the existing channel — no new transport needed |
| Hard-coded agent personas in UI | Persisted agent rows with role/icon/name | Phase 21 schema | Brainstormer is a real agent row; existing ChatAgentBadge renders it for free |
| Issues created only from dashboard | Issues created from any origin (originKind field exists in schema) | Pre-existing | `originKind: "manual"` works for chat-originated issues; no schema change needed |
**No deprecated approaches identified for this phase.**
---
## Open Questions
1. **Should the Brainstormer be auto-created on company bootstrap, or lazily on first use?**
- What we know: Phase 21 creates PM and Engineer agents on company bootstrap (see `server/src/services/companies.ts` — not inspected, but inferred from "auto-creates PM + Engineer" in STATE.md).
- What's unclear: Whether `POST /api/companies` already provisions a fixed set of agents and whether adding Brainstormer there would cause issues for existing companies on migration.
- Recommendation: Use lazy `ensureBrainstormerAgent(db, companyId)` on first conversation creation. This is safe for existing deployments. Add it to the company bootstrap path in a follow-up.
2. **How should the Brainstormer know what questions to ask?**
- What we know: The Phase 22 stream endpoint uses a generic system prompt. The `adapterConfig` on the Brainstormer agent row can carry a custom `systemPrompt` field.
- What's unclear: Whether the Phase 22 stream implementation reads `agent.adapterConfig.systemPrompt` or uses a hardcoded generic prompt.
- Recommendation: Store the Brainstormer's structured questioning prompt in `adapterConfig.systemPrompt` on the agent row. The Phase 22 stream endpoint should read this and pass it as the LLM system message. If Phase 22 does not yet do this, Phase 23 plan 01 adds it.
3. **How does the server know a conversation is in "spec generation" phase vs "question" phase?**
- What we know: `chat_messages` has the full message history. The server can inspect the last N messages to determine flow phase.
- What's unclear: Whether to track `brainstormerPhase` as a column on `chat_conversations` or infer it from message history.
- Recommendation: Infer from message history (count of user turns since Brainstormer opened). Simple, no new column needed. If the conversation has 0-2 user messages, stream clarifying questions. If it has 3+, produce the spec card. This threshold is configurable in `brainstormer-flow.ts`.
4. **Which PM agent receives the handoff?**
- What we know: `createIssueSchema` has `assigneeAgentId` (optional). The `agents` table has `role = "ceo"` (which AGENT_ROLE_LABELS maps to "Project Manager").
- What's unclear: Whether to pick the PM by role lookup at handoff time or let the user choose in the spec card.
- Recommendation: Auto-resolve PM agent by `role = "ceo"` within the company at handoff time. Include the PM's name in the handoff indicator message. Expose an optional override in the spec card UI.
---
## Environment Availability
Step 2.6: SKIPPED — Phase 23 is purely code/schema changes building on Phase 21/22 infrastructure. No new external tools, databases, or CLI utilities are required.
---
## Validation Architecture
### Test Framework
| Property | Value |
|----------|-------|
| Framework | Vitest 3.0.5 |
| Config file | `server/vitest.config.ts` and `ui/vite.config.ts` |
| Quick run command | `pnpm --filter @paperclipai/server test --run brainstormer` |
| Full suite command | `pnpm --filter @paperclipai/server test --run && pnpm --filter @paperclipai/ui test --run` |
### Phase Requirements → Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| AGENT-01 | New conversation defaults agentId to Brainstormer row | unit | `pnpm --filter @paperclipai/server test --run brainstormer-routes` | ❌ Wave 0 |
| AGENT-02 | Handoff route inserts spec card metadata + handoff indicator | unit | `pnpm --filter @paperclipai/server test --run brainstormer-routes` | ❌ Wave 0 |
| AGENT-03 | Handoff route creates issues via issueService and returns IDs | unit | `pnpm --filter @paperclipai/server test --run brainstormer-routes` | ❌ Wave 0 |
| AGENT-05 | Handoff indicator system message persisted with correct metadata.type | unit | `pnpm --filter @paperclipai/server test --run brainstormer-routes` | ❌ Wave 0 |
| AGENT-06 | `/task <title>` slash command creates issue and returns link | unit | `pnpm --filter @paperclipai/server test --run brainstormer-routes` | ❌ Wave 0 |
| AGENT-07 | Issue status transition to "done" publishes chat.status_update live event | unit | `pnpm --filter @paperclipai/server test --run brainstormer-routes` | ❌ Wave 0 |
| CHAT-09 | System-role message with handoff_indicator type visible in message list | component | `pnpm --filter @paperclipai/ui test --run ChatHandoffIndicator` | ❌ Wave 0 |
### Sampling Rate
- **Per task commit:** `pnpm --filter @paperclipai/server test --run brainstormer-routes`
- **Per wave merge:** `pnpm --filter @paperclipai/server test --run && pnpm --filter @paperclipai/ui test --run`
- **Phase gate:** Full suite green before `/gsd:verify-work`
### Wave 0 Gaps
- [ ] `server/src/__tests__/brainstormer-routes.test.ts` — covers AGENT-01 through AGENT-07
- [ ] `ui/src/components/ChatHandoffIndicator.test.tsx` — covers CHAT-09
---
## Sources
### Primary (HIGH confidence)
- Codebase direct inspection — `packages/db/src/schema/chat_messages.ts`, `chat_conversations.ts`, `agents.ts`, `issues.ts`
- Codebase direct inspection — `server/src/services/chat.ts`, `live-events.ts`, `issues.ts`
- Codebase direct inspection — `packages/shared/src/constants.ts` (LIVE_EVENT_TYPES, AGENT_ROLES)
- Codebase direct inspection — `packages/shared/src/validators/issue.ts` (createIssueSchema)
- Codebase direct inspection — `server/src/routes/chat.ts`, `issues.ts` (route factory patterns)
- `.planning/phases/22-agent-streaming/22-RESEARCH.md` — confirmed Phase 22 deliverables
- `.planning/phases/22-agent-streaming/22-01-PLAN.md`, `22-03-PLAN.md` — confirmed SSE stream route and Phase 22 artifacts
- `.planning/REQUIREMENTS.md` — confirmed AGENT-01 through AGENT-07, CHAT-09 scope and traceability
- `.planning/ROADMAP.md` — confirmed Phase 23 success criteria
### Secondary (MEDIUM confidence)
- STATE.md — "auto-creates PM + Engineer" on company onboarding (not verified in companies.ts, inferred)
### Tertiary (LOW confidence)
- None
---
## Metadata
**Confidence breakdown:**
- Standard stack: HIGH — no new packages required; all libraries verified by direct inspection of package.json files
- Architecture: HIGH — all patterns derived from existing codebase; no external library assumptions
- Pitfalls: HIGH — derived from direct schema inspection and known race conditions in upsert patterns
- Test map: HIGH — Vitest pattern confirmed from existing `chat-routes.test.ts`
**Research date:** 2026-04-01
**Valid until:** 2026-05-01 (stable stack; Phase 22 must complete before Phase 23 executes)

View file

@ -0,0 +1,284 @@
import express from "express";
import request from "supertest";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { errorHandler } from "../middleware/index.js";
import { chatRoutes } from "../routes/chat.js";
const mockChatService = vi.hoisted(() => ({
listConversations: vi.fn(),
createConversation: vi.fn(),
getConversation: vi.fn(),
updateConversation: vi.fn(),
softDeleteConversation: vi.fn(),
archiveConversation: vi.fn(),
unarchiveConversation: vi.fn(),
pinConversation: vi.fn(),
unpinConversation: vi.fn(),
listMessages: vi.fn(),
addMessage: vi.fn(),
editMessage: vi.fn(),
getMessageHistory: vi.fn(),
}));
vi.mock("../services/chat.js", () => ({
chatService: () => mockChatService,
}));
function createApp() {
const app = express();
app.use(express.json());
app.use((req, _res, next) => {
(req as any).actor = {
type: "board",
userId: "user-1",
companyIds: ["company-1"],
source: "session",
isInstanceAdmin: false,
};
next();
});
app.use("/api", chatRoutes({} as any));
app.use(errorHandler);
return app;
}
const baseConversation = {
id: "conv-1",
companyId: "company-1",
title: "Test",
agentId: null,
pinnedAt: null,
archivedAt: null,
deletedAt: null,
createdAt: "2024-01-01T00:00:00.000Z",
updatedAt: "2024-01-01T00:00:00.000Z",
};
describe("chat stream routes", () => {
beforeEach(() => {
vi.clearAllMocks();
});
describe("GET /api/conversations/:id/stream", () => {
it("returns 404 when conversation not found", async () => {
mockChatService.getConversation.mockResolvedValue(null);
const res = await request(createApp()).get("/api/conversations/nonexistent/stream");
expect(res.status).toBe(404);
});
it("returns text/event-stream Content-Type", async () => {
mockChatService.getConversation.mockResolvedValue(baseConversation);
mockChatService.getMessageHistory.mockResolvedValue([
{
id: "msg-1",
conversationId: "conv-1",
role: "user",
content: "hello world",
agentId: null,
editedContent: null,
editedAt: null,
createdAt: "2024-01-01T00:00:00.000Z",
effectiveContent: "hello world",
},
]);
mockChatService.addMessage.mockResolvedValue({
id: "msg-2",
conversationId: "conv-1",
role: "assistant",
content: "Echo from agent: hello world",
agentId: null,
editedContent: null,
editedAt: null,
createdAt: "2024-01-01T00:00:01.000Z",
});
const res = await request(createApp())
.get("/api/conversations/conv-1/stream")
.buffer(true)
.parse((res, callback) => {
let data = "";
res.on("data", (chunk: Buffer) => { data += chunk.toString(); });
res.on("end", () => callback(null, data));
});
expect(res.status).toBe(200);
expect(res.headers["content-type"]).toContain("text/event-stream");
});
it("returns X-Accel-Buffering: no header", async () => {
mockChatService.getConversation.mockResolvedValue(baseConversation);
mockChatService.getMessageHistory.mockResolvedValue([]);
mockChatService.addMessage.mockResolvedValue({
id: "msg-2",
conversationId: "conv-1",
role: "assistant",
content: "No message to echo.",
agentId: null,
editedContent: null,
editedAt: null,
createdAt: "2024-01-01T00:00:01.000Z",
});
const res = await request(createApp())
.get("/api/conversations/conv-1/stream")
.buffer(true)
.parse((res, callback) => {
let data = "";
res.on("data", (chunk: Buffer) => { data += chunk.toString(); });
res.on("end", () => callback(null, data));
});
expect(res.headers["x-accel-buffering"]).toBe("no");
});
it("sends token events and a done event", async () => {
mockChatService.getConversation.mockResolvedValue(baseConversation);
mockChatService.getMessageHistory.mockResolvedValue([
{
id: "msg-1",
conversationId: "conv-1",
role: "user",
content: "hi",
agentId: null,
editedContent: null,
editedAt: null,
createdAt: "2024-01-01T00:00:00.000Z",
effectiveContent: "hi",
},
]);
const assistantMsgId = "msg-assistant-1";
mockChatService.addMessage.mockResolvedValue({
id: assistantMsgId,
conversationId: "conv-1",
role: "assistant",
content: "Echo from agent: hi",
agentId: null,
editedContent: null,
editedAt: null,
createdAt: "2024-01-01T00:00:01.000Z",
});
let rawBody = "";
const res = await request(createApp())
.get("/api/conversations/conv-1/stream")
.buffer(true)
.parse((res, callback) => {
res.on("data", (chunk: Buffer) => { rawBody += chunk.toString(); });
res.on("end", () => callback(null, rawBody));
});
expect(res.status).toBe(200);
// Extract data events
const dataLines = rawBody
.split("\n")
.filter((line) => line.startsWith("data: "));
const events = dataLines.map((line) => JSON.parse(line.slice(6)));
// Should have at least one token event
const tokenEvents = events.filter((e) => e.type === "token");
expect(tokenEvents.length).toBeGreaterThan(0);
// Last event should be done with messageId
const lastEvent = events[events.length - 1];
expect(lastEvent).toMatchObject({ type: "done", messageId: assistantMsgId });
});
it("persists assistant message after stream completes", async () => {
mockChatService.getConversation.mockResolvedValue(baseConversation);
mockChatService.getMessageHistory.mockResolvedValue([
{
id: "msg-1",
conversationId: "conv-1",
role: "user",
content: "test message",
agentId: null,
editedContent: null,
editedAt: null,
createdAt: "2024-01-01T00:00:00.000Z",
effectiveContent: "test message",
},
]);
mockChatService.addMessage.mockResolvedValue({
id: "msg-2",
conversationId: "conv-1",
role: "assistant",
content: "Echo from agent: test message",
agentId: null,
editedContent: null,
editedAt: null,
createdAt: "2024-01-01T00:00:01.000Z",
});
await request(createApp())
.get("/api/conversations/conv-1/stream")
.buffer(true)
.parse((res, callback) => {
let data = "";
res.on("data", (chunk: Buffer) => { data += chunk.toString(); });
res.on("end", () => callback(null, data));
});
// addMessage should have been called once (for the assistant response)
expect(mockChatService.addMessage).toHaveBeenCalledWith(
"conv-1",
expect.objectContaining({ role: "assistant" }),
);
});
it("sends :ok SSE comment at start of stream", async () => {
mockChatService.getConversation.mockResolvedValue(baseConversation);
mockChatService.getMessageHistory.mockResolvedValue([]);
mockChatService.addMessage.mockResolvedValue({
id: "msg-2",
conversationId: "conv-1",
role: "assistant",
content: "No message to echo.",
agentId: null,
editedContent: null,
editedAt: null,
createdAt: "2024-01-01T00:00:01.000Z",
});
let rawBody = "";
await request(createApp())
.get("/api/conversations/conv-1/stream")
.buffer(true)
.parse((res, callback) => {
res.on("data", (chunk: Buffer) => { rawBody += chunk.toString(); });
res.on("end", () => callback(null, rawBody));
});
// Should start with SSE comment
expect(rawBody).toMatch(/^:ok\n\n/);
});
});
describe("PUT /api/conversations/:id/messages/:messageId (stream test file)", () => {
it("returns 200 with editedContent", async () => {
const editedMessage = {
id: "msg-1",
conversationId: "conv-1",
role: "user",
content: "original",
agentId: null,
editedContent: "edited content",
editedAt: "2024-01-01T01:00:00.000Z",
createdAt: "2024-01-01T00:00:00.000Z",
};
mockChatService.editMessage.mockResolvedValue(editedMessage);
const res = await request(createApp())
.put("/api/conversations/conv-1/messages/msg-1")
.send({ content: "edited content" });
expect(res.status).toBe(200);
expect(res.body).toMatchObject({
editedContent: "edited content",
editedAt: expect.any(String),
});
});
});
});

View file

@ -108,5 +108,69 @@ export function chatRoutes(db: Db) {
res.json(message);
});
// GET /api/conversations/:id/stream
router.get("/conversations/:id/stream", async (req, res) => {
assertBoard(req);
const conversationId = req.params.id as string;
const conversation = await svc.getConversation(conversationId);
if (!conversation) {
res.status(404).json({ error: "Not found" });
return;
}
// Set SSE headers -- copied from plugins.ts pattern
res.writeHead(200, {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no",
});
res.flushHeaders();
res.write(":ok\n\n");
let aborted = false;
req.on("close", () => { aborted = true; });
// Resolve the agent for this conversation
const agentId = conversation.agentId;
// Get message history for LLM context
const history = await svc.getMessageHistory(conversationId);
// ECHO-STREAM PLACEHOLDER (Phase 22):
// Streams the user's last message back word-by-word to fully exercise the SSE
// pipeline. Phase 23 replaces this block with:
// const adapter = resolveAdapter(agentId);
// for await (const token of adapter.stream(history)) { ... }
const lastUserMsg = history.filter((m) => m.role === "user").at(-1);
const echoContent = lastUserMsg
? `Echo from agent: ${lastUserMsg.content}`
: "No message to echo.";
const tokens = echoContent.split(/(\s+)/);
let accumulated = "";
for (const token of tokens) {
if (aborted) break;
accumulated += token;
res.write(`data: ${JSON.stringify({ type: "token", content: token })}\n\n`);
// Tiny yield to allow abort detection
await new Promise<void>((resolve) => setTimeout(resolve, 5));
}
// Persist assistant message only if stream completed (not aborted)
if (!aborted && accumulated.trim()) {
const assistantMsg = await svc.addMessage(conversationId, {
role: "assistant",
content: accumulated,
agentId,
});
res.write(`data: ${JSON.stringify({ type: "done", messageId: assistantMsg.id })}\n\n`);
}
// Do NOT persist partial messages when aborted (per RESEARCH.md pitfall 4)
res.end();
});
return router;
}