homelabby/.planning/phases/02-ai-pipeline/02-RESEARCH.md

# Phase 2: AI Pipeline - Research

**Researched:** 2026-04-10
**Domain:** Go AI client interface, multipart photo intake, multimodal vision with Gemma 4 via oMLX, three-tier orchestrator, confidence-based quality gate wiring
**Confidence:** HIGH (core patterns from training knowledge, verified against codebase and stack decisions)

---

<user_constraints>
## User Constraints (from CONTEXT.md)

### Locked Decisions

- Single `go-openai` client with configurable BaseURL per tier
- Tier 1: oMLX at http://localhost:8000/v1 (Gemma 4 E4B default)
- Tier 2: OpenRouter at https://openrouter.ai/api/v1 (research agent)
- Tier 3: OpenRouter (Opus for Lab Advisor — deferred to Phase 6)
- Config JSON drives tier routing — no code changes to swap providers
- POST /api/intake accepts multipart/form-data with 1-3 photo files
- Photos encoded as base64 and sent to Gemma 4 vision endpoint
- AI extracts: serial number, model, manufacturer, specs, category, suggested tags
- Confidence score determines catalog_status: high → indexed, low → needs_research
- Config flag enables skip-review flow for high-confidence items (Quick Add mode)
- oMLX may not be installed on dev machine — use mock AI client for unit tests
- Integration tests skip gracefully when oMLX unreachable
- Expose `AIClient` interface so production uses oMLX, tests use mock
- AI config lives in ai_config.json (separate from main config.json)
- Intake handler should use write-ahead queue if NetBox unreachable
- SearXNG function calling deferred to Phase 7

### Claude's Discretion

All implementation details are at Claude's discretion. Use Phase 1 artifacts (NetBox client, quality gate, HW-ID) as building blocks.

### Deferred Ideas (OUT OF SCOPE)

- SearXNG function calling (Phase 7)
- Lab Advisor tier 3 (Phase 6)
- Natural language search (Phase 7)
- Actual Gemma 4 model tuning/fine-tuning
- React UI for intake (Phase 3)
</user_constraints>

---

<phase_requirements>
## Phase Requirements

| ID | Description | Research Support |
|----|-------------|------------------|
| AI-01 | oMLX installed on Mac Mini M4 with Gemma 4 model serving OpenAI-compatible API | oMLX setup guide + mock pattern for dev |
| AI-02 | User can upload 1-3 photos and AI extracts serial number, model, manufacturer, specs via multimodal vision | Multipart form handling + base64 vision message pattern |
| AI-03 | AI suggests category, tags, and location for each item | Structured JSON response from vision prompt |
| AI-04 | AI calls SearXNG via function calling to research product specs (STUB only this phase) | Stub interface only; real impl Phase 7 |
| AI-05 | Orchestrator reviews Tier 1 output for completeness and flags gaps as needs_research | Confidence extraction + quality gate transition |
| AI-06 | Tier 2 research agent (OpenRouter) automatically enriches items flagged needs_research | go-openai BaseURL swap pattern |
| AI-07 | Quick add mode skips review screen for items with high AI confidence | Config flag + threshold comparison |
| AI-08 | All AI tiers accessed via single OpenAI-compatible client with configurable base URLs | go-openai ClientConfig.BaseURL |
| AI-09 | Provider routing configured via JSON file — swap any tier without code changes | ai_config.json schema + factory pattern |
</phase_requirements>

---

## Summary

Phase 2 builds the AI backbone of HWLab: a Go interface hierarchy that decouples test-time mocks from production oMLX/OpenRouter calls, a multipart photo intake handler that encodes images as base64 vision messages, a structured-output extractor that parses Gemma 4 JSON responses into typed `IntakeResult` values, and a three-tier orchestrator that escalates to OpenRouter when Tier 1 confidence falls below threshold.

The key design challenge is keeping the `AIClient` interface minimal enough to mock cleanly while capturing the full vision + JSON-mode call pattern used by go-openai. The confidence score must be embedded in the model's structured output (not inferred post-hoc) because Gemma 4 / OpenAI-compatible APIs do not expose logprobs for vision tasks reliably.

The orchestrator plugs directly into Phase 1's `CatalogUpdater`, `AllocateNextHWID`, `PatchCustomFields`, and `SyncTags` — all four are stable and tested. The WAQ from Phase 1 (Plan 05) is already wired into main.go and is the fallback path when NetBox is unreachable during intake.

**Primary recommendation:** Build the `AIClient` interface and mock first, then the intake handler, then the orchestrator. Keep confidence scoring self-contained inside the AI package — do not leak `float64` confidence values into the service layer; instead expose a typed `CatalogStatus` decision from the orchestrator.

---

## Standard Stack

### Core (Phase 2 additions)

| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| github.com/sashabaranov/go-openai | v1.x | OpenAI-compatible HTTP client | Single client for oMLX + OpenRouter; BaseURL swap is the tier-routing mechanism; already recommended in STACK.md |

**Version verification:**
```bash
go get github.com/sashabaranov/go-openai@latest
# As of 2026-04 training knowledge: v1.36+ is current — verify before install
```
[ASSUMED: exact latest version; run `npm view` equivalent: `go list -m github.com/sashabaranov/go-openai@latest` to confirm]

### Already in go.mod (no new dependencies needed)

| Package | Current Version | Used By Phase 2 |
|---------|-----------------|-----------------|
| github.com/go-chi/chi/v5 | v5.2.5 | POST /api/intake route |
| github.com/spf13/viper | v1.21.0 | ai_config.json loading |
| github.com/google/uuid | v1.6.0 | Intake job ID (already indirect) |
| github.com/redis/go-redis/v9 | v9.18.0 | WAQ fallback on NetBox failure |

### Installation

```bash
cd /home/mikkel/homelabby
go get github.com/sashabaranov/go-openai@latest
```

---

## Architecture Patterns

### Recommended Package Structure (Phase 2 additions)

```
internal/
├── ai/
│   ├── client.go          # AIClient interface + TierClient concrete type
│   ├── mock.go            # MockAIClient for unit tests
│   ├── orchestrator.go    # Three-tier routing + escalation logic
│   ├── types.go           # IntakeRequest, IntakeResult, ConfidenceLevel
│   └── prompts/
│       └── intake.go      # Prompt templates for hardware analysis
├── api/
│   ├── handlers/
│   │   └── intake.go      # POST /api/intake multipart handler (new)
│   └── router.go          # Add intake route (modify existing)
└── config/
    └── config.go          # Add AIConfig fields (modify existing)
```

---

### Pattern 1: AIClient Interface + TierClient

**What:** A minimal Go interface that captures the one call shape Phase 2 needs. `TierClient` wraps `*openai.Client` from go-openai. `MockAIClient` implements the same interface deterministically.

**Why minimal interface:** The interface should expose the behavior, not the library. If the interface requires `*openai.ChatCompletionRequest`, tests must import go-openai. A domain-typed interface (`AnalyzePhotos`) keeps mocks simple.

```go
// Source: training knowledge — standard Go interface pattern [ASSUMED]
// internal/ai/client.go

package ai

import "context"

// AIClient is the single abstraction over any OpenAI-compatible inference backend.
// Production: TierClient wrapping sashabaranov/go-openai.
// Tests: MockAIClient with canned responses.
type AIClient interface {
    AnalyzePhotos(ctx context.Context, req IntakeRequest) (*IntakeResult, error)
}

// TierConfig holds provider configuration for one AI tier.
type TierConfig struct {
    BaseURL  string  `json:"base_url"`
    APIKey   string  `json:"api_key"`
    Model    string  `json:"model"`
    TimeoutS int     `json:"timeout_seconds"`
}

// TierClient is the production AIClient backed by go-openai.
type TierClient struct {
    client *openai.Client
    model  string
}

func NewTierClient(cfg TierConfig) *TierClient {
    config := openai.DefaultConfig(cfg.APIKey)
    config.BaseURL = cfg.BaseURL
    return &TierClient{
        client: openai.NewClientWithConfig(config),
        model:  cfg.Model,
    }
}
```

[VERIFIED: go-openai BaseURL override via `openai.DefaultConfig` + `config.BaseURL` — confirmed pattern from STACK.md and ARCHITECTURE.md]

---

### Pattern 2: Multipart Photo Upload → Base64 Vision Message

**What:** chi handler reads up to 3 files from multipart form, reads each into `[]byte`, encodes to base64 data URL, assembles a `ChatCompletionRequest` with `ImageURL` content parts.

**go-openai vision message shape:** [ASSUMED: standard pattern, consistent with OpenAI API]

```go
// internal/api/handlers/intake.go
// Source: go-openai vision pattern [ASSUMED — matches OpenAI API spec]

func (h *IntakeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    // Parse multipart — 32MB max
    if err := r.ParseMultipartForm(32 << 20); err != nil {
        http.Error(w, "bad multipart", http.StatusBadRequest)
        return
    }

    files := r.MultipartForm.File["photos"]
    if len(files) == 0 || len(files) > 3 {
        http.Error(w, "1-3 photos required", http.StatusBadRequest)
        return
    }

    var photosB64 []string
    for _, fh := range files {
        f, err := fh.Open()
        if err != nil { /* handle */ }
        defer f.Close()
        data, err := io.ReadAll(f)
        if err != nil { /* handle */ }
        // Detect MIME type from first 512 bytes
        mime := http.DetectContentType(data[:min(512, len(data))])
        photosB64 = append(photosB64, fmt.Sprintf("data:%s;base64,%s",
            mime, base64.StdEncoding.EncodeToString(data)))
    }

    result, err := h.ai.AnalyzePhotos(r.Context(), ai.IntakeRequest{
        PhotosBase64: photosB64,
    })
    // ...
}
```

**go-openai vision content parts:** [ASSUMED]

```go
// internal/ai/client.go — TierClient.AnalyzePhotos
func (c *TierClient) AnalyzePhotos(ctx context.Context, req IntakeRequest) (*IntakeResult, error) {
    // Build image content parts
    parts := []openai.ChatMessagePart{
        {
            Type: openai.ChatMessagePartTypeText,
            Text: buildIntakePrompt(),
        },
    }
    for _, b64 := range req.PhotosBase64 {
        parts = append(parts, openai.ChatMessagePart{
            Type: openai.ChatMessagePartTypeImageURL,
            ImageURL: &openai.ChatMessageImageURL{
                URL:    b64,   // data:image/jpeg;base64,...
                Detail: openai.ImageURLDetailAuto,
            },
        })
    }

    resp, err := c.client.CreateChatCompletion(ctx, openai.ChatCompletionRequest{
        Model: c.model,
        Messages: []openai.ChatCompletionMessage{
            {Role: openai.ChatMessageRoleUser, MultiContent: parts},
        },
        // ResponseFormat for JSON mode — see Pattern 3
    })
    // parse resp.Choices[0].Message.Content as JSON
}
```

[ASSUMED: `MultiContent` field name in go-openai ChatCompletionMessage — verify against actual go-openai source after install. Some versions use `Content` string OR `MultiContent []ChatMessagePart`]

**CRITICAL NOTE:** Verify the exact `ChatCompletionMessage` field for multi-content vision after `go get`. The field has been `MultiContent` in v1.20+ but naming may differ. Check with:
```bash
go doc github.com/sashabaranov/go-openai ChatCompletionMessage
```

---

### Pattern 3: Structured JSON Output from Gemma 4

**What:** Instruct the model to return a specific JSON schema via prompt engineering. Use `ResponseFormat` with `JSONObject` type when the endpoint supports it (oMLX/Gemma 4 may not support strict JSON schema mode — fall back to prompt-only).

**IntakeResult schema:**

```go
// internal/ai/types.go
package ai

// IntakeResult is the structured output from any AI tier's photo analysis.
// The model is instructed to return this JSON shape verbatim.
type IntakeResult struct {
    SerialNumber   string   `json:"serial_number"`    // empty string if not visible
    Model          string   `json:"model"`
    Manufacturer   string   `json:"manufacturer"`
    Category       string   `json:"category"`         // e.g. "networking", "cable", "compute"
    Specs          map[string]string `json:"specs"`   // key-value hardware specs
    SuggestedTags  []string `json:"suggested_tags"`
    AINotes        string   `json:"ai_notes"`          // free-form observations
    Confidence     float64  `json:"confidence"`        // 0.0–1.0, self-reported by model
    ConfidenceNote string   `json:"confidence_note"`   // why confidence is low (if < threshold)
}
```

**Prompt pattern for JSON output:**

```go
// internal/ai/prompts/intake.go
func buildIntakePrompt() string {
    return `Analyze the hardware in the provided photo(s) and return ONLY valid JSON matching this schema:
{
  "serial_number": "<string or empty>",
  "model": "<string>",
  "manufacturer": "<string>",
  "category": "<one of: compute, networking, storage, cable, peripheral, component, unknown>",
  "specs": {"<key>": "<value>"},
  "suggested_tags": ["<tag1>", "<tag2>"],
  "ai_notes": "<observations>",
  "confidence": <float 0.0-1.0>,
  "confidence_note": "<reason if confidence < 0.75>"
}
Return ONLY the JSON object. No markdown, no explanation.`
}
```

**JSON mode ResponseFormat (use if supported by endpoint):** [ASSUMED]

```go
// Only set if oMLX / OpenRouter model supports JSON mode
ResponseFormat: &openai.ChatCompletionResponseFormat{
    Type: openai.ChatCompletionResponseFormatTypeJSONObject,
},
```

[ASSUMED: Gemma 4 via oMLX may not support `response_format: json_object` — implement with prompt-only fallback and parse `json.Unmarshal` on the raw response string. If JSON parse fails, treat as low-confidence and escalate.]

---

### Pattern 4: Three-Tier Orchestrator

**What:** Orchestrator holds two `AIClient` instances (tier1, tier2). For each intake request: call tier1, parse result, check confidence. If confidence < threshold OR parse failed, call tier2 with same request. Map confidence to `CatalogStatus` for quality gate.

```go
// internal/ai/orchestrator.go
package ai

type Orchestrator struct {
    tier1     AIClient
    tier2     AIClient
    threshold float64 // from config — default 0.75
}

func NewOrchestrator(tier1, tier2 AIClient, threshold float64) *Orchestrator {
    return &Orchestrator{tier1: tier1, tier2: tier2, threshold: threshold}
}

// Analyze runs tier1, escalates to tier2 if needed, returns result + catalog decision.
func (o *Orchestrator) Analyze(ctx context.Context, req IntakeRequest) (*IntakeResult, inventory.CatalogStatus, error) {
    result, err := o.tier1.AnalyzePhotos(ctx, req)
    if err != nil || result == nil || result.Confidence < o.threshold {
        // Escalate to tier2
        result2, err2 := o.tier2.AnalyzePhotos(ctx, req)
        if err2 == nil && result2 != nil {
            result = result2
        }
        // If tier2 also fails, use tier1 result (or zero result) with NeedsResearch status
    }

    status := inventory.StatusIndexed
    if result == nil || result.Confidence < o.threshold {
        status = inventory.StatusNeedsResearch
    }
    return result, status, nil
}
```

---

### Pattern 5: MockAIClient for Unit Tests

**What:** A deterministic mock that returns canned `IntakeResult` values. Implements `AIClient` interface. Configurable to return high-confidence or low-confidence responses, and optionally errors.

```go
// internal/ai/mock.go
package ai

import "context"

// MockAIClient is a test double for AIClient.
// Configure FixedResult and/or FixedError before use.
type MockAIClient struct {
    FixedResult *IntakeResult
    FixedError  error
    Calls       []IntakeRequest // record of calls for assertions
}

func (m *MockAIClient) AnalyzePhotos(_ context.Context, req IntakeRequest) (*IntakeResult, error) {
    m.Calls = append(m.Calls, req)
    return m.FixedResult, m.FixedError
}

// HighConfidenceResult returns a fixture IntakeResult with confidence 0.95.
func HighConfidenceResult() *IntakeResult {
    return &IntakeResult{
        Model:        "Raspberry Pi 4 Model B",
        Manufacturer: "Raspberry Pi Foundation",
        Category:     "compute",
        Specs:        map[string]string{"ram": "4GB", "cpu": "BCM2711"},
        SuggestedTags: []string{"raspberry-pi", "compute", "arm"},
        Confidence:   0.95,
    }
}

// LowConfidenceResult returns a fixture with confidence 0.40 (below threshold).
func LowConfidenceResult() *IntakeResult {
    return &IntakeResult{
        Model:          "Unknown Device",
        Category:       "unknown",
        Confidence:     0.40,
        ConfidenceNote: "Cannot identify markings clearly",
    }
}
```

---

### Pattern 6: AI Config Schema (ai_config.json)

**What:** Separate JSON config file for AI provider settings. Loaded by viper alongside main config.json. Keeps provider credentials out of the main config.

```json
{
  "tier1": {
    "base_url": "http://localhost:8000/v1",
    "api_key": "local",
    "model": "gemma-4-e4b",
    "timeout_seconds": 30
  },
  "tier2": {
    "base_url": "https://openrouter.ai/api/v1",
    "api_key": "sk-or-...",
    "model": "google/gemma-2-27b-it",
    "timeout_seconds": 60
  },
  "confidence_threshold": 0.75,
  "quick_add_enabled": false,
  "quick_add_threshold": 0.90
}
```

**Config struct extension** (extend existing `internal/config/config.go`):

```go
type AIConfig struct {
    Tier1               TierConfig `mapstructure:"tier1"`
    Tier2               TierConfig `mapstructure:"tier2"`
    ConfidenceThreshold float64    `mapstructure:"confidence_threshold"`
    QuickAddEnabled     bool       `mapstructure:"quick_add_enabled"`
    QuickAddThreshold   float64    `mapstructure:"quick_add_threshold"`
}

// Add to Config struct:
AI AIConfig `mapstructure:"ai"`
```

**Viper loads ai_config.json** by merging it into the same viper instance using `v.MergeInConfig()` with a second config name, or by embedding the AI fields directly in config.json under an `"ai"` key. Simplest: use a single config.json with an `"ai"` section and add `ai_config.json` as an override file via `v.MergeConfigMap`.

[ASSUMED: viper MergeInConfig pattern for secondary config file — standard viper v1 capability]

---

### Pattern 7: Intake Handler Wiring to Phase 1 Components

**What:** The intake handler coordinates: orchestrator (AI analysis) → `AllocateNextHWID` (ID) → `BuildFullCustomFieldsPatch` (fields) → `NetboxClient.CreateDevice` or `PatchCustomFields` → `SyncTags` → `CatalogUpdater.UpdateCatalogStatus` → WAQ fallback.

**Existing Phase 1 APIs the handler calls:**

| Phase 1 Function | Package | Handler Usage |
|-----------------|---------|---------------|
| `AllocateNextHWID(ctx)` | `internal/netbox` | Assign HW-XXXXX ID to new record |
| `BuildFullCustomFieldsPatch(cf)` | `internal/netbox` | Populate custom fields from IntakeResult |
| `PatchCustomFields(ctx, id, patch)` | `internal/netbox` | Write AI data to NetBox device |
| `SyncTags(ctx, tags)` | `internal/netbox` | Create and assign AI-suggested tags |
| `UpdateCatalogStatus(ctx, id, current, next)` | `internal/inventory` | Set indexed or needs_research |
| `waq.Enqueue(ctx, op)` | `internal/queue` | Buffer NetBox write if unreachable |

**Note:** Phase 1's `client.go` has `ListDevices` and `GetDevice` but no `CreateDevice`. The intake handler will need `CreateDevice` — this is a new method on `internal/netbox.Client`. Plan must include this task.

---

### Pattern 8: SearXNG Stub (AI-04)

**What:** AI-04 is listed as "Phase 7" in REQUIREMENTS.md but the CONTEXT.md says "stub only" this phase. Implement a `ResearchClient` interface with a `Search(ctx, query)` method, and a `NoOpResearchClient` that returns empty results. This satisfies the interface requirement without Phase 7 scope creep.

```go
// internal/ai/research.go (stub)
type ResearchClient interface {
    Search(ctx context.Context, query string) ([]SearchResult, error)
}

type NoOpResearchClient struct{}

func (n *NoOpResearchClient) Search(_ context.Context, _ string) ([]SearchResult, error) {
    return nil, nil // Phase 7 will provide real implementation
}
```

---

### Anti-Patterns to Avoid

- **Don't extract confidence from logprobs:** Gemma 4 vision via oMLX does not expose per-token logprobs reliably. Embed `confidence: float` in the JSON output schema and instruct the model to self-report it. [ASSUMED: oMLX logprobs availability is uncertain]
- **Don't store photos:** Per CLAUDE.md stack patterns: "Store the original photo in a local temp directory only until the NetBox record is created; do not persist photos in HWLab itself." Photos are transient.
- **Don't call NetBox from the AI package:** `internal/ai` should not import `internal/netbox`. The intake handler (service layer) orchestrates both. Keep the AI package focused on inference only.
- **Don't share a single go-openai client across tiers:** Each tier gets its own `*openai.Client` instance with its own `BaseURL` and `APIKey`. Mutating a shared client's config is a race condition.
- **Don't block the HTTP response on AI inference:** AI calls take 2-30 seconds. The intake handler should return a job ID immediately and push the result via SSE. (Phase 3 will add SSE — for Phase 2, a synchronous response is acceptable since there's no UI yet, but design the handler to support async promotion.)

---

## Don't Hand-Roll

| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| OpenAI-compatible HTTP client | Custom HTTP calls to oMLX | `sashabaranov/go-openai` | Handles auth headers, retry, streaming, vision content parts |
| Base64 encoding | Custom encoder | `encoding/base64` stdlib | Already in Go stdlib |
| MIME type detection | File extension parsing | `net/http.DetectContentType` | Magic bytes detection from stdlib |
| JSON structured output parsing | Regex extraction | `encoding/json.Unmarshal` | Model output is well-formed JSON when prompted correctly |
| Multipart form parsing | Manual `--boundary` parsing | `r.ParseMultipartForm()` | stdlib net/http handles multipart |

---

## Common Pitfalls

### Pitfall 1: go-openai Vision MultiContent Field Name

**What goes wrong:** Code compiles but `ChatCompletionMessage.MultiContent` field doesn't exist or is named differently in the installed version.

**Why it happens:** go-openai API evolved; older versions used a single `Content string`, newer versions added `MultiContent []ChatMessagePart` for vision. The exact field name depends on the version.

**How to avoid:** After `go get github.com/sashabaranov/go-openai@latest`, run `go doc github.com/sashabaranov/go-openai ChatCompletionMessage` and verify the vision field name before writing handler code.

**Warning signs:** Compiler error "unknown field MultiContent" or images silently not being sent (text-only response from model).

---

### Pitfall 2: oMLX JSON Mode Not Supported

**What goes wrong:** Setting `ResponseFormat: {Type: "json_object"}` causes a 400 error from oMLX because Gemma 4 E4B via oMLX may not support the `response_format` parameter.

**Why it happens:** The `response_format` JSON schema enforcement is an OpenAI-specific feature not universally implemented across all OpenAI-compatible servers.

**How to avoid:** Implement JSON parsing with a fallback: try `json.Unmarshal(content)` on the raw string. If parse fails, treat result as zero-confidence and escalate to tier2. Do not set `ResponseFormat` unless tested against live oMLX.

**Warning signs:** 400 Bad Request from oMLX at inference time with "unsupported parameter" in body.

---

### Pitfall 3: Data URL MIME Type vs go-openai Image URL

**What goes wrong:** Some OpenAI-compatible servers reject `data:image/jpeg;base64,...` data URLs in vision requests and require a `https://` URL instead.

**Why it happens:** The OpenAI spec allows data URLs in `image_url.url` but not all providers implement this.

**How to avoid:** oMLX (local, Gemma 4) should accept data URLs since it's processing locally. Test with a minimal integration test against live oMLX before building the full intake flow. Keep the base64 path for oMLX (tier1) and note that tier2 (OpenRouter) may require a different approach if it doesn't accept data URLs.

**Warning signs:** 400 or inference-time error from oMLX with "invalid image_url".

---

### Pitfall 4: CreateDevice Not in Phase 1 NetBox Client

**What goes wrong:** Intake handler tries to call `netboxClient.CreateDevice(...)` but that method was not built in Phase 1 (only ListDevices, GetDevice, PatchCustomFields were built).

**Why it happens:** Phase 1 was scoped to read/patch existing devices for the quality gate workflow. Intake requires creating new records.

**How to avoid:** Plan must include a Wave 0 task to add `CreateDevice(ctx, name, assetTag) (int, error)` to `internal/netbox/client.go` before the intake handler can be completed.

**go-netbox v4 create pattern:** [ASSUMED — matches observed PATCH pattern from 01-02-SUMMARY]
```go
req := nb.WritableDeviceWithConfigContextRequest{}
req.SetName(name)
req.SetAssetTag(assetTag)
// DeviceRole and DeviceType are required by NetBox — plan must handle defaults
resp, _, err := c.api.DcimAPI.DcimDevicesCreate(ctx).
    WritableDeviceWithConfigContextRequest(req).Execute()
```

**Note:** NetBox `DcimDevicesCreate` requires `device_role` and `device_type` to be set (they are non-nullable FK fields in NetBox v4). The intake handler must either pick sensible defaults or require them to exist in NetBox as pre-provisioned "Unknown" role/type records.

---

### Pitfall 5: Confidence Self-Reporting Calibration

**What goes wrong:** Model returns `"confidence": 0.95` for every item regardless of actual uncertainty, making the threshold useless.

**Why it happens:** LLMs tend to be overconfident in self-reporting. Without explicit calibration prompting, models bias toward high confidence.

**How to avoid:** Add calibration guidance to the intake prompt: "Return confidence < 0.75 if: serial number not visible, item is partially obscured, or manufacturer/model cannot be determined from visual inspection alone." This nudges the model toward honest low-confidence responses for ambiguous photos.

---

### Pitfall 6: WAQ Integration — PendingOp Payload Schema

**What goes wrong:** Intake handler enqueues a `PendingOp` with a payload, but Phase 1's `NoOpHandler` (the WAQ worker) is still installed — it drains the queue silently. Phase 2 must replace `NoOpHandler` with a real NetBox retry handler.

**Why it happens:** Phase 1 explicitly left `NoOpHandler` as a stub: "Phase 2 will replace this with a real retry handler."

**How to avoid:** Phase 2 plan must include a task to implement the real WAQ handler that retries failed NetBox `CreateDevice` / `PatchCustomFields` calls. Define `PendingOp.OpType` constants (e.g., `"netbox.create_device"`, `"netbox.patch_custom_fields"`) and the payload structs for each.

---

## Code Examples

### go-openai Client Configuration for oMLX

```go
// Source: go-openai README pattern, confirmed in STACK.md [ASSUMED version specifics]
import openai "github.com/sashabaranov/go-openai"

cfg := openai.DefaultConfig("local")   // API key "local" for oMLX (no auth)
cfg.BaseURL = "http://localhost:8000/v1"
client := openai.NewClientWithConfig(cfg)
```

### go-openai Client Configuration for OpenRouter

```go
cfg := openai.DefaultConfig("sk-or-your-key-here")
cfg.BaseURL = "https://openrouter.ai/api/v1"
client := openai.NewClientWithConfig(cfg)
```

### Multipart File Reading in chi Handler

```go
// Source: Go stdlib net/http [VERIFIED: stdlib pattern]
r.ParseMultipartForm(32 << 20)  // 32MB max memory
files := r.MultipartForm.File["photos"]
for _, fh := range files {
    f, err := fh.Open()
    defer f.Close()
    data, _ := io.ReadAll(f)
    mime := http.DetectContentType(data[:min(512, len(data))])
    b64 := base64.StdEncoding.EncodeToString(data)
    dataURL := fmt.Sprintf("data:%s;base64,%s", mime, b64)
}
```

### JSON Parse with Fallback

```go
// Source: Go stdlib encoding/json [VERIFIED: stdlib pattern]
var result ai.IntakeResult
content := resp.Choices[0].Message.Content
if err := json.Unmarshal([]byte(content), &result); err != nil {
    // Model returned non-JSON — treat as low confidence, escalate
    return &ai.IntakeResult{Confidence: 0.0}, nil
}
```

### Integration Test Skip Guard (consistent with Phase 1 pattern)

```go
// Source: Phase 1 established pattern (01-02-SUMMARY.md) [VERIFIED: codebase]
func TestAnalyzePhotosLive(t *testing.T) {
    endpoint := os.Getenv("HWLAB_OMLX_ENDPOINT")
    if endpoint == "" {
        t.Skip("HWLAB_OMLX_ENDPOINT not set — skipping live oMLX test")
    }
    // ...
}
```

---

## Validation Architecture

### Test Framework

| Property | Value |
|----------|-------|
| Framework | Go testing stdlib (`go test ./...`) |
| Config file | none — test flags via env vars |
| Quick run command | `go test ./internal/ai/... -run "^Test[^L]" -timeout 30s` |
| Full suite command | `go test ./...` |

### Phase Requirements → Test Map

| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| AI-02 | Photo upload multipart parsing | unit | `go test ./internal/api/handlers/... -run TestIntakeHandler` | Wave 0 |
| AI-02 | Base64 encoding of JPEG | unit | `go test ./internal/ai/... -run TestEncodePhoto` | Wave 0 |
| AI-03 | JSON parse of structured output | unit | `go test ./internal/ai/... -run TestParseIntakeResult` | Wave 0 |
| AI-05 | Confidence below threshold → needs_research | unit | `go test ./internal/ai/... -run TestOrchestratorEscalation` | Wave 0 |
| AI-05 | Confidence above threshold → indexed | unit | `go test ./internal/ai/... -run TestOrchestratorHighConf` | Wave 0 |
| AI-06 | Tier 2 called on tier 1 failure | unit | `go test ./internal/ai/... -run TestOrchestratorTier2Fallback` | Wave 0 |
| AI-07 | Quick add flag honors threshold | unit | `go test ./internal/ai/... -run TestQuickAddMode` | Wave 0 |
| AI-08 | TierClient uses configured BaseURL | unit | `go test ./internal/ai/... -run TestTierClientConfig` | Wave 0 |
| AI-09 | ai_config.json loaded via viper | unit | `go test ./internal/config/... -run TestAIConfig` | Wave 0 |
| AI-01 | oMLX live inference smoke test | integration | `go test ./internal/ai/... -run TestAnalyzePhotosLive` (skip if env unset) | Wave 0 |

### Sampling Rate

- **Per task commit:** `go test ./internal/ai/... ./internal/api/handlers/... -timeout 30s`
- **Per wave merge:** `go test ./...`
- **Phase gate:** Full suite green before `/gsd-verify-work`

### Wave 0 Gaps

- [ ] `internal/ai/client_test.go` — covers AI-08, AI-09 (TierClient config)
- [ ] `internal/ai/orchestrator_test.go` — covers AI-05, AI-06, AI-07
- [ ] `internal/ai/types_test.go` — covers AI-03 (JSON parse)
- [ ] `internal/api/handlers/intake_test.go` — covers AI-02

---

## Security Domain

### Applicable ASVS Categories

| ASVS Category | Applies | Standard Control |
|---------------|---------|-----------------|
| V2 Authentication | no | No auth in solo homelab tool |
| V3 Session Management | no | Stateless REST |
| V4 Access Control | no | Solo operator, no roles |
| V5 Input Validation | yes | Validate photo count (1-3), file size cap, MIME type check |
| V6 Cryptography | no | API keys in config, not in code |

### Known Threat Patterns

| Pattern | STRIDE | Standard Mitigation |
|---------|--------|---------------------|
| Oversized photo upload (DoS) | Denial of Service | `ParseMultipartForm(32 << 20)` caps memory; add explicit per-file size check (e.g., 10MB/photo) |
| AI prompt injection via filename | Tampering | Do not include original filename in AI prompt; use only image bytes |
| API key leakage in logs | Info Disclosure | Never log `TierConfig.APIKey`; use `***` redaction in any debug output |
| Malformed JSON from model | Tampering | Always `json.Unmarshal` into typed struct; ignore extra fields; treat parse failure as low confidence |

---

## Environment Availability

| Dependency | Required By | Available | Version | Fallback |
|------------|------------|-----------|---------|----------|
| oMLX on localhost:8000 | AI-01, Tier 1 inference | Unknown (dev machine) | — | MockAIClient for unit tests; integration tests skip with env guard |
| OpenRouter API key | AI-06, Tier 2 | Unknown | — | Integration tests skip; tier2 returns error, orchestrator falls back to needs_research |
| DragonFlyDB (10.5.0.10) | WAQ fallback | VERIFIED reachable (from 01-05-SUMMARY) | — | WAQ init is non-fatal; see 01-05 pattern |
| NetBox (10.5.0.130:8000) | CreateDevice, PatchCustomFields | Available (integration tests skip on placeholder token) | — | WAQ enqueues ops; real token needed for integration tests |

**Missing dependencies with no fallback:**
- None — all dependencies have mock/skip fallbacks for unit tests.

**Missing dependencies with fallback:**
- oMLX: MockAIClient covers unit tests; integration test skips with `HWLAB_OMLX_ENDPOINT` guard.
- OpenRouter key: Same skip guard pattern.

---

## Open Questions

1. **NetBox device_role and device_type for CreateDevice**
   - What we know: NetBox v4 requires both to be non-null FKs on device creation
   - What's unclear: Should intake auto-create "Unknown" role/type records if absent, or require them pre-provisioned?
   - Recommendation: Phase 1 (Plan 03, provision.go) may have already provisioned these. Check `internal/netbox/provision.go` before planning the CreateDevice task.

2. **Gemma 4 E4B model ID string in oMLX**
   - What we know: CONTEXT.md says `model: "gemma-4-e4b"` as default; oMLX uses the model filename/ID
   - What's unclear: The exact model ID string oMLX uses for Gemma 4 E4B (may be `mlx-community/gemma-4-e4b` or similar)
   - Recommendation: Leave as a config value; user sets the correct model ID once oMLX is installed. Default to `"gemma-4-e4b"` in ai_config.json with a comment.

3. **Synchronous vs async intake response**
   - What we know: AI inference takes 2-30 seconds; Phase 3 adds SSE; no UI in Phase 2
   - What's unclear: Should Phase 2 implement async job IDs now (for Phase 3 to build on) or keep synchronous for simplicity?
   - Recommendation: Implement synchronous for Phase 2 (no UI yet); design the handler to accept a `?async=true` query param stub that returns "not yet implemented" — this reserves the API surface for Phase 3 without blocking Phase 2.

---

## Assumptions Log

| # | Claim | Section | Risk if Wrong |
|---|-------|---------|---------------|
| A1 | go-openai vision content uses `MultiContent []ChatMessagePart` field on `ChatCompletionMessage` | Pattern 2 | Compile error; verify with `go doc` after install |
| A2 | oMLX supports data URL base64 images in vision requests | Pattern 2 | 400 error at inference time; may need to write image to temp file and use URL instead |
| A3 | oMLX may not support `response_format: json_object` | Pattern 3 | Must use prompt-only JSON mode; 400 if ResponseFormat is set |
| A4 | go-openai latest version is v1.36+ | Standard Stack | Run `go get` to verify; version is only needed to confirm stability |
| A5 | Gemma 4 E4B self-reports honest confidence scores with calibration prompting | Pattern 5 pitfall | Threshold becomes useless if model is always overconfident; may need threshold tuning |
| A6 | viper `MergeInConfig` can load ai_config.json as secondary config | Pattern 6 | Config loading fails silently; test config loading in Wave 0 |

---

## Sources

### Primary (HIGH confidence)

- CONTEXT.md `02-CONTEXT.md` — locked decisions for Phase 2 (this session)
- `01-02-SUMMARY.md`, `01-04-SUMMARY.md`, `01-05-SUMMARY.md` — Phase 1 actual implementation (verified codebase state)
- `internal/config/config.go` — existing config struct to extend
- `internal/api/router.go` — existing chi router to add route to
- `go.mod` — confirmed go-openai not yet installed

### Secondary (MEDIUM confidence)

- `ARCHITECTURE.md`, `STACK.md` — project research documents (verified at research time)
- CLAUDE.md stack patterns section — photo intake pattern, AI tier routing pattern

### Tertiary (LOW/ASSUMED)

- go-openai `ChatCompletionMessage.MultiContent` field name — training knowledge, verify post-install
- oMLX `response_format` support status — not tested; marked ASSUMED
- go-openai latest version number — marked ASSUMED

---

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH — go-openai is the decided library; already in STACK.md; pattern for BaseURL swap is verified
- Architecture (interface/mock pattern): HIGH — standard Go interface idiom, consistent with Phase 1 patterns
- go-openai vision API field names: LOW — exact field names require post-install verification
- oMLX JSON mode support: LOW — not tested against live oMLX

**Research date:** 2026-04-10
**Valid until:** 2026-05-10 (go-openai API is stable; oMLX is fast-moving — re-verify JSON mode if oMLX version changes)