15 KiB
15 KiB
| phase | verified | status | score | overrides_applied | human_verification | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 02-ai-pipeline | 2026-04-10T07:00:00Z | human_needed | 4/5 must-haves verified | 0 |
|
Phase 2: AI Pipeline Verification Report
Phase Goal: Users can submit 1-3 photos and receive a structured NetBox-ready record with AI-extracted specs, suggested category/tags, and a quality gate status reflecting confidence Verified: 2026-04-10T07:00:00Z Status: human_needed Re-verification: No — initial verification
Goal Achievement
Observable Truths (from ROADMAP.md success criteria)
| # | Truth | Status | Evidence |
|---|---|---|---|
| 1 | oMLX serves Gemma 4 on Mac Mini M4 with measured memory budget documented | ? HUMAN NEEDED | Plan 02-04 deferred — oMLX hardware not available on dev machine. Integration test file not created. docs/omlx-setup.md does not exist. |
| 2 | POST /api/intake with 1-3 photos returns serial number, model, manufacturer, specs, category, and tags extracted by AI | ✓ VERIFIED | internal/api/handlers/intake.go — ServeHTTP parses multipart, calls orchestrator.Analyze, returns IntakeResponse with all fields. 6/6 unit tests pass including high/low confidence and rejection of 0 or 4+ photos. |
| 3 | Items with AI confidence below threshold are automatically set to needs_research; high-confidence items advance to indexed | ✓ VERIFIED | internal/ai/orchestrator.go — Analyze() maps confidence < threshold → StatusNeedsResearch, >= threshold → StatusIndexed. 5/5 orchestrator tests cover all escalation paths. TestIntakeHandlerHighConfidence (201, indexed) and TestIntakeHandlerLowConfidence (201, needs_research) confirm handler propagates status correctly. |
| 4 | Quick add mode skips review for high-confidence items and creates the NetBox record in one step | ✓ VERIFIED | internal/api/handlers/intake.go lines 54-77: quickAddEnabled + quickAddThresh fields. TestIntakeHandlerQuickAdd confirms: quick_add_enabled=true, confidence 0.95 → CreateDevice called once, 201 response. Config-driven via cfg.AI.QuickAddEnabled and cfg.AI.QuickAddThreshold in main.go. |
| 5 | Any AI tier (local oMLX, OpenRouter) can be swapped by changing a config JSON value with no code changes | ✓ VERIFIED | internal/ai/client.go:31-32 — NewTierClient uses openai.DefaultConfig(key) + oCfg.BaseURL = cfg.BaseURL. ai_config.json has tier1 (localhost:8000) and tier2 (openrouter.ai) independently configurable. TierConfig.BaseURL and TierConfig.Model both have mapstructure tags wired to viper. Changing ai_config.json values requires no code changes. |
Score: 4/5 truths verified (1 requires human with Mac Mini M4 hardware)
Required Artifacts
| Artifact | Expected | Status | Details |
|---|---|---|---|
internal/ai/types.go |
IntakeRequest, IntakeResult, TierConfig, AIConfig domain types | ✓ VERIFIED | Exists, all four types present with JSON + mapstructure tags |
internal/ai/client.go |
AIClient interface + TierClient production implementation | ✓ VERIFIED | AIClient interface at line 17, TierClient at line 22, NewTierClient at line 30 |
internal/ai/mock.go |
MockAIClient test double with fixture constructors | ✓ VERIFIED | MockAIClient, HighConfidenceResult(), LowConfidenceResult() all present |
internal/ai/prompts/intake.go |
BuildIntakePrompt() returning JSON-extraction prompt template | ✓ VERIFIED | File exists with BuildIntakePrompt(photoCount int) |
internal/ai/orchestrator.go |
Orchestrator with Analyze(ctx, IntakeRequest) → (*IntakeResult, CatalogStatus, error) | ✓ VERIFIED | NewOrchestrator and Analyze both present; all 5 tests pass |
internal/ai/research.go |
ResearchClient interface + NoOpResearchClient stub | ✓ VERIFIED | Both present; NoOpResearchClient returns nil, nil (Phase 7 placeholder) |
internal/queue/handler.go |
NetBoxOpHandler for create_device and patch_custom_fields | ✓ VERIFIED | NewNetBoxOpHandler, OpNetBoxCreateDevice, OpNetBoxPatchCustomFields constants, NetBoxOpsClient interface all present; 6 tests pass |
internal/api/handlers/intake.go |
POST /api/intake multipart handler | ✓ VERIFIED | IntakeHandler, NewIntakeHandler, ServeHTTP with full flow |
internal/api/router.go |
POST /api/intake route registered | ✓ VERIFIED | r.Post("/intake", intakeHandler.ServeHTTP) at line 44 |
cmd/hwlab/main.go |
NewNetBoxOpHandler wired as WAQ handler | ✓ VERIFIED | queue.NewNetBoxOpHandler(nbClient) at line 59; NoOpHandler absent |
internal/config/config.go |
Config struct with AI AIConfig and NetBoxDefault* fields | ✓ VERIFIED | AI ai.AIConfig at line 31; NetBoxDefaultDeviceTypeID/RoleID/SiteID at lines 27-29 |
ai_config.json |
Template config with tier1/tier2/threshold/quick_add settings | ✓ VERIFIED | File exists with all expected fields |
internal/ai/omlx_integration_test.go |
Integration test that skips unless HWLAB_OMLX_URL is set | ✗ MISSING | Plan 02-04 was fully deferred — file not created. Required for AI-01 validation. |
docs/omlx-setup.md |
oMLX installation steps, model tier selection, measured memory budget | ✗ MISSING | Deferred with plan 02-04 — docs/ directory does not exist. |
Key Link Verification
| From | To | Via | Status | Details |
|---|---|---|---|---|
internal/config/config.go |
internal/ai/types.go |
Config.AI ai.AIConfig embeds TierConfig |
✓ WIRED | AI ai.AIConfig \mapstructure:"ai"`` at line 31; AIConfig contains Tier1, Tier2 TierConfig |
internal/ai/client.go |
github.com/sashabaranov/go-openai |
TierClient wraps openai.Client; BaseURL from TierConfig | ✓ WIRED | oCfg := openai.DefaultConfig(cfg.APIKey); oCfg.BaseURL = cfg.BaseURL in NewTierClient |
internal/ai/orchestrator.go |
internal/inventory/quality_gate.go |
Returns inventory.CatalogStatus — StatusIndexed or StatusNeedsResearch | ✓ WIRED | inventory.StatusIndexed and inventory.StatusNeedsResearch used in Analyze() |
internal/queue/handler.go |
internal/netbox/client.go |
NetBoxOpHandler calls CreateDevice or PatchCustomFields based on op.Type | ✓ WIRED | NetBoxOpsClient interface matches *netbox.Client methods; routing via switch op.Type |
internal/api/handlers/intake.go |
internal/ai/orchestrator.go |
IntakeHandler calls orchestrator.Analyze with base64-encoded photos | ✓ WIRED | result, status, err := h.orchestrator.Analyze(r.Context(), ai.IntakeRequest{...}) at line 146 |
internal/api/handlers/intake.go |
internal/netbox/hwid.go |
AllocateNextHWID called after successful AI analysis | ✓ WIRED | hwid, err := h.netboxClient.AllocateNextHWID(r.Context()) at line 156 |
internal/api/handlers/intake.go |
internal/queue/handler.go |
WAQ.Enqueue called with OpNetBoxCreateDevice payload when NetBox unreachable | ✓ WIRED | queue.NewPendingOp(queue.OpNetBoxCreateDevice, ...) at line 193; TestIntakeHandlerNetBoxDown confirms 202 + WAQ enqueue |
internal/ai/omlx_integration_test.go |
http://localhost:8000/v1 |
TierClient with real oMLX endpoint; skips when HWLAB_OMLX_URL unset | ✗ NOT WIRED | File not created (plan 02-04 deferred) |
Data-Flow Trace (Level 4)
| Artifact | Data Variable | Source | Produces Real Data | Status |
|---|---|---|---|---|
intake.go ServeHTTP |
result *ai.IntakeResult |
orchestrator.Analyze() → tier1/tier2 AnalyzePhotos() |
Yes — real HTTP call to oMLX/OpenRouter in production; MockAIClient in tests | ✓ FLOWING (mock in tests, real in prod) |
intake.go ServeHTTP |
hwid string |
netboxClient.AllocateNextHWID() → NetBox API call |
Yes — NetBox assigns sequential HW-XXXXX IDs | ✓ FLOWING |
orchestrator.go Analyze |
result *ai.IntakeResult |
tier1.AnalyzePhotos() then optional tier2.AnalyzePhotos() |
Yes — go-openai calls real LLM endpoint | ✓ FLOWING |
Behavioral Spot-Checks
| Behavior | Command | Result | Status |
|---|---|---|---|
go build ./... compiles clean |
go build ./... && echo BUILD OK |
BUILD OK | ✓ PASS |
| All unit tests pass (no FAIL) | go test ./... -count=1 |
6 packages ok, 0 FAIL | ✓ PASS |
| POST /api/intake rejects 0 photos (400) | go test ./internal/api/handlers/... -run TestIntakeHandlerRejectsZeroPhotos -v |
PASS | ✓ PASS |
| POST /api/intake rejects 4 photos (400) | go test ./internal/api/handlers/... -run TestIntakeHandlerRejectsFourPhotos -v |
PASS | ✓ PASS |
| Orchestrator escalates tier1→tier2 on low confidence | go test ./internal/ai/... -run TestOrchestratorLowConfidenceEscalates -v |
PASS | ✓ PASS |
| WAQ enqueues on NetBox failure (202 response) | go test ./internal/api/handlers/... -run TestIntakeHandlerNetBoxDown -v |
PASS | ✓ PASS |
| NoOpHandler replaced in main.go | grep NoOpHandler cmd/hwlab/main.go |
no output | ✓ PASS |
| oMLX integration test on Mac Mini | requires Mac Mini M4 hardware + oMLX installed | N/A | ? SKIP (hardware) |
Requirements Coverage
| Requirement | Source Plan | Description | Status | Evidence |
|---|---|---|---|---|
| AI-01 | 02-04 | oMLX installed on Mac Mini M4 with Gemma 4 serving OpenAI-compatible API | ? NEEDS HUMAN | Integration test file not created; oMLX hardware setup deferred to human UAT |
| AI-02 | 02-03 | User can upload 1-3 photos and AI extracts serial, model, manufacturer, specs | ✓ SATISFIED | intake.go ServeHTTP; 6 handler tests; IntakeResponse includes all fields |
| AI-03 | 02-03 | AI suggests category, tags, and location for each item | ✓ SATISFIED | IntakeResult.Category, SuggestedTags in response; SyncTags called in handler |
| AI-04 | 02-02 (stub) | AI calls SearXNG via function calling to research product specs | ✓ SATISFIED (stub) | ResearchClient interface + NoOpResearchClient in research.go. REQUIREMENTS.md traceability maps AI-04 to Phase 7 — stub satisfies Phase 2 scope. |
| AI-05 | 02-02 | Orchestrator reviews Tier 1 output for completeness and flags gaps as needs_research | ✓ SATISFIED | orchestrator.Analyze escalates low-confidence results; confidence < threshold → StatusNeedsResearch |
| AI-06 | 02-02 | Tier 2 research agent (OpenRouter) automatically enriches items flagged needs_research | ✓ SATISFIED | Orchestrator escalates to tier2 when tier1 confidence below threshold; tier2 configured as OpenRouter in ai_config.json |
| AI-07 | 02-03 | Quick add mode skips review screen for items with high AI confidence | ✓ SATISFIED | quickAddEnabled + quickAddThresh in IntakeHandler; TestIntakeHandlerQuickAdd confirms one-step NetBox create |
| AI-08 | 02-01 | All AI tiers accessed via single OpenAI-compatible client with configurable base URLs | ✓ SATISFIED | AIClient interface, TierClient wraps go-openai with BaseURL override |
| AI-09 | 02-01 | Provider routing configured via JSON file — swap any tier without code changes | ✓ SATISFIED | ai_config.json drives tier1/tier2 BaseURL + Model; mapstructure bindings confirmed |
Anti-Patterns Found
| File | Line | Pattern | Severity | Impact |
|---|---|---|---|---|
internal/ai/research.go |
22-24 | NoOpResearchClient.Search returns nil, nil |
ℹ️ Info | Intentional Phase 2 stub for Phase 7 SearXNG implementation. ResearchClient interface is not wired to any production path yet — no data flows through this path. Not a blocker. |
Human Verification Required
1. oMLX Memory Validation and Integration Test
Test: On Mac Mini M4, install oMLX and serve Gemma 4 E4B on port 8000. Then create internal/ai/omlx_integration_test.go (template in Plan 02-04) and run:
HWLAB_OMLX_URL=http://localhost:8000/v1 go test -tags integration ./internal/ai/... -run TestOMLXIntegration -v
While the test runs, open Activity Monitor and note the oMLX process "Real Memory" peak.
Document peak memory in docs/omlx-setup.md: "Gemma 4 E4B: X GB real memory on Mac Mini M4 16GB".
Expected: Test PASS. Peak memory for E4B expected ~8-10 GB, leaving sufficient headroom for Go backend (~200 MB) and macOS overhead.
Why human: Requires Apple Silicon Mac Mini M4 hardware. oMLX does not run on Intel/Linux. The integration test scaffold was part of deferred Plan 02-04.
2. Live End-to-End Intake with Real Photo
Test: Start server (go run cmd/hwlab/main.go) and send a real hardware photo:
curl -s -X POST http://localhost:8080/api/intake \
-F "photos=@/path/to/hardware-photo.jpg" | python3 -m json.tool
Expected: JSON response with hw_id (HW-XXXXX format), model, manufacturer, category, specs (non-empty), suggested_tags, confidence score, and catalog_status of "indexed" or "needs_research" depending on AI confidence.
Why human: Requires live oMLX inference on Mac Mini and a running NetBox instance. All dependencies are mocked in unit tests.
3. Config-Driven Tier Swap Smoke Test
Test: Edit ai_config.json to change tier1 model, restart server, send intake request. No code changes should be needed.
Expected: Intake endpoint continues to respond with valid JSON. Tier1 uses the new model name from config.
Why human: Code inspection confirms the mechanism (BaseURL + Model from TierConfig), but live smoke test confirms the full config parse → client construction → API call path with a real endpoint.
Gaps Summary
No blocking gaps in the code artifacts. The phase delivered all planned code for Plans 02-01, 02-02, and 02-03 with all unit tests passing. Plan 02-04 (oMLX integration validation) was explicitly deferred due to hardware unavailability and is tracked in HUMAN-UAT.md.
The two missing artifacts (internal/ai/omlx_integration_test.go and docs/omlx-setup.md) are gated on Mac Mini M4 availability and should be created as part of the human UAT process described above.
AI-04 (SearXNG function calling) is correctly stubbed — REQUIREMENTS.md maps AI-04 to Phase 7, and the ResearchClient interface is in place for that implementation.
Verified: 2026-04-10T07:00:00Z Verifier: Claude (gsd-verifier)