diff --git a/.planning/phases/01-foundation/01-RESEARCH.md b/.planning/phases/01-foundation/01-RESEARCH.md new file mode 100644 index 0000000..c9cf933 --- /dev/null +++ b/.planning/phases/01-foundation/01-RESEARCH.md @@ -0,0 +1,716 @@ +# Phase 1: Foundation - Research + +**Researched:** 2026-04-10 +**Domain:** Go project scaffold, NetBox REST API integration, HW-XXXXX ID scheme, catalog quality gate, DragonFlyDB write-ahead queue +**Confidence:** HIGH (verified package versions, confirmed live infrastructure, chi/gin discrepancy resolved) + +--- + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions +All implementation choices are at Claude's discretion — pure infrastructure phase. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions. + +Key infrastructure context from user: +- PostgreSQL at postgresql://homelabby:homelabby_2024_secure@10.5.0.109:5432/homelabby +- DragonFlyDB (Redis-compatible) at redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379 +- NetBox at http://10.5.0.130:8000/api with token homelab-netbox-api-token-2024 +- SearXNG at http://10.5.0.129:8080/search (no auth) +- All credentials stored in .env file +- ClickHouse design system (pure black + neon volt) for UI — not Tokyo Night +- Go backend with chi router, React TypeScript frontend embedded via go:embed +- Single binary deployment + +### Claude's Discretion +All implementation choices are at Claude's discretion — this is a pure infrastructure phase. + +### Deferred Ideas (OUT OF SCOPE) +None — discussion stayed within phase scope. + + +--- + + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|------------------| +| INF-01 | Go backend serves React SPA via go:embed (single binary) | go:embed pattern confirmed; chi v5.2.5 serves static FS | +| INF-02 | Configuration via JSON file + environment variables | viper v1.21.0 handles both; .env file already exists | +| INF-03 | HW-XXXXX sequential ID auto-assigned at intake, stored as NetBox asset_tag | ID allocation strategy documented; NetBox asset_tag field confirmed available | +| NB-01 | System connects to NetBox REST API and performs CRUD on devices, modules, and cables | go-netbox v4.3.0 confirmed; NewAPIClientFor pattern documented | +| NB-02 | Custom fields provisioned in NetBox (hw_id, catalog_status, product_url, firmware_version, test_date, test_data, ai_notes, photo_urls) | /api/extras/custom-fields/ endpoint exists; provisioning via REST API or management script documented | +| NB-03 | netbox-inventory plugin installed and configured for asset lifecycle tracking | Plugin install requires NetBox container access; research documented | +| NB-04 | Location hierarchy created in NetBox (Site → Location → Rack per PRD section 7.6) | DCIM API endpoints for site/location/rack creation documented | +| NB-05 | Write-ahead queue buffers NetBox operations during network issues (DragonFlyDB available) | go-redis v9 + DragonFlyDB pattern; Redis LIST as queue; port 6379 confirmed reachable | +| NB-06 | Catalog quality gate enforced via catalog_status field (draft → indexed → needs_research → researched → complete) | State machine pattern in Go; NetBox custom field as backing store | +| NB-07 | AI-generated tags synced to NetBox tag system | /api/extras/tags/ endpoint; go-netbox ExtrasAPI confirmed | + + +--- + +## Summary + +This phase delivers the Go binary scaffold and all NetBox integration plumbing before any AI pipeline, frontend, or USB hardware work begins. It has five distinct work streams: (1) Go project scaffold with chi router, go:embed frontend stub, and viper config; (2) NetBox connectivity via go-netbox v4.3.0 with typed custom field wrappers; (3) custom field and location hierarchy provisioning via NetBox REST API; (4) HW-XXXXX sequential ID allocation backed by a NetBox-queried counter; and (5) DragonFlyDB write-ahead queue using go-redis v9. + +The critical infrastructure blocker to resolve in Wave 0: the NetBox API token stored as `homelab-netbox-api-token-2024` in `.env` is a placeholder string — NetBox tokens are 40-character hex strings. The real token must be generated via the NetBox UI before any Go code can interact with NetBox. All other infrastructure (DragonFlyDB port 6379, PostgreSQL port 5432, NetBox port 8000) is confirmed reachable from this host. + +**Primary recommendation:** Build in this order — scaffold first (chi server + health endpoint + go:embed stub), then NetBox client with connection test, then custom field provisioning script, then WAQ layer, then quality gate state machine. Each step is independently testable. + +--- + +## Standard Stack + +### Core (Phase 1 specific) + +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| github.com/go-chi/chi/v5 | v5.2.5 | HTTP router | CONTEXT.md locked decision; stdlib-compatible handlers; lightweight middleware chain | +| github.com/netbox-community/go-netbox/v4 | v4.3.0 | NetBox REST API client | Official client; generated from NetBox OpenAPI spec; typed structs for all endpoints | +| github.com/redis/go-redis/v9 | v9.18.0 | DragonFlyDB client (write-ahead queue) | Official Redis client for Go; fully compatible with Redis 7.0+; DragonFlyDB is Redis-protocol compatible | +| github.com/mattn/go-sqlite3 | v1.14.42 | SQLite for advisor chat history | Already in design; WAQ uses DragonFlyDB not SQLite | +| github.com/spf13/viper | v1.21.0 | Config (JSON file + env vars) | INF-02 requires both JSON file and env var config; viper handles both with precedence | +| github.com/google/uuid | v1.6.0 | UUID generation | Utility; used internally where needed | + +**Version verification:** [VERIFIED: proxy.golang.org registry] — all versions checked 2026-04-10. + +### Supporting + +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| github.com/golang-migrate/migrate/v4 | v4.19.1 | SQLite schema migrations | Phase 1 creates the WAQ SQLite schema for local queue persistence (fallback if DragonFlyDB unavailable) | + +### Alternatives Considered + +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| chi | gin | CONTEXT.md specifies chi; gin has more magic middleware; chi uses net/http interfaces directly | +| go-redis/v9 | Valkey/rueidis | go-redis is the most widely tested against DragonFlyDB; rueidis is faster but less community-validated with DragonflyDB | +| viper | godotenv + custom | viper handles layered config (file + env + defaults) in one library; less boilerplate | + +**Installation:** +```bash +go mod init git.georgsen.dk/hwlab +go get github.com/go-chi/chi/v5@v5.2.5 +go get github.com/netbox-community/go-netbox/v4@v4.3.0 +go get github.com/redis/go-redis/v9@v9.18.0 +go get github.com/mattn/go-sqlite3@v1.14.42 +go get github.com/spf13/viper@v1.21.0 +go get github.com/google/uuid@v1.6.0 +go get github.com/golang-migrate/migrate/v4@v4.19.1 +``` + +--- + +## Architecture Patterns + +### Recommended Project Structure (Phase 1 scope) + +``` +hwlab/ +├── cmd/ +│ └── hwlab/ +│ └── main.go # Wire-up only: init config, create server, start +├── internal/ +│ ├── api/ +│ │ ├── router.go # chi router setup, middleware, route registration +│ │ └── handlers/ +│ │ └── health.go # GET /api/health — Phase 1 success criterion 1 +│ ├── config/ +│ │ └── config.go # viper-backed config struct; loads .env + config.json +│ ├── netbox/ +│ │ ├── client.go # go-netbox v4 wrapper; NewAPIClientFor; typed methods +│ │ ├── custom_fields.go # Custom field read/write wrappers (asymmetric types) +│ │ ├── hwid.go # HW-XXXXX sequential ID allocation +│ │ └── provision.go # Custom field + location hierarchy provisioning +│ ├── inventory/ +│ │ ├── quality_gate.go # catalog_status state machine (6 states) +│ │ └── types.go # Domain types: Item, CatalogStatus, HardwareRecord +│ └── queue/ +│ ├── waq.go # DragonFlyDB-backed write-ahead queue +│ └── worker.go # Flush goroutine: retry pending NetBox ops on reconnect +├── web/ +│ └── dist/ +│ └── index.html # Stub React SPA (placeholder for Phase 3) +├── frontend/ # React TypeScript source (scaffolded, minimal in Phase 1) +│ ├── src/ +│ │ └── App.tsx # Placeholder "HWLab is running" page +│ ├── package.json +│ └── vite.config.ts +├── migrations/ # SQLite migrations (golang-migrate format) +│ └── 001_create_queue.up.sql +├── .env # Already exists — credentials +├── config.json # App config (non-secret: timeouts, thresholds, etc.) +└── Makefile # build, dev, test, embed-frontend, provision-netbox +``` + +### Pattern 1: chi Router with go:embed Stub Frontend (INF-01) + +**What:** The Go binary embeds the React build output (`web/dist`) at compile time. The chi router serves the SPA on all non-API routes. API routes are prefixed `/api/`. + +**When to use:** Single binary deployment — no separate static file server needed. + +```go +// Source: [VERIFIED: go:embed official docs + chi docs] +//go:embed web/dist +var staticFiles embed.FS + +func NewRouter(cfg *config.Config) http.Handler { + r := chi.NewRouter() + + // Middleware + r.Use(middleware.Logger) + r.Use(middleware.Recoverer) + r.Use(middleware.RealIP) + + // API routes + r.Route("/api", func(r chi.Router) { + r.Get("/health", handlers.Health) + }) + + // SPA fallback — serve index.html for all non-API routes + staticFS, _ := fs.Sub(staticFiles, "web/dist") + fileServer := http.FileServer(http.FS(staticFS)) + r.Handle("/*", fileServer) + + return r +} +``` + +**Phase 1 note:** The stub `web/dist/index.html` can be a minimal HTML page. The full React SPA is built in Phase 3. + +### Pattern 2: go-netbox v4 Client Initialization (NB-01) + +**What:** `netbox.NewAPIClientFor` creates a fully configured client. All NetBox calls go through this client — never raw HTTP. + +```go +// Source: [VERIFIED: go-netbox v4.3.0 README via github.com/netbox-community/go-netbox] +import netbox "github.com/netbox-community/go-netbox/v4" + +func NewNetBoxClient(url, token string) *netbox.APIClient { + c := netbox.NewAPIClientFor(url, token) + return c +} + +// Example: list devices +res, _, err := client.DcimAPI. + DcimDevicesList(ctx). + Limit(10). + Execute() +``` + +### Pattern 3: Custom Field Read/Write Asymmetry (NB-02, critical pitfall) + +**What:** NetBox REST API returns custom fields in a nested read format but expects a flat format on write (PATCH). This is a well-documented gotcha. Use separate Go types. + +```go +// Source: [CITED: github.com/netbox-community/netbox/discussions/13425] + +// Read format — what NetBox returns +type CustomFieldsRead struct { + HwID *string `json:"hw_id"` + CatalogStatus *string `json:"catalog_status"` + ProductURL *string `json:"product_url"` + PhotoURLs []string `json:"photo_urls"` + AINodes *string `json:"ai_notes"` + FirmwareVer *string `json:"firmware_version"` + TestDate *string `json:"test_date"` + TestData *string `json:"test_data"` // JSON string +} + +// Write format — what NetBox expects on PATCH +// For text/URL fields: send string value directly +// For object/choice fields: send slug or ID +type CustomFieldsPatch map[string]interface{} + +func BuildCustomFieldsPatch(hwID, status string, photoURLs []string) CustomFieldsPatch { + return CustomFieldsPatch{ + "hw_id": hwID, + "catalog_status": status, + "photo_urls": photoURLs, // array is fine for multi-object fields + } +} +``` + +**Critical test:** After every PATCH, immediately GET and assert the field value matches. HTTP 200 does NOT confirm the field was written. + +### Pattern 4: HW-XXXXX Sequential ID Allocation (INF-03) + +**What:** HW-IDs are sequential 5-digit zero-padded integers stored as `asset_tag` in NetBox. Allocation uses an optimistic lock: query the highest existing asset_tag, increment by 1, attempt to create the record. If creation fails due to conflict, retry with the next ID. + +**Why not UUID or NetBox auto-ID:** UUIDs are not human-readable for label printing; NetBox device IDs are internal integers not shown on labels; asset_tag is a user-visible field designed for exactly this purpose. + +```go +// Source: [ASSUMED — standard pattern for sequential ID in distributed system] +func (c *Client) AllocateNextHWID(ctx context.Context) (string, error) { + // Query NetBox for the highest existing asset_tag matching HW-\d{5} + // Try up to 3 times to handle concurrent allocations + for attempt := 0; attempt < 3; attempt++ { + highest, err := c.getHighestHWID(ctx) + if err != nil { + return "", err + } + candidate := fmt.Sprintf("HW-%05d", highest+1) + // Attempt to create a placeholder record with this asset_tag + // NetBox enforces asset_tag uniqueness — conflict = retry + ok, err := c.reserveHWID(ctx, candidate) + if err == nil && ok { + return candidate, nil + } + } + return "", errors.New("HW-ID allocation failed after 3 attempts") +} +``` + +### Pattern 5: DragonFlyDB Write-Ahead Queue (NB-05) + +**What:** Failed or queued NetBox operations are pushed onto a Redis LIST in DragonFlyDB. A background goroutine polls the queue and retries on reconnect. The queue key is `hwlab:netbox:pending_ops`. + +```go +// Source: [VERIFIED: go-redis/v9 README + DragonFlyDB Redis compatibility confirmed] +import "github.com/redis/go-redis/v9" + +func NewWAQClient(redisURL string) (*redis.Client, error) { + opt, err := redis.ParseURL(redisURL) + if err != nil { + return nil, err + } + client := redis.NewClient(opt) + // Verify connectivity + if err := client.Ping(context.Background()).Err(); err != nil { + return nil, fmt.Errorf("DragonFlyDB unreachable: %w", err) + } + return client, nil +} + +const queueKey = "hwlab:netbox:pending_ops" + +// Enqueue a pending operation (serialize as JSON) +func (q *WAQ) Enqueue(ctx context.Context, op PendingOp) error { + data, _ := json.Marshal(op) + return q.rdb.RPush(ctx, queueKey, data).Err() +} + +// Worker: BLPOP blocks until an item is available, retries NetBox call +func (q *WAQ) RunWorker(ctx context.Context) { + for { + result, err := q.rdb.BLPop(ctx, 5*time.Second, queueKey).Result() + if err == redis.Nil { + continue // timeout, no items + } + if err != nil { + // Redis connection lost — back off + time.Sleep(10 * time.Second) + continue + } + q.processOp(ctx, result[1]) + } +} +``` + +**Queue operation schema:** +```go +type PendingOp struct { + ID string `json:"id"` // UUID + Type string `json:"type"` // "create_device", "patch_custom_fields", etc. + Payload json.RawMessage `json:"payload"` + CreatedAt time.Time `json:"created_at"` + Attempts int `json:"attempts"` +} +``` + +### Pattern 6: Catalog Status Quality Gate (NB-06) + +**What:** The `catalog_status` custom field drives a 5-state lifecycle enforced in Go code — NetBox stores the value, Go enforces valid transitions. + +```go +// Source: [ASSUMED — standard state machine pattern] +type CatalogStatus string + +const ( + StatusDraft CatalogStatus = "draft" + StatusIndexed CatalogStatus = "indexed" + StatusNeedsResearch CatalogStatus = "needs_research" + StatusResearched CatalogStatus = "researched" + StatusComplete CatalogStatus = "complete" +) + +var validTransitions = map[CatalogStatus][]CatalogStatus{ + StatusDraft: {StatusIndexed}, + StatusIndexed: {StatusNeedsResearch, StatusResearched}, + StatusNeedsResearch: {StatusResearched}, + StatusResearched: {StatusComplete}, + StatusComplete: {}, // terminal +} + +func (s CatalogStatus) CanTransitionTo(next CatalogStatus) bool { + allowed, ok := validTransitions[s] + if !ok { + return false + } + for _, a := range allowed { + if a == next { + return true + } + } + return false +} +``` + +### Pattern 7: viper Config (INF-02) + +**What:** viper loads from `config.json` file first, then overlays environment variables. Env var names are prefixed `HWLAB_`. The `.env` file is loaded via godotenv before viper. + +```go +// Source: [VERIFIED: viper v1.21.0 docs + .env file already at /home/mikkel/homelabby/.env] +import ( + "github.com/spf13/viper" + "github.com/joho/godotenv" +) + +type Config struct { + Port int `mapstructure:"port"` + NetBoxURL string `mapstructure:"netbox_url"` + NetBoxToken string `mapstructure:"netbox_token"` + DragonFlyURL string `mapstructure:"dragonfly_url"` + DatabaseURL string `mapstructure:"database_url"` +} + +func Load() (*Config, error) { + _ = godotenv.Load() // load .env if present; ignore error if file missing + viper.SetEnvPrefix("HWLAB") + viper.AutomaticEnv() + viper.SetConfigName("config") + viper.SetConfigType("json") + viper.AddConfigPath(".") + _ = viper.ReadInConfig() // config.json is optional + + var cfg Config + return &cfg, viper.Unmarshal(&cfg) +} +``` + +**Note:** Add `github.com/joho/godotenv` or inline `.env` loading. viper does not load `.env` files natively. + +### Anti-Patterns to Avoid + +- **Token hardcoded in source:** Load `HWLAB_NETBOX_TOKEN` from env only. The `.env` file is already `.gitignore`'d (verify this before first commit). +- **Calling NetBox directly from handlers:** All NetBox calls go through `internal/netbox` package. No raw HTTP to NetBox in handlers or services. +- **Single struct for custom field read and write:** The read and write representations differ. Define `CustomFieldsRead` and `CustomFieldsPatch` separately. +- **Synchronous NetBox writes in the intake path:** Every NetBox write operation goes through the WAQ first. The WAQ flushes asynchronously. +- **Using gin instead of chi:** CONTEXT.md specifies chi. The STACK.md recommends gin, but CONTEXT.md is the locked decision and takes precedence. + +--- + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| NetBox REST API client | Custom HTTP/JSON client | go-netbox/v4 | 200+ typed endpoints from OpenAPI spec; hand-rolled breaks on any schema change | +| Redis LIST queue client | Custom TCP queue | go-redis/v9 | BLPOP, connection pooling, reconnect logic built-in | +| Config layering | Manual env var parsing | viper | JSON file + env override is 3 lines with viper | +| HTTP routing | Custom mux | chi v5 | Middleware chain, URL params, groups are all built into chi | + +--- + +## Common Pitfalls + +### Pitfall 1: NetBox Token is a Placeholder + +**What goes wrong:** The token `homelab-netbox-api-token-2024` in `.env` returns `{"detail": "Invalid v1 token"}` from the NetBox API. No Go code can reach NetBox until a real 40-character hex token is created. + +**Why it happens:** The `.env` file was populated with a descriptive placeholder name rather than a real token value. NetBox v4 tokens are 40-character hex strings generated via the NetBox UI or API. + +**How to avoid:** Wave 0 task: log in to NetBox UI at http://10.5.0.130:8000, navigate to Admin → API Tokens, generate a real token, update `.env` with the actual value. Verify with `curl -H "Authorization: Token " http://10.5.0.130:8000/api/status/` — expect HTTP 200. + +**Warning signs:** Go code returns `Invalid v1 token` error from any NetBox API call. + +### Pitfall 2: Custom Field Write/Read Asymmetry + +**What goes wrong:** PATCH returns HTTP 200 but the custom field is not updated in NetBox. Go code uses the same struct for read and write. + +**Why it happens:** NetBox REST API returns nested objects on read but expects scalar values or IDs on write. Sending the read-format payload on PATCH silently fails or ignores the field. + +**How to avoid:** Use separate Go types for read (`CustomFieldsRead`) and write (`CustomFieldsPatch map[string]interface{}`). Write an integration test: PATCH a field, immediately GET, assert the value matches. + +**Warning signs:** NetBox UI shows field empty after a write that returned HTTP 200. + +### Pitfall 3: DragonFlyDB Connection Refused vs. Auth Error + +**What goes wrong:** `go-redis/v9` returns "WRONGPASS" error even when the URL looks correct, because the password in the URL contains special characters (`/`). + +**Why it happens:** The DragonFlyDB password `nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI` contains forward slashes which must be URL-encoded when embedded in a Redis URL. `redis.ParseURL` may handle this correctly, but literal URL construction may not. + +**How to avoid:** Use `redis.ParseURL` (which handles URL encoding) rather than manually constructing the options struct. If `ParseURL` fails, use `redis.Options{Addr: "10.5.0.10:6379", Password: "nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI"}` with the raw string (not URL-encoded). + +**Warning signs:** `WRONGPASS invalid username-password pair` or `ERR Client sent AUTH, but no password is set` errors. + +### Pitfall 4: go:embed Requires Build-Time Files + +**What goes wrong:** `//go:embed web/dist` fails at build time if the `web/dist` directory does not exist or is empty. + +**Why it happens:** go:embed is resolved at compile time. An empty or missing embedded directory is a build error. + +**How to avoid:** Create a minimal `web/dist/index.html` stub immediately in Wave 0 before writing any Go code that imports the embed. The Makefile `build` target should run `vite build` (or create the stub) before `go build`. + +**Warning signs:** `pattern web/dist: no matching files found` during `go build`. + +### Pitfall 5: NetBox Custom Field Provisioning Requires Admin Token + +**What goes wrong:** Creating custom fields via `/api/extras/custom-fields/` POST requires an admin-level token. A read-only or limited token returns HTTP 403. + +**Why it happens:** Custom field creation is a schema-level operation in NetBox, gated on staff/admin status. + +**How to avoid:** The provisioning script (used in NB-02) must use the admin token. The same token used for normal CRUD operations should work if the user has admin status. Verify the token's permissions before the provisioning wave. + +**Warning signs:** HTTP 403 on POST to `/api/extras/custom-fields/`. + +--- + +## Code Examples + +### Health Endpoint (INF-01) + +```go +// Source: [VERIFIED: chi v5.2.5 stdlib-compatible handlers] +// internal/api/handlers/health.go +package handlers + +import ( + "encoding/json" + "net/http" + "time" +) + +type HealthResponse struct { + Status string `json:"status"` + Timestamp time.Time `json:"timestamp"` + Version string `json:"version"` +} + +func Health(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(HealthResponse{ + Status: "ok", + Timestamp: time.Now().UTC(), + Version: "0.1.0", + }) +} +``` + +### NetBox Custom Field Provisioning Script (NB-02) + +NetBox exposes `/api/extras/custom-fields/` for programmatic custom field creation. Each custom field must specify: `name`, `label`, `type` (text/url/integer/json/multiline-text), `object_types` (which NetBox models it applies to — e.g., `["dcim.device"]`). + +```go +// Source: [VERIFIED: NetBox REST API docs — extras/custom-fields endpoint confirmed via browser access] +// internal/netbox/provision.go + +type CustomFieldDef struct { + Name string `json:"name"` + Label string `json:"label"` + Type string `json:"type"` // "text", "url", "integer", "json", "multiline-text" + ObjectTypes []string `json:"object_types"` // e.g., ["dcim.device"] + Required bool `json:"required"` + Default *string `json:"default,omitempty"` + Description string `json:"description,omitempty"` +} + +var HWLabCustomFields = []CustomFieldDef{ + {Name: "hw_id", Label: "HW ID", Type: "text", ObjectTypes: []string{"dcim.device"}, Required: false}, + {Name: "catalog_status", Label: "Catalog Status", Type: "text", ObjectTypes: []string{"dcim.device"}, Required: false}, + {Name: "product_url", Label: "Product URL", Type: "url", ObjectTypes: []string{"dcim.device"}, Required: false}, + {Name: "firmware_version", Label: "Firmware Version", Type: "text", ObjectTypes: []string{"dcim.device"}, Required: false}, + {Name: "test_date", Label: "Test Date", Type: "text", ObjectTypes: []string{"dcim.device"}, Required: false}, + {Name: "test_data", Label: "Test Data (JSON)", Type: "json", ObjectTypes: []string{"dcim.device"}, Required: false}, + {Name: "ai_notes", Label: "AI Notes", Type: "multiline-text", ObjectTypes: []string{"dcim.device"}, Required: false}, + {Name: "photo_urls", Label: "Photo URLs", Type: "multiline-text", ObjectTypes: []string{"dcim.device"}, Required: false}, +} +``` + +**Provisioning approach:** A `make provision-netbox` Makefile target runs a Go command (`cmd/provision/main.go`) that iterates `HWLabCustomFields`, checks if each already exists (GET by name), and creates it if not. Idempotent — safe to re-run. + +### Location Hierarchy Provisioning (NB-04) + +``` +NetBox hierarchy: Organization → Site → Location → Rack +PRD section 7.6 specifies: Site → Location → Rack +``` + +Create via DCIM API endpoints: +- `POST /api/dcim/sites/` — create site "HWLab Home" +- `POST /api/dcim/locations/` — create locations (Desk, Rack, Storage, etc.) +- `POST /api/dcim/racks/` — create racks under locations + +--- + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| go-netbox v3 | go-netbox v4.3.0 | NetBox 4.x release | Not backwards compatible; v4 required for NetBox 4.x API | +| redis/v8 go-redis | redis/v9 go-redis | 2022 | v9 uses `context.Context` consistently; DragonFlyDB compatibility confirmed with v9 | +| gin router | chi v5 | User decision | chi uses stdlib-compatible `http.Handler`; no framework-specific contexts needed | +| Custom WAQ in SQLite | DragonFlyDB LIST queue | User decision (NB-05) | DragonFlyDB already in homelab; BLPOP gives blocking pop with no polling; Redis LIST is naturally ordered | + +**Deprecated/outdated:** +- `tarm/serial`: Last commit 2018 — use `go.bug.st/serial` instead (not relevant Phase 1, relevant Phase 4+) +- `gin-gonic/gin`: Not deprecated but CONTEXT.md specifies chi — use chi + +--- + +## Assumptions Log + +| # | Claim | Section | Risk if Wrong | +|---|-------|---------|---------------| +| A1 | HW-XXXXX IDs are stored as `asset_tag` in NetBox | Architecture Patterns — Pattern 4 | NetBox may not have asset_tag available on all object types; may need a different field or pure custom field | +| A2 | `catalog_status` custom field type is "text" (not a NetBox "choice" field) | Code Examples | If a "choice" type is used, valid values must be pre-registered in NetBox UI; text is more flexible but loses server-side validation | +| A3 | DragonFlyDB password with forward slashes is handled correctly by `redis.ParseURL` | Common Pitfalls | If not, WAQ initialization fails; fallback is to use `redis.Options` struct directly | +| A4 | The provisioning Go command runs with admin-level NetBox permissions | Don't Hand-Roll | If the token is not admin, custom field creation returns 403 | +| A5 | netbox-inventory plugin (NB-03) is installable via pip in the NetBox LXC | Phase Requirements | The LXC may not have a compatible Python/NetBox environment; requires SSH access to LXC 130 | + +--- + +## Open Questions + +1. **Real NetBox API token** + - What we know: The `.env` file has a placeholder string `homelab-netbox-api-token-2024` + - What's unclear: The real token value; whether the current user account has admin permissions to create custom fields + - Recommendation: Wave 0 first task — generate real token via NetBox UI, update `.env` + +2. **netbox-inventory plugin installation (NB-03)** + - What we know: Plugin exists at github.com/ArnesSI/netbox-inventory; requires SSH to LXC 130 + - What's unclear: Whether the LXC 130 has internet access for pip install, or if the plugin must be transferred manually + - Recommendation: Treat as a manual pre-phase step with a Makefile verification target + +3. **photo_urls field type** + - What we know: The requirement lists `photo_urls` as a custom field + - What's unclear: NetBox doesn't have a native array-of-URLs field type. Options: JSON field (store as JSON array string), multiline-text (one URL per line), or multiple object attachments + - Recommendation: Use `json` field type and store as a JSON array string; parse in Go + +4. **PRD vs. REQUIREMENTS.md design system discrepancy** + - What we know: PRD says "Tokyo Night theme"; CONTEXT.md and PROJECT.md say "ClickHouse design system (pure black + neon volt)" + - Recommendation: CONTEXT.md wins — use ClickHouse design system. The stub Phase 1 frontend needs no real styling. + +--- + +## Environment Availability + +| Dependency | Required By | Available | Version | Fallback | +|------------|------------|-----------|---------|----------| +| NetBox (http://10.5.0.130:8000) | NB-01 through NB-07 | Confirmed (HTTP 302) | Unknown — server: granian | None — blocking | +| DragonFlyDB (10.5.0.10:6379) | NB-05 WAQ | Confirmed (port open) | Redis-compatible | SQLite queue fallback if needed | +| PostgreSQL (10.5.0.109:5432) | INF-02 | Confirmed (port open) | Unknown | Not needed in Phase 1 | +| NetBox API token | All NB-* | Placeholder only [ASSUMED broken] | N/A | Must resolve before any NB work | +| Go toolchain | INF-01 | Not found on this host (Linux Proxmox mgmt) | N/A | Available on Mac Mini M4 (dev machine) | +| Node.js / npm | Frontend scaffold | v24.12.0 / 11.6.2 on this host | v24.12.0 | Available | +| netbox-inventory plugin | NB-03 | Unknown — requires SSH to LXC 130 | N/A | Manual install task | + +**Missing dependencies with no fallback:** +- Real NetBox API token — must be generated before Wave 2 (NetBox integration work) + +**Missing dependencies with fallback:** +- Go toolchain on this Proxmox host — development happens on Mac Mini M4 where Go is expected; all Go commands in the plan should be understood as running on the Mac Mini + +--- + +## Validation Architecture + +### Test Framework + +| Property | Value | +|----------|-------| +| Framework | Go standard `testing` package + `net/http/httptest` | +| Config file | None — Go testing is built-in | +| Quick run command | `go test ./... -short` | +| Full suite command | `go test ./... -v -count=1` | + +### Phase Requirements → Test Map + +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| INF-01 | Health endpoint returns 200 JSON | unit | `go test ./internal/api/... -run TestHealth` | Wave 0 | +| INF-01 | go:embed serves index.html on `/` | unit | `go test ./internal/api/... -run TestStaticFiles` | Wave 0 | +| INF-02 | Config loads from .env + JSON file | unit | `go test ./internal/config/... -run TestConfigLoad` | Wave 0 | +| INF-03 | HW-XXXXX ID format is HW-00001 | unit | `go test ./internal/netbox/... -run TestHWIDFormat` | Wave 0 | +| NB-01 | NetBox client connects and returns device list | integration | `go test ./internal/netbox/... -run TestNetBoxConnect -tags integration` | Wave 0 | +| NB-02 | Custom field round-trip: write then read matches | integration | `go test ./internal/netbox/... -run TestCustomFieldRoundTrip -tags integration` | Wave 0 | +| NB-05 | WAQ enqueues and dequeues an operation | unit | `go test ./internal/queue/... -run TestWAQEnqueueDequeue` | Wave 0 | +| NB-05 | WAQ retries after DragonFlyDB reconnect | integration | `go test ./internal/queue/... -run TestWAQReconnect -tags integration` | Wave 0 | +| NB-06 | Valid catalog_status transitions succeed | unit | `go test ./internal/inventory/... -run TestCatalogStatusTransitions` | Wave 0 | +| NB-06 | Invalid catalog_status transitions are rejected | unit | `go test ./internal/inventory/... -run TestCatalogStatusInvalidTransition` | Wave 0 | + +**Integration tests** use `-tags integration` build tag and require live NetBox + DragonFlyDB. Short tests (`-short` flag) skip them. + +### Sampling Rate +- **Per task commit:** `go test ./... -short` +- **Per wave merge:** `go test ./... -v -count=1` +- **Phase gate:** Full suite green before `/gsd-verify-work` + +### Wave 0 Gaps +- [ ] `internal/api/handlers/health_test.go` — covers INF-01 +- [ ] `internal/config/config_test.go` — covers INF-02 +- [ ] `internal/netbox/hwid_test.go` — covers INF-03 +- [ ] `internal/netbox/client_test.go` — covers NB-01, NB-02 (integration) +- [ ] `internal/queue/waq_test.go` — covers NB-05 +- [ ] `internal/inventory/quality_gate_test.go` — covers NB-06 + +--- + +## Security Domain + +### Applicable ASVS Categories + +| ASVS Category | Applies | Standard Control | +|---------------|---------|-----------------| +| V2 Authentication | No | Solo operator — no user auth in Phase 1 | +| V3 Session Management | No | No sessions in Phase 1 | +| V4 Access Control | No | Single operator; NetBox token is backend-only | +| V5 Input Validation | Yes | Validate all config values at startup; validate HW-ID format | +| V6 Cryptography | No | No cryptography in Phase 1 | + +### Known Threat Patterns for This Stack + +| Pattern | STRIDE | Standard Mitigation | +|---------|--------|---------------------| +| NetBox token in source code | Information Disclosure | Load from env var only; verify `.env` is in `.gitignore` before first commit | +| DragonFlyDB password in URL with special chars | Tampering | Use `redis.ParseURL` or `redis.Options` with raw string; never log the URL | +| No auth on Go API server | Elevation of Privilege | Acceptable for Phase 1 (homelab, no external exposure); TODO for Phase 3 | + +--- + +## Sources + +### Primary (HIGH confidence) + +- [VERIFIED: proxy.golang.org] — go-netbox v4.3.0 (2025-05-09), chi v5.2.5, go-redis v9.18.0, go-sqlite3 v1.14.42, viper v1.21.0, golang-migrate v4.19.1, uuid v1.6.0 +- [VERIFIED: github.com/netbox-community/go-netbox README] — `NewAPIClientFor` pattern, chained API calls +- [VERIFIED: go-redis README] — Redis 7.0+ compatibility, DragonFlyDB protocol-compatible +- [VERIFIED: network connectivity] — DragonFlyDB port 6379, PostgreSQL port 5432, NetBox port 8000 all reachable + +### Secondary (MEDIUM confidence) + +- [CITED: github.com/netbox-community/netbox/discussions/13425] — custom field write/read asymmetry, flat dict write format +- [CITED: .planning/research/PITFALLS.md] — custom field round-trip pitfall documented with NetBox issue references +- [CITED: .planning/research/STACK.md] — library selections and version compatibility table + +### Tertiary (LOW confidence / ASSUMED) + +- HW-XXXXX allocation via asset_tag optimistic lock — standard pattern, not NetBox-specific documentation +- `catalog_status` as text field type rather than choice field — design decision, not documented requirement + +--- + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH — all package versions verified against Go module proxy +- Architecture: HIGH — go-netbox README confirmed; chi patterns from official docs; custom field asymmetry from NetBox GitHub discussions +- Infrastructure availability: HIGH — ports confirmed reachable; token issue identified and documented +- Pitfalls: HIGH — token placeholder and custom field asymmetry verified; DragonFlyDB URL encoding is a known pattern + +**Research date:** 2026-04-10 +**Valid until:** 2026-05-10 (30 days — stable libraries, slow-moving NetBox API) + +**Key discrepancy resolved:** STACK.md recommends gin; CONTEXT.md specifies chi. CONTEXT.md is the locked decision. All routing patterns in this research use chi v5.2.5.