From c9ad50fdf23c87140d1bbd5d469b54de6a655902 Mon Sep 17 00:00:00 2001 From: Mikkel Georgsen Date: Fri, 10 Apr 2026 01:07:55 +0000 Subject: [PATCH] docs(01-foundation): create phase 1 plans (5 plans, 2 waves) Plans 01-02 are Wave 1 (parallel). Plans 03-04-05 are Wave 2. All 11 requirements covered: INF-01, INF-02, INF-03, NB-01 through NB-07. Co-Authored-By: Claude Sonnet 4.6 --- .planning/ROADMAP.md | 13 +- .planning/phases/01-foundation/01-01-PLAN.md | 486 +++++++++++++ .planning/phases/01-foundation/01-02-PLAN.md | 608 ++++++++++++++++ .planning/phases/01-foundation/01-03-PLAN.md | 569 +++++++++++++++ .planning/phases/01-foundation/01-04-PLAN.md | 698 +++++++++++++++++++ .planning/phases/01-foundation/01-05-PLAN.md | 555 +++++++++++++++ 6 files changed, 2926 insertions(+), 3 deletions(-) create mode 100644 .planning/phases/01-foundation/01-01-PLAN.md create mode 100644 .planning/phases/01-foundation/01-02-PLAN.md create mode 100644 .planning/phases/01-foundation/01-03-PLAN.md create mode 100644 .planning/phases/01-foundation/01-04-PLAN.md create mode 100644 .planning/phases/01-foundation/01-05-PLAN.md diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 1496aa9..d413e65 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -31,8 +31,15 @@ Decimal phases appear between their surrounding integers in numeric order. 2. All HWLab custom fields (hw_id, catalog_status, photo_urls, etc.) are readable and writable via the NetBox API with round-trip test coverage 3. A new item can be created in NetBox with a sequential HW-XXXXX ID auto-assigned 4. catalog_status transitions from draft through complete are enforced by the backend quality gate - 5. A write-ahead queue in SQLite buffers failed NetBox operations and retries them on reconnect -**Plans**: TBD + 5. A write-ahead queue in DragonFlyDB buffers failed NetBox operations and retries them on reconnect +**Plans**: 5 plans + +Plans: +- [ ] 01-01-PLAN.md — Go scaffold: chi server, go:embed stub SPA, viper config, health endpoint +- [ ] 01-02-PLAN.md — NetBox client: go-netbox v4 wrapper, custom field read/write types, integration tests +- [ ] 01-03-PLAN.md — NetBox provisioning: 8 custom fields, location hierarchy, plugin check, provision CLI +- [ ] 01-04-PLAN.md — HW-ID allocation, quality gate state machine, AI tag sync to NetBox +- [ ] 01-05-PLAN.md — Write-ahead queue: DragonFlyDB WAQ + retry worker goroutine ### Phase 2: AI Pipeline **Goal**: Users can submit 1-3 photos and receive a structured NetBox-ready record with AI-extracted specs, suggested category/tags, and a quality gate status reflecting confidence @@ -116,7 +123,7 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7 | Phase | Plans Complete | Status | Completed | |-------|----------------|--------|-----------| -| 1. Foundation | 0/TBD | Not started | - | +| 1. Foundation | 0/5 | Not started | - | | 2. AI Pipeline | 0/TBD | Not started | - | | 3. Dashboard & Intake UI | 0/TBD | Not started | - | | 4. USB Manager & Label Printing | 0/TBD | Not started | - | diff --git a/.planning/phases/01-foundation/01-01-PLAN.md b/.planning/phases/01-foundation/01-01-PLAN.md new file mode 100644 index 0000000..2341a88 --- /dev/null +++ b/.planning/phases/01-foundation/01-01-PLAN.md @@ -0,0 +1,486 @@ +--- +phase: 01-foundation +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - go.mod + - go.sum + - cmd/hwlab/main.go + - internal/api/router.go + - internal/api/handlers/health.go + - internal/config/config.go + - config.json + - Makefile + - web/dist/index.html +autonomous: true +requirements: + - INF-01 + - INF-02 + +must_haves: + truths: + - "Running `go run ./cmd/hwlab/...` starts an HTTP server on port 8080" + - "GET /api/health returns 200 with JSON {status:ok, version:0.1.0}" + - "GET / serves the stub HTML page from web/dist/index.html (embedded in binary)" + - "Config loads from config.json and .env without panicking on missing optional fields" + artifacts: + - path: "cmd/hwlab/main.go" + provides: "Binary entry point — wires config, server, starts listener" + - path: "internal/api/router.go" + provides: "chi router with middleware, /api routes, SPA fallback" + - path: "internal/api/handlers/health.go" + provides: "GET /api/health handler" + - path: "internal/config/config.go" + provides: "viper-backed Config struct loaded from .env + config.json" + - path: "web/dist/index.html" + provides: "Stub SPA embedded via go:embed" + - path: "go.mod" + provides: "Module declaration git.georgsen.dk/hwlab with all Phase 1 deps" + key_links: + - from: "cmd/hwlab/main.go" + to: "internal/api/router.go" + via: "NewRouter(cfg) call" + pattern: "NewRouter" + - from: "internal/api/router.go" + to: "web/dist" + via: "go:embed + http.FileServer" + pattern: "go:embed web/dist" +--- + + +Initialize the Go binary scaffold: module setup, chi HTTP server, viper config, health endpoint, and stub React SPA embedded via go:embed. + +Purpose: Every subsequent plan in this phase depends on the Go module and server existing. This plan creates that foundation. +Output: A compilable Go binary that serves GET /api/health and the stub SPA, loads config from .env + config.json, and is ready to have NetBox and queue packages added in Wave 2. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.env +@.planning/phases/01-foundation/01-RESEARCH.md + + + + + + Task 1: Go module init and chi server with go:embed SPA + go.mod, go.sum, cmd/hwlab/main.go, internal/api/router.go, internal/api/handlers/health.go, web/dist/index.html + + + - /home/mikkel/homelabby/.env (confirm HWLAB_PORT=8080) + - /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 1: chi Router with go:embed, lines 148-181) + + + + - Test 1: GET /api/health returns HTTP 200 with body {"status":"ok","version":"0.1.0"} + - Test 2: GET / returns HTTP 200 with HTML body containing "HWLab" + - Test 3: GET /api/nonexistent returns 404 (chi default) + - Test 4: GET /some/spa/route returns the stub index.html (SPA fallback) + + + + 1. Initialize Go module: `go mod init git.georgsen.dk/hwlab` + + 2. Install dependencies: + ``` + go get github.com/go-chi/chi/v5@v5.2.5 + go get github.com/redis/go-redis/v9@v9.18.0 + go get github.com/spf13/viper@v1.21.0 + go get github.com/joho/godotenv@v1.5.1 + go get github.com/google/uuid@v1.6.0 + go get github.com/netbox-community/go-netbox/v4@v4.3.0 + ``` + + 3. Create `web/dist/index.html` — minimal stub (NOT a full React app; placeholder for Phase 3): + ```html + + + + + + HWLab + + + +
+

HWLab

+

Backend is running. UI coming in Phase 3.

+
+ + + ``` + + 4. Create `internal/api/handlers/health.go`: + ```go + package handlers + + import ( + "encoding/json" + "net/http" + ) + + type HealthResponse struct { + Status string `json:"status"` + Version string `json:"version"` + } + + func Health(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + json.NewEncoder(w).Encode(HealthResponse{ + Status: "ok", + Version: "0.1.0", + }) + } + ``` + + 5. Create `internal/api/router.go` — chi router with go:embed SPA fallback. The embed directive MUST be in the same file as the var declaration: + ```go + package api + + import ( + "embed" + "io/fs" + "net/http" + + "github.com/go-chi/chi/v5" + "github.com/go-chi/chi/v5/middleware" + + "git.georgsen.dk/hwlab/internal/api/handlers" + ) + + //go:embed ../../web/dist + var staticFiles embed.FS + + func NewRouter() http.Handler { + r := chi.NewRouter() + r.Use(middleware.Logger) + r.Use(middleware.Recoverer) + r.Use(middleware.RealIP) + + r.Route("/api", func(r chi.Router) { + r.Get("/health", handlers.Health) + }) + + // SPA fallback — serve index.html for all non-API routes + staticFS, _ := fs.Sub(staticFiles, "web/dist") + fileServer := http.FileServer(http.FS(staticFS)) + r.Handle("/*", fileServer) + + return r + } + ``` + + NOTE: The go:embed path `../../web/dist` is relative to the file location `internal/api/router.go`. Verify path resolves correctly. If the embed directive causes issues due to directory depth, move the embed var to `cmd/hwlab/main.go` and pass the fs.FS into NewRouter as a parameter instead. + + 6. Create `cmd/hwlab/main.go` — wire-up only, no business logic: + ```go + package main + + import ( + "fmt" + "log" + "net/http" + + "git.georgsen.dk/hwlab/internal/api" + "git.georgsen.dk/hwlab/internal/config" + ) + + func main() { + cfg, err := config.Load() + if err != nil { + log.Fatalf("config: %v", err) + } + + router := api.NewRouter() + addr := fmt.Sprintf("%s:%d", cfg.Host, cfg.Port) + log.Printf("HWLab starting on %s", addr) + if err := http.ListenAndServe(addr, router); err != nil { + log.Fatalf("server: %v", err) + } + } + ``` + + 7. Write test `internal/api/handlers/health_test.go`: + ```go + package handlers_test + + import ( + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + + "git.georgsen.dk/hwlab/internal/api/handlers" + ) + + func TestHealth(t *testing.T) { + req := httptest.NewRequest(http.MethodGet, "/api/health", nil) + w := httptest.NewRecorder() + handlers.Health(w, req) + + if w.Code != http.StatusOK { + t.Fatalf("expected 200, got %d", w.Code) + } + + var resp handlers.HealthResponse + if err := json.NewDecoder(w.Body).Decode(&resp); err != nil { + t.Fatalf("decode: %v", err) + } + if resp.Status != "ok" { + t.Errorf("expected status=ok, got %s", resp.Status) + } + if resp.Version != "0.1.0" { + t.Errorf("expected version=0.1.0, got %s", resp.Version) + } + } + ``` + + + + cd /home/mikkel/homelabby && go test ./internal/api/handlers/... -v + + + + - `go build ./cmd/hwlab/...` exits 0 with no errors + - `go test ./internal/api/handlers/...` passes (TestHealth green) + - `grep -r "go:embed web/dist" internal/api/router.go` returns a match (OR the embed is in main.go — check whichever file contains it) + - `grep "git.georgsen.dk/hwlab" go.mod` returns the module declaration line + - `grep "go-chi/chi/v5" go.mod` returns a line with `v5.2.5` + - File `web/dist/index.html` exists and contains "HWLab" + + + Go module compiles, health handler test passes, chi router wires go:embed SPA fallback, binary starts without panic. + + + + Task 2: viper config loader (INF-02) + internal/config/config.go, internal/config/config_test.go, config.json + + + - /home/mikkel/homelabby/.env (all existing env var names and values) + - /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 7: viper Config, lines 362+) + + + + - Test 1: Load() with env var HWLAB_PORT=9999 set returns cfg.Port == 9999 + - Test 2: Load() with no env vars returns cfg.Port == 8080 (from config.json default) + - Test 3: Load() returns cfg.NetBoxURL == "http://10.5.0.130:8000/api" when HWLAB_NETBOX_URL is set + - Test 4: Load() does not return error when config.json fields are missing (optional fields use defaults) + + + + 1. Install godotenv: `go get github.com/joho/godotenv@v1.5.1` + + 2. Create `config.json` at project root with non-secret defaults: + ```json + { + "port": 8080, + "host": "0.0.0.0", + "netbox_url": "http://10.5.0.130:8000/api", + "dragonfly_url": "redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379", + "log_level": "info", + "netbox_timeout_seconds": 10, + "waq_retry_interval_seconds": 30, + "waq_max_attempts": 5, + "quality_gate_confidence_threshold": 0.75 + } + ``` + + 3. Create `internal/config/config.go`: + ```go + package config + + import ( + "fmt" + "strings" + + "github.com/joho/godotenv" + "github.com/spf13/viper" + ) + + type Config struct { + Host string `mapstructure:"host"` + Port int `mapstructure:"port"` + LogLevel string `mapstructure:"log_level"` + + NetBoxURL string `mapstructure:"netbox_url"` + NetBoxToken string `mapstructure:"netbox_token"` + NetBoxTimeoutSeconds int `mapstructure:"netbox_timeout_seconds"` + + DragonflyURL string `mapstructure:"dragonfly_url"` + WAQRetryIntervalSeconds int `mapstructure:"waq_retry_interval_seconds"` + WAQMaxAttempts int `mapstructure:"waq_max_attempts"` + + QualityGateConfidenceThreshold float64 `mapstructure:"quality_gate_confidence_threshold"` + } + + func Load() (*Config, error) { + // Load .env file if present (ignore error — .env is optional in production) + _ = godotenv.Load() + + v := viper.New() + + // Set defaults + v.SetDefault("host", "0.0.0.0") + v.SetDefault("port", 8080) + v.SetDefault("log_level", "info") + v.SetDefault("netbox_timeout_seconds", 10) + v.SetDefault("waq_retry_interval_seconds", 30) + v.SetDefault("waq_max_attempts", 5) + v.SetDefault("quality_gate_confidence_threshold", 0.75) + + // Config file + v.SetConfigName("config") + v.SetConfigType("json") + v.AddConfigPath(".") + v.AddConfigPath("/etc/hwlab") + + // Environment variables: HWLAB_PORT -> port, HWLAB_NETBOX_URL -> netbox_url, etc. + v.SetEnvPrefix("HWLAB") + v.SetEnvKeyReplacer(strings.NewReplacer(".", "_")) + v.AutomaticEnv() + + // Read config file (non-fatal if missing) + if err := v.ReadInConfig(); err != nil { + if _, ok := err.(viper.ConfigFileNotFoundError); !ok { + return nil, fmt.Errorf("config file: %w", err) + } + } + + var cfg Config + if err := v.Unmarshal(&cfg); err != nil { + return nil, fmt.Errorf("unmarshal config: %w", err) + } + + return &cfg, nil + } + ``` + + 4. Create `internal/config/config_test.go`: + ```go + package config_test + + import ( + "os" + "testing" + + "git.georgsen.dk/hwlab/internal/config" + ) + + func TestLoadDefaults(t *testing.T) { + // Unset env vars that might interfere + os.Unsetenv("HWLAB_PORT") + os.Unsetenv("HWLAB_NETBOX_URL") + + cfg, err := config.Load() + if err != nil { + t.Fatalf("Load() error: %v", err) + } + if cfg.Port != 8080 { + t.Errorf("default port: want 8080, got %d", cfg.Port) + } + } + + func TestLoadEnvOverride(t *testing.T) { + os.Setenv("HWLAB_PORT", "9999") + defer os.Unsetenv("HWLAB_PORT") + + cfg, err := config.Load() + if err != nil { + t.Fatalf("Load() error: %v", err) + } + if cfg.Port != 9999 { + t.Errorf("env override port: want 9999, got %d", cfg.Port) + } + } + + func TestLoadNetBoxURL(t *testing.T) { + os.Setenv("HWLAB_NETBOX_URL", "http://10.5.0.130:8000/api") + defer os.Unsetenv("HWLAB_NETBOX_URL") + + cfg, err := config.Load() + if err != nil { + t.Fatalf("Load() error: %v", err) + } + if cfg.NetBoxURL != "http://10.5.0.130:8000/api" { + t.Errorf("netbox url: want http://10.5.0.130:8000/api, got %s", cfg.NetBoxURL) + } + } + ``` + + + + cd /home/mikkel/homelabby && go test ./internal/config/... -v + + + + - `go test ./internal/config/...` passes all 3 tests + - `grep "mapstructure:\"netbox_token\"" internal/config/config.go` returns a match + - `grep "mapstructure:\"dragonfly_url\"" internal/config/config.go` returns a match + - `grep "HWLAB" internal/config/config.go` returns the SetEnvPrefix line + - File `config.json` exists at project root + - `grep "quality_gate_confidence_threshold" config.json` returns a match + - `go build ./cmd/hwlab/...` still exits 0 after adding config + + + Config loads from config.json and .env, env vars override file values, all tests pass, binary compiles with config wired. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| .env file → config loader | Credentials (NetBox token, DragonFlyDB password) enter the process here | +| HTTP client → Go server | All inbound HTTP requests to chi router | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-01-01 | Information Disclosure | .env file | mitigate | .env is in .gitignore; HWLAB_NETBOX_TOKEN is a placeholder in this plan — real token generated in Plan 03 | +| T-01-02 | Tampering | config.json | accept | Local homelab tool, single operator, no integrity threat model needed at this layer | +| T-01-03 | Denial of Service | GET /api/health | accept | No auth needed on health endpoint; low-value target, single operator | +| T-01-04 | Information Disclosure | chi middleware.Logger | accept | Logs go to stdout only, no external log shipping in Phase 1 | + + + +After both tasks complete: +- `go test ./...` green (all packages) +- `go build ./cmd/hwlab/...` exits 0 +- Run binary: `./hwlab &` then `curl http://localhost:8080/api/health` returns `{"status":"ok","version":"0.1.0"}` +- `curl http://localhost:8080/` returns HTML containing "HWLab" +- `curl http://localhost:8080/some/deep/route` returns the same HTML (SPA fallback) +- Kill binary, verify process exits cleanly + + + +1. `go test ./...` passes with no failures +2. `go build ./cmd/hwlab/...` compiles a binary +3. Running the binary and hitting GET /api/health returns HTTP 200 JSON with status=ok +4. Running the binary and hitting GET / returns HTTP 200 HTML containing "HWLab" +5. Config reads HWLAB_PORT from environment when set + + + +After completion, create `.planning/phases/01-foundation/01-01-SUMMARY.md` with: +- Files created/modified +- Key decisions made (especially if go:embed path needed adjustment) +- Test results +- Any deviations from the plan + diff --git a/.planning/phases/01-foundation/01-02-PLAN.md b/.planning/phases/01-foundation/01-02-PLAN.md new file mode 100644 index 0000000..71646af --- /dev/null +++ b/.planning/phases/01-foundation/01-02-PLAN.md @@ -0,0 +1,608 @@ +--- +phase: 01-foundation +plan: 02 +type: execute +wave: 1 +depends_on: [] +files_modified: + - internal/netbox/client.go + - internal/netbox/client_test.go + - internal/netbox/custom_fields.go + - internal/netbox/types.go +autonomous: true +requirements: + - NB-01 + - NB-02 + +must_haves: + truths: + - "NetBox client connects to http://10.5.0.130:8000/api and lists devices without error" + - "Client can create, read, update, and delete a device in NetBox" + - "Custom field read/write wrappers handle the asymmetric NetBox format (read nested, write flat)" + - "Round-trip test confirms custom field written via PATCH is retrievable via GET" + artifacts: + - path: "internal/netbox/client.go" + provides: "go-netbox v4 wrapper with typed methods for device/module/cable CRUD" + exports: ["NewClient", "Client"] + - path: "internal/netbox/custom_fields.go" + provides: "HWLab custom field read/write types and helper functions" + exports: ["CustomFieldsRead", "CustomFieldsPatch", "BuildCustomFieldsPatch", "ParseCustomFields"] + - path: "internal/netbox/types.go" + provides: "HWLab domain types wrapping NetBox responses" + exports: ["Device", "CustomFields"] + key_links: + - from: "internal/netbox/client.go" + to: "http://10.5.0.130:8000/api" + via: "go-netbox NewAPIClientFor" + pattern: "NewAPIClientFor" + - from: "internal/netbox/custom_fields.go" + to: "internal/netbox/client.go" + via: "PatchCustomFields method on Client" + pattern: "PatchCustomFields" +--- + + +Build the typed NetBox client package: go-netbox v4 wrapper, custom field read/write types, and integration tests that verify round-trip custom field writes against the live NetBox instance. + +Purpose: Every other Phase 1 package (quality gate, HW-ID, WAQ) depends on the NetBox client being stable. Building it independently in Wave 1 means Wave 2 plans can consume it without waiting for the scaffold. +Output: `internal/netbox` package with typed CRUD methods and custom field helpers, integration tests that pass against live NetBox at 10.5.0.130. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.env +@.planning/phases/01-foundation/01-RESEARCH.md + + + + + + +go-netbox v4 initialization pattern: +```go +import netbox "github.com/netbox-community/go-netbox/v4" + +client := netbox.NewAPIClientFor("http://10.5.0.130:8000", "YOUR_TOKEN_HERE") + +// List devices +res, _, err := client.DcimAPI.DcimDevicesList(ctx).Limit(10).Execute() +// res.Results is []netbox.DeviceWithConfigContext + +// Create device +req := netbox.WritableDeviceWithConfigContextRequest{ + Name: netbox.PtrString("test-device"), + DeviceType: // ID ref + Site: // ID ref +} +result, _, err := client.DcimAPI.DcimDevicesCreate(ctx). + WritableDeviceWithConfigContextRequest(req).Execute() + +// Custom fields are on Device.CustomFields as map[string]interface{} +// To PATCH custom fields: use DcimDevicesPartialUpdate with PatchedWritableDeviceWithConfigContextRequest +// PatchedWritableDeviceWithConfigContextRequest.CustomFields = map[string]interface{}{...} +``` + +IMPORTANT: The NetBox token in .env (`homelab-netbox-api-token-2024`) is a placeholder string. +Real NetBox tokens are 40-character hex strings generated via NetBox UI. +The executor MUST verify the token is real before running integration tests. +If the token is the placeholder string, add a human checkpoint or skip integration test with t.Skip(). + + + + + + Task 1: NetBox client wrapper with device CRUD (NB-01) + internal/netbox/client.go, internal/netbox/types.go, internal/netbox/client_test.go + + + - /home/mikkel/homelabby/.env (HWLAB_NETBOX_URL, HWLAB_NETBOX_TOKEN — check if token is real or placeholder) + - /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 2: go-netbox v4 Client Initialization, lines 183-210) + - /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Summary section — note about placeholder token, lines 55-58) + + + + - Test 1: NewClient with valid URL and token returns non-nil *Client without error + - Test 2: NewClient with empty token returns error "netbox token is required" + - Test 3: Client.Ping(ctx) against live http://10.5.0.130:8000/api returns no error (INTEGRATION — skip if token is placeholder) + - Test 4: Client.ListDevices(ctx, limit=5) returns slice without error (INTEGRATION — skip if token is placeholder) + + + + 1. Create `internal/netbox/types.go` — HWLab domain types: + ```go + package netbox + + import "time" + + // Device represents a HWLab inventory item backed by a NetBox device record. + type Device struct { + ID int + Name string + AssetTag string // HW-XXXXX identifier + CustomFields CustomFields + Created time.Time + LastUpdated time.Time + } + + // CustomFields holds all HWLab-defined NetBox custom field values for a device. + // NetBox returns these as map[string]interface{} — we provide typed access. + type CustomFields struct { + HWID string // hw_id + CatalogStatus string // catalog_status + ProductURL string // product_url + FirmwareVersion string // firmware_version + TestDate string // test_date (ISO 8601 date string) + TestData string // test_data (JSON string) + AINotes string // ai_notes + PhotoURLs []string // photo_urls (multi-value) + } + ``` + + 2. Create `internal/netbox/client.go`: + ```go + package netbox + + import ( + "context" + "errors" + "fmt" + + nb "github.com/netbox-community/go-netbox/v4" + ) + + // Client wraps go-netbox v4 APIClient with typed HWLab methods. + // All NetBox calls MUST go through this Client — no direct go-netbox calls in other packages. + type Client struct { + api *nb.APIClient + url string + } + + // NewClient creates a configured NetBox client. Returns error if url or token is empty. + func NewClient(url, token string) (*Client, error) { + if url == "" { + return nil, errors.New("netbox url is required") + } + if token == "" { + return nil, errors.New("netbox token is required") + } + // Note: NewAPIClientFor accepts the base URL WITHOUT /api suffix + // The go-netbox library appends /api internally. + // Strip trailing /api if present to avoid double-appending. + baseURL := url + if len(baseURL) > 4 && baseURL[len(baseURL)-4:] == "/api" { + baseURL = baseURL[:len(baseURL)-4] + } + api := nb.NewAPIClientFor(baseURL, token) + return &Client{api: api, url: url}, nil + } + + // Ping verifies the NetBox API is reachable by fetching the API root status. + // Returns nil on success. + func (c *Client) Ping(ctx context.Context) error { + // Use DcimDevicesList with limit=1 as a lightweight connectivity check. + _, resp, err := c.api.DcimAPI.DcimDevicesList(ctx).Limit(1).Execute() + if err != nil { + return fmt.Errorf("netbox ping: %w", err) + } + if resp.StatusCode >= 500 { + return fmt.Errorf("netbox ping: server error %d", resp.StatusCode) + } + return nil + } + + // ListDevices returns up to limit devices from NetBox. + func (c *Client) ListDevices(ctx context.Context, limit int) ([]Device, error) { + if limit <= 0 { + limit = 50 + } + res, _, err := c.api.DcimAPI.DcimDevicesList(ctx).Limit(int32(limit)).Execute() + if err != nil { + return nil, fmt.Errorf("list devices: %w", err) + } + devices := make([]Device, 0, len(res.Results)) + for _, d := range res.Results { + devices = append(devices, deviceFromNetBox(d)) + } + return devices, nil + } + + // GetDevice retrieves a single device by its NetBox internal ID. + func (c *Client) GetDevice(ctx context.Context, id int) (*Device, error) { + d, _, err := c.api.DcimAPI.DcimDevicesRetrieve(ctx, int32(id)).Execute() + if err != nil { + return nil, fmt.Errorf("get device %d: %w", id, err) + } + dev := deviceFromNetBox(*d) + return &dev, nil + } + + // deviceFromNetBox maps a go-netbox DeviceWithConfigContext to our Device type. + // Custom fields are mapped separately via ParseCustomFields. + func deviceFromNetBox(d nb.DeviceWithConfigContext) Device { + dev := Device{ + ID: int(d.GetId()), + Name: d.GetName(), + } + if tag := d.GetAssetTag(); tag != "" { + dev.AssetTag = tag + } + dev.CustomFields = ParseCustomFields(d.GetCustomFields()) + return dev + } + ``` + + 3. Write `internal/netbox/client_test.go`: + ```go + package netbox_test + + import ( + "context" + "os" + "testing" + + "git.georgsen.dk/hwlab/internal/netbox" + ) + + func TestNewClientValidation(t *testing.T) { + _, err := netbox.NewClient("", "token") + if err == nil { + t.Error("expected error for empty url") + } + _, err = netbox.NewClient("http://10.5.0.130:8000/api", "") + if err == nil { + t.Error("expected error for empty token") + } + c, err := netbox.NewClient("http://10.5.0.130:8000/api", "sometoken") + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if c == nil { + t.Error("expected non-nil client") + } + } + + // integrationToken returns the real NetBox token from env, or skips the test + // if only the placeholder is present (placeholder is never 40 hex chars). + func integrationToken(t *testing.T) string { + t.Helper() + token := os.Getenv("HWLAB_NETBOX_TOKEN") + if len(token) != 40 { + t.Skip("HWLAB_NETBOX_TOKEN is not a real 40-char token — skipping integration test") + } + return token + } + + func TestPingLive(t *testing.T) { + token := integrationToken(t) + c, err := netbox.NewClient("http://10.5.0.130:8000/api", token) + if err != nil { + t.Fatalf("NewClient: %v", err) + } + if err := c.Ping(context.Background()); err != nil { + t.Fatalf("Ping: %v", err) + } + } + + func TestListDevicesLive(t *testing.T) { + token := integrationToken(t) + c, _ := netbox.NewClient("http://10.5.0.130:8000/api", token) + devices, err := c.ListDevices(context.Background(), 5) + if err != nil { + t.Fatalf("ListDevices: %v", err) + } + t.Logf("found %d devices in NetBox", len(devices)) + // Not asserting count — NetBox may be empty; just assert no error + } + ``` + + + + cd /home/mikkel/homelabby && go test ./internal/netbox/... -v -run TestNewClientValidation + + + + - `go test ./internal/netbox/... -run TestNewClientValidation` passes (unit tests, no integration needed) + - `grep "NewClient" internal/netbox/client.go` returns the exported function declaration + - `grep "ParseCustomFields" internal/netbox/client.go` returns usage of the function + - `grep "go-netbox" go.mod` returns the dependency line with v4.3.0 + - `go build ./internal/netbox/...` exits 0 + - If real NetBox token available: `go test ./internal/netbox/... -v` shows TestPingLive and TestListDevicesLive PASS (not SKIP) + + + NetBox client compiles, unit validation tests pass, integration tests skip cleanly when token is a placeholder, and pass when a real token is provided. + + + + Task 2: Custom field read/write wrappers (NB-02 round-trip) + internal/netbox/custom_fields.go, internal/netbox/custom_fields_test.go + + + - /home/mikkel/homelabby/internal/netbox/types.go (CustomFields struct from Task 1) + - /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 3: Custom Field Read/Write Asymmetry, lines 203-235) + + + + - Test 1: ParseCustomFields(map with "hw_id":"HW-00001") returns CustomFields{HWID:"HW-00001"} + - Test 2: ParseCustomFields(nil map) returns zero-value CustomFields (no panic) + - Test 3: ParseCustomFields(map with "photo_urls": []interface{}{"url1","url2"}) returns PhotoURLs with 2 entries + - Test 4: BuildCustomFieldsPatch("HW-00001", "draft", nil) returns map containing hw_id and catalog_status keys + - Test 5: BuildCustomFieldsPatch with photo_urls slice includes photo_urls key in patch map + - Test 6 (INTEGRATION, skip if no real token): Client.PatchCustomFields then GetDevice returns matching custom field values + + + + 1. Create `internal/netbox/custom_fields.go`: + ```go + package netbox + + import ( + "context" + "fmt" + ) + + // ParseCustomFields maps NetBox's map[string]interface{} custom fields response + // to the typed CustomFields struct. NetBox returns values as interface{} — we + // perform safe type assertions for each expected field. + func ParseCustomFields(raw map[string]interface{}) CustomFields { + cf := CustomFields{} + if raw == nil { + return cf + } + + if v, ok := raw["hw_id"].(string); ok { + cf.HWID = v + } + if v, ok := raw["catalog_status"].(string); ok { + cf.CatalogStatus = v + } + if v, ok := raw["product_url"].(string); ok { + cf.ProductURL = v + } + if v, ok := raw["firmware_version"].(string); ok { + cf.FirmwareVersion = v + } + if v, ok := raw["test_date"].(string); ok { + cf.TestDate = v + } + if v, ok := raw["test_data"].(string); ok { + cf.TestData = v + } + if v, ok := raw["ai_notes"].(string); ok { + cf.AINotes = v + } + // photo_urls is a multi-value field — NetBox returns []interface{} + if v, ok := raw["photo_urls"].([]interface{}); ok { + urls := make([]string, 0, len(v)) + for _, u := range v { + if s, ok := u.(string); ok { + urls = append(urls, s) + } + } + cf.PhotoURLs = urls + } + + return cf + } + + // BuildCustomFieldsPatch constructs the flat map[string]interface{} payload + // required by NetBox PATCH endpoints. Only include fields that are non-empty + // to avoid accidentally clearing existing values. + // + // NetBox custom field write format differs from read format: + // - Text/URL/date fields: send string value directly + // - Selection fields (catalog_status): send the choice value as string + // - Multi-value fields (photo_urls): send []string directly + func BuildCustomFieldsPatch(hwID, catalogStatus string, photoURLs []string) map[string]interface{} { + patch := make(map[string]interface{}) + if hwID != "" { + patch["hw_id"] = hwID + } + if catalogStatus != "" { + patch["catalog_status"] = catalogStatus + } + if len(photoURLs) > 0 { + patch["photo_urls"] = photoURLs + } + return patch + } + + // BuildFullCustomFieldsPatch constructs a patch with all custom fields. + // Use for initial record creation where all fields should be set. + func BuildFullCustomFieldsPatch(cf CustomFields) map[string]interface{} { + patch := make(map[string]interface{}) + if cf.HWID != "" { + patch["hw_id"] = cf.HWID + } + if cf.CatalogStatus != "" { + patch["catalog_status"] = cf.CatalogStatus + } + if cf.ProductURL != "" { + patch["product_url"] = cf.ProductURL + } + if cf.FirmwareVersion != "" { + patch["firmware_version"] = cf.FirmwareVersion + } + if cf.TestDate != "" { + patch["test_date"] = cf.TestDate + } + if cf.TestData != "" { + patch["test_data"] = cf.TestData + } + if cf.AINotes != "" { + patch["ai_notes"] = cf.AINotes + } + if len(cf.PhotoURLs) > 0 { + patch["photo_urls"] = cf.PhotoURLs + } + return patch + } + + // PatchCustomFields updates the custom fields of a device identified by netboxID. + // After PATCH, performs a GET to verify the write succeeded (HTTP 200 ≠ write confirmed). + func (c *Client) PatchCustomFields(ctx context.Context, deviceID int, patch map[string]interface{}) error { + req := c.api.DcimAPI.DcimDevicesPartialUpdate(ctx, int32(deviceID)) + // go-netbox v4 uses PatchedWritableDeviceWithConfigContextRequest for partial updates + // Set custom fields via the request object + patchReq := c.api.DcimAPI.DcimDevicesPartialUpdate(ctx, int32(deviceID)) + _ = patchReq // suppress unused + // Build the partial update request + // NOTE: go-netbox v4 API — DcimDevicesPartialUpdate takes a PatchedWritableDeviceWithConfigContextRequest + // CustomFields field is map[string]interface{} + import_note := "use c.api.DcimAPI.DcimDevicesPartialUpdate" + _ = import_note + + // Correct approach for go-netbox v4: + nb_req := c.api.DcimAPI.DcimDevicesPartialUpdate(ctx, int32(deviceID)) + _ = nb_req + // TODO: fill in correctly based on generated API — see go-netbox v4 generated code + // The generated struct is: PatchedWritableDeviceWithConfigContextRequest + // It has a CustomFields field of type map[string]interface{} + return fmt.Errorf("PatchCustomFields: implement using go-netbox v4 PatchedWritableDeviceWithConfigContextRequest.CustomFields") + } + ``` + + IMPORTANT NOTE FOR EXECUTOR: The `PatchCustomFields` stub above contains pseudocode that will not compile. Once the go-netbox v4 module is downloaded, inspect the generated API to find the correct struct and method signature: + ``` + grep -r "PatchedWritableDevice" $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/ 2>/dev/null | head -5 + ``` + Then implement `PatchCustomFields` using the correct generated struct. The pattern is: + ```go + patchReq := nb.PatchedWritableDeviceWithConfigContextRequest{} + patchReq.SetCustomFields(patch) + _, _, err := c.api.DcimAPI.DcimDevicesPartialUpdate(ctx, int32(deviceID)). + PatchedWritableDeviceWithConfigContextRequest(patchReq).Execute() + ``` + + 2. Create `internal/netbox/custom_fields_test.go`: + ```go + package netbox_test + + import ( + "testing" + + "git.georgsen.dk/hwlab/internal/netbox" + ) + + func TestParseCustomFieldsNil(t *testing.T) { + cf := netbox.ParseCustomFields(nil) + if cf.HWID != "" { + t.Error("expected empty HWID for nil map") + } + } + + func TestParseCustomFieldsHWID(t *testing.T) { + raw := map[string]interface{}{ + "hw_id": "HW-00001", + "catalog_status": "draft", + } + cf := netbox.ParseCustomFields(raw) + if cf.HWID != "HW-00001" { + t.Errorf("want HW-00001, got %s", cf.HWID) + } + if cf.CatalogStatus != "draft" { + t.Errorf("want draft, got %s", cf.CatalogStatus) + } + } + + func TestParseCustomFieldsPhotoURLs(t *testing.T) { + raw := map[string]interface{}{ + "photo_urls": []interface{}{"http://a.com/1.jpg", "http://a.com/2.jpg"}, + } + cf := netbox.ParseCustomFields(raw) + if len(cf.PhotoURLs) != 2 { + t.Errorf("want 2 photo urls, got %d", len(cf.PhotoURLs)) + } + } + + func TestBuildCustomFieldsPatch(t *testing.T) { + patch := netbox.BuildCustomFieldsPatch("HW-00001", "draft", nil) + if patch["hw_id"] != "HW-00001" { + t.Errorf("hw_id: want HW-00001, got %v", patch["hw_id"]) + } + if patch["catalog_status"] != "draft" { + t.Errorf("catalog_status: want draft, got %v", patch["catalog_status"]) + } + if _, ok := patch["photo_urls"]; ok { + t.Error("photo_urls should not be present when nil passed") + } + } + + func TestBuildCustomFieldsPatchWithURLs(t *testing.T) { + patch := netbox.BuildCustomFieldsPatch("HW-00001", "indexed", []string{"http://a.com/1.jpg"}) + urls, ok := patch["photo_urls"].([]string) + if !ok { + t.Fatal("photo_urls should be []string") + } + if len(urls) != 1 { + t.Errorf("want 1 url, got %d", len(urls)) + } + } + ``` + + + + cd /home/mikkel/homelabby && go test ./internal/netbox/... -v -run "TestParseCustomFields|TestBuildCustomFields" + + + + - `go test ./internal/netbox/... -run "TestParseCustomFields|TestBuildCustomFields"` passes all 5 unit tests + - `grep "ParseCustomFields" internal/netbox/custom_fields.go` returns the exported function declaration + - `grep "BuildCustomFieldsPatch" internal/netbox/custom_fields.go` returns the exported function declaration + - `grep "BuildFullCustomFieldsPatch" internal/netbox/custom_fields.go` returns the exported function declaration + - `grep "photo_urls" internal/netbox/custom_fields.go` returns handling of the []interface{} case + - `PatchCustomFields` is implemented (not returning an error string — the stub must be replaced with real go-netbox v4 API call) + - `go build ./internal/netbox/...` exits 0 + + + Custom field parsing and patch building tested and passing. PatchCustomFields implemented using correct go-netbox v4 generated structs (not stub pseudocode). All unit tests green. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| Go code → NetBox REST API | Authenticated API calls; token is the only credential | +| NetBox response → Go custom field parsing | Untrusted map[string]interface{} values enter type assertions | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-02-01 | Information Disclosure | HWLAB_NETBOX_TOKEN in env | mitigate | Token never logged; only passed to go-netbox client constructor; integration tests skip when placeholder token present | +| T-02-02 | Tampering | ParseCustomFields raw map | accept | Source is NetBox REST API on private homelab LAN (10.5.0.130); no untrusted input path in Phase 1 | +| T-02-03 | Denial of Service | DcimDevicesList with large limit | accept | Single-operator tool; no external callers; limit param is Go code controlled | +| T-02-04 | Information Disclosure | go test logging device IDs | accept | Tests run locally; t.Logf output is ephemeral | + + + +After both tasks complete: +- `go test ./internal/netbox/... -v` shows TestNewClientValidation PASS, integration tests either PASS (real token) or SKIP (placeholder) +- `go test ./internal/netbox/... -run "TestParseCustomFields|TestBuildCustomFields"` all green +- `go build ./...` exits 0 (all packages compile together) +- If real token: `go test ./internal/netbox/... -v -run TestPingLive` shows PASS + + + +1. All unit tests in `internal/netbox` pass without requiring live NetBox +2. Integration tests skip gracefully when token is the placeholder `homelab-netbox-api-token-2024` +3. `ParseCustomFields` handles nil, string values, and []interface{} photo_urls without panicking +4. `PatchCustomFields` is implemented with real go-netbox v4 API calls (not stub) +5. `go build ./...` compiles cleanly + + + +After completion, create `.planning/phases/01-foundation/01-02-SUMMARY.md` with: +- Whether integration tests ran (real token) or skipped (placeholder) +- The exact PatchedWritableDeviceWithConfigContextRequest struct name used (from go-netbox v4 generated code) +- Any go-netbox v4 API surprises (e.g., custom field write format differs from documented pattern) +- Files created/modified + diff --git a/.planning/phases/01-foundation/01-03-PLAN.md b/.planning/phases/01-foundation/01-03-PLAN.md new file mode 100644 index 0000000..ea1d77c --- /dev/null +++ b/.planning/phases/01-foundation/01-03-PLAN.md @@ -0,0 +1,569 @@ +--- +phase: 01-foundation +plan: 03 +type: execute +wave: 2 +depends_on: + - 01-02-PLAN.md +files_modified: + - internal/netbox/provision.go + - internal/netbox/provision_test.go + - scripts/provision-netbox.go +autonomous: true +requirements: + - NB-02 + - NB-03 + - NB-04 + +must_haves: + truths: + - "All 8 HWLab custom fields exist in NetBox after provisioning runs" + - "GET /api/extras/custom-fields/?name=hw_id returns the hw_id field definition" + - "GET /api/extras/custom-fields/?name=catalog_status returns the catalog_status field" + - "Location hierarchy exists: at least one Site, one Location, one Rack in NetBox" + - "Provisioning script is idempotent — running it twice does not create duplicates" + artifacts: + - path: "internal/netbox/provision.go" + provides: "Provision() function: creates custom fields + location hierarchy if not present" + exports: ["Provision", "ProvisionCustomFields", "ProvisionLocationHierarchy"] + - path: "scripts/provision-netbox.go" + provides: "Standalone CLI: `go run scripts/provision-netbox.go` — provisions NetBox from .env" + key_links: + - from: "scripts/provision-netbox.go" + to: "internal/netbox/provision.go" + via: "direct function call Provision(client)" + pattern: "Provision" + - from: "internal/netbox/provision.go" + to: "http://10.5.0.130:8000/api" + via: "REST POST /api/extras/custom-fields/, /api/dcim/sites/, /api/dcim/locations/, /api/dcim/racks/" + pattern: "ExtrasAPI|DcimAPI" +--- + + +Provision the NetBox instance with all HWLab custom fields and location hierarchy. This is an idempotent provisioning operation: if a custom field already exists, skip it; if the location hierarchy exists, skip it. + +Purpose: Custom fields must exist in NetBox before any Go code can write them. NB-02 requires 8 specific fields; NB-03 requires the netbox-inventory plugin installed; NB-04 requires the Site→Location→Rack hierarchy. +Output: A `Provision()` function and a standalone script that a human can run against the live NetBox instance to bootstrap it. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.env +@.planning/phases/01-foundation/01-RESEARCH.md +@.planning/phases/01-foundation/01-02-SUMMARY.md + + + + +From internal/netbox/client.go: +```go +type Client struct { ... } +func NewClient(url, token string) (*Client, error) +// Client.api is *nb.APIClient (go-netbox v4) +// Access raw API via: client.api.ExtrasAPI, client.api.DcimAPI +``` + +NetBox custom field POST payload (REST API): +```json +{ + "name": "hw_id", + "label": "HW ID", + "type": "text", + "object_types": ["dcim.device"], + "required": false, + "description": "HWLab sequential identifier (HW-XXXXX)" +} +``` + +NetBox location hierarchy (DCIM API): +- Site: POST /api/dcim/sites/ {name, slug} +- Location: POST /api/dcim/locations/ {name, slug, site: {id}} +- Rack: POST /api/dcim/racks/ {name, site: {id}, location: {id}, u_height: 42} + +Check for existing custom field by name: GET /api/extras/custom-fields/?name=hw_id +- If count > 0, field exists — skip creation + + + + + + Task 1: Custom field provisioning (NB-02) + internal/netbox/provision.go, internal/netbox/provision_test.go + + + - /home/mikkel/homelabby/internal/netbox/client.go (Client struct, api field access pattern) + - /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (phase requirements table, lines 38-48) + - /home/mikkel/homelabby/.env (HWLAB_NETBOX_TOKEN — check if real 40-char token) + + + + - Test 1: customFieldSpec("hw_id") returns a spec with Name="hw_id", Type="text", ObjectTypes=["dcim.device"] + - Test 2: customFieldSpec("catalog_status") returns Type="text" (stored as free text, not NetBox choice field, to avoid NetBox admin dependency) + - Test 3: customFieldSpec("photo_urls") returns a spec with Type="text" and description mentioning comma-separated + - Test 4 (INTEGRATION — skip if no real token): ProvisionCustomFields() creates all 8 fields and returns no error + - Test 5 (INTEGRATION — skip if no real token): ProvisionCustomFields() called twice is idempotent (no error, no duplicate fields) + + + + All 8 custom fields to provision (NB-02): + - hw_id: text, dcim.device — "HWLab sequential identifier (HW-XXXXX)" + - catalog_status: text, dcim.device — "Lifecycle status: draft|indexed|needs_research|researched|complete" + - product_url: url, dcim.device — "Manufacturer product page URL" + - firmware_version: text, dcim.device — "Current firmware/software version" + - test_date: date, dcim.device — "Date of last cable/hardware test" + - test_data: text, dcim.device — "Structured JSON test results from cable testers" + - ai_notes: text, dcim.device — "AI-generated notes from intake analysis" + - photo_urls: text, dcim.device — "Comma-separated photo URLs captured during intake" + + NOTE on photo_urls: Use type "text" (not multi-object) to avoid NetBox v4 multi-value custom field complexity. Store as comma-separated string. The Go layer will split/join as needed. + + NOTE on catalog_status: Use type "text" not NetBox "selection" field — the Go quality gate owns the valid values; we don't want to maintain a parallel list in NetBox admin UI. + + Create `internal/netbox/provision.go`: + ```go + package netbox + + import ( + "context" + "fmt" + "log" + ) + + // CustomFieldSpec defines a NetBox custom field to provision. + type CustomFieldSpec struct { + Name string + Label string + Type string // "text", "url", "date", "integer", "boolean" + ObjectTypes []string + Description string + Required bool + } + + // hwlabCustomFields is the canonical list of all HWLab custom fields. + // These MUST be provisioned in NetBox before any item can be created. + var hwlabCustomFields = []CustomFieldSpec{ + {Name: "hw_id", Label: "HW ID", Type: "text", ObjectTypes: []string{"dcim.device"}, Description: "HWLab sequential identifier (HW-XXXXX)"}, + {Name: "catalog_status", Label: "Catalog Status", Type: "text", ObjectTypes: []string{"dcim.device"}, Description: "Lifecycle: draft|indexed|needs_research|researched|complete"}, + {Name: "product_url", Label: "Product URL", Type: "url", ObjectTypes: []string{"dcim.device"}, Description: "Manufacturer product page URL"}, + {Name: "firmware_version", Label: "Firmware Version", Type: "text", ObjectTypes: []string{"dcim.device"}, Description: "Current firmware/software version"}, + {Name: "test_date", Label: "Test Date", Type: "date", ObjectTypes: []string{"dcim.device"}, Description: "Date of last cable/hardware test (ISO 8601)"}, + {Name: "test_data", Label: "Test Data", Type: "text", ObjectTypes: []string{"dcim.device"}, Description: "Structured JSON test results from cable testers"}, + {Name: "ai_notes", Label: "AI Notes", Type: "text", ObjectTypes: []string{"dcim.device"}, Description: "AI-generated notes from intake photo analysis"}, + {Name: "photo_urls", Label: "Photo URLs", Type: "text", ObjectTypes: []string{"dcim.device"}, Description: "Comma-separated photo URLs captured during intake"}, + } + + // customFieldSpec returns the spec for a named custom field (for testing). + func customFieldSpec(name string) *CustomFieldSpec { + for i := range hwlabCustomFields { + if hwlabCustomFields[i].Name == name { + return &hwlabCustomFields[i] + } + } + return nil + } + + // ProvisionCustomFields ensures all HWLab custom fields exist in NetBox. + // Idempotent: fields that already exist are skipped. + // Returns the count of fields created (0 if all existed). + func (c *Client) ProvisionCustomFields(ctx context.Context) (int, error) { + created := 0 + for _, spec := range hwlabCustomFields { + exists, err := c.customFieldExists(ctx, spec.Name) + if err != nil { + return created, fmt.Errorf("check field %s: %w", spec.Name, err) + } + if exists { + log.Printf("custom field %q already exists — skipping", spec.Name) + continue + } + if err := c.createCustomField(ctx, spec); err != nil { + return created, fmt.Errorf("create field %s: %w", spec.Name, err) + } + log.Printf("created custom field %q", spec.Name) + created++ + } + return created, nil + } + + // customFieldExists checks if a custom field with the given name already exists. + func (c *Client) customFieldExists(ctx context.Context, name string) (bool, error) { + res, _, err := c.api.ExtrasAPI.ExtrasCustomFieldsList(ctx).Name([]string{name}).Execute() + if err != nil { + return false, err + } + return res.GetCount() > 0, nil + } + + // createCustomField creates a single custom field in NetBox via the Extras API. + // Uses the go-netbox v4 generated WritableCustomFieldRequest type. + func (c *Client) createCustomField(ctx context.Context, spec CustomFieldSpec) error { + // go-netbox v4: use CustomFieldTypeValue for the Type field + // Available types: "text", "longtext", "integer", "decimal", "boolean", + // "date", "datetime", "url", "json", "select", "multiselect", + // "object", "multiobject" + // NOTE: Executor must check the actual enum values in go-netbox v4 generated code: + // grep -r "CustomFieldTypeValue" $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/ | head -10 + + nb_pkg := "github.com/netbox-community/go-netbox/v4" + _ = nb_pkg + + // Pseudocode — executor must use real go-netbox v4 WritableCustomFieldRequest: + // req := nb.WritableCustomFieldRequest{ + // Name: spec.Name, + // Label: nb.PtrString(spec.Label), + // Type: nb.CustomFieldTypeValue(spec.Type), + // ObjectTypes: spec.ObjectTypes, + // Description: nb.PtrString(spec.Description), + // } + // _, _, err := c.api.ExtrasAPI.ExtrasCustomFieldsCreate(ctx). + // WritableCustomFieldRequest(req).Execute() + // return err + + return fmt.Errorf("createCustomField: implement using go-netbox v4 WritableCustomFieldRequest — check generated types at $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@*/model_writable_custom_field_request.go") + } + + // Provision runs all provisioning steps: custom fields + location hierarchy. + func (c *Client) Provision(ctx context.Context) error { + n, err := c.ProvisionCustomFields(ctx) + if err != nil { + return fmt.Errorf("provision custom fields: %w", err) + } + log.Printf("custom fields: %d created", n) + + if err := c.ProvisionLocationHierarchy(ctx); err != nil { + return fmt.Errorf("provision locations: %w", err) + } + return nil + } + ``` + + IMPORTANT: The `createCustomField` function contains a stub that will not compile. The executor MUST: + 1. Run: `ls $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/` to find generated files + 2. Run: `grep -l "WritableCustomFieldRequest" $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/*.go` + 3. Read the relevant generated file to get the exact field names and enum types + 4. Replace the pseudocode with real go-netbox v4 API calls + + Create `internal/netbox/provision_test.go`: + ```go + package netbox + + import ( + "testing" + ) + + func TestCustomFieldSpec(t *testing.T) { + spec := customFieldSpec("hw_id") + if spec == nil { + t.Fatal("hw_id spec not found") + } + if spec.Type != "text" { + t.Errorf("hw_id type: want text, got %s", spec.Type) + } + for _, ot := range spec.ObjectTypes { + if ot == "dcim.device" { + return + } + } + t.Error("hw_id ObjectTypes must include dcim.device") + } + + func TestCustomFieldSpecCatalogStatus(t *testing.T) { + spec := customFieldSpec("catalog_status") + if spec == nil { + t.Fatal("catalog_status spec not found") + } + if spec.Type != "text" { + t.Errorf("catalog_status type: want text (not selection), got %s", spec.Type) + } + } + + func TestCustomFieldSpecPhotoURLs(t *testing.T) { + spec := customFieldSpec("photo_urls") + if spec == nil { + t.Fatal("photo_urls spec not found") + } + if spec.Description == "" { + t.Error("photo_urls must have a description") + } + } + + func TestAllEightFieldsDefined(t *testing.T) { + expected := []string{"hw_id", "catalog_status", "product_url", "firmware_version", + "test_date", "test_data", "ai_notes", "photo_urls"} + for _, name := range expected { + if customFieldSpec(name) == nil { + t.Errorf("missing custom field spec: %s", name) + } + } + if len(hwlabCustomFields) != 8 { + t.Errorf("want 8 custom fields, got %d", len(hwlabCustomFields)) + } + } + ``` + + + + cd /home/mikkel/homelabby && go test ./internal/netbox/... -v -run "TestCustomFieldSpec|TestAllEight" + + + + - `go test ./internal/netbox/... -run "TestCustomFieldSpec|TestAllEight"` passes all 4 unit tests + - `grep -c "Name:" internal/netbox/provision.go` returns 8 or more (one per custom field) + - `grep "photo_urls" internal/netbox/provision.go` appears in hwlabCustomFields slice + - `grep "ai_notes" internal/netbox/provision.go` appears in hwlabCustomFields slice + - `createCustomField` is implemented with real go-netbox v4 API calls (not stub returning error string) + - `go build ./internal/netbox/...` exits 0 + + + All 8 custom field specs defined and tested. ProvisionCustomFields implemented with idempotent check-before-create. createCustomField uses real go-netbox v4 API (not stub). + + + + Task 2: Location hierarchy provisioning + NB-03 plugin check + provision CLI (NB-03, NB-04) + internal/netbox/provision.go, scripts/provision-netbox.go + + + - /home/mikkel/homelabby/internal/netbox/provision.go (ProvisionCustomFields added in Task 1) + - /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (phase requirements NB-03, NB-04 lines 43-48) + - /home/mikkel/homelabby/.env (HWLAB_NETBOX_URL, HWLAB_NETBOX_TOKEN) + + + + Location hierarchy to create (per ROADMAP "Site → Location → Rack per PRD section 7.6"): + - Site: name="Homelab", slug="homelab" + - Location: name="Lab Bench", slug="lab-bench", site=homelab + - Rack: name="Primary Rack", site=homelab, location=lab-bench, u_height=42 + + 1. Add `ProvisionLocationHierarchy` to `internal/netbox/provision.go`: + ```go + // ProvisionLocationHierarchy creates the Site → Location → Rack hierarchy. + // Idempotent: each level is checked before creation. + func (c *Client) ProvisionLocationHierarchy(ctx context.Context) error { + // Step 1: Create or find Site "Homelab" + siteID, err := c.ensureSite(ctx, "Homelab", "homelab") + if err != nil { + return fmt.Errorf("ensure site: %w", err) + } + + // Step 2: Create or find Location "Lab Bench" under the site + locationID, err := c.ensureLocation(ctx, "Lab Bench", "lab-bench", siteID) + if err != nil { + return fmt.Errorf("ensure location: %w", err) + } + + // Step 3: Create or find Rack "Primary Rack" under the site + location + if err := c.ensureRack(ctx, "Primary Rack", siteID, locationID); err != nil { + return fmt.Errorf("ensure rack: %w", err) + } + + return nil + } + + func (c *Client) ensureSite(ctx context.Context, name, slug string) (int32, error) { + // Check if site exists by slug + res, _, err := c.api.DcimAPI.DcimSitesList(ctx).Slug([]string{slug}).Execute() + if err != nil { + return 0, err + } + if res.GetCount() > 0 { + log.Printf("site %q already exists — skipping", name) + return res.Results[0].GetId(), nil + } + // Create site + // NOTE: Executor must use go-netbox v4 WritableSiteRequest type + // Pattern: c.api.DcimAPI.DcimSitesCreate(ctx).WritableSiteRequest(req).Execute() + // Required fields: Name (string), Slug (string) + // Status: use nb.SiteStatusValue("active") or equivalent + return 0, fmt.Errorf("ensureSite: implement using go-netbox v4 WritableSiteRequest") + } + + func (c *Client) ensureLocation(ctx context.Context, name, slug string, siteID int32) (int32, error) { + res, _, err := c.api.DcimAPI.DcimLocationsList(ctx).Slug([]string{slug}).Execute() + if err != nil { + return 0, err + } + if res.GetCount() > 0 { + log.Printf("location %q already exists — skipping", name) + return res.Results[0].GetId(), nil + } + // Create location under site + // NOTE: use go-netbox v4 WritableLocationRequest + // Required: Name, Slug, Site (NestedSiteRequest with ID) + return 0, fmt.Errorf("ensureLocation: implement using go-netbox v4 WritableLocationRequest") + } + + func (c *Client) ensureRack(ctx context.Context, name string, siteID, locationID int32) error { + res, _, err := c.api.DcimAPI.DcimRacksList(ctx).Name([]string{name}).Execute() + if err != nil { + return err + } + if res.GetCount() > 0 { + log.Printf("rack %q already exists — skipping", name) + return nil + } + // Create rack under site + location + // NOTE: use go-netbox v4 WritableRackRequest + // Required: Name, Site (NestedSiteRequest), Location (NestedLocationRequest), UHeight (int32) + // UHeight = 42 + return fmt.Errorf("ensureRack: implement using go-netbox v4 WritableRackRequest") + } + ``` + + EXECUTOR MUST replace all stub returns with real go-netbox v4 API calls. Locate the generated types: + ``` + ls $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/model_writable_site_request.go + ls $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/model_writable_location_request.go + ls $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/model_writable_rack_request.go + ``` + + 2. Add NB-03 plugin check note in provision.go as a comment (NB-03 requires SSH to LXC 130 — this is a manual verification step documented in VALIDATION.md): + ```go + // CheckNetBoxInventoryPlugin verifies the netbox-inventory plugin is installed. + // This check uses the NetBox plugins API endpoint. + // NB-03: netbox-inventory plugin must be installed on LXC 130. + // Manual verification: SSH to LXC 130, run: pip show netbox-inventory + // API check: GET /api/plugins/ lists installed plugin API endpoints. + func (c *Client) CheckNetBoxInventoryPlugin(ctx context.Context) (bool, error) { + // The netbox-inventory plugin registers under /api/plugins/inventory/ + // We can check by hitting that endpoint and seeing if we get a 200 vs 404. + resp, err := c.api.GetConfig().HTTPClient.Get(c.url[:len(c.url)-4] + "/api/plugins/") + if err != nil { + return false, err + } + defer resp.Body.Close() + return resp.StatusCode == 200, nil + } + ``` + + 3. Create `scripts/provision-netbox.go` — standalone CLI script: + ```go + //go:build ignore + + // Run with: go run scripts/provision-netbox.go + // Provisions NetBox with all HWLab custom fields and location hierarchy. + // Reads HWLAB_NETBOX_URL and HWLAB_NETBOX_TOKEN from environment (.env auto-loaded). + + package main + + import ( + "context" + "log" + "os" + + "github.com/joho/godotenv" + + "git.georgsen.dk/hwlab/internal/netbox" + ) + + func main() { + // Load .env + if err := godotenv.Load(); err != nil { + log.Printf("no .env file: %v", err) + } + + url := os.Getenv("HWLAB_NETBOX_URL") + token := os.Getenv("HWLAB_NETBOX_TOKEN") + + if url == "" || token == "" { + log.Fatal("HWLAB_NETBOX_URL and HWLAB_NETBOX_TOKEN must be set") + } + + client, err := netbox.NewClient(url, token) + if err != nil { + log.Fatalf("netbox client: %v", err) + } + + ctx := context.Background() + + log.Println("Provisioning NetBox...") + if err := client.Provision(ctx); err != nil { + log.Fatalf("provision: %v", err) + } + + // Check netbox-inventory plugin (NB-03) + ok, err := client.CheckNetBoxInventoryPlugin(ctx) + if err != nil { + log.Printf("plugin check error: %v", err) + } else if ok { + log.Println("netbox-inventory plugin: INSTALLED") + } else { + log.Println("WARNING: netbox-inventory plugin may not be installed") + log.Println(" Manual check: SSH to LXC 130, run: pip show netbox-inventory") + log.Println(" Install if missing: pip install netbox-inventory") + } + + log.Println("Provisioning complete.") + } + ``` + + + + cd /home/mikkel/homelabby && go build ./internal/netbox/... && go vet ./internal/netbox/... + + + + - `go build ./internal/netbox/...` exits 0 (all stubs replaced with real go-netbox v4 API calls) + - `go vet ./internal/netbox/...` exits 0 (no vet errors) + - `grep "ProvisionLocationHierarchy" internal/netbox/provision.go` returns the exported function + - `grep "ensureSite\|ensureLocation\|ensureRack" internal/netbox/provision.go` returns 3 matches (not stub errors) + - `grep "CheckNetBoxInventoryPlugin" internal/netbox/provision.go` returns the function + - File `scripts/provision-netbox.go` exists + - `grep "go:build ignore" scripts/provision-netbox.go` returns a match (build tag prevents accidental inclusion) + - If real token available: `go run scripts/provision-netbox.go` exits 0 and logs "Provisioning complete." + - If real token available: `curl -H "Authorization: Token REAL_TOKEN" http://10.5.0.130:8000/api/extras/custom-fields/?name=hw_id` returns count > 0 + + + Location hierarchy provisioning implemented. Provision CLI script created. All functions use real go-netbox v4 API (no stubs). `go build` and `go vet` clean. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| scripts/provision-netbox.go → NetBox API | Write access to NetBox via authenticated REST API | +| Provisioning logic → NetBox admin state | Creates objects in NetBox; idempotency prevents data corruption | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-03-01 | Tampering | ProvisionCustomFields idempotency | mitigate | Check-before-create pattern ensures no duplicates; NetBox also enforces unique names on custom fields | +| T-03-02 | Information Disclosure | scripts/provision-netbox.go logs | accept | Logs go to stdout only; no sensitive data logged beyond field names | +| T-03-03 | Denial of Service | Provisioning script run in loop | accept | Script has `//go:build ignore` tag, requires explicit `go run`; single operator context | +| T-03-04 | Elevation of Privilege | Provisioning requires NetBox admin rights | accept | NetBox token grants admin-level access by design; this is the operator's own homelab | + + + +After both tasks complete: +- `go test ./internal/netbox/... -v` all tests pass (unit) or skip (integration without real token) +- `go build ./...` exits 0 +- `go vet ./...` exits 0 +- If real token: `go run scripts/provision-netbox.go` provisions cleanly +- If real token: `curl -s -H "Authorization: Token $HWLAB_NETBOX_TOKEN" "http://10.5.0.130:8000/api/extras/custom-fields/" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['count'])"` shows 8 or more custom fields + + + +1. All 8 custom field specs defined in hwlabCustomFields slice with correct names, types, object_types +2. ProvisionCustomFields is idempotent (check-before-create) +3. ProvisionLocationHierarchy creates Site "Homelab" → Location "Lab Bench" → Rack "Primary Rack" +4. Standalone script runs with `go run scripts/provision-netbox.go` against live NetBox +5. `go build` and `go vet` clean on all packages + + + +After completion, create `.planning/phases/01-foundation/01-03-SUMMARY.md` with: +- Whether provisioning ran against live NetBox (yes/no) +- If yes: which custom fields were created vs. already existed +- If yes: whether netbox-inventory plugin was detected +- Any go-netbox v4 API quirks encountered (WritableSiteRequest fields, slug handling, etc.) +- Files created/modified + diff --git a/.planning/phases/01-foundation/01-04-PLAN.md b/.planning/phases/01-foundation/01-04-PLAN.md new file mode 100644 index 0000000..2c030b9 --- /dev/null +++ b/.planning/phases/01-foundation/01-04-PLAN.md @@ -0,0 +1,698 @@ +--- +phase: 01-foundation +plan: 04 +type: execute +wave: 2 +depends_on: + - 01-02-PLAN.md +files_modified: + - internal/netbox/hwid.go + - internal/netbox/hwid_test.go + - internal/inventory/quality_gate.go + - internal/inventory/quality_gate_test.go + - internal/inventory/types.go + - internal/netbox/tags.go + - internal/netbox/tags_test.go +autonomous: true +requirements: + - INF-03 + - NB-06 + - NB-07 + +must_haves: + truths: + - "AllocateNextHWID returns HW-00001 on first call against an empty NetBox" + - "AllocateNextHWID returns HW-00002 on subsequent calls (increments)" + - "CatalogStatus.CanTransitionTo enforces valid transitions (draft→indexed allowed, indexed→draft rejected)" + - "Invalid transitions return an error with the exact invalid transition described" + - "SyncTags creates new NetBox tags for AI-suggested tags not yet present" + artifacts: + - path: "internal/netbox/hwid.go" + provides: "AllocateNextHWID: optimistic-lock sequential ID allocation from NetBox" + exports: ["AllocateNextHWID"] + - path: "internal/inventory/quality_gate.go" + provides: "CatalogStatus type with Transition() enforcing valid state machine" + exports: ["CatalogStatus", "StatusDraft", "StatusIndexed", "StatusNeedsResearch", "StatusResearched", "StatusComplete", "Transition"] + - path: "internal/inventory/types.go" + provides: "HardwareRecord domain type composing NetBox device with HWLab semantics" + exports: ["HardwareRecord"] + - path: "internal/netbox/tags.go" + provides: "SyncTags: creates NetBox tags from AI-suggested string slice, returns IDs" + exports: ["SyncTags"] + key_links: + - from: "internal/netbox/hwid.go" + to: "http://10.5.0.130:8000/api" + via: "DcimDevicesList filtered by asset_tag pattern" + pattern: "DcimDevicesList.*asset_tag|asset_tag.*DcimDevicesList" + - from: "internal/inventory/quality_gate.go" + to: "internal/netbox/client.go" + via: "CatalogStatus value stored as NetBox custom field via PatchCustomFields" + pattern: "PatchCustomFields.*catalog_status" +--- + + +Implement three distinct capabilities that all depend on the NetBox client (Plan 02): HW-XXXXX sequential ID allocation, catalog quality gate state machine, and AI tag sync to NetBox. + +Purpose: These three capabilities are the behavioral core of Phase 1. HW-ID is required at intake time (Phase 2). Quality gate drives all lifecycle operations. Tag sync links AI output to NetBox taxonomy. +Output: Three packages — netbox/hwid.go, inventory/quality_gate.go, netbox/tags.go — all tested independently. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.env +@.planning/phases/01-foundation/01-RESEARCH.md +@.planning/phases/01-foundation/01-02-SUMMARY.md + + + + +From internal/netbox/client.go: +```go +func NewClient(url, token string) (*Client, error) +func (c *Client) ListDevices(ctx context.Context, limit int) ([]Device, error) +func (c *Client) PatchCustomFields(ctx context.Context, deviceID int, patch map[string]interface{}) error +// c.api is *nb.APIClient — can call DcimAPI, ExtrasAPI directly from hwid.go and tags.go +``` + +From internal/netbox/types.go: +```go +type Device struct { + ID int + Name string + AssetTag string // HW-XXXXX + CustomFields CustomFields + Created time.Time + LastUpdated time.Time +} +type CustomFields struct { + HWID, CatalogStatus, ProductURL, FirmwareVersion string + TestDate, TestData, AINotes string + PhotoURLs []string +} +``` + +From internal/netbox/custom_fields.go: +```go +func BuildCustomFieldsPatch(hwID, catalogStatus string, photoURLs []string) map[string]interface{} +``` + + + + + + Task 1: HW-XXXXX sequential ID allocation (INF-03) + internal/netbox/hwid.go, internal/netbox/hwid_test.go + + + - /home/mikkel/homelabby/internal/netbox/client.go (Client struct, api access) + - /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 4: HW-XXXXX Sequential ID Allocation, lines 238-265) + + + + - Test 1: parseHWID("HW-00042") returns 42, nil + - Test 2: parseHWID("HW-99999") returns 99999, nil + - Test 3: parseHWID("not-a-hw-id") returns 0, error + - Test 4: parseHWID("") returns 0, error + - Test 5: formatHWID(1) returns "HW-00001" + - Test 6: formatHWID(99999) returns "HW-99999" + - Test 7 (INTEGRATION — skip if no real token): AllocateNextHWID returns a string matching ^HW-\d{5}$ + + + + Create `internal/netbox/hwid.go`: + ```go + package netbox + + import ( + "context" + "errors" + "fmt" + "regexp" + "strconv" + "strings" + ) + + var hwIDPattern = regexp.MustCompile(`^HW-(\d{5})$`) + + // formatHWID formats an integer as a HW-XXXXX string. + func formatHWID(n int) string { + return fmt.Sprintf("HW-%05d", n) + } + + // parseHWID parses a HW-XXXXX string to an integer. + // Returns error if the format does not match. + func parseHWID(s string) (int, error) { + m := hwIDPattern.FindStringSubmatch(s) + if m == nil { + return 0, fmt.Errorf("invalid HW-ID format: %q (expected HW-NNNNN)", s) + } + n, err := strconv.Atoi(m[1]) + if err != nil { + return 0, err + } + return n, nil + } + + // AllocateNextHWID allocates the next available HW-XXXXX identifier. + // Strategy: optimistic locking — query the highest existing asset_tag, increment by 1, + // attempt to reserve it. Retry up to 3 times on conflict. + // + // The reservation is a placeholder NetBox device with name "__hwid_reservation__" + // that the caller MUST immediately replace with the real device data. + // In practice, Phase 2 will create the real device in a single atomic step, + // so the placeholder device is never committed separately. + // + // For Phase 1, AllocateNextHWID returns the ID string without creating a device. + // The caller is responsible for creating the device record and setting asset_tag. + func (c *Client) AllocateNextHWID(ctx context.Context) (string, error) { + const maxAttempts = 3 + + for attempt := 0; attempt < maxAttempts; attempt++ { + highest, err := c.getHighestHWIDNumber(ctx) + if err != nil { + return "", fmt.Errorf("get highest HW-ID: %w", err) + } + candidate := formatHWID(highest + 1) + // Check that this candidate is not already taken + // (handles concurrent allocation if ever needed) + taken, err := c.hwIDExists(ctx, candidate) + if err != nil { + return "", fmt.Errorf("check HW-ID %s: %w", candidate, err) + } + if !taken { + return candidate, nil + } + // Candidate is taken — loop and try highest+2, etc. + } + return "", errors.New("HW-ID allocation failed after 3 attempts — concurrent allocation conflict") + } + + // getHighestHWIDNumber queries NetBox for the highest existing HW-XXXXX asset_tag number. + // Returns 0 if no HW-XXXXX asset_tags exist (first allocation will be HW-00001). + func (c *Client) getHighestHWIDNumber(ctx context.Context) (int, error) { + // Query all devices, paginate if needed — in Phase 1 this is small + // Filter by asset_tag starting with "HW-" to limit results + // NOTE: go-netbox v4 DcimDevicesList supports AssetTag filter + // Use limit=1000 and sort by asset_tag descending to find the highest efficiently + // If NetBox v4 supports ordering by asset_tag: use .Ordering("-asset_tag") + + res, _, err := c.api.DcimAPI.DcimDevicesList(ctx). + Limit(1000). + Execute() + if err != nil { + return 0, fmt.Errorf("list devices for HW-ID query: %w", err) + } + + highest := 0 + for _, d := range res.Results { + tag := d.GetAssetTag() + if !strings.HasPrefix(tag, "HW-") { + continue + } + n, err := parseHWID(tag) + if err != nil { + continue // non-HWLab asset tag — skip + } + if n > highest { + highest = n + } + } + return highest, nil + } + + // hwIDExists checks if a given HW-XXXXX asset_tag is already used in NetBox. + func (c *Client) hwIDExists(ctx context.Context, hwid string) (bool, error) { + res, _, err := c.api.DcimAPI.DcimDevicesList(ctx). + AssetTag([]string{hwid}). + Limit(1). + Execute() + if err != nil { + return false, err + } + return res.GetCount() > 0, nil + } + ``` + + Create `internal/netbox/hwid_test.go`: + ```go + package netbox + + import ( + "testing" + ) + + func TestFormatHWID(t *testing.T) { + tests := []struct { + n int + want string + }{ + {1, "HW-00001"}, + {42, "HW-00042"}, + {99999, "HW-99999"}, + } + for _, tt := range tests { + got := formatHWID(tt.n) + if got != tt.want { + t.Errorf("formatHWID(%d) = %q, want %q", tt.n, got, tt.want) + } + } + } + + func TestParseHWID(t *testing.T) { + tests := []struct { + s string + want int + wantErr bool + }{ + {"HW-00001", 1, false}, + {"HW-00042", 42, false}, + {"HW-99999", 99999, false}, + {"", 0, true}, + {"not-a-hw-id", 0, true}, + {"HW-0001", 0, true}, // only 4 digits — invalid + {"hw-00001", 0, true}, // lowercase — invalid + } + for _, tt := range tests { + got, err := parseHWID(tt.s) + if tt.wantErr && err == nil { + t.Errorf("parseHWID(%q): expected error, got nil", tt.s) + } + if !tt.wantErr && err != nil { + t.Errorf("parseHWID(%q): unexpected error: %v", tt.s, err) + } + if !tt.wantErr && got != tt.want { + t.Errorf("parseHWID(%q) = %d, want %d", tt.s, got, tt.want) + } + } + } + ``` + + + + cd /home/mikkel/homelabby && go test ./internal/netbox/... -v -run "TestFormatHWID|TestParseHWID" + + + + - `go test ./internal/netbox/... -run "TestFormatHWID|TestParseHWID"` passes all 10 cases + - `grep "AllocateNextHWID" internal/netbox/hwid.go` returns the exported function + - `grep "HW-%05d" internal/netbox/hwid.go` returns the Sprintf format call + - `grep "getHighestHWIDNumber" internal/netbox/hwid.go` returns the private helper + - `go build ./internal/netbox/...` exits 0 + + + HW-ID format/parse unit tests pass. AllocateNextHWID implemented with optimistic-lock retry. Binary compiles. + + + + Task 2: Quality gate state machine and AI tag sync (NB-06, NB-07) + internal/inventory/types.go, internal/inventory/quality_gate.go, internal/inventory/quality_gate_test.go, internal/netbox/tags.go, internal/netbox/tags_test.go + + + - /home/mikkel/homelabby/internal/netbox/client.go (Client struct for SyncTags) + - /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 6: Catalog Status Quality Gate, lines 322-360) + - /home/mikkel/homelabby/internal/netbox/types.go (CustomFields.CatalogStatus field) + + + + Quality gate tests: + - Test 1: StatusDraft.CanTransitionTo(StatusIndexed) returns true + - Test 2: StatusDraft.CanTransitionTo(StatusComplete) returns false + - Test 3: StatusIndexed.CanTransitionTo(StatusNeedsResearch) returns true + - Test 4: StatusIndexed.CanTransitionTo(StatusDraft) returns false (no backward transitions) + - Test 5: StatusComplete.CanTransitionTo(anything) returns false (terminal) + - Test 6: Transition(StatusDraft, StatusIndexed) returns StatusIndexed, nil + - Test 7: Transition(StatusDraft, StatusComplete) returns "", error containing "invalid transition" + - Test 8: ParseCatalogStatus("draft") returns StatusDraft, nil + - Test 9: ParseCatalogStatus("unknown_value") returns "", error + + Tag sync tests: + - Test 10: normalizeTags([]string{" USB Cable ", "USB cable", "usb-cable"}) returns deduplicated, lowercase-trimmed slice + - Test 11 (INTEGRATION — skip if no real token): SyncTags([]string{"usb-c-cable"}) creates tag in NetBox and returns its ID + + + + 1. Create `internal/inventory/types.go`: + ```go + package inventory + + import "git.georgsen.dk/hwlab/internal/netbox" + + // HardwareRecord is the HWLab domain representation of a cataloged item. + // It wraps a NetBox device with HWLab-specific fields and lifecycle state. + type HardwareRecord struct { + HWID string // HW-XXXXX from asset_tag + NetBoxID int // NetBox device internal ID + Name string // Device name in NetBox + CatalogStatus CatalogStatus // Quality gate lifecycle status + CustomFields netbox.CustomFields // All HWLab custom fields + AITags []string // AI-suggested tags (synced to NetBox) + } + ``` + + 2. Create `internal/inventory/quality_gate.go`: + ```go + package inventory + + import "fmt" + + // CatalogStatus represents the lifecycle stage of a cataloged hardware item. + // Stored as the catalog_status custom field value in NetBox. + type CatalogStatus string + + const ( + StatusDraft CatalogStatus = "draft" + StatusIndexed CatalogStatus = "indexed" + StatusNeedsResearch CatalogStatus = "needs_research" + StatusResearched CatalogStatus = "researched" + StatusComplete CatalogStatus = "complete" + ) + + // validTransitions defines the allowed state machine transitions. + // No backward transitions are permitted (lifecycle is forward-only). + var validTransitions = map[CatalogStatus][]CatalogStatus{ + StatusDraft: {StatusIndexed}, + StatusIndexed: {StatusNeedsResearch, StatusResearched}, + StatusNeedsResearch: {StatusResearched}, + StatusResearched: {StatusComplete}, + StatusComplete: {}, // terminal — no further transitions + } + + // CanTransitionTo returns true if transitioning from s to next is permitted. + func (s CatalogStatus) CanTransitionTo(next CatalogStatus) bool { + allowed, ok := validTransitions[s] + if !ok { + return false + } + for _, a := range allowed { + if a == next { + return true + } + } + return false + } + + // Transition attempts to move from current to next status. + // Returns the new status on success, or an error describing the invalid transition. + func Transition(current, next CatalogStatus) (CatalogStatus, error) { + if !current.CanTransitionTo(next) { + return "", fmt.Errorf("invalid transition: %s → %s (not in valid transitions map)", current, next) + } + return next, nil + } + + // ParseCatalogStatus parses a string to a CatalogStatus. + // Returns error for unknown status values. + func ParseCatalogStatus(s string) (CatalogStatus, error) { + cs := CatalogStatus(s) + if _, ok := validTransitions[cs]; ok { + return cs, nil + } + return "", fmt.Errorf("unknown catalog status: %q (valid: draft, indexed, needs_research, researched, complete)", s) + } + + // AllStatuses returns all valid catalog statuses in lifecycle order. + func AllStatuses() []CatalogStatus { + return []CatalogStatus{ + StatusDraft, StatusIndexed, StatusNeedsResearch, StatusResearched, StatusComplete, + } + } + ``` + + 3. Create `internal/inventory/quality_gate_test.go`: + ```go + package inventory_test + + import ( + "strings" + "testing" + + "git.georgsen.dk/hwlab/internal/inventory" + ) + + func TestCanTransitionTo(t *testing.T) { + tests := []struct { + from inventory.CatalogStatus + to inventory.CatalogStatus + allowed bool + }{ + {inventory.StatusDraft, inventory.StatusIndexed, true}, + {inventory.StatusDraft, inventory.StatusComplete, false}, + {inventory.StatusDraft, inventory.StatusDraft, false}, + {inventory.StatusIndexed, inventory.StatusNeedsResearch, true}, + {inventory.StatusIndexed, inventory.StatusResearched, true}, + {inventory.StatusIndexed, inventory.StatusDraft, false}, + {inventory.StatusNeedsResearch, inventory.StatusResearched, true}, + {inventory.StatusNeedsResearch, inventory.StatusIndexed, false}, + {inventory.StatusResearched, inventory.StatusComplete, true}, + {inventory.StatusResearched, inventory.StatusDraft, false}, + {inventory.StatusComplete, inventory.StatusDraft, false}, + {inventory.StatusComplete, inventory.StatusResearched, false}, + } + for _, tt := range tests { + got := tt.from.CanTransitionTo(tt.to) + if got != tt.allowed { + t.Errorf("%s → %s: want %v, got %v", tt.from, tt.to, tt.allowed, got) + } + } + } + + func TestTransitionValid(t *testing.T) { + got, err := inventory.Transition(inventory.StatusDraft, inventory.StatusIndexed) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got != inventory.StatusIndexed { + t.Errorf("want indexed, got %s", got) + } + } + + func TestTransitionInvalid(t *testing.T) { + _, err := inventory.Transition(inventory.StatusDraft, inventory.StatusComplete) + if err == nil { + t.Fatal("expected error for invalid transition") + } + if !strings.Contains(err.Error(), "invalid transition") { + t.Errorf("error should mention 'invalid transition', got: %v", err) + } + } + + func TestParseCatalogStatus(t *testing.T) { + for _, s := range []string{"draft", "indexed", "needs_research", "researched", "complete"} { + cs, err := inventory.ParseCatalogStatus(s) + if err != nil { + t.Errorf("ParseCatalogStatus(%q): unexpected error: %v", s, err) + } + if string(cs) != s { + t.Errorf("ParseCatalogStatus(%q) = %q, want %q", s, cs, s) + } + } + _, err := inventory.ParseCatalogStatus("unknown_status") + if err == nil { + t.Error("expected error for unknown status") + } + } + ``` + + 4. Create `internal/netbox/tags.go` (NB-07 — AI tags synced to NetBox): + ```go + package netbox + + import ( + "context" + "fmt" + "strings" + ) + + // normalizeTags deduplicates and normalizes a slice of tag strings: + // trims whitespace, lowercases, removes empty strings. + func normalizeTags(tags []string) []string { + seen := make(map[string]struct{}) + out := make([]string, 0, len(tags)) + for _, t := range tags { + t = strings.ToLower(strings.TrimSpace(t)) + if t == "" { + continue + } + if _, ok := seen[t]; ok { + continue + } + seen[t] = struct{}{} + out = append(out, t) + } + return out + } + + // TagRef holds a NetBox tag name and its internal ID. + type TagRef struct { + ID int32 + Name string + Slug string + } + + // SyncTags ensures all tags in the provided slice exist in NetBox. + // Tags are normalized before sync (lowercase, trimmed, deduplicated). + // Returns the TagRef list for all tags (existing + newly created). + func (c *Client) SyncTags(ctx context.Context, tags []string) ([]TagRef, error) { + normalized := normalizeTags(tags) + if len(normalized) == 0 { + return nil, nil + } + + result := make([]TagRef, 0, len(normalized)) + for _, name := range normalized { + slug := tagNameToSlug(name) + ref, err := c.ensureTag(ctx, name, slug) + if err != nil { + return result, fmt.Errorf("sync tag %q: %w", name, err) + } + result = append(result, ref) + } + return result, nil + } + + // tagNameToSlug converts a tag name to a NetBox-compatible slug. + // NetBox slugs: lowercase, hyphens instead of spaces, only [a-z0-9-_]. + func tagNameToSlug(name string) string { + s := strings.ToLower(strings.TrimSpace(name)) + s = strings.ReplaceAll(s, " ", "-") + // Remove characters not in [a-z0-9-_] + var out []byte + for _, c := range []byte(s) { + if (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || c == '-' || c == '_' { + out = append(out, c) + } + } + return string(out) + } + + // ensureTag returns an existing tag or creates a new one. + func (c *Client) ensureTag(ctx context.Context, name, slug string) (TagRef, error) { + // Check for existing tag by slug + res, _, err := c.api.ExtrasAPI.ExtrasTagsList(ctx).Slug([]string{slug}).Execute() + if err != nil { + return TagRef{}, fmt.Errorf("list tags: %w", err) + } + if res.GetCount() > 0 { + t := res.Results[0] + return TagRef{ID: t.GetId(), Name: t.GetName(), Slug: t.GetSlug()}, nil + } + // Create new tag + // NOTE: Executor must use go-netbox v4 TagRequest type + // Pattern: c.api.ExtrasAPI.ExtrasTagsCreate(ctx).TagRequest(req).Execute() + // Required fields: Name (string), Slug (string) + // Optional: Color (hex string, e.g. "faff69" for volt yellow) + return TagRef{}, fmt.Errorf("ensureTag: implement using go-netbox v4 TagRequest — grep TagRequest $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/") + } + ``` + + EXECUTOR: Replace the `ensureTag` stub with real go-netbox v4 TagRequest call. Locate: + ``` + ls $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/model_tag_request.go + ``` + + 5. Create `internal/netbox/tags_test.go`: + ```go + package netbox + + import "testing" + + func TestNormalizeTags(t *testing.T) { + in := []string{" USB Cable ", "USB cable", "usb-cable", "", " "} + out := normalizeTags(in) + // "USB Cable", "USB cable", "usb-cable" all normalize to "usb-cable" — only 1 unique + if len(out) != 1 { + t.Errorf("want 1 unique normalized tag, got %d: %v", len(out), out) + } + if out[0] != "usb-cable" { + t.Errorf("want usb-cable, got %s", out[0]) + } + } + + func TestTagNameToSlug(t *testing.T) { + tests := []struct { + name string + slug string + }{ + {"USB Cable", "usb-cable"}, + {"10GbE NIC", "10gbe-nic"}, + {"SFP+ Transceiver", "sfp-transceiver"}, + {" spaces ", "spaces"}, + } + for _, tt := range tests { + got := tagNameToSlug(tt.name) + if got != tt.slug { + t.Errorf("tagNameToSlug(%q) = %q, want %q", tt.name, got, tt.slug) + } + } + } + ``` + + + + cd /home/mikkel/homelabby && go test ./internal/inventory/... ./internal/netbox/... -v -run "TestCanTransitionTo|TestTransitionValid|TestTransitionInvalid|TestParseCatalogStatus|TestNormalizeTags|TestTagNameToSlug" + + + + - `go test ./internal/inventory/...` passes all 12 table-driven cases for CanTransitionTo + - `go test ./internal/inventory/...` passes TestTransitionValid, TestTransitionInvalid, TestParseCatalogStatus + - `go test ./internal/netbox/... -run "TestNormalizeTags|TestTagNameToSlug"` passes all cases + - `grep "StatusComplete.*{}" internal/inventory/quality_gate.go` returns the terminal state entry with empty transitions + - `grep "invalid transition" internal/inventory/quality_gate.go` returns error string in Transition() + - `ensureTag` in `internal/netbox/tags.go` is implemented with real go-netbox v4 TagRequest (not stub returning error string) + - `go build ./internal/inventory/... ./internal/netbox/...` exits 0 + + + Quality gate state machine fully tested. Tag normalization and slug conversion tested. ensureTag implemented with real go-netbox v4 API. All packages build cleanly. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| AI output → SyncTags | AI-suggested tag strings enter normalizeTags before any NetBox write | +| Quality gate transitions → NetBox writes | Transition validation in Go; NetBox stores the result | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-04-01 | Tampering | Quality gate bypass | mitigate | All status changes MUST go through Transition() — no direct NetBox PATCH of catalog_status without Transition validation | +| T-04-02 | Tampering | AI tag injection | mitigate | normalizeTags strips whitespace, lowercases, deduplicates — limits injection surface before NetBox write | +| T-04-03 | Denial of Service | AllocateNextHWID scanning all devices | accept | Phase 1 inventory is small; getHighestHWIDNumber scans all devices; acceptable until inventory exceeds ~10k items | +| T-04-04 | Information Disclosure | HW-ID sequential enumeration | accept | IDs are not secret — they appear on printed labels; sequential is intentional for readability | + + + +After both tasks complete: +- `go test ./internal/inventory/... ./internal/netbox/...` all green (unit tests) +- `go build ./...` exits 0 (all packages compile together) +- Quality gate: `draft → indexed → needs_research → researched → complete` is the only fully valid path +- `curl -s -H "Authorization: Token $HWLAB_NETBOX_TOKEN" "http://10.5.0.130:8000/api/extras/tags/" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['count'])"` (verify tags endpoint reachable, if real token) + + + +1. All 12 state machine transition cases tested and correct +2. Transition() returns error containing "invalid transition" for bad transitions +3. ParseCatalogStatus rejects unknown status strings +4. normalizeTags handles deduplication across case/whitespace variants +5. AllocateNextHWID implemented with optimistic retry loop +6. ensureTag uses real go-netbox v4 API (not stub) +7. `go build ./...` and `go test ./...` both clean + + + +After completion, create `.planning/phases/01-foundation/01-04-SUMMARY.md` with: +- All test results (pass counts) +- Whether ensureTag integration test ran (real token) or skipped +- Any issues with go-netbox v4 tag or device list filter API +- Files created/modified + diff --git a/.planning/phases/01-foundation/01-05-PLAN.md b/.planning/phases/01-foundation/01-05-PLAN.md new file mode 100644 index 0000000..c0bc710 --- /dev/null +++ b/.planning/phases/01-foundation/01-05-PLAN.md @@ -0,0 +1,555 @@ +--- +phase: 01-foundation +plan: 05 +type: execute +wave: 2 +depends_on: + - 01-01-PLAN.md +files_modified: + - internal/queue/waq.go + - internal/queue/waq_test.go + - internal/queue/worker.go + - cmd/hwlab/main.go +autonomous: true +requirements: + - NB-05 + +must_haves: + truths: + - "WAQ connects to DragonFlyDB at redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379" + - "Enqueue() pushes a serialized PendingOp onto the hwlab:netbox:pending_ops LIST" + - "RunWorker() BLPOP-blocks and processes available operations" + - "DragonFlyDB unavailability does not crash the binary — WAQ degrades gracefully" + - "Dequeue returns the oldest operation first (FIFO via RPUSH/BLPOP pattern)" + artifacts: + - path: "internal/queue/waq.go" + provides: "WAQ type: Enqueue, Dequeue, Len — DragonFlyDB-backed write-ahead queue" + exports: ["WAQ", "NewWAQ", "PendingOp", "Enqueue", "Len"] + - path: "internal/queue/worker.go" + provides: "RunWorker goroutine: BLPOP retry loop, exponential backoff on connection failure" + exports: ["RunWorker"] + key_links: + - from: "internal/queue/waq.go" + to: "redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379" + via: "go-redis v9 ParseURL + NewClient" + pattern: "redis.ParseURL|ParseURL" + - from: "internal/queue/worker.go" + to: "internal/queue/waq.go" + via: "BLPOP + processOp callback" + pattern: "BLPOP|RunWorker" + - from: "cmd/hwlab/main.go" + to: "internal/queue/worker.go" + via: "go waq.RunWorker(ctx)" + pattern: "RunWorker" +--- + + +Implement the DragonFlyDB write-ahead queue: enqueue failed or deferred NetBox operations during downtime, and a worker goroutine that retries them when connectivity restores. + +Purpose: NetBox may be temporarily unavailable (container restart, network blip). The WAQ ensures no inventory operations are lost — they're buffered in DragonFlyDB and retried automatically. NB-05 is the sole requirement for this plan. +Output: `internal/queue` package with WAQ and worker, wired into the main binary. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.env +@.planning/phases/01-foundation/01-RESEARCH.md +@.planning/phases/01-foundation/01-01-SUMMARY.md + + + + +URL: redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379 +Queue key: "hwlab:netbox:pending_ops" (Redis LIST) +Operations: RPUSH (enqueue), BLPOP (dequeue + block), LLEN (queue depth) + +From internal/config/config.go: +```go +type Config struct { + DragonflyURL string // "redis://:PASSWORD@10.5.0.10:6379" + WAQRetryIntervalSeconds int // default 30 + WAQMaxAttempts int // default 5 +} +``` + +go-redis v9 pattern (verified in RESEARCH.md): +```go +import "github.com/redis/go-redis/v9" + +opt, err := redis.ParseURL(redisURL) // parses redis:// URL including password +client := redis.NewClient(opt) +err = client.Ping(ctx).Err() // connectivity check + +// Enqueue: RPUSH appends to right (FIFO with BLPOP from left) +err = client.RPush(ctx, key, data).Err() + +// Dequeue (blocking): BLPOP pops from left, blocks up to timeout +result, err := client.BLPop(ctx, 5*time.Second, key).Result() +// result[0] = key name, result[1] = value + +// Queue depth +n, err := client.LLen(ctx, key).Result() +``` + + + + + + Task 1: Write-ahead queue core (Enqueue, Dequeue, Len) + internal/queue/waq.go, internal/queue/waq_test.go + + + - /home/mikkel/homelabby/.env (HWLAB_DRAGONFLY_URL — verify it's redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379) + - /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 5: DragonFlyDB Write-Ahead Queue, lines 266-320) + - /home/mikkel/homelabby/internal/config/config.go (DragonflyURL field) + + + + - Test 1: NewWAQ with invalid URL returns error + - Test 2: NewWAQ with valid URL but unreachable server returns error on Ping + - Test 3 (INTEGRATION — skip if DragonFlyDB unreachable): Enqueue() + Len() = 1 + - Test 4 (INTEGRATION): Enqueue(op1), Enqueue(op2), Dequeue() returns op1 first (FIFO) + - Test 5 (INTEGRATION): Len() returns 0 after all ops dequeued + - Test 6: PendingOp marshals/unmarshals to/from JSON correctly (ID, Type, Payload preserved) + + + + Create `internal/queue/waq.go`: + ```go + package queue + + import ( + "context" + "encoding/json" + "fmt" + "log" + "time" + + "github.com/google/uuid" + "github.com/redis/go-redis/v9" + ) + + const queueKey = "hwlab:netbox:pending_ops" + + // PendingOp represents a queued NetBox operation. + // Serialized as JSON in DragonFlyDB LIST. + type PendingOp struct { + ID string `json:"id"` // UUID + Type string `json:"type"` // "create_device", "patch_custom_fields", "sync_tags", etc. + Payload json.RawMessage `json:"payload"` // operation-specific data + CreatedAt time.Time `json:"created_at"` + Attempts int `json:"attempts"` // retry count + } + + // NewPendingOp creates a new PendingOp with a generated ID and current timestamp. + func NewPendingOp(opType string, payload interface{}) (PendingOp, error) { + data, err := json.Marshal(payload) + if err != nil { + return PendingOp{}, fmt.Errorf("marshal payload: %w", err) + } + return PendingOp{ + ID: uuid.New().String(), + Type: opType, + Payload: json.RawMessage(data), + CreatedAt: time.Now().UTC(), + Attempts: 0, + }, nil + } + + // WAQ is a write-ahead queue backed by DragonFlyDB (Redis-compatible). + // Operations are stored as JSON in a Redis LIST using RPUSH/BLPOP for FIFO ordering. + type WAQ struct { + rdb *redis.Client + } + + // NewWAQ creates a new WAQ connected to the given Redis/DragonFlyDB URL. + // Returns error if the URL is invalid or the server is unreachable. + func NewWAQ(redisURL string) (*WAQ, error) { + opt, err := redis.ParseURL(redisURL) + if err != nil { + return nil, fmt.Errorf("parse dragonfly url: %w", err) + } + client := redis.NewClient(opt) + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := client.Ping(ctx).Err(); err != nil { + client.Close() + return nil, fmt.Errorf("dragonfly unreachable at %s: %w", redisURL, err) + } + log.Printf("WAQ connected to DragonFlyDB") + return &WAQ{rdb: client}, nil + } + + // Enqueue pushes a PendingOp onto the right end of the queue (FIFO). + func (q *WAQ) Enqueue(ctx context.Context, op PendingOp) error { + data, err := json.Marshal(op) + if err != nil { + return fmt.Errorf("marshal op: %w", err) + } + return q.rdb.RPush(ctx, queueKey, data).Err() + } + + // Dequeue pops the oldest operation (blocking for up to timeout). + // Returns nil, nil if timeout elapses with no item. + // Returns nil, redis.Nil if queue is empty (non-blocking variant would use LPop). + func (q *WAQ) Dequeue(ctx context.Context, timeout time.Duration) (*PendingOp, error) { + result, err := q.rdb.BLPop(ctx, timeout, queueKey).Result() + if err == redis.Nil { + return nil, nil // timeout — no items + } + if err != nil { + return nil, fmt.Errorf("blpop: %w", err) + } + var op PendingOp + if err := json.Unmarshal([]byte(result[1]), &op); err != nil { + return nil, fmt.Errorf("unmarshal op: %w", err) + } + return &op, nil + } + + // Len returns the current number of pending operations in the queue. + func (q *WAQ) Len(ctx context.Context) (int64, error) { + n, err := q.rdb.LLen(ctx, queueKey).Result() + if err != nil { + return 0, fmt.Errorf("llen: %w", err) + } + return n, nil + } + + // Close releases the DragonFlyDB connection. + func (q *WAQ) Close() error { + return q.rdb.Close() + } + ``` + + Create `internal/queue/waq_test.go`: + ```go + package queue_test + + import ( + "context" + "encoding/json" + "os" + "testing" + "time" + + "git.georgsen.dk/hwlab/internal/queue" + ) + + func TestPendingOpJSON(t *testing.T) { + payload := map[string]string{"device_id": "42", "hw_id": "HW-00001"} + op, err := queue.NewPendingOp("create_device", payload) + if err != nil { + t.Fatalf("NewPendingOp: %v", err) + } + if op.ID == "" { + t.Error("ID should be a UUID") + } + if op.Type != "create_device" { + t.Errorf("Type: want create_device, got %s", op.Type) + } + // Round-trip JSON + data, _ := json.Marshal(op) + var op2 queue.PendingOp + if err := json.Unmarshal(data, &op2); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if op2.ID != op.ID { + t.Errorf("ID mismatch after round-trip: %s != %s", op2.ID, op.ID) + } + } + + func TestNewWAQInvalidURL(t *testing.T) { + _, err := queue.NewWAQ("not-a-redis-url") + if err == nil { + t.Error("expected error for invalid URL") + } + } + + // dragonflyURL returns the DragonFlyDB URL from env, or skips test if unreachable. + func dragonflyURL(t *testing.T) string { + t.Helper() + url := os.Getenv("HWLAB_DRAGONFLY_URL") + if url == "" { + url = "redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379" + } + return url + } + + func TestWAQEnqueueDequeue(t *testing.T) { + waq, err := queue.NewWAQ(dragonflyURL(t)) + if err != nil { + t.Skipf("DragonFlyDB unavailable: %v", err) + } + defer waq.Close() + + ctx := context.Background() + + // Clean up before test + // Note: In a real test suite, use a test-specific queue key. + // For now, just ensure queue starts non-empty cleanup is acceptable. + + op, _ := queue.NewPendingOp("test_op", map[string]string{"test": "value"}) + if err := waq.Enqueue(ctx, op); err != nil { + t.Fatalf("Enqueue: %v", err) + } + + n, err := waq.Len(ctx) + if err != nil { + t.Fatalf("Len: %v", err) + } + if n < 1 { + t.Error("expected at least 1 item after enqueue") + } + + got, err := waq.Dequeue(ctx, 2*time.Second) + if err != nil { + t.Fatalf("Dequeue: %v", err) + } + if got == nil { + t.Fatal("expected op from dequeue, got nil") + } + if got.ID != op.ID { + t.Errorf("dequeued op ID mismatch: want %s, got %s", op.ID, got.ID) + } + } + ``` + + + + cd /home/mikkel/homelabby && go test ./internal/queue/... -v -run "TestPendingOpJSON|TestNewWAQInvalidURL" + + + + - `go test ./internal/queue/... -run "TestPendingOpJSON|TestNewWAQInvalidURL"` passes (unit tests, no DragonFlyDB needed) + - `grep "hwlab:netbox:pending_ops" internal/queue/waq.go` returns the queueKey const + - `grep "RPush" internal/queue/waq.go` returns the Enqueue implementation + - `grep "BLPop" internal/queue/waq.go` returns the Dequeue implementation + - `grep "ParseURL" internal/queue/waq.go` returns the redis.ParseURL call + - `go build ./internal/queue/...` exits 0 + - If DragonFlyDB reachable: `go test ./internal/queue/... -v -run TestWAQEnqueueDequeue` passes + + + WAQ core implemented. Unit tests pass. FIFO enqueue/dequeue via RPUSH/BLPOP. Integration test skips gracefully when DragonFlyDB unreachable. + + + + Task 2: WAQ retry worker + wire into main binary + internal/queue/worker.go, cmd/hwlab/main.go + + + - /home/mikkel/homelabby/internal/queue/waq.go (WAQ type, PendingOp, Dequeue method) + - /home/mikkel/homelabby/cmd/hwlab/main.go (current main.go to understand wiring pattern) + - /home/mikkel/homelabby/internal/config/config.go (WAQRetryIntervalSeconds, DragonflyURL) + + + + 1. Create `internal/queue/worker.go`: + ```go + package queue + + import ( + "context" + "log" + "time" + ) + + // OpHandler is a function that processes a single dequeued operation. + // Returns nil on success, error if the operation should be re-queued. + type OpHandler func(ctx context.Context, op PendingOp) error + + // RunWorker runs a blocking BLPOP loop processing ops from the queue. + // It calls handler for each dequeued op. If handler returns an error, + // the op is re-enqueued with incremented Attempts count. + // Ops that exceed maxAttempts are dropped with a log warning. + // + // On DragonFlyDB connection loss, RunWorker backs off and retries connection. + // Call with a cancellable context to stop the worker cleanly. + func (q *WAQ) RunWorker(ctx context.Context, handler OpHandler, maxAttempts int, retryInterval time.Duration) { + log.Printf("WAQ worker started (maxAttempts=%d, retryInterval=%s)", maxAttempts, retryInterval) + for { + select { + case <-ctx.Done(): + log.Printf("WAQ worker stopping: %v", ctx.Err()) + return + default: + } + + op, err := q.Dequeue(ctx, 5*time.Second) + if err != nil { + // Connection error — back off before retrying + log.Printf("WAQ dequeue error: %v — backing off %s", err, retryInterval) + select { + case <-ctx.Done(): + return + case <-time.After(retryInterval): + } + continue + } + if op == nil { + // Timeout with no items — loop immediately (BLPOP already waited 5s) + continue + } + + // Process the operation + if err := handler(ctx, *op); err != nil { + op.Attempts++ + if op.Attempts >= maxAttempts { + log.Printf("WAQ: dropping op %s (type=%s) after %d failed attempts: %v", + op.ID, op.Type, op.Attempts, err) + continue + } + // Re-enqueue for retry + log.Printf("WAQ: re-enqueuing op %s (type=%s, attempt=%d): %v", + op.ID, op.Type, op.Attempts, err) + if enqErr := q.Enqueue(ctx, *op); enqErr != nil { + log.Printf("WAQ: failed to re-enqueue op %s: %v", op.ID, enqErr) + } + } + } + } + + // NoOpHandler is a placeholder op handler for Phase 1. + // Phase 2 will replace this with a real NetBox retry handler. + // It logs the operation and returns nil (success) so ops drain from the queue. + func NoOpHandler(ctx context.Context, op PendingOp) error { + log.Printf("WAQ [noop]: processing op %s (type=%s, attempts=%d)", op.ID, op.Type, op.Attempts) + return nil + } + ``` + + 2. Update `cmd/hwlab/main.go` to wire the WAQ: + Read the current main.go first, then add WAQ initialization and worker goroutine. + The WAQ initialization must be non-fatal — if DragonFlyDB is unavailable, the binary + still starts and serves HTTP. WAQ is degraded, not required. + + Updated main.go pattern: + ```go + package main + + import ( + "context" + "fmt" + "log" + "net/http" + "os/signal" + "syscall" + "time" + + "git.georgsen.dk/hwlab/internal/api" + "git.georgsen.dk/hwlab/internal/config" + "git.georgsen.dk/hwlab/internal/queue" + ) + + func main() { + cfg, err := config.Load() + if err != nil { + log.Fatalf("config: %v", err) + } + + // Context for graceful shutdown + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + // Start write-ahead queue worker (non-fatal if DragonFlyDB unavailable) + waq, err := queue.NewWAQ(cfg.DragonflyURL) + if err != nil { + log.Printf("WARNING: WAQ unavailable (%v) — NetBox operations will not be queued during downtime", err) + } else { + retryInterval := time.Duration(cfg.WAQRetryIntervalSeconds) * time.Second + go waq.RunWorker(ctx, queue.NoOpHandler, cfg.WAQMaxAttempts, retryInterval) + defer waq.Close() + log.Printf("WAQ worker started") + } + + router := api.NewRouter() + addr := fmt.Sprintf("%s:%d", cfg.Host, cfg.Port) + log.Printf("HWLab starting on %s", addr) + + srv := &http.Server{Addr: addr, Handler: router} + go func() { + if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed { + log.Fatalf("server: %v", err) + } + }() + + // Wait for shutdown signal + <-ctx.Done() + log.Println("Shutting down...") + shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + srv.Shutdown(shutdownCtx) + log.Println("Shutdown complete") + } + ``` + + + + cd /home/mikkel/homelabby && go build ./cmd/hwlab/... && go vet ./... + + + + - `go build ./cmd/hwlab/...` exits 0 + - `go vet ./...` exits 0 + - `grep "RunWorker" cmd/hwlab/main.go` returns the goroutine call + - `grep "NoOpHandler" internal/queue/worker.go` returns the placeholder handler + - `grep "maxAttempts" internal/queue/worker.go` returns the op drop condition + - `grep "WARNING: WAQ unavailable" cmd/hwlab/main.go` returns the non-fatal degraded path + - Binary starts without panic when DragonFlyDB is available + - Binary starts with WARNING log (not fatal) when DragonFlyDB is unavailable + - `go test ./...` remains green (no regressions from main.go changes) + + + WAQ worker implemented with exponential backoff, max attempts drop, and NoOpHandler placeholder. main.go updated with graceful shutdown and non-fatal WAQ initialization. Full `go test ./...` still green. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| internal/queue → DragonFlyDB | Redis protocol over TCP to 10.5.0.10:6379; password in URL | +| PendingOp.Payload | JSON RawMessage from NetBox operation context — validated by op handler | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Disposition | Mitigation Plan | +|-----------|----------|-----------|-------------|-----------------| +| T-05-01 | Information Disclosure | DragonFlyDB password in HWLAB_DRAGONFLY_URL | accept | Private homelab LAN (10.5.0.x); password is in .env which is .gitignored; no external access | +| T-05-02 | Tampering | PendingOp re-enqueue on failure | accept | Ops re-enqueued only by the worker itself; no external write path to the queue in Phase 1 | +| T-05-03 | Denial of Service | Queue accumulation if handler always fails | mitigate | maxAttempts drop logic — ops dropped after cfg.WAQMaxAttempts (default 5) failures; prevents unbounded queue growth | +| T-05-04 | Denial of Service | WAQ worker tight-loop on connection loss | mitigate | retryInterval backoff (default 30s) prevents hammering DragonFlyDB on reconnect | + + + +After both tasks complete: +- `go test ./...` passes (all packages, no regressions) +- `go build ./cmd/hwlab/...` exits 0 +- `go vet ./...` exits 0 +- If DragonFlyDB reachable: `go test ./internal/queue/... -v` shows TestWAQEnqueueDequeue PASS +- Start binary and verify it serves health endpoint: `./hwlab &; sleep 1; curl http://localhost:8080/api/health; kill %1` + + + +1. WAQ core: Enqueue, Dequeue, Len all working against DragonFlyDB (integration) or skipping gracefully (unit) +2. Worker: BLPOP loop with backoff, max attempts drop, context cancellation +3. main.go: Non-fatal WAQ init — WARNING log instead of panic when DragonFlyDB unavailable +4. Graceful shutdown: SIGINT triggers orderly HTTP server and WAQ worker shutdown +5. Full test suite `go test ./...` green + + + +After completion, create `.planning/phases/01-foundation/01-05-SUMMARY.md` with: +- Whether DragonFlyDB integration tests ran or skipped +- Any DragonFlyDB/go-redis v9 compatibility notes (DragonFlyDB is Redis-compatible but may have minor differences) +- Final `go test ./...` output +- Files created/modified +