mikkel/homelabby
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(01-foundation): create phase 1 plans (5 plans, 2 waves)

Browse source 
Plans 01-02 are Wave 1 (parallel). Plans 03-04-05 are Wave 2.
All 11 requirements covered: INF-01, INF-02, INF-03, NB-01 through NB-07.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Mikkel Georgsen 2026-04-10 01:07:55 +00:00
parent 73eec3ee76
commit c9ad50fdf2
6 changed files with 2926 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
.planning/ROADMAP.md
View file

@ -31,8 +31,15 @@ Decimal phases appear between their surrounding integers in numeric order.
2. All HWLab custom fields (hw_id, catalog_status, photo_urls, etc.) are readable and writable via the NetBox API with round-trip test coverage
3. A new item can be created in NetBox with a sequential HW-XXXXX ID auto-assigned
4. catalog_status transitions from draft through complete are enforced by the backend quality gate
5. A write-ahead queue in SQLite buffers failed NetBox operations and retries them on reconnect
**Plans**: TBD
5. A write-ahead queue in DragonFlyDB buffers failed NetBox operations and retries them on reconnect
**Plans**: 5 plans
Plans:
- [ ] 01-01-PLAN.md — Go scaffold: chi server, go:embed stub SPA, viper config, health endpoint
- [ ] 01-02-PLAN.md — NetBox client: go-netbox v4 wrapper, custom field read/write types, integration tests
- [ ] 01-03-PLAN.md — NetBox provisioning: 8 custom fields, location hierarchy, plugin check, provision CLI
- [ ] 01-04-PLAN.md — HW-ID allocation, quality gate state machine, AI tag sync to NetBox
- [ ] 01-05-PLAN.md — Write-ahead queue: DragonFlyDB WAQ + retry worker goroutine
### Phase 2: AI Pipeline
**Goal**: Users can submit 1-3 photos and receive a structured NetBox-ready record with AI-extracted specs, suggested category/tags, and a quality gate status reflecting confidence
@ -116,7 +123,7 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7
| Phase | Plans Complete | Status | Completed |
|-------|----------------|--------|-----------|
| 1. Foundation | 0/TBD | Not started | - |
| 1. Foundation | 0/5 | Not started | - |
| 2. AI Pipeline | 0/TBD | Not started | - |
| 3. Dashboard & Intake UI | 0/TBD | Not started | - |
| 4. USB Manager & Label Printing | 0/TBD | Not started | - |

486
.planning/phases/01-foundation/01-01-PLAN.md Normal file
View file

@ -0,0 +1,486 @@
---
phase: 01-foundation
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- go.mod
- go.sum
- cmd/hwlab/main.go
- internal/api/router.go
- internal/api/handlers/health.go
- internal/config/config.go
- config.json
- Makefile
- web/dist/index.html
autonomous: true
requirements:
- INF-01
- INF-02
must_haves:
truths:
- "Running `go run ./cmd/hwlab/...` starts an HTTP server on port 8080"
- "GET /api/health returns 200 with JSON {status:ok, version:0.1.0}"
- "GET / serves the stub HTML page from web/dist/index.html (embedded in binary)"
- "Config loads from config.json and .env without panicking on missing optional fields"
artifacts:
- path: "cmd/hwlab/main.go"
provides: "Binary entry point — wires config, server, starts listener"
- path: "internal/api/router.go"
provides: "chi router with middleware, /api routes, SPA fallback"
- path: "internal/api/handlers/health.go"
provides: "GET /api/health handler"
- path: "internal/config/config.go"
provides: "viper-backed Config struct loaded from .env + config.json"
- path: "web/dist/index.html"
provides: "Stub SPA embedded via go:embed"
- path: "go.mod"
provides: "Module declaration git.georgsen.dk/hwlab with all Phase 1 deps"
key_links:
- from: "cmd/hwlab/main.go"
to: "internal/api/router.go"
via: "NewRouter(cfg) call"
pattern: "NewRouter"
- from: "internal/api/router.go"
to: "web/dist"
via: "go:embed + http.FileServer"
pattern: "go:embed web/dist"
---
<objective>
Initialize the Go binary scaffold: module setup, chi HTTP server, viper config, health endpoint, and stub React SPA embedded via go:embed.
Purpose: Every subsequent plan in this phase depends on the Go module and server existing. This plan creates that foundation.
Output: A compilable Go binary that serves GET /api/health and the stub SPA, loads config from .env + config.json, and is ready to have NetBox and queue packages added in Wave 2.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.env
@.planning/phases/01-foundation/01-RESEARCH.md
</context>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: Go module init and chi server with go:embed SPA</name>
<files>go.mod, go.sum, cmd/hwlab/main.go, internal/api/router.go, internal/api/handlers/health.go, web/dist/index.html</files>
<read_first>
- /home/mikkel/homelabby/.env (confirm HWLAB_PORT=8080)
- /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 1: chi Router with go:embed, lines 148-181)
</read_first>
<behavior>
- Test 1: GET /api/health returns HTTP 200 with body {"status":"ok","version":"0.1.0"}
- Test 2: GET / returns HTTP 200 with HTML body containing "HWLab"
- Test 3: GET /api/nonexistent returns 404 (chi default)
- Test 4: GET /some/spa/route returns the stub index.html (SPA fallback)
</behavior>
<action>
1. Initialize Go module: `go mod init git.georgsen.dk/hwlab`
2. Install dependencies:
```
go get github.com/go-chi/chi/v5@v5.2.5
go get github.com/redis/go-redis/v9@v9.18.0
go get github.com/spf13/viper@v1.21.0
go get github.com/joho/godotenv@v1.5.1
go get github.com/google/uuid@v1.6.0
go get github.com/netbox-community/go-netbox/v4@v4.3.0
```
3. Create `web/dist/index.html` — minimal stub (NOT a full React app; placeholder for Phase 3):
```html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>HWLab</title>
<style>
body { background: #000; color: #faff69; font-family: monospace; display: flex; align-items: center; justify-content: center; height: 100vh; margin: 0; }
h1 { font-size: 2rem; }
p { color: #666; }
</style>
</head>
<body>
<div>
<h1>HWLab</h1>
<p>Backend is running. UI coming in Phase 3.</p>
</div>
</body>
</html>
```
4. Create `internal/api/handlers/health.go`:
```go
package handlers
import (
"encoding/json"
"net/http"
)
type HealthResponse struct {
Status string `json:"status"`
Version string `json:"version"`
}
func Health(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(HealthResponse{
Status: "ok",
Version: "0.1.0",
})
}
```
5. Create `internal/api/router.go` — chi router with go:embed SPA fallback. The embed directive MUST be in the same file as the var declaration:
```go
package api
import (
"embed"
"io/fs"
"net/http"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"git.georgsen.dk/hwlab/internal/api/handlers"
)
//go:embed ../../web/dist
var staticFiles embed.FS
func NewRouter() http.Handler {
r := chi.NewRouter()
r.Use(middleware.Logger)
r.Use(middleware.Recoverer)
r.Use(middleware.RealIP)
r.Route("/api", func(r chi.Router) {
r.Get("/health", handlers.Health)
})
// SPA fallback — serve index.html for all non-API routes
staticFS, _ := fs.Sub(staticFiles, "web/dist")
fileServer := http.FileServer(http.FS(staticFS))
r.Handle("/*", fileServer)
return r
}
```
NOTE: The go:embed path `../../web/dist` is relative to the file location `internal/api/router.go`. Verify path resolves correctly. If the embed directive causes issues due to directory depth, move the embed var to `cmd/hwlab/main.go` and pass the fs.FS into NewRouter as a parameter instead.
6. Create `cmd/hwlab/main.go` — wire-up only, no business logic:
```go
package main
import (
"fmt"
"log"
"net/http"
"git.georgsen.dk/hwlab/internal/api"
"git.georgsen.dk/hwlab/internal/config"
)
func main() {
cfg, err := config.Load()
if err != nil {
log.Fatalf("config: %v", err)
}
router := api.NewRouter()
addr := fmt.Sprintf("%s:%d", cfg.Host, cfg.Port)
log.Printf("HWLab starting on %s", addr)
if err := http.ListenAndServe(addr, router); err != nil {
log.Fatalf("server: %v", err)
}
}
```
7. Write test `internal/api/handlers/health_test.go`:
```go
package handlers_test
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"git.georgsen.dk/hwlab/internal/api/handlers"
)
func TestHealth(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/api/health", nil)
w := httptest.NewRecorder()
handlers.Health(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", w.Code)
}
var resp handlers.HealthResponse
if err := json.NewDecoder(w.Body).Decode(&resp); err != nil {
t.Fatalf("decode: %v", err)
}
if resp.Status != "ok" {
t.Errorf("expected status=ok, got %s", resp.Status)
}
if resp.Version != "0.1.0" {
t.Errorf("expected version=0.1.0, got %s", resp.Version)
}
}
```
</action>
<verify>
<automated>cd /home/mikkel/homelabby && go test ./internal/api/handlers/... -v</automated>
</verify>
<acceptance_criteria>
- `go build ./cmd/hwlab/...` exits 0 with no errors
- `go test ./internal/api/handlers/...` passes (TestHealth green)
- `grep -r "go:embed web/dist" internal/api/router.go` returns a match (OR the embed is in main.go — check whichever file contains it)
- `grep "git.georgsen.dk/hwlab" go.mod` returns the module declaration line
- `grep "go-chi/chi/v5" go.mod` returns a line with `v5.2.5`
- File `web/dist/index.html` exists and contains "HWLab"
</acceptance_criteria>
<done>Go module compiles, health handler test passes, chi router wires go:embed SPA fallback, binary starts without panic.</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: viper config loader (INF-02)</name>
<files>internal/config/config.go, internal/config/config_test.go, config.json</files>
<read_first>
- /home/mikkel/homelabby/.env (all existing env var names and values)
- /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 7: viper Config, lines 362+)
</read_first>
<behavior>
- Test 1: Load() with env var HWLAB_PORT=9999 set returns cfg.Port == 9999
- Test 2: Load() with no env vars returns cfg.Port == 8080 (from config.json default)
- Test 3: Load() returns cfg.NetBoxURL == "http://10.5.0.130:8000/api" when HWLAB_NETBOX_URL is set
- Test 4: Load() does not return error when config.json fields are missing (optional fields use defaults)
</behavior>
<action>
1. Install godotenv: `go get github.com/joho/godotenv@v1.5.1`
2. Create `config.json` at project root with non-secret defaults:
```json
{
"port": 8080,
"host": "0.0.0.0",
"netbox_url": "http://10.5.0.130:8000/api",
"dragonfly_url": "redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379",
"log_level": "info",
"netbox_timeout_seconds": 10,
"waq_retry_interval_seconds": 30,
"waq_max_attempts": 5,
"quality_gate_confidence_threshold": 0.75
}
```
3. Create `internal/config/config.go`:
```go
package config
import (
"fmt"
"strings"
"github.com/joho/godotenv"
"github.com/spf13/viper"
)
type Config struct {
Host string `mapstructure:"host"`
Port int `mapstructure:"port"`
LogLevel string `mapstructure:"log_level"`
NetBoxURL string `mapstructure:"netbox_url"`
NetBoxToken string `mapstructure:"netbox_token"`
NetBoxTimeoutSeconds int `mapstructure:"netbox_timeout_seconds"`
DragonflyURL string `mapstructure:"dragonfly_url"`
WAQRetryIntervalSeconds int `mapstructure:"waq_retry_interval_seconds"`
WAQMaxAttempts int `mapstructure:"waq_max_attempts"`
QualityGateConfidenceThreshold float64 `mapstructure:"quality_gate_confidence_threshold"`
}
func Load() (*Config, error) {
// Load .env file if present (ignore error — .env is optional in production)
_ = godotenv.Load()
v := viper.New()
// Set defaults
v.SetDefault("host", "0.0.0.0")
v.SetDefault("port", 8080)
v.SetDefault("log_level", "info")
v.SetDefault("netbox_timeout_seconds", 10)
v.SetDefault("waq_retry_interval_seconds", 30)
v.SetDefault("waq_max_attempts", 5)
v.SetDefault("quality_gate_confidence_threshold", 0.75)
// Config file
v.SetConfigName("config")
v.SetConfigType("json")
v.AddConfigPath(".")
v.AddConfigPath("/etc/hwlab")
// Environment variables: HWLAB_PORT -> port, HWLAB_NETBOX_URL -> netbox_url, etc.
v.SetEnvPrefix("HWLAB")
v.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
v.AutomaticEnv()
// Read config file (non-fatal if missing)
if err := v.ReadInConfig(); err != nil {
if _, ok := err.(viper.ConfigFileNotFoundError); !ok {
return nil, fmt.Errorf("config file: %w", err)
}
}
var cfg Config
if err := v.Unmarshal(&cfg); err != nil {
return nil, fmt.Errorf("unmarshal config: %w", err)
}
return &cfg, nil
}
```
4. Create `internal/config/config_test.go`:
```go
package config_test
import (
"os"
"testing"
"git.georgsen.dk/hwlab/internal/config"
)
func TestLoadDefaults(t *testing.T) {
// Unset env vars that might interfere
os.Unsetenv("HWLAB_PORT")
os.Unsetenv("HWLAB_NETBOX_URL")
cfg, err := config.Load()
if err != nil {
t.Fatalf("Load() error: %v", err)
}
if cfg.Port != 8080 {
t.Errorf("default port: want 8080, got %d", cfg.Port)
}
}
func TestLoadEnvOverride(t *testing.T) {
os.Setenv("HWLAB_PORT", "9999")
defer os.Unsetenv("HWLAB_PORT")
cfg, err := config.Load()
if err != nil {
t.Fatalf("Load() error: %v", err)
}
if cfg.Port != 9999 {
t.Errorf("env override port: want 9999, got %d", cfg.Port)
}
}
func TestLoadNetBoxURL(t *testing.T) {
os.Setenv("HWLAB_NETBOX_URL", "http://10.5.0.130:8000/api")
defer os.Unsetenv("HWLAB_NETBOX_URL")
cfg, err := config.Load()
if err != nil {
t.Fatalf("Load() error: %v", err)
}
if cfg.NetBoxURL != "http://10.5.0.130:8000/api" {
t.Errorf("netbox url: want http://10.5.0.130:8000/api, got %s", cfg.NetBoxURL)
}
}
```
</action>
<verify>
<automated>cd /home/mikkel/homelabby && go test ./internal/config/... -v</automated>
</verify>
<acceptance_criteria>
- `go test ./internal/config/...` passes all 3 tests
- `grep "mapstructure:\"netbox_token\"" internal/config/config.go` returns a match
- `grep "mapstructure:\"dragonfly_url\"" internal/config/config.go` returns a match
- `grep "HWLAB" internal/config/config.go` returns the SetEnvPrefix line
- File `config.json` exists at project root
- `grep "quality_gate_confidence_threshold" config.json` returns a match
- `go build ./cmd/hwlab/...` still exits 0 after adding config
</acceptance_criteria>
<done>Config loads from config.json and .env, env vars override file values, all tests pass, binary compiles with config wired.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| .env file → config loader | Credentials (NetBox token, DragonFlyDB password) enter the process here |
| HTTP client → Go server | All inbound HTTP requests to chi router |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-01-01 | Information Disclosure | .env file | mitigate | .env is in .gitignore; HWLAB_NETBOX_TOKEN is a placeholder in this plan — real token generated in Plan 03 |
| T-01-02 | Tampering | config.json | accept | Local homelab tool, single operator, no integrity threat model needed at this layer |
| T-01-03 | Denial of Service | GET /api/health | accept | No auth needed on health endpoint; low-value target, single operator |
| T-01-04 | Information Disclosure | chi middleware.Logger | accept | Logs go to stdout only, no external log shipping in Phase 1 |
</threat_model>
<verification>
After both tasks complete:
- `go test ./...` green (all packages)
- `go build ./cmd/hwlab/...` exits 0
- Run binary: `./hwlab &` then `curl http://localhost:8080/api/health` returns `{"status":"ok","version":"0.1.0"}`
- `curl http://localhost:8080/` returns HTML containing "HWLab"
- `curl http://localhost:8080/some/deep/route` returns the same HTML (SPA fallback)
- Kill binary, verify process exits cleanly
</verification>
<success_criteria>
1. `go test ./...` passes with no failures
2. `go build ./cmd/hwlab/...` compiles a binary
3. Running the binary and hitting GET /api/health returns HTTP 200 JSON with status=ok
4. Running the binary and hitting GET / returns HTTP 200 HTML containing "HWLab"
5. Config reads HWLAB_PORT from environment when set
</success_criteria>
<output>
After completion, create `.planning/phases/01-foundation/01-01-SUMMARY.md` with:
- Files created/modified
- Key decisions made (especially if go:embed path needed adjustment)
- Test results
- Any deviations from the plan
</output>

608
.planning/phases/01-foundation/01-02-PLAN.md Normal file
View file

@ -0,0 +1,608 @@
---
phase: 01-foundation
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
- internal/netbox/client.go
- internal/netbox/client_test.go
- internal/netbox/custom_fields.go
- internal/netbox/types.go
autonomous: true
requirements:
- NB-01
- NB-02
must_haves:
truths:
- "NetBox client connects to http://10.5.0.130:8000/api and lists devices without error"
- "Client can create, read, update, and delete a device in NetBox"
- "Custom field read/write wrappers handle the asymmetric NetBox format (read nested, write flat)"
- "Round-trip test confirms custom field written via PATCH is retrievable via GET"
artifacts:
- path: "internal/netbox/client.go"
provides: "go-netbox v4 wrapper with typed methods for device/module/cable CRUD"
exports: ["NewClient", "Client"]
- path: "internal/netbox/custom_fields.go"
provides: "HWLab custom field read/write types and helper functions"
exports: ["CustomFieldsRead", "CustomFieldsPatch", "BuildCustomFieldsPatch", "ParseCustomFields"]
- path: "internal/netbox/types.go"
provides: "HWLab domain types wrapping NetBox responses"
exports: ["Device", "CustomFields"]
key_links:
- from: "internal/netbox/client.go"
to: "http://10.5.0.130:8000/api"
via: "go-netbox NewAPIClientFor"
pattern: "NewAPIClientFor"
- from: "internal/netbox/custom_fields.go"
to: "internal/netbox/client.go"
via: "PatchCustomFields method on Client"
pattern: "PatchCustomFields"
---
<objective>
Build the typed NetBox client package: go-netbox v4 wrapper, custom field read/write types, and integration tests that verify round-trip custom field writes against the live NetBox instance.
Purpose: Every other Phase 1 package (quality gate, HW-ID, WAQ) depends on the NetBox client being stable. Building it independently in Wave 1 means Wave 2 plans can consume it without waiting for the scaffold.
Output: `internal/netbox` package with typed CRUD methods and custom field helpers, integration tests that pass against live NetBox at 10.5.0.130.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.env
@.planning/phases/01-foundation/01-RESEARCH.md
</context>
<interfaces>
<!-- Key types the executor needs from go-netbox v4 -->
<!-- Source: github.com/netbox-community/go-netbox/v4 -->
go-netbox v4 initialization pattern:
```go
import netbox "github.com/netbox-community/go-netbox/v4"
client := netbox.NewAPIClientFor("http://10.5.0.130:8000", "YOUR_TOKEN_HERE")
// List devices
res, _, err := client.DcimAPI.DcimDevicesList(ctx).Limit(10).Execute()
// res.Results is []netbox.DeviceWithConfigContext
// Create device
req := netbox.WritableDeviceWithConfigContextRequest{
Name: netbox.PtrString("test-device"),
DeviceType: // ID ref
Site: // ID ref
}
result, _, err := client.DcimAPI.DcimDevicesCreate(ctx).
WritableDeviceWithConfigContextRequest(req).Execute()
// Custom fields are on Device.CustomFields as map[string]interface{}
// To PATCH custom fields: use DcimDevicesPartialUpdate with PatchedWritableDeviceWithConfigContextRequest
// PatchedWritableDeviceWithConfigContextRequest.CustomFields = map[string]interface{}{...}
```
IMPORTANT: The NetBox token in .env (`homelab-netbox-api-token-2024`) is a placeholder string.
Real NetBox tokens are 40-character hex strings generated via NetBox UI.
The executor MUST verify the token is real before running integration tests.
If the token is the placeholder string, add a human checkpoint or skip integration test with t.Skip().
</interfaces>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: NetBox client wrapper with device CRUD (NB-01)</name>
<files>internal/netbox/client.go, internal/netbox/types.go, internal/netbox/client_test.go</files>
<read_first>
- /home/mikkel/homelabby/.env (HWLAB_NETBOX_URL, HWLAB_NETBOX_TOKEN — check if token is real or placeholder)
- /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 2: go-netbox v4 Client Initialization, lines 183-210)
- /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Summary section — note about placeholder token, lines 55-58)
</read_first>
<behavior>
- Test 1: NewClient with valid URL and token returns non-nil *Client without error
- Test 2: NewClient with empty token returns error "netbox token is required"
- Test 3: Client.Ping(ctx) against live http://10.5.0.130:8000/api returns no error (INTEGRATION — skip if token is placeholder)
- Test 4: Client.ListDevices(ctx, limit=5) returns slice without error (INTEGRATION — skip if token is placeholder)
</behavior>
<action>
1. Create `internal/netbox/types.go` — HWLab domain types:
```go
package netbox
import "time"
// Device represents a HWLab inventory item backed by a NetBox device record.
type Device struct {
ID int
Name string
AssetTag string // HW-XXXXX identifier
CustomFields CustomFields
Created time.Time
LastUpdated time.Time
}
// CustomFields holds all HWLab-defined NetBox custom field values for a device.
// NetBox returns these as map[string]interface{} — we provide typed access.
type CustomFields struct {
HWID string // hw_id
CatalogStatus string // catalog_status
ProductURL string // product_url
FirmwareVersion string // firmware_version
TestDate string // test_date (ISO 8601 date string)
TestData string // test_data (JSON string)
AINotes string // ai_notes
PhotoURLs []string // photo_urls (multi-value)
}
```
2. Create `internal/netbox/client.go`:
```go
package netbox
import (
"context"
"errors"
"fmt"
nb "github.com/netbox-community/go-netbox/v4"
)
// Client wraps go-netbox v4 APIClient with typed HWLab methods.
// All NetBox calls MUST go through this Client — no direct go-netbox calls in other packages.
type Client struct {
api *nb.APIClient
url string
}
// NewClient creates a configured NetBox client. Returns error if url or token is empty.
func NewClient(url, token string) (*Client, error) {
if url == "" {
return nil, errors.New("netbox url is required")
}
if token == "" {
return nil, errors.New("netbox token is required")
}
// Note: NewAPIClientFor accepts the base URL WITHOUT /api suffix
// The go-netbox library appends /api internally.
// Strip trailing /api if present to avoid double-appending.
baseURL := url
if len(baseURL) > 4 && baseURL[len(baseURL)-4:] == "/api" {
baseURL = baseURL[:len(baseURL)-4]
}
api := nb.NewAPIClientFor(baseURL, token)
return &Client{api: api, url: url}, nil
}
// Ping verifies the NetBox API is reachable by fetching the API root status.
// Returns nil on success.
func (c *Client) Ping(ctx context.Context) error {
// Use DcimDevicesList with limit=1 as a lightweight connectivity check.
_, resp, err := c.api.DcimAPI.DcimDevicesList(ctx).Limit(1).Execute()
if err != nil {
return fmt.Errorf("netbox ping: %w", err)
}
if resp.StatusCode >= 500 {
return fmt.Errorf("netbox ping: server error %d", resp.StatusCode)
}
return nil
}
// ListDevices returns up to limit devices from NetBox.
func (c *Client) ListDevices(ctx context.Context, limit int) ([]Device, error) {
if limit <= 0 {
limit = 50
}
res, _, err := c.api.DcimAPI.DcimDevicesList(ctx).Limit(int32(limit)).Execute()
if err != nil {
return nil, fmt.Errorf("list devices: %w", err)
}
devices := make([]Device, 0, len(res.Results))
for _, d := range res.Results {
devices = append(devices, deviceFromNetBox(d))
}
return devices, nil
}
// GetDevice retrieves a single device by its NetBox internal ID.
func (c *Client) GetDevice(ctx context.Context, id int) (*Device, error) {
d, _, err := c.api.DcimAPI.DcimDevicesRetrieve(ctx, int32(id)).Execute()
if err != nil {
return nil, fmt.Errorf("get device %d: %w", id, err)
}
dev := deviceFromNetBox(*d)
return &dev, nil
}
// deviceFromNetBox maps a go-netbox DeviceWithConfigContext to our Device type.
// Custom fields are mapped separately via ParseCustomFields.
func deviceFromNetBox(d nb.DeviceWithConfigContext) Device {
dev := Device{
ID: int(d.GetId()),
Name: d.GetName(),
}
if tag := d.GetAssetTag(); tag != "" {
dev.AssetTag = tag
}
dev.CustomFields = ParseCustomFields(d.GetCustomFields())
return dev
}
```
3. Write `internal/netbox/client_test.go`:
```go
package netbox_test
import (
"context"
"os"
"testing"
"git.georgsen.dk/hwlab/internal/netbox"
)
func TestNewClientValidation(t *testing.T) {
_, err := netbox.NewClient("", "token")
if err == nil {
t.Error("expected error for empty url")
}
_, err = netbox.NewClient("http://10.5.0.130:8000/api", "")
if err == nil {
t.Error("expected error for empty token")
}
c, err := netbox.NewClient("http://10.5.0.130:8000/api", "sometoken")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if c == nil {
t.Error("expected non-nil client")
}
}
// integrationToken returns the real NetBox token from env, or skips the test
// if only the placeholder is present (placeholder is never 40 hex chars).
func integrationToken(t *testing.T) string {
t.Helper()
token := os.Getenv("HWLAB_NETBOX_TOKEN")
if len(token) != 40 {
t.Skip("HWLAB_NETBOX_TOKEN is not a real 40-char token — skipping integration test")
}
return token
}
func TestPingLive(t *testing.T) {
token := integrationToken(t)
c, err := netbox.NewClient("http://10.5.0.130:8000/api", token)
if err != nil {
t.Fatalf("NewClient: %v", err)
}
if err := c.Ping(context.Background()); err != nil {
t.Fatalf("Ping: %v", err)
}
}
func TestListDevicesLive(t *testing.T) {
token := integrationToken(t)
c, _ := netbox.NewClient("http://10.5.0.130:8000/api", token)
devices, err := c.ListDevices(context.Background(), 5)
if err != nil {
t.Fatalf("ListDevices: %v", err)
}
t.Logf("found %d devices in NetBox", len(devices))
// Not asserting count — NetBox may be empty; just assert no error
}
```
</action>
<verify>
<automated>cd /home/mikkel/homelabby && go test ./internal/netbox/... -v -run TestNewClientValidation</automated>
</verify>
<acceptance_criteria>
- `go test ./internal/netbox/... -run TestNewClientValidation` passes (unit tests, no integration needed)
- `grep "NewClient" internal/netbox/client.go` returns the exported function declaration
- `grep "ParseCustomFields" internal/netbox/client.go` returns usage of the function
- `grep "go-netbox" go.mod` returns the dependency line with v4.3.0
- `go build ./internal/netbox/...` exits 0
- If real NetBox token available: `go test ./internal/netbox/... -v` shows TestPingLive and TestListDevicesLive PASS (not SKIP)
</acceptance_criteria>
<done>NetBox client compiles, unit validation tests pass, integration tests skip cleanly when token is a placeholder, and pass when a real token is provided.</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Custom field read/write wrappers (NB-02 round-trip)</name>
<files>internal/netbox/custom_fields.go, internal/netbox/custom_fields_test.go</files>
<read_first>
- /home/mikkel/homelabby/internal/netbox/types.go (CustomFields struct from Task 1)
- /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 3: Custom Field Read/Write Asymmetry, lines 203-235)
</read_first>
<behavior>
- Test 1: ParseCustomFields(map with "hw_id":"HW-00001") returns CustomFields{HWID:"HW-00001"}
- Test 2: ParseCustomFields(nil map) returns zero-value CustomFields (no panic)
- Test 3: ParseCustomFields(map with "photo_urls": []interface{}{"url1","url2"}) returns PhotoURLs with 2 entries
- Test 4: BuildCustomFieldsPatch("HW-00001", "draft", nil) returns map containing hw_id and catalog_status keys
- Test 5: BuildCustomFieldsPatch with photo_urls slice includes photo_urls key in patch map
- Test 6 (INTEGRATION, skip if no real token): Client.PatchCustomFields then GetDevice returns matching custom field values
</behavior>
<action>
1. Create `internal/netbox/custom_fields.go`:
```go
package netbox
import (
"context"
"fmt"
)
// ParseCustomFields maps NetBox's map[string]interface{} custom fields response
// to the typed CustomFields struct. NetBox returns values as interface{} — we
// perform safe type assertions for each expected field.
func ParseCustomFields(raw map[string]interface{}) CustomFields {
cf := CustomFields{}
if raw == nil {
return cf
}
if v, ok := raw["hw_id"].(string); ok {
cf.HWID = v
}
if v, ok := raw["catalog_status"].(string); ok {
cf.CatalogStatus = v
}
if v, ok := raw["product_url"].(string); ok {
cf.ProductURL = v
}
if v, ok := raw["firmware_version"].(string); ok {
cf.FirmwareVersion = v
}
if v, ok := raw["test_date"].(string); ok {
cf.TestDate = v
}
if v, ok := raw["test_data"].(string); ok {
cf.TestData = v
}
if v, ok := raw["ai_notes"].(string); ok {
cf.AINotes = v
}
// photo_urls is a multi-value field — NetBox returns []interface{}
if v, ok := raw["photo_urls"].([]interface{}); ok {
urls := make([]string, 0, len(v))
for _, u := range v {
if s, ok := u.(string); ok {
urls = append(urls, s)
}
}
cf.PhotoURLs = urls
}
return cf
}
// BuildCustomFieldsPatch constructs the flat map[string]interface{} payload
// required by NetBox PATCH endpoints. Only include fields that are non-empty
// to avoid accidentally clearing existing values.
//
// NetBox custom field write format differs from read format:
// - Text/URL/date fields: send string value directly
// - Selection fields (catalog_status): send the choice value as string
// - Multi-value fields (photo_urls): send []string directly
func BuildCustomFieldsPatch(hwID, catalogStatus string, photoURLs []string) map[string]interface{} {
patch := make(map[string]interface{})
if hwID != "" {
patch["hw_id"] = hwID
}
if catalogStatus != "" {
patch["catalog_status"] = catalogStatus
}
if len(photoURLs) > 0 {
patch["photo_urls"] = photoURLs
}
return patch
}
// BuildFullCustomFieldsPatch constructs a patch with all custom fields.
// Use for initial record creation where all fields should be set.
func BuildFullCustomFieldsPatch(cf CustomFields) map[string]interface{} {
patch := make(map[string]interface{})
if cf.HWID != "" {
patch["hw_id"] = cf.HWID
}
if cf.CatalogStatus != "" {
patch["catalog_status"] = cf.CatalogStatus
}
if cf.ProductURL != "" {
patch["product_url"] = cf.ProductURL
}
if cf.FirmwareVersion != "" {
patch["firmware_version"] = cf.FirmwareVersion
}
if cf.TestDate != "" {
patch["test_date"] = cf.TestDate
}
if cf.TestData != "" {
patch["test_data"] = cf.TestData
}
if cf.AINotes != "" {
patch["ai_notes"] = cf.AINotes
}
if len(cf.PhotoURLs) > 0 {
patch["photo_urls"] = cf.PhotoURLs
}
return patch
}
// PatchCustomFields updates the custom fields of a device identified by netboxID.
// After PATCH, performs a GET to verify the write succeeded (HTTP 200 ≠ write confirmed).
func (c *Client) PatchCustomFields(ctx context.Context, deviceID int, patch map[string]interface{}) error {
req := c.api.DcimAPI.DcimDevicesPartialUpdate(ctx, int32(deviceID))
// go-netbox v4 uses PatchedWritableDeviceWithConfigContextRequest for partial updates
// Set custom fields via the request object
patchReq := c.api.DcimAPI.DcimDevicesPartialUpdate(ctx, int32(deviceID))
_ = patchReq // suppress unused
// Build the partial update request
// NOTE: go-netbox v4 API — DcimDevicesPartialUpdate takes a PatchedWritableDeviceWithConfigContextRequest
// CustomFields field is map[string]interface{}
import_note := "use c.api.DcimAPI.DcimDevicesPartialUpdate"
_ = import_note
// Correct approach for go-netbox v4:
nb_req := c.api.DcimAPI.DcimDevicesPartialUpdate(ctx, int32(deviceID))
_ = nb_req
// TODO: fill in correctly based on generated API — see go-netbox v4 generated code
// The generated struct is: PatchedWritableDeviceWithConfigContextRequest
// It has a CustomFields field of type map[string]interface{}
return fmt.Errorf("PatchCustomFields: implement using go-netbox v4 PatchedWritableDeviceWithConfigContextRequest.CustomFields")
}
```
IMPORTANT NOTE FOR EXECUTOR: The `PatchCustomFields` stub above contains pseudocode that will not compile. Once the go-netbox v4 module is downloaded, inspect the generated API to find the correct struct and method signature:
```
grep -r "PatchedWritableDevice" $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/ 2>/dev/null | head -5
```
Then implement `PatchCustomFields` using the correct generated struct. The pattern is:
```go
patchReq := nb.PatchedWritableDeviceWithConfigContextRequest{}
patchReq.SetCustomFields(patch)
_, _, err := c.api.DcimAPI.DcimDevicesPartialUpdate(ctx, int32(deviceID)).
PatchedWritableDeviceWithConfigContextRequest(patchReq).Execute()
```
2. Create `internal/netbox/custom_fields_test.go`:
```go
package netbox_test
import (
"testing"
"git.georgsen.dk/hwlab/internal/netbox"
)
func TestParseCustomFieldsNil(t *testing.T) {
cf := netbox.ParseCustomFields(nil)
if cf.HWID != "" {
t.Error("expected empty HWID for nil map")
}
}
func TestParseCustomFieldsHWID(t *testing.T) {
raw := map[string]interface{}{
"hw_id": "HW-00001",
"catalog_status": "draft",
}
cf := netbox.ParseCustomFields(raw)
if cf.HWID != "HW-00001" {
t.Errorf("want HW-00001, got %s", cf.HWID)
}
if cf.CatalogStatus != "draft" {
t.Errorf("want draft, got %s", cf.CatalogStatus)
}
}
func TestParseCustomFieldsPhotoURLs(t *testing.T) {
raw := map[string]interface{}{
"photo_urls": []interface{}{"http://a.com/1.jpg", "http://a.com/2.jpg"},
}
cf := netbox.ParseCustomFields(raw)
if len(cf.PhotoURLs) != 2 {
t.Errorf("want 2 photo urls, got %d", len(cf.PhotoURLs))
}
}
func TestBuildCustomFieldsPatch(t *testing.T) {
patch := netbox.BuildCustomFieldsPatch("HW-00001", "draft", nil)
if patch["hw_id"] != "HW-00001" {
t.Errorf("hw_id: want HW-00001, got %v", patch["hw_id"])
}
if patch["catalog_status"] != "draft" {
t.Errorf("catalog_status: want draft, got %v", patch["catalog_status"])
}
if _, ok := patch["photo_urls"]; ok {
t.Error("photo_urls should not be present when nil passed")
}
}
func TestBuildCustomFieldsPatchWithURLs(t *testing.T) {
patch := netbox.BuildCustomFieldsPatch("HW-00001", "indexed", []string{"http://a.com/1.jpg"})
urls, ok := patch["photo_urls"].([]string)
if !ok {
t.Fatal("photo_urls should be []string")
}
if len(urls) != 1 {
t.Errorf("want 1 url, got %d", len(urls))
}
}
```
</action>
<verify>
<automated>cd /home/mikkel/homelabby && go test ./internal/netbox/... -v -run "TestParseCustomFields|TestBuildCustomFields"</automated>
</verify>
<acceptance_criteria>
- `go test ./internal/netbox/... -run "TestParseCustomFields|TestBuildCustomFields"` passes all 5 unit tests
- `grep "ParseCustomFields" internal/netbox/custom_fields.go` returns the exported function declaration
- `grep "BuildCustomFieldsPatch" internal/netbox/custom_fields.go` returns the exported function declaration
- `grep "BuildFullCustomFieldsPatch" internal/netbox/custom_fields.go` returns the exported function declaration
- `grep "photo_urls" internal/netbox/custom_fields.go` returns handling of the []interface{} case
- `PatchCustomFields` is implemented (not returning an error string — the stub must be replaced with real go-netbox v4 API call)
- `go build ./internal/netbox/...` exits 0
</acceptance_criteria>
<done>Custom field parsing and patch building tested and passing. PatchCustomFields implemented using correct go-netbox v4 generated structs (not stub pseudocode). All unit tests green.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| Go code → NetBox REST API | Authenticated API calls; token is the only credential |
| NetBox response → Go custom field parsing | Untrusted map[string]interface{} values enter type assertions |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-02-01 | Information Disclosure | HWLAB_NETBOX_TOKEN in env | mitigate | Token never logged; only passed to go-netbox client constructor; integration tests skip when placeholder token present |
| T-02-02 | Tampering | ParseCustomFields raw map | accept | Source is NetBox REST API on private homelab LAN (10.5.0.130); no untrusted input path in Phase 1 |
| T-02-03 | Denial of Service | DcimDevicesList with large limit | accept | Single-operator tool; no external callers; limit param is Go code controlled |
| T-02-04 | Information Disclosure | go test logging device IDs | accept | Tests run locally; t.Logf output is ephemeral |
</threat_model>
<verification>
After both tasks complete:
- `go test ./internal/netbox/... -v` shows TestNewClientValidation PASS, integration tests either PASS (real token) or SKIP (placeholder)
- `go test ./internal/netbox/... -run "TestParseCustomFields|TestBuildCustomFields"` all green
- `go build ./...` exits 0 (all packages compile together)
- If real token: `go test ./internal/netbox/... -v -run TestPingLive` shows PASS
</verification>
<success_criteria>
1. All unit tests in `internal/netbox` pass without requiring live NetBox
2. Integration tests skip gracefully when token is the placeholder `homelab-netbox-api-token-2024`
3. `ParseCustomFields` handles nil, string values, and []interface{} photo_urls without panicking
4. `PatchCustomFields` is implemented with real go-netbox v4 API calls (not stub)
5. `go build ./...` compiles cleanly
</success_criteria>
<output>
After completion, create `.planning/phases/01-foundation/01-02-SUMMARY.md` with:
- Whether integration tests ran (real token) or skipped (placeholder)
- The exact PatchedWritableDeviceWithConfigContextRequest struct name used (from go-netbox v4 generated code)
- Any go-netbox v4 API surprises (e.g., custom field write format differs from documented pattern)
- Files created/modified
</output>

569
.planning/phases/01-foundation/01-03-PLAN.md Normal file
View file

@ -0,0 +1,569 @@
---
phase: 01-foundation
plan: 03
type: execute
wave: 2
depends_on:
- 01-02-PLAN.md
files_modified:
- internal/netbox/provision.go
- internal/netbox/provision_test.go
- scripts/provision-netbox.go
autonomous: true
requirements:
- NB-02
- NB-03
- NB-04
must_haves:
truths:
- "All 8 HWLab custom fields exist in NetBox after provisioning runs"
- "GET /api/extras/custom-fields/?name=hw_id returns the hw_id field definition"
- "GET /api/extras/custom-fields/?name=catalog_status returns the catalog_status field"
- "Location hierarchy exists: at least one Site, one Location, one Rack in NetBox"
- "Provisioning script is idempotent — running it twice does not create duplicates"
artifacts:
- path: "internal/netbox/provision.go"
provides: "Provision() function: creates custom fields + location hierarchy if not present"
exports: ["Provision", "ProvisionCustomFields", "ProvisionLocationHierarchy"]
- path: "scripts/provision-netbox.go"
provides: "Standalone CLI: `go run scripts/provision-netbox.go` — provisions NetBox from .env"
key_links:
- from: "scripts/provision-netbox.go"
to: "internal/netbox/provision.go"
via: "direct function call Provision(client)"
pattern: "Provision"
- from: "internal/netbox/provision.go"
to: "http://10.5.0.130:8000/api"
via: "REST POST /api/extras/custom-fields/, /api/dcim/sites/, /api/dcim/locations/, /api/dcim/racks/"
pattern: "ExtrasAPI|DcimAPI"
---
<objective>
Provision the NetBox instance with all HWLab custom fields and location hierarchy. This is an idempotent provisioning operation: if a custom field already exists, skip it; if the location hierarchy exists, skip it.
Purpose: Custom fields must exist in NetBox before any Go code can write them. NB-02 requires 8 specific fields; NB-03 requires the netbox-inventory plugin installed; NB-04 requires the Site→Location→Rack hierarchy.
Output: A `Provision()` function and a standalone script that a human can run against the live NetBox instance to bootstrap it.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.env
@.planning/phases/01-foundation/01-RESEARCH.md
@.planning/phases/01-foundation/01-02-SUMMARY.md
</context>
<interfaces>
<!-- Types from Plan 02 that this plan uses -->
From internal/netbox/client.go:
```go
type Client struct { ... }
func NewClient(url, token string) (*Client, error)
// Client.api is *nb.APIClient (go-netbox v4)
// Access raw API via: client.api.ExtrasAPI, client.api.DcimAPI
```
NetBox custom field POST payload (REST API):
```json
{
"name": "hw_id",
"label": "HW ID",
"type": "text",
"object_types": ["dcim.device"],
"required": false,
"description": "HWLab sequential identifier (HW-XXXXX)"
}
```
NetBox location hierarchy (DCIM API):
- Site: POST /api/dcim/sites/ {name, slug}
- Location: POST /api/dcim/locations/ {name, slug, site: {id}}
- Rack: POST /api/dcim/racks/ {name, site: {id}, location: {id}, u_height: 42}
Check for existing custom field by name: GET /api/extras/custom-fields/?name=hw_id
- If count > 0, field exists — skip creation
</interfaces>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: Custom field provisioning (NB-02)</name>
<files>internal/netbox/provision.go, internal/netbox/provision_test.go</files>
<read_first>
- /home/mikkel/homelabby/internal/netbox/client.go (Client struct, api field access pattern)
- /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (phase requirements table, lines 38-48)
- /home/mikkel/homelabby/.env (HWLAB_NETBOX_TOKEN — check if real 40-char token)
</read_first>
<behavior>
- Test 1: customFieldSpec("hw_id") returns a spec with Name="hw_id", Type="text", ObjectTypes=["dcim.device"]
- Test 2: customFieldSpec("catalog_status") returns Type="text" (stored as free text, not NetBox choice field, to avoid NetBox admin dependency)
- Test 3: customFieldSpec("photo_urls") returns a spec with Type="text" and description mentioning comma-separated
- Test 4 (INTEGRATION — skip if no real token): ProvisionCustomFields() creates all 8 fields and returns no error
- Test 5 (INTEGRATION — skip if no real token): ProvisionCustomFields() called twice is idempotent (no error, no duplicate fields)
</behavior>
<action>
All 8 custom fields to provision (NB-02):
- hw_id: text, dcim.device — "HWLab sequential identifier (HW-XXXXX)"
- catalog_status: text, dcim.device — "Lifecycle status: draft|indexed|needs_research|researched|complete"
- product_url: url, dcim.device — "Manufacturer product page URL"
- firmware_version: text, dcim.device — "Current firmware/software version"
- test_date: date, dcim.device — "Date of last cable/hardware test"
- test_data: text, dcim.device — "Structured JSON test results from cable testers"
- ai_notes: text, dcim.device — "AI-generated notes from intake analysis"
- photo_urls: text, dcim.device — "Comma-separated photo URLs captured during intake"
NOTE on photo_urls: Use type "text" (not multi-object) to avoid NetBox v4 multi-value custom field complexity. Store as comma-separated string. The Go layer will split/join as needed.
NOTE on catalog_status: Use type "text" not NetBox "selection" field — the Go quality gate owns the valid values; we don't want to maintain a parallel list in NetBox admin UI.
Create `internal/netbox/provision.go`:
```go
package netbox
import (
"context"
"fmt"
"log"
)
// CustomFieldSpec defines a NetBox custom field to provision.
type CustomFieldSpec struct {
Name string
Label string
Type string // "text", "url", "date", "integer", "boolean"
ObjectTypes []string
Description string
Required bool
}
// hwlabCustomFields is the canonical list of all HWLab custom fields.
// These MUST be provisioned in NetBox before any item can be created.
var hwlabCustomFields = []CustomFieldSpec{
{Name: "hw_id", Label: "HW ID", Type: "text", ObjectTypes: []string{"dcim.device"}, Description: "HWLab sequential identifier (HW-XXXXX)"},
{Name: "catalog_status", Label: "Catalog Status", Type: "text", ObjectTypes: []string{"dcim.device"}, Description: "Lifecycle: draft|indexed|needs_research|researched|complete"},
{Name: "product_url", Label: "Product URL", Type: "url", ObjectTypes: []string{"dcim.device"}, Description: "Manufacturer product page URL"},
{Name: "firmware_version", Label: "Firmware Version", Type: "text", ObjectTypes: []string{"dcim.device"}, Description: "Current firmware/software version"},
{Name: "test_date", Label: "Test Date", Type: "date", ObjectTypes: []string{"dcim.device"}, Description: "Date of last cable/hardware test (ISO 8601)"},
{Name: "test_data", Label: "Test Data", Type: "text", ObjectTypes: []string{"dcim.device"}, Description: "Structured JSON test results from cable testers"},
{Name: "ai_notes", Label: "AI Notes", Type: "text", ObjectTypes: []string{"dcim.device"}, Description: "AI-generated notes from intake photo analysis"},
{Name: "photo_urls", Label: "Photo URLs", Type: "text", ObjectTypes: []string{"dcim.device"}, Description: "Comma-separated photo URLs captured during intake"},
}
// customFieldSpec returns the spec for a named custom field (for testing).
func customFieldSpec(name string) *CustomFieldSpec {
for i := range hwlabCustomFields {
if hwlabCustomFields[i].Name == name {
return &hwlabCustomFields[i]
}
}
return nil
}
// ProvisionCustomFields ensures all HWLab custom fields exist in NetBox.
// Idempotent: fields that already exist are skipped.
// Returns the count of fields created (0 if all existed).
func (c *Client) ProvisionCustomFields(ctx context.Context) (int, error) {
created := 0
for _, spec := range hwlabCustomFields {
exists, err := c.customFieldExists(ctx, spec.Name)
if err != nil {
return created, fmt.Errorf("check field %s: %w", spec.Name, err)
}
if exists {
log.Printf("custom field %q already exists — skipping", spec.Name)
continue
}
if err := c.createCustomField(ctx, spec); err != nil {
return created, fmt.Errorf("create field %s: %w", spec.Name, err)
}
log.Printf("created custom field %q", spec.Name)
created++
}
return created, nil
}
// customFieldExists checks if a custom field with the given name already exists.
func (c *Client) customFieldExists(ctx context.Context, name string) (bool, error) {
res, _, err := c.api.ExtrasAPI.ExtrasCustomFieldsList(ctx).Name([]string{name}).Execute()
if err != nil {
return false, err
}
return res.GetCount() > 0, nil
}
// createCustomField creates a single custom field in NetBox via the Extras API.
// Uses the go-netbox v4 generated WritableCustomFieldRequest type.
func (c *Client) createCustomField(ctx context.Context, spec CustomFieldSpec) error {
// go-netbox v4: use CustomFieldTypeValue for the Type field
// Available types: "text", "longtext", "integer", "decimal", "boolean",
// "date", "datetime", "url", "json", "select", "multiselect",
// "object", "multiobject"
// NOTE: Executor must check the actual enum values in go-netbox v4 generated code:
// grep -r "CustomFieldTypeValue" $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/ | head -10
nb_pkg := "github.com/netbox-community/go-netbox/v4"
_ = nb_pkg
// Pseudocode — executor must use real go-netbox v4 WritableCustomFieldRequest:
// req := nb.WritableCustomFieldRequest{
// Name: spec.Name,
// Label: nb.PtrString(spec.Label),
// Type: nb.CustomFieldTypeValue(spec.Type),
// ObjectTypes: spec.ObjectTypes,
// Description: nb.PtrString(spec.Description),
// }
// _, _, err := c.api.ExtrasAPI.ExtrasCustomFieldsCreate(ctx).
// WritableCustomFieldRequest(req).Execute()
// return err
return fmt.Errorf("createCustomField: implement using go-netbox v4 WritableCustomFieldRequest — check generated types at $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@*/model_writable_custom_field_request.go")
}
// Provision runs all provisioning steps: custom fields + location hierarchy.
func (c *Client) Provision(ctx context.Context) error {
n, err := c.ProvisionCustomFields(ctx)
if err != nil {
return fmt.Errorf("provision custom fields: %w", err)
}
log.Printf("custom fields: %d created", n)
if err := c.ProvisionLocationHierarchy(ctx); err != nil {
return fmt.Errorf("provision locations: %w", err)
}
return nil
}
```
IMPORTANT: The `createCustomField` function contains a stub that will not compile. The executor MUST:
1. Run: `ls $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/` to find generated files
2. Run: `grep -l "WritableCustomFieldRequest" $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/*.go`
3. Read the relevant generated file to get the exact field names and enum types
4. Replace the pseudocode with real go-netbox v4 API calls
Create `internal/netbox/provision_test.go`:
```go
package netbox
import (
"testing"
)
func TestCustomFieldSpec(t *testing.T) {
spec := customFieldSpec("hw_id")
if spec == nil {
t.Fatal("hw_id spec not found")
}
if spec.Type != "text" {
t.Errorf("hw_id type: want text, got %s", spec.Type)
}
for _, ot := range spec.ObjectTypes {
if ot == "dcim.device" {
return
}
}
t.Error("hw_id ObjectTypes must include dcim.device")
}
func TestCustomFieldSpecCatalogStatus(t *testing.T) {
spec := customFieldSpec("catalog_status")
if spec == nil {
t.Fatal("catalog_status spec not found")
}
if spec.Type != "text" {
t.Errorf("catalog_status type: want text (not selection), got %s", spec.Type)
}
}
func TestCustomFieldSpecPhotoURLs(t *testing.T) {
spec := customFieldSpec("photo_urls")
if spec == nil {
t.Fatal("photo_urls spec not found")
}
if spec.Description == "" {
t.Error("photo_urls must have a description")
}
}
func TestAllEightFieldsDefined(t *testing.T) {
expected := []string{"hw_id", "catalog_status", "product_url", "firmware_version",
"test_date", "test_data", "ai_notes", "photo_urls"}
for _, name := range expected {
if customFieldSpec(name) == nil {
t.Errorf("missing custom field spec: %s", name)
}
}
if len(hwlabCustomFields) != 8 {
t.Errorf("want 8 custom fields, got %d", len(hwlabCustomFields))
}
}
```
</action>
<verify>
<automated>cd /home/mikkel/homelabby && go test ./internal/netbox/... -v -run "TestCustomFieldSpec|TestAllEight"</automated>
</verify>
<acceptance_criteria>
- `go test ./internal/netbox/... -run "TestCustomFieldSpec|TestAllEight"` passes all 4 unit tests
- `grep -c "Name:" internal/netbox/provision.go` returns 8 or more (one per custom field)
- `grep "photo_urls" internal/netbox/provision.go` appears in hwlabCustomFields slice
- `grep "ai_notes" internal/netbox/provision.go` appears in hwlabCustomFields slice
- `createCustomField` is implemented with real go-netbox v4 API calls (not stub returning error string)
- `go build ./internal/netbox/...` exits 0
</acceptance_criteria>
<done>All 8 custom field specs defined and tested. ProvisionCustomFields implemented with idempotent check-before-create. createCustomField uses real go-netbox v4 API (not stub).</done>
</task>
<task type="auto">
<name>Task 2: Location hierarchy provisioning + NB-03 plugin check + provision CLI (NB-03, NB-04)</name>
<files>internal/netbox/provision.go, scripts/provision-netbox.go</files>
<read_first>
- /home/mikkel/homelabby/internal/netbox/provision.go (ProvisionCustomFields added in Task 1)
- /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (phase requirements NB-03, NB-04 lines 43-48)
- /home/mikkel/homelabby/.env (HWLAB_NETBOX_URL, HWLAB_NETBOX_TOKEN)
</read_first>
<action>
Location hierarchy to create (per ROADMAP "Site → Location → Rack per PRD section 7.6"):
- Site: name="Homelab", slug="homelab"
- Location: name="Lab Bench", slug="lab-bench", site=homelab
- Rack: name="Primary Rack", site=homelab, location=lab-bench, u_height=42
1. Add `ProvisionLocationHierarchy` to `internal/netbox/provision.go`:
```go
// ProvisionLocationHierarchy creates the Site → Location → Rack hierarchy.
// Idempotent: each level is checked before creation.
func (c *Client) ProvisionLocationHierarchy(ctx context.Context) error {
// Step 1: Create or find Site "Homelab"
siteID, err := c.ensureSite(ctx, "Homelab", "homelab")
if err != nil {
return fmt.Errorf("ensure site: %w", err)
}
// Step 2: Create or find Location "Lab Bench" under the site
locationID, err := c.ensureLocation(ctx, "Lab Bench", "lab-bench", siteID)
if err != nil {
return fmt.Errorf("ensure location: %w", err)
}
// Step 3: Create or find Rack "Primary Rack" under the site + location
if err := c.ensureRack(ctx, "Primary Rack", siteID, locationID); err != nil {
return fmt.Errorf("ensure rack: %w", err)
}
return nil
}
func (c *Client) ensureSite(ctx context.Context, name, slug string) (int32, error) {
// Check if site exists by slug
res, _, err := c.api.DcimAPI.DcimSitesList(ctx).Slug([]string{slug}).Execute()
if err != nil {
return 0, err
}
if res.GetCount() > 0 {
log.Printf("site %q already exists — skipping", name)
return res.Results[0].GetId(), nil
}
// Create site
// NOTE: Executor must use go-netbox v4 WritableSiteRequest type
// Pattern: c.api.DcimAPI.DcimSitesCreate(ctx).WritableSiteRequest(req).Execute()
// Required fields: Name (string), Slug (string)
// Status: use nb.SiteStatusValue("active") or equivalent
return 0, fmt.Errorf("ensureSite: implement using go-netbox v4 WritableSiteRequest")
}
func (c *Client) ensureLocation(ctx context.Context, name, slug string, siteID int32) (int32, error) {
res, _, err := c.api.DcimAPI.DcimLocationsList(ctx).Slug([]string{slug}).Execute()
if err != nil {
return 0, err
}
if res.GetCount() > 0 {
log.Printf("location %q already exists — skipping", name)
return res.Results[0].GetId(), nil
}
// Create location under site
// NOTE: use go-netbox v4 WritableLocationRequest
// Required: Name, Slug, Site (NestedSiteRequest with ID)
return 0, fmt.Errorf("ensureLocation: implement using go-netbox v4 WritableLocationRequest")
}
func (c *Client) ensureRack(ctx context.Context, name string, siteID, locationID int32) error {
res, _, err := c.api.DcimAPI.DcimRacksList(ctx).Name([]string{name}).Execute()
if err != nil {
return err
}
if res.GetCount() > 0 {
log.Printf("rack %q already exists — skipping", name)
return nil
}
// Create rack under site + location
// NOTE: use go-netbox v4 WritableRackRequest
// Required: Name, Site (NestedSiteRequest), Location (NestedLocationRequest), UHeight (int32)
// UHeight = 42
return fmt.Errorf("ensureRack: implement using go-netbox v4 WritableRackRequest")
}
```
EXECUTOR MUST replace all stub returns with real go-netbox v4 API calls. Locate the generated types:
```
ls $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/model_writable_site_request.go
ls $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/model_writable_location_request.go
ls $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/model_writable_rack_request.go
```
2. Add NB-03 plugin check note in provision.go as a comment (NB-03 requires SSH to LXC 130 — this is a manual verification step documented in VALIDATION.md):
```go
// CheckNetBoxInventoryPlugin verifies the netbox-inventory plugin is installed.
// This check uses the NetBox plugins API endpoint.
// NB-03: netbox-inventory plugin must be installed on LXC 130.
// Manual verification: SSH to LXC 130, run: pip show netbox-inventory
// API check: GET /api/plugins/ lists installed plugin API endpoints.
func (c *Client) CheckNetBoxInventoryPlugin(ctx context.Context) (bool, error) {
// The netbox-inventory plugin registers under /api/plugins/inventory/
// We can check by hitting that endpoint and seeing if we get a 200 vs 404.
resp, err := c.api.GetConfig().HTTPClient.Get(c.url[:len(c.url)-4] + "/api/plugins/")
if err != nil {
return false, err
}
defer resp.Body.Close()
return resp.StatusCode == 200, nil
}
```
3. Create `scripts/provision-netbox.go` — standalone CLI script:
```go
//go:build ignore
// Run with: go run scripts/provision-netbox.go
// Provisions NetBox with all HWLab custom fields and location hierarchy.
// Reads HWLAB_NETBOX_URL and HWLAB_NETBOX_TOKEN from environment (.env auto-loaded).
package main
import (
"context"
"log"
"os"
"github.com/joho/godotenv"
"git.georgsen.dk/hwlab/internal/netbox"
)
func main() {
// Load .env
if err := godotenv.Load(); err != nil {
log.Printf("no .env file: %v", err)
}
url := os.Getenv("HWLAB_NETBOX_URL")
token := os.Getenv("HWLAB_NETBOX_TOKEN")
if url == "" || token == "" {
log.Fatal("HWLAB_NETBOX_URL and HWLAB_NETBOX_TOKEN must be set")
}
client, err := netbox.NewClient(url, token)
if err != nil {
log.Fatalf("netbox client: %v", err)
}
ctx := context.Background()
log.Println("Provisioning NetBox...")
if err := client.Provision(ctx); err != nil {
log.Fatalf("provision: %v", err)
}
// Check netbox-inventory plugin (NB-03)
ok, err := client.CheckNetBoxInventoryPlugin(ctx)
if err != nil {
log.Printf("plugin check error: %v", err)
} else if ok {
log.Println("netbox-inventory plugin: INSTALLED")
} else {
log.Println("WARNING: netbox-inventory plugin may not be installed")
log.Println(" Manual check: SSH to LXC 130, run: pip show netbox-inventory")
log.Println(" Install if missing: pip install netbox-inventory")
}
log.Println("Provisioning complete.")
}
```
</action>
<verify>
<automated>cd /home/mikkel/homelabby && go build ./internal/netbox/... && go vet ./internal/netbox/...</automated>
</verify>
<acceptance_criteria>
- `go build ./internal/netbox/...` exits 0 (all stubs replaced with real go-netbox v4 API calls)
- `go vet ./internal/netbox/...` exits 0 (no vet errors)
- `grep "ProvisionLocationHierarchy" internal/netbox/provision.go` returns the exported function
- `grep "ensureSite\|ensureLocation\|ensureRack" internal/netbox/provision.go` returns 3 matches (not stub errors)
- `grep "CheckNetBoxInventoryPlugin" internal/netbox/provision.go` returns the function
- File `scripts/provision-netbox.go` exists
- `grep "go:build ignore" scripts/provision-netbox.go` returns a match (build tag prevents accidental inclusion)
- If real token available: `go run scripts/provision-netbox.go` exits 0 and logs "Provisioning complete."
- If real token available: `curl -H "Authorization: Token REAL_TOKEN" http://10.5.0.130:8000/api/extras/custom-fields/?name=hw_id` returns count > 0
</acceptance_criteria>
<done>Location hierarchy provisioning implemented. Provision CLI script created. All functions use real go-netbox v4 API (no stubs). `go build` and `go vet` clean.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| scripts/provision-netbox.go → NetBox API | Write access to NetBox via authenticated REST API |
| Provisioning logic → NetBox admin state | Creates objects in NetBox; idempotency prevents data corruption |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-03-01 | Tampering | ProvisionCustomFields idempotency | mitigate | Check-before-create pattern ensures no duplicates; NetBox also enforces unique names on custom fields |
| T-03-02 | Information Disclosure | scripts/provision-netbox.go logs | accept | Logs go to stdout only; no sensitive data logged beyond field names |
| T-03-03 | Denial of Service | Provisioning script run in loop | accept | Script has `//go:build ignore` tag, requires explicit `go run`; single operator context |
| T-03-04 | Elevation of Privilege | Provisioning requires NetBox admin rights | accept | NetBox token grants admin-level access by design; this is the operator's own homelab |
</threat_model>
<verification>
After both tasks complete:
- `go test ./internal/netbox/... -v` all tests pass (unit) or skip (integration without real token)
- `go build ./...` exits 0
- `go vet ./...` exits 0
- If real token: `go run scripts/provision-netbox.go` provisions cleanly
- If real token: `curl -s -H "Authorization: Token $HWLAB_NETBOX_TOKEN" "http://10.5.0.130:8000/api/extras/custom-fields/" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['count'])"` shows 8 or more custom fields
</verification>
<success_criteria>
1. All 8 custom field specs defined in hwlabCustomFields slice with correct names, types, object_types
2. ProvisionCustomFields is idempotent (check-before-create)
3. ProvisionLocationHierarchy creates Site "Homelab" → Location "Lab Bench" → Rack "Primary Rack"
4. Standalone script runs with `go run scripts/provision-netbox.go` against live NetBox
5. `go build` and `go vet` clean on all packages
</success_criteria>
<output>
After completion, create `.planning/phases/01-foundation/01-03-SUMMARY.md` with:
- Whether provisioning ran against live NetBox (yes/no)
- If yes: which custom fields were created vs. already existed
- If yes: whether netbox-inventory plugin was detected
- Any go-netbox v4 API quirks encountered (WritableSiteRequest fields, slug handling, etc.)
- Files created/modified
</output>

698
.planning/phases/01-foundation/01-04-PLAN.md Normal file
View file

@ -0,0 +1,698 @@
---
phase: 01-foundation
plan: 04
type: execute
wave: 2
depends_on:
- 01-02-PLAN.md
files_modified:
- internal/netbox/hwid.go
- internal/netbox/hwid_test.go
- internal/inventory/quality_gate.go
- internal/inventory/quality_gate_test.go
- internal/inventory/types.go
- internal/netbox/tags.go
- internal/netbox/tags_test.go
autonomous: true
requirements:
- INF-03
- NB-06
- NB-07
must_haves:
truths:
- "AllocateNextHWID returns HW-00001 on first call against an empty NetBox"
- "AllocateNextHWID returns HW-00002 on subsequent calls (increments)"
- "CatalogStatus.CanTransitionTo enforces valid transitions (draft→indexed allowed, indexed→draft rejected)"
- "Invalid transitions return an error with the exact invalid transition described"
- "SyncTags creates new NetBox tags for AI-suggested tags not yet present"
artifacts:
- path: "internal/netbox/hwid.go"
provides: "AllocateNextHWID: optimistic-lock sequential ID allocation from NetBox"
exports: ["AllocateNextHWID"]
- path: "internal/inventory/quality_gate.go"
provides: "CatalogStatus type with Transition() enforcing valid state machine"
exports: ["CatalogStatus", "StatusDraft", "StatusIndexed", "StatusNeedsResearch", "StatusResearched", "StatusComplete", "Transition"]
- path: "internal/inventory/types.go"
provides: "HardwareRecord domain type composing NetBox device with HWLab semantics"
exports: ["HardwareRecord"]
- path: "internal/netbox/tags.go"
provides: "SyncTags: creates NetBox tags from AI-suggested string slice, returns IDs"
exports: ["SyncTags"]
key_links:
- from: "internal/netbox/hwid.go"
to: "http://10.5.0.130:8000/api"
via: "DcimDevicesList filtered by asset_tag pattern"
pattern: "DcimDevicesList.*asset_tag|asset_tag.*DcimDevicesList"
- from: "internal/inventory/quality_gate.go"
to: "internal/netbox/client.go"
via: "CatalogStatus value stored as NetBox custom field via PatchCustomFields"
pattern: "PatchCustomFields.*catalog_status"
---
<objective>
Implement three distinct capabilities that all depend on the NetBox client (Plan 02): HW-XXXXX sequential ID allocation, catalog quality gate state machine, and AI tag sync to NetBox.
Purpose: These three capabilities are the behavioral core of Phase 1. HW-ID is required at intake time (Phase 2). Quality gate drives all lifecycle operations. Tag sync links AI output to NetBox taxonomy.
Output: Three packages — netbox/hwid.go, inventory/quality_gate.go, netbox/tags.go — all tested independently.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.env
@.planning/phases/01-foundation/01-RESEARCH.md
@.planning/phases/01-foundation/01-02-SUMMARY.md
</context>
<interfaces>
<!-- Types from Plan 02 available to this plan -->
From internal/netbox/client.go:
```go
func NewClient(url, token string) (*Client, error)
func (c *Client) ListDevices(ctx context.Context, limit int) ([]Device, error)
func (c *Client) PatchCustomFields(ctx context.Context, deviceID int, patch map[string]interface{}) error
// c.api is *nb.APIClient — can call DcimAPI, ExtrasAPI directly from hwid.go and tags.go
```
From internal/netbox/types.go:
```go
type Device struct {
ID int
Name string
AssetTag string // HW-XXXXX
CustomFields CustomFields
Created time.Time
LastUpdated time.Time
}
type CustomFields struct {
HWID, CatalogStatus, ProductURL, FirmwareVersion string
TestDate, TestData, AINotes string
PhotoURLs []string
}
```
From internal/netbox/custom_fields.go:
```go
func BuildCustomFieldsPatch(hwID, catalogStatus string, photoURLs []string) map[string]interface{}
```
</interfaces>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: HW-XXXXX sequential ID allocation (INF-03)</name>
<files>internal/netbox/hwid.go, internal/netbox/hwid_test.go</files>
<read_first>
- /home/mikkel/homelabby/internal/netbox/client.go (Client struct, api access)
- /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 4: HW-XXXXX Sequential ID Allocation, lines 238-265)
</read_first>
<behavior>
- Test 1: parseHWID("HW-00042") returns 42, nil
- Test 2: parseHWID("HW-99999") returns 99999, nil
- Test 3: parseHWID("not-a-hw-id") returns 0, error
- Test 4: parseHWID("") returns 0, error
- Test 5: formatHWID(1) returns "HW-00001"
- Test 6: formatHWID(99999) returns "HW-99999"
- Test 7 (INTEGRATION — skip if no real token): AllocateNextHWID returns a string matching ^HW-\d{5}$
</behavior>
<action>
Create `internal/netbox/hwid.go`:
```go
package netbox
import (
"context"
"errors"
"fmt"
"regexp"
"strconv"
"strings"
)
var hwIDPattern = regexp.MustCompile(`^HW-(\d{5})$`)
// formatHWID formats an integer as a HW-XXXXX string.
func formatHWID(n int) string {
return fmt.Sprintf("HW-%05d", n)
}
// parseHWID parses a HW-XXXXX string to an integer.
// Returns error if the format does not match.
func parseHWID(s string) (int, error) {
m := hwIDPattern.FindStringSubmatch(s)
if m == nil {
return 0, fmt.Errorf("invalid HW-ID format: %q (expected HW-NNNNN)", s)
}
n, err := strconv.Atoi(m[1])
if err != nil {
return 0, err
}
return n, nil
}
// AllocateNextHWID allocates the next available HW-XXXXX identifier.
// Strategy: optimistic locking — query the highest existing asset_tag, increment by 1,
// attempt to reserve it. Retry up to 3 times on conflict.
//
// The reservation is a placeholder NetBox device with name "__hwid_reservation__"
// that the caller MUST immediately replace with the real device data.
// In practice, Phase 2 will create the real device in a single atomic step,
// so the placeholder device is never committed separately.
//
// For Phase 1, AllocateNextHWID returns the ID string without creating a device.
// The caller is responsible for creating the device record and setting asset_tag.
func (c *Client) AllocateNextHWID(ctx context.Context) (string, error) {
const maxAttempts = 3
for attempt := 0; attempt < maxAttempts; attempt++ {
highest, err := c.getHighestHWIDNumber(ctx)
if err != nil {
return "", fmt.Errorf("get highest HW-ID: %w", err)
}
candidate := formatHWID(highest + 1)
// Check that this candidate is not already taken
// (handles concurrent allocation if ever needed)
taken, err := c.hwIDExists(ctx, candidate)
if err != nil {
return "", fmt.Errorf("check HW-ID %s: %w", candidate, err)
}
if !taken {
return candidate, nil
}
// Candidate is taken — loop and try highest+2, etc.
}
return "", errors.New("HW-ID allocation failed after 3 attempts — concurrent allocation conflict")
}
// getHighestHWIDNumber queries NetBox for the highest existing HW-XXXXX asset_tag number.
// Returns 0 if no HW-XXXXX asset_tags exist (first allocation will be HW-00001).
func (c *Client) getHighestHWIDNumber(ctx context.Context) (int, error) {
// Query all devices, paginate if needed — in Phase 1 this is small
// Filter by asset_tag starting with "HW-" to limit results
// NOTE: go-netbox v4 DcimDevicesList supports AssetTag filter
// Use limit=1000 and sort by asset_tag descending to find the highest efficiently
// If NetBox v4 supports ordering by asset_tag: use .Ordering("-asset_tag")
res, _, err := c.api.DcimAPI.DcimDevicesList(ctx).
Limit(1000).
Execute()
if err != nil {
return 0, fmt.Errorf("list devices for HW-ID query: %w", err)
}
highest := 0
for _, d := range res.Results {
tag := d.GetAssetTag()
if !strings.HasPrefix(tag, "HW-") {
continue
}
n, err := parseHWID(tag)
if err != nil {
continue // non-HWLab asset tag — skip
}
if n > highest {
highest = n
}
}
return highest, nil
}
// hwIDExists checks if a given HW-XXXXX asset_tag is already used in NetBox.
func (c *Client) hwIDExists(ctx context.Context, hwid string) (bool, error) {
res, _, err := c.api.DcimAPI.DcimDevicesList(ctx).
AssetTag([]string{hwid}).
Limit(1).
Execute()
if err != nil {
return false, err
}
return res.GetCount() > 0, nil
}
```
Create `internal/netbox/hwid_test.go`:
```go
package netbox
import (
"testing"
)
func TestFormatHWID(t *testing.T) {
tests := []struct {
n int
want string
}{
{1, "HW-00001"},
{42, "HW-00042"},
{99999, "HW-99999"},
}
for _, tt := range tests {
got := formatHWID(tt.n)
if got != tt.want {
t.Errorf("formatHWID(%d) = %q, want %q", tt.n, got, tt.want)
}
}
}
func TestParseHWID(t *testing.T) {
tests := []struct {
s string
want int
wantErr bool
}{
{"HW-00001", 1, false},
{"HW-00042", 42, false},
{"HW-99999", 99999, false},
{"", 0, true},
{"not-a-hw-id", 0, true},
{"HW-0001", 0, true}, // only 4 digits — invalid
{"hw-00001", 0, true}, // lowercase — invalid
}
for _, tt := range tests {
got, err := parseHWID(tt.s)
if tt.wantErr && err == nil {
t.Errorf("parseHWID(%q): expected error, got nil", tt.s)
}
if !tt.wantErr && err != nil {
t.Errorf("parseHWID(%q): unexpected error: %v", tt.s, err)
}
if !tt.wantErr && got != tt.want {
t.Errorf("parseHWID(%q) = %d, want %d", tt.s, got, tt.want)
}
}
}
```
</action>
<verify>
<automated>cd /home/mikkel/homelabby && go test ./internal/netbox/... -v -run "TestFormatHWID|TestParseHWID"</automated>
</verify>
<acceptance_criteria>
- `go test ./internal/netbox/... -run "TestFormatHWID|TestParseHWID"` passes all 10 cases
- `grep "AllocateNextHWID" internal/netbox/hwid.go` returns the exported function
- `grep "HW-%05d" internal/netbox/hwid.go` returns the Sprintf format call
- `grep "getHighestHWIDNumber" internal/netbox/hwid.go` returns the private helper
- `go build ./internal/netbox/...` exits 0
</acceptance_criteria>
<done>HW-ID format/parse unit tests pass. AllocateNextHWID implemented with optimistic-lock retry. Binary compiles.</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Quality gate state machine and AI tag sync (NB-06, NB-07)</name>
<files>internal/inventory/types.go, internal/inventory/quality_gate.go, internal/inventory/quality_gate_test.go, internal/netbox/tags.go, internal/netbox/tags_test.go</files>
<read_first>
- /home/mikkel/homelabby/internal/netbox/client.go (Client struct for SyncTags)
- /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 6: Catalog Status Quality Gate, lines 322-360)
- /home/mikkel/homelabby/internal/netbox/types.go (CustomFields.CatalogStatus field)
</read_first>
<behavior>
Quality gate tests:
- Test 1: StatusDraft.CanTransitionTo(StatusIndexed) returns true
- Test 2: StatusDraft.CanTransitionTo(StatusComplete) returns false
- Test 3: StatusIndexed.CanTransitionTo(StatusNeedsResearch) returns true
- Test 4: StatusIndexed.CanTransitionTo(StatusDraft) returns false (no backward transitions)
- Test 5: StatusComplete.CanTransitionTo(anything) returns false (terminal)
- Test 6: Transition(StatusDraft, StatusIndexed) returns StatusIndexed, nil
- Test 7: Transition(StatusDraft, StatusComplete) returns "", error containing "invalid transition"
- Test 8: ParseCatalogStatus("draft") returns StatusDraft, nil
- Test 9: ParseCatalogStatus("unknown_value") returns "", error
Tag sync tests:
- Test 10: normalizeTags([]string{" USB Cable ", "USB cable", "usb-cable"}) returns deduplicated, lowercase-trimmed slice
- Test 11 (INTEGRATION — skip if no real token): SyncTags([]string{"usb-c-cable"}) creates tag in NetBox and returns its ID
</behavior>
<action>
1. Create `internal/inventory/types.go`:
```go
package inventory
import "git.georgsen.dk/hwlab/internal/netbox"
// HardwareRecord is the HWLab domain representation of a cataloged item.
// It wraps a NetBox device with HWLab-specific fields and lifecycle state.
type HardwareRecord struct {
HWID string // HW-XXXXX from asset_tag
NetBoxID int // NetBox device internal ID
Name string // Device name in NetBox
CatalogStatus CatalogStatus // Quality gate lifecycle status
CustomFields netbox.CustomFields // All HWLab custom fields
AITags []string // AI-suggested tags (synced to NetBox)
}
```
2. Create `internal/inventory/quality_gate.go`:
```go
package inventory
import "fmt"
// CatalogStatus represents the lifecycle stage of a cataloged hardware item.
// Stored as the catalog_status custom field value in NetBox.
type CatalogStatus string
const (
StatusDraft CatalogStatus = "draft"
StatusIndexed CatalogStatus = "indexed"
StatusNeedsResearch CatalogStatus = "needs_research"
StatusResearched CatalogStatus = "researched"
StatusComplete CatalogStatus = "complete"
)
// validTransitions defines the allowed state machine transitions.
// No backward transitions are permitted (lifecycle is forward-only).
var validTransitions = map[CatalogStatus][]CatalogStatus{
StatusDraft: {StatusIndexed},
StatusIndexed: {StatusNeedsResearch, StatusResearched},
StatusNeedsResearch: {StatusResearched},
StatusResearched: {StatusComplete},
StatusComplete: {}, // terminal — no further transitions
}
// CanTransitionTo returns true if transitioning from s to next is permitted.
func (s CatalogStatus) CanTransitionTo(next CatalogStatus) bool {
allowed, ok := validTransitions[s]
if !ok {
return false
}
for _, a := range allowed {
if a == next {
return true
}
}
return false
}
// Transition attempts to move from current to next status.
// Returns the new status on success, or an error describing the invalid transition.
func Transition(current, next CatalogStatus) (CatalogStatus, error) {
if !current.CanTransitionTo(next) {
return "", fmt.Errorf("invalid transition: %s → %s (not in valid transitions map)", current, next)
}
return next, nil
}
// ParseCatalogStatus parses a string to a CatalogStatus.
// Returns error for unknown status values.
func ParseCatalogStatus(s string) (CatalogStatus, error) {
cs := CatalogStatus(s)
if _, ok := validTransitions[cs]; ok {
return cs, nil
}
return "", fmt.Errorf("unknown catalog status: %q (valid: draft, indexed, needs_research, researched, complete)", s)
}
// AllStatuses returns all valid catalog statuses in lifecycle order.
func AllStatuses() []CatalogStatus {
return []CatalogStatus{
StatusDraft, StatusIndexed, StatusNeedsResearch, StatusResearched, StatusComplete,
}
}
```
3. Create `internal/inventory/quality_gate_test.go`:
```go
package inventory_test
import (
"strings"
"testing"
"git.georgsen.dk/hwlab/internal/inventory"
)
func TestCanTransitionTo(t *testing.T) {
tests := []struct {
from inventory.CatalogStatus
to inventory.CatalogStatus
allowed bool
}{
{inventory.StatusDraft, inventory.StatusIndexed, true},
{inventory.StatusDraft, inventory.StatusComplete, false},
{inventory.StatusDraft, inventory.StatusDraft, false},
{inventory.StatusIndexed, inventory.StatusNeedsResearch, true},
{inventory.StatusIndexed, inventory.StatusResearched, true},
{inventory.StatusIndexed, inventory.StatusDraft, false},
{inventory.StatusNeedsResearch, inventory.StatusResearched, true},
{inventory.StatusNeedsResearch, inventory.StatusIndexed, false},
{inventory.StatusResearched, inventory.StatusComplete, true},
{inventory.StatusResearched, inventory.StatusDraft, false},
{inventory.StatusComplete, inventory.StatusDraft, false},
{inventory.StatusComplete, inventory.StatusResearched, false},
}
for _, tt := range tests {
got := tt.from.CanTransitionTo(tt.to)
if got != tt.allowed {
t.Errorf("%s → %s: want %v, got %v", tt.from, tt.to, tt.allowed, got)
}
}
}
func TestTransitionValid(t *testing.T) {
got, err := inventory.Transition(inventory.StatusDraft, inventory.StatusIndexed)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got != inventory.StatusIndexed {
t.Errorf("want indexed, got %s", got)
}
}
func TestTransitionInvalid(t *testing.T) {
_, err := inventory.Transition(inventory.StatusDraft, inventory.StatusComplete)
if err == nil {
t.Fatal("expected error for invalid transition")
}
if !strings.Contains(err.Error(), "invalid transition") {
t.Errorf("error should mention 'invalid transition', got: %v", err)
}
}
func TestParseCatalogStatus(t *testing.T) {
for _, s := range []string{"draft", "indexed", "needs_research", "researched", "complete"} {
cs, err := inventory.ParseCatalogStatus(s)
if err != nil {
t.Errorf("ParseCatalogStatus(%q): unexpected error: %v", s, err)
}
if string(cs) != s {
t.Errorf("ParseCatalogStatus(%q) = %q, want %q", s, cs, s)
}
}
_, err := inventory.ParseCatalogStatus("unknown_status")
if err == nil {
t.Error("expected error for unknown status")
}
}
```
4. Create `internal/netbox/tags.go` (NB-07 — AI tags synced to NetBox):
```go
package netbox
import (
"context"
"fmt"
"strings"
)
// normalizeTags deduplicates and normalizes a slice of tag strings:
// trims whitespace, lowercases, removes empty strings.
func normalizeTags(tags []string) []string {
seen := make(map[string]struct{})
out := make([]string, 0, len(tags))
for _, t := range tags {
t = strings.ToLower(strings.TrimSpace(t))
if t == "" {
continue
}
if _, ok := seen[t]; ok {
continue
}
seen[t] = struct{}{}
out = append(out, t)
}
return out
}
// TagRef holds a NetBox tag name and its internal ID.
type TagRef struct {
ID int32
Name string
Slug string
}
// SyncTags ensures all tags in the provided slice exist in NetBox.
// Tags are normalized before sync (lowercase, trimmed, deduplicated).
// Returns the TagRef list for all tags (existing + newly created).
func (c *Client) SyncTags(ctx context.Context, tags []string) ([]TagRef, error) {
normalized := normalizeTags(tags)
if len(normalized) == 0 {
return nil, nil
}
result := make([]TagRef, 0, len(normalized))
for _, name := range normalized {
slug := tagNameToSlug(name)
ref, err := c.ensureTag(ctx, name, slug)
if err != nil {
return result, fmt.Errorf("sync tag %q: %w", name, err)
}
result = append(result, ref)
}
return result, nil
}
// tagNameToSlug converts a tag name to a NetBox-compatible slug.
// NetBox slugs: lowercase, hyphens instead of spaces, only [a-z0-9-_].
func tagNameToSlug(name string) string {
s := strings.ToLower(strings.TrimSpace(name))
s = strings.ReplaceAll(s, " ", "-")
// Remove characters not in [a-z0-9-_]
var out []byte
for _, c := range []byte(s) {
if (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || c == '-' || c == '_' {
out = append(out, c)
}
}
return string(out)
}
// ensureTag returns an existing tag or creates a new one.
func (c *Client) ensureTag(ctx context.Context, name, slug string) (TagRef, error) {
// Check for existing tag by slug
res, _, err := c.api.ExtrasAPI.ExtrasTagsList(ctx).Slug([]string{slug}).Execute()
if err != nil {
return TagRef{}, fmt.Errorf("list tags: %w", err)
}
if res.GetCount() > 0 {
t := res.Results[0]
return TagRef{ID: t.GetId(), Name: t.GetName(), Slug: t.GetSlug()}, nil
}
// Create new tag
// NOTE: Executor must use go-netbox v4 TagRequest type
// Pattern: c.api.ExtrasAPI.ExtrasTagsCreate(ctx).TagRequest(req).Execute()
// Required fields: Name (string), Slug (string)
// Optional: Color (hex string, e.g. "faff69" for volt yellow)
return TagRef{}, fmt.Errorf("ensureTag: implement using go-netbox v4 TagRequest — grep TagRequest $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/")
}
```
EXECUTOR: Replace the `ensureTag` stub with real go-netbox v4 TagRequest call. Locate:
```
ls $(go env GOPATH)/pkg/mod/github.com/netbox-community/go-netbox/v4@v4.3.0/model_tag_request.go
```
5. Create `internal/netbox/tags_test.go`:
```go
package netbox
import "testing"
func TestNormalizeTags(t *testing.T) {
in := []string{" USB Cable ", "USB cable", "usb-cable", "", " "}
out := normalizeTags(in)
// "USB Cable", "USB cable", "usb-cable" all normalize to "usb-cable" — only 1 unique
if len(out) != 1 {
t.Errorf("want 1 unique normalized tag, got %d: %v", len(out), out)
}
if out[0] != "usb-cable" {
t.Errorf("want usb-cable, got %s", out[0])
}
}
func TestTagNameToSlug(t *testing.T) {
tests := []struct {
name string
slug string
}{
{"USB Cable", "usb-cable"},
{"10GbE NIC", "10gbe-nic"},
{"SFP+ Transceiver", "sfp-transceiver"},
{" spaces ", "spaces"},
}
for _, tt := range tests {
got := tagNameToSlug(tt.name)
if got != tt.slug {
t.Errorf("tagNameToSlug(%q) = %q, want %q", tt.name, got, tt.slug)
}
}
}
```
</action>
<verify>
<automated>cd /home/mikkel/homelabby && go test ./internal/inventory/... ./internal/netbox/... -v -run "TestCanTransitionTo|TestTransitionValid|TestTransitionInvalid|TestParseCatalogStatus|TestNormalizeTags|TestTagNameToSlug"</automated>
</verify>
<acceptance_criteria>
- `go test ./internal/inventory/...` passes all 12 table-driven cases for CanTransitionTo
- `go test ./internal/inventory/...` passes TestTransitionValid, TestTransitionInvalid, TestParseCatalogStatus
- `go test ./internal/netbox/... -run "TestNormalizeTags|TestTagNameToSlug"` passes all cases
- `grep "StatusComplete.*{}" internal/inventory/quality_gate.go` returns the terminal state entry with empty transitions
- `grep "invalid transition" internal/inventory/quality_gate.go` returns error string in Transition()
- `ensureTag` in `internal/netbox/tags.go` is implemented with real go-netbox v4 TagRequest (not stub returning error string)
- `go build ./internal/inventory/... ./internal/netbox/...` exits 0
</acceptance_criteria>
<done>Quality gate state machine fully tested. Tag normalization and slug conversion tested. ensureTag implemented with real go-netbox v4 API. All packages build cleanly.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| AI output → SyncTags | AI-suggested tag strings enter normalizeTags before any NetBox write |
| Quality gate transitions → NetBox writes | Transition validation in Go; NetBox stores the result |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-04-01 | Tampering | Quality gate bypass | mitigate | All status changes MUST go through Transition() — no direct NetBox PATCH of catalog_status without Transition validation |
| T-04-02 | Tampering | AI tag injection | mitigate | normalizeTags strips whitespace, lowercases, deduplicates — limits injection surface before NetBox write |
| T-04-03 | Denial of Service | AllocateNextHWID scanning all devices | accept | Phase 1 inventory is small; getHighestHWIDNumber scans all devices; acceptable until inventory exceeds ~10k items |
| T-04-04 | Information Disclosure | HW-ID sequential enumeration | accept | IDs are not secret — they appear on printed labels; sequential is intentional for readability |
</threat_model>
<verification>
After both tasks complete:
- `go test ./internal/inventory/... ./internal/netbox/...` all green (unit tests)
- `go build ./...` exits 0 (all packages compile together)
- Quality gate: `draft → indexed → needs_research → researched → complete` is the only fully valid path
- `curl -s -H "Authorization: Token $HWLAB_NETBOX_TOKEN" "http://10.5.0.130:8000/api/extras/tags/" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['count'])"` (verify tags endpoint reachable, if real token)
</verification>
<success_criteria>
1. All 12 state machine transition cases tested and correct
2. Transition() returns error containing "invalid transition" for bad transitions
3. ParseCatalogStatus rejects unknown status strings
4. normalizeTags handles deduplication across case/whitespace variants
5. AllocateNextHWID implemented with optimistic retry loop
6. ensureTag uses real go-netbox v4 API (not stub)
7. `go build ./...` and `go test ./...` both clean
</success_criteria>
<output>
After completion, create `.planning/phases/01-foundation/01-04-SUMMARY.md` with:
- All test results (pass counts)
- Whether ensureTag integration test ran (real token) or skipped
- Any issues with go-netbox v4 tag or device list filter API
- Files created/modified
</output>

555
.planning/phases/01-foundation/01-05-PLAN.md Normal file
View file

@ -0,0 +1,555 @@
---
phase: 01-foundation
plan: 05
type: execute
wave: 2
depends_on:
- 01-01-PLAN.md
files_modified:
- internal/queue/waq.go
- internal/queue/waq_test.go
- internal/queue/worker.go
- cmd/hwlab/main.go
autonomous: true
requirements:
- NB-05
must_haves:
truths:
- "WAQ connects to DragonFlyDB at redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379"
- "Enqueue() pushes a serialized PendingOp onto the hwlab:netbox:pending_ops LIST"
- "RunWorker() BLPOP-blocks and processes available operations"
- "DragonFlyDB unavailability does not crash the binary — WAQ degrades gracefully"
- "Dequeue returns the oldest operation first (FIFO via RPUSH/BLPOP pattern)"
artifacts:
- path: "internal/queue/waq.go"
provides: "WAQ type: Enqueue, Dequeue, Len — DragonFlyDB-backed write-ahead queue"
exports: ["WAQ", "NewWAQ", "PendingOp", "Enqueue", "Len"]
- path: "internal/queue/worker.go"
provides: "RunWorker goroutine: BLPOP retry loop, exponential backoff on connection failure"
exports: ["RunWorker"]
key_links:
- from: "internal/queue/waq.go"
to: "redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379"
via: "go-redis v9 ParseURL + NewClient"
pattern: "redis.ParseURL|ParseURL"
- from: "internal/queue/worker.go"
to: "internal/queue/waq.go"
via: "BLPOP + processOp callback"
pattern: "BLPOP|RunWorker"
- from: "cmd/hwlab/main.go"
to: "internal/queue/worker.go"
via: "go waq.RunWorker(ctx)"
pattern: "RunWorker"
---
<objective>
Implement the DragonFlyDB write-ahead queue: enqueue failed or deferred NetBox operations during downtime, and a worker goroutine that retries them when connectivity restores.
Purpose: NetBox may be temporarily unavailable (container restart, network blip). The WAQ ensures no inventory operations are lost — they're buffered in DragonFlyDB and retried automatically. NB-05 is the sole requirement for this plan.
Output: `internal/queue` package with WAQ and worker, wired into the main binary.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.env
@.planning/phases/01-foundation/01-RESEARCH.md
@.planning/phases/01-foundation/01-01-SUMMARY.md
</context>
<interfaces>
<!-- DragonFlyDB connection info -->
URL: redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379
Queue key: "hwlab:netbox:pending_ops" (Redis LIST)
Operations: RPUSH (enqueue), BLPOP (dequeue + block), LLEN (queue depth)
From internal/config/config.go:
```go
type Config struct {
DragonflyURL string // "redis://:PASSWORD@10.5.0.10:6379"
WAQRetryIntervalSeconds int // default 30
WAQMaxAttempts int // default 5
}
```
go-redis v9 pattern (verified in RESEARCH.md):
```go
import "github.com/redis/go-redis/v9"
opt, err := redis.ParseURL(redisURL) // parses redis:// URL including password
client := redis.NewClient(opt)
err = client.Ping(ctx).Err() // connectivity check
// Enqueue: RPUSH appends to right (FIFO with BLPOP from left)
err = client.RPush(ctx, key, data).Err()
// Dequeue (blocking): BLPOP pops from left, blocks up to timeout
result, err := client.BLPop(ctx, 5*time.Second, key).Result()
// result[0] = key name, result[1] = value
// Queue depth
n, err := client.LLen(ctx, key).Result()
```
</interfaces>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: Write-ahead queue core (Enqueue, Dequeue, Len)</name>
<files>internal/queue/waq.go, internal/queue/waq_test.go</files>
<read_first>
- /home/mikkel/homelabby/.env (HWLAB_DRAGONFLY_URL — verify it's redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379)
- /home/mikkel/homelabby/.planning/phases/01-foundation/01-RESEARCH.md (Pattern 5: DragonFlyDB Write-Ahead Queue, lines 266-320)
- /home/mikkel/homelabby/internal/config/config.go (DragonflyURL field)
</read_first>
<behavior>
- Test 1: NewWAQ with invalid URL returns error
- Test 2: NewWAQ with valid URL but unreachable server returns error on Ping
- Test 3 (INTEGRATION — skip if DragonFlyDB unreachable): Enqueue() + Len() = 1
- Test 4 (INTEGRATION): Enqueue(op1), Enqueue(op2), Dequeue() returns op1 first (FIFO)
- Test 5 (INTEGRATION): Len() returns 0 after all ops dequeued
- Test 6: PendingOp marshals/unmarshals to/from JSON correctly (ID, Type, Payload preserved)
</behavior>
<action>
Create `internal/queue/waq.go`:
```go
package queue
import (
"context"
"encoding/json"
"fmt"
"log"
"time"
"github.com/google/uuid"
"github.com/redis/go-redis/v9"
)
const queueKey = "hwlab:netbox:pending_ops"
// PendingOp represents a queued NetBox operation.
// Serialized as JSON in DragonFlyDB LIST.
type PendingOp struct {
ID string `json:"id"` // UUID
Type string `json:"type"` // "create_device", "patch_custom_fields", "sync_tags", etc.
Payload json.RawMessage `json:"payload"` // operation-specific data
CreatedAt time.Time `json:"created_at"`
Attempts int `json:"attempts"` // retry count
}
// NewPendingOp creates a new PendingOp with a generated ID and current timestamp.
func NewPendingOp(opType string, payload interface{}) (PendingOp, error) {
data, err := json.Marshal(payload)
if err != nil {
return PendingOp{}, fmt.Errorf("marshal payload: %w", err)
}
return PendingOp{
ID: uuid.New().String(),
Type: opType,
Payload: json.RawMessage(data),
CreatedAt: time.Now().UTC(),
Attempts: 0,
}, nil
}
// WAQ is a write-ahead queue backed by DragonFlyDB (Redis-compatible).
// Operations are stored as JSON in a Redis LIST using RPUSH/BLPOP for FIFO ordering.
type WAQ struct {
rdb *redis.Client
}
// NewWAQ creates a new WAQ connected to the given Redis/DragonFlyDB URL.
// Returns error if the URL is invalid or the server is unreachable.
func NewWAQ(redisURL string) (*WAQ, error) {
opt, err := redis.ParseURL(redisURL)
if err != nil {
return nil, fmt.Errorf("parse dragonfly url: %w", err)
}
client := redis.NewClient(opt)
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := client.Ping(ctx).Err(); err != nil {
client.Close()
return nil, fmt.Errorf("dragonfly unreachable at %s: %w", redisURL, err)
}
log.Printf("WAQ connected to DragonFlyDB")
return &WAQ{rdb: client}, nil
}
// Enqueue pushes a PendingOp onto the right end of the queue (FIFO).
func (q *WAQ) Enqueue(ctx context.Context, op PendingOp) error {
data, err := json.Marshal(op)
if err != nil {
return fmt.Errorf("marshal op: %w", err)
}
return q.rdb.RPush(ctx, queueKey, data).Err()
}
// Dequeue pops the oldest operation (blocking for up to timeout).
// Returns nil, nil if timeout elapses with no item.
// Returns nil, redis.Nil if queue is empty (non-blocking variant would use LPop).
func (q *WAQ) Dequeue(ctx context.Context, timeout time.Duration) (*PendingOp, error) {
result, err := q.rdb.BLPop(ctx, timeout, queueKey).Result()
if err == redis.Nil {
return nil, nil // timeout — no items
}
if err != nil {
return nil, fmt.Errorf("blpop: %w", err)
}
var op PendingOp
if err := json.Unmarshal([]byte(result[1]), &op); err != nil {
return nil, fmt.Errorf("unmarshal op: %w", err)
}
return &op, nil
}
// Len returns the current number of pending operations in the queue.
func (q *WAQ) Len(ctx context.Context) (int64, error) {
n, err := q.rdb.LLen(ctx, queueKey).Result()
if err != nil {
return 0, fmt.Errorf("llen: %w", err)
}
return n, nil
}
// Close releases the DragonFlyDB connection.
func (q *WAQ) Close() error {
return q.rdb.Close()
}
```
Create `internal/queue/waq_test.go`:
```go
package queue_test
import (
"context"
"encoding/json"
"os"
"testing"
"time"
"git.georgsen.dk/hwlab/internal/queue"
)
func TestPendingOpJSON(t *testing.T) {
payload := map[string]string{"device_id": "42", "hw_id": "HW-00001"}
op, err := queue.NewPendingOp("create_device", payload)
if err != nil {
t.Fatalf("NewPendingOp: %v", err)
}
if op.ID == "" {
t.Error("ID should be a UUID")
}
if op.Type != "create_device" {
t.Errorf("Type: want create_device, got %s", op.Type)
}
// Round-trip JSON
data, _ := json.Marshal(op)
var op2 queue.PendingOp
if err := json.Unmarshal(data, &op2); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if op2.ID != op.ID {
t.Errorf("ID mismatch after round-trip: %s != %s", op2.ID, op.ID)
}
}
func TestNewWAQInvalidURL(t *testing.T) {
_, err := queue.NewWAQ("not-a-redis-url")
if err == nil {
t.Error("expected error for invalid URL")
}
}
// dragonflyURL returns the DragonFlyDB URL from env, or skips test if unreachable.
func dragonflyURL(t *testing.T) string {
t.Helper()
url := os.Getenv("HWLAB_DRAGONFLY_URL")
if url == "" {
url = "redis://:nUq/IfoIQJf/kouckKHRQOk7vV0NwCuI@10.5.0.10:6379"
}
return url
}
func TestWAQEnqueueDequeue(t *testing.T) {
waq, err := queue.NewWAQ(dragonflyURL(t))
if err != nil {
t.Skipf("DragonFlyDB unavailable: %v", err)
}
defer waq.Close()
ctx := context.Background()
// Clean up before test
// Note: In a real test suite, use a test-specific queue key.
// For now, just ensure queue starts non-empty cleanup is acceptable.
op, _ := queue.NewPendingOp("test_op", map[string]string{"test": "value"})
if err := waq.Enqueue(ctx, op); err != nil {
t.Fatalf("Enqueue: %v", err)
}
n, err := waq.Len(ctx)
if err != nil {
t.Fatalf("Len: %v", err)
}
if n < 1 {
t.Error("expected at least 1 item after enqueue")
}
got, err := waq.Dequeue(ctx, 2*time.Second)
if err != nil {
t.Fatalf("Dequeue: %v", err)
}
if got == nil {
t.Fatal("expected op from dequeue, got nil")
}
if got.ID != op.ID {
t.Errorf("dequeued op ID mismatch: want %s, got %s", op.ID, got.ID)
}
}
```
</action>
<verify>
<automated>cd /home/mikkel/homelabby && go test ./internal/queue/... -v -run "TestPendingOpJSON|TestNewWAQInvalidURL"</automated>
</verify>
<acceptance_criteria>
- `go test ./internal/queue/... -run "TestPendingOpJSON|TestNewWAQInvalidURL"` passes (unit tests, no DragonFlyDB needed)
- `grep "hwlab:netbox:pending_ops" internal/queue/waq.go` returns the queueKey const
- `grep "RPush" internal/queue/waq.go` returns the Enqueue implementation
- `grep "BLPop" internal/queue/waq.go` returns the Dequeue implementation
- `grep "ParseURL" internal/queue/waq.go` returns the redis.ParseURL call
- `go build ./internal/queue/...` exits 0
- If DragonFlyDB reachable: `go test ./internal/queue/... -v -run TestWAQEnqueueDequeue` passes
</acceptance_criteria>
<done>WAQ core implemented. Unit tests pass. FIFO enqueue/dequeue via RPUSH/BLPOP. Integration test skips gracefully when DragonFlyDB unreachable.</done>
</task>
<task type="auto">
<name>Task 2: WAQ retry worker + wire into main binary</name>
<files>internal/queue/worker.go, cmd/hwlab/main.go</files>
<read_first>
- /home/mikkel/homelabby/internal/queue/waq.go (WAQ type, PendingOp, Dequeue method)
- /home/mikkel/homelabby/cmd/hwlab/main.go (current main.go to understand wiring pattern)
- /home/mikkel/homelabby/internal/config/config.go (WAQRetryIntervalSeconds, DragonflyURL)
</read_first>
<action>
1. Create `internal/queue/worker.go`:
```go
package queue
import (
"context"
"log"
"time"
)
// OpHandler is a function that processes a single dequeued operation.
// Returns nil on success, error if the operation should be re-queued.
type OpHandler func(ctx context.Context, op PendingOp) error
// RunWorker runs a blocking BLPOP loop processing ops from the queue.
// It calls handler for each dequeued op. If handler returns an error,
// the op is re-enqueued with incremented Attempts count.
// Ops that exceed maxAttempts are dropped with a log warning.
//
// On DragonFlyDB connection loss, RunWorker backs off and retries connection.
// Call with a cancellable context to stop the worker cleanly.
func (q *WAQ) RunWorker(ctx context.Context, handler OpHandler, maxAttempts int, retryInterval time.Duration) {
log.Printf("WAQ worker started (maxAttempts=%d, retryInterval=%s)", maxAttempts, retryInterval)
for {
select {
case <-ctx.Done():
log.Printf("WAQ worker stopping: %v", ctx.Err())
return
default:
}
op, err := q.Dequeue(ctx, 5*time.Second)
if err != nil {
// Connection error — back off before retrying
log.Printf("WAQ dequeue error: %v — backing off %s", err, retryInterval)
select {
case <-ctx.Done():
return
case <-time.After(retryInterval):
}
continue
}
if op == nil {
// Timeout with no items — loop immediately (BLPOP already waited 5s)
continue
}
// Process the operation
if err := handler(ctx, *op); err != nil {
op.Attempts++
if op.Attempts >= maxAttempts {
log.Printf("WAQ: dropping op %s (type=%s) after %d failed attempts: %v",
op.ID, op.Type, op.Attempts, err)
continue
}
// Re-enqueue for retry
log.Printf("WAQ: re-enqueuing op %s (type=%s, attempt=%d): %v",
op.ID, op.Type, op.Attempts, err)
if enqErr := q.Enqueue(ctx, *op); enqErr != nil {
log.Printf("WAQ: failed to re-enqueue op %s: %v", op.ID, enqErr)
}
}
}
}
// NoOpHandler is a placeholder op handler for Phase 1.
// Phase 2 will replace this with a real NetBox retry handler.
// It logs the operation and returns nil (success) so ops drain from the queue.
func NoOpHandler(ctx context.Context, op PendingOp) error {
log.Printf("WAQ [noop]: processing op %s (type=%s, attempts=%d)", op.ID, op.Type, op.Attempts)
return nil
}
```
2. Update `cmd/hwlab/main.go` to wire the WAQ:
Read the current main.go first, then add WAQ initialization and worker goroutine.
The WAQ initialization must be non-fatal — if DragonFlyDB is unavailable, the binary
still starts and serves HTTP. WAQ is degraded, not required.
Updated main.go pattern:
```go
package main
import (
"context"
"fmt"
"log"
"net/http"
"os/signal"
"syscall"
"time"
"git.georgsen.dk/hwlab/internal/api"
"git.georgsen.dk/hwlab/internal/config"
"git.georgsen.dk/hwlab/internal/queue"
)
func main() {
cfg, err := config.Load()
if err != nil {
log.Fatalf("config: %v", err)
}
// Context for graceful shutdown
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
// Start write-ahead queue worker (non-fatal if DragonFlyDB unavailable)
waq, err := queue.NewWAQ(cfg.DragonflyURL)
if err != nil {
log.Printf("WARNING: WAQ unavailable (%v) — NetBox operations will not be queued during downtime", err)
} else {
retryInterval := time.Duration(cfg.WAQRetryIntervalSeconds) * time.Second
go waq.RunWorker(ctx, queue.NoOpHandler, cfg.WAQMaxAttempts, retryInterval)
defer waq.Close()
log.Printf("WAQ worker started")
}
router := api.NewRouter()
addr := fmt.Sprintf("%s:%d", cfg.Host, cfg.Port)
log.Printf("HWLab starting on %s", addr)
srv := &http.Server{Addr: addr, Handler: router}
go func() {
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatalf("server: %v", err)
}
}()
// Wait for shutdown signal
<-ctx.Done()
log.Println("Shutting down...")
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
srv.Shutdown(shutdownCtx)
log.Println("Shutdown complete")
}
```
</action>
<verify>
<automated>cd /home/mikkel/homelabby && go build ./cmd/hwlab/... && go vet ./...</automated>
</verify>
<acceptance_criteria>
- `go build ./cmd/hwlab/...` exits 0
- `go vet ./...` exits 0
- `grep "RunWorker" cmd/hwlab/main.go` returns the goroutine call
- `grep "NoOpHandler" internal/queue/worker.go` returns the placeholder handler
- `grep "maxAttempts" internal/queue/worker.go` returns the op drop condition
- `grep "WARNING: WAQ unavailable" cmd/hwlab/main.go` returns the non-fatal degraded path
- Binary starts without panic when DragonFlyDB is available
- Binary starts with WARNING log (not fatal) when DragonFlyDB is unavailable
- `go test ./...` remains green (no regressions from main.go changes)
</acceptance_criteria>
<done>WAQ worker implemented with exponential backoff, max attempts drop, and NoOpHandler placeholder. main.go updated with graceful shutdown and non-fatal WAQ initialization. Full `go test ./...` still green.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| internal/queue → DragonFlyDB | Redis protocol over TCP to 10.5.0.10:6379; password in URL |
| PendingOp.Payload | JSON RawMessage from NetBox operation context — validated by op handler |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-05-01 | Information Disclosure | DragonFlyDB password in HWLAB_DRAGONFLY_URL | accept | Private homelab LAN (10.5.0.x); password is in .env which is .gitignored; no external access |
| T-05-02 | Tampering | PendingOp re-enqueue on failure | accept | Ops re-enqueued only by the worker itself; no external write path to the queue in Phase 1 |
| T-05-03 | Denial of Service | Queue accumulation if handler always fails | mitigate | maxAttempts drop logic — ops dropped after cfg.WAQMaxAttempts (default 5) failures; prevents unbounded queue growth |
| T-05-04 | Denial of Service | WAQ worker tight-loop on connection loss | mitigate | retryInterval backoff (default 30s) prevents hammering DragonFlyDB on reconnect |
</threat_model>
<verification>
After both tasks complete:
- `go test ./...` passes (all packages, no regressions)
- `go build ./cmd/hwlab/...` exits 0
- `go vet ./...` exits 0
- If DragonFlyDB reachable: `go test ./internal/queue/... -v` shows TestWAQEnqueueDequeue PASS
- Start binary and verify it serves health endpoint: `./hwlab &; sleep 1; curl http://localhost:8080/api/health; kill %1`
</verification>
<success_criteria>
1. WAQ core: Enqueue, Dequeue, Len all working against DragonFlyDB (integration) or skipping gracefully (unit)
2. Worker: BLPOP loop with backoff, max attempts drop, context cancellation
3. main.go: Non-fatal WAQ init — WARNING log instead of panic when DragonFlyDB unavailable
4. Graceful shutdown: SIGINT triggers orderly HTTP server and WAQ worker shutdown
5. Full test suite `go test ./...` green
</success_criteria>
<output>
After completion, create `.planning/phases/01-foundation/01-05-SUMMARY.md` with:
- Whether DragonFlyDB integration tests ran or skipped
- Any DragonFlyDB/go-redis v9 compatibility notes (DragonFlyDB is Redis-compatible but may have minor differences)
- Final `go test ./...` output
- Files created/modified
</output>