mikkel/homelab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
homelab/.planning/codebase/CONCERNS.md
Mikkel Georgsen a639a53b0b docs: add codebase map and domain research 
Codebase: 7 documents (stack, architecture, structure, conventions, testing, integrations, concerns)
Research: 5 documents (stack, features, architecture, pitfalls, summary)
2026-02-04 13:50:03 +00:00

17 KiB
Raw Permalink Blame History

Codebase Concerns

Analysis Date: 2026-02-04

Tech Debt

IP Addressing Scheme Inconsistency:

  • Issue: Container IPs don't follow VMID convention. NPM (VMID 100) is at .1, Dockge (VMID 101) at .10, PBS (VMID 106) at .6, instead of matching .100, .101, .106
  • Files: homelab-documentation.md (lines 139-159)
  • Impact: Manual IP tracking required, DNS records must be maintained separately, new containers require manual IP assignment planning, documentation drift risk
  • Fix approach: Execute TODO task to reorganize vmbr1 to VMID=IP scheme (.100-.253 range), update NPM proxy hosts, DNS records (lab.georgsen.dk), and documentation

DNS Record Maintenance Manual:

  • Issue: Internal DNS (Technitium) and external DNS (dns.services) require manual updates when IPs/domains change
  • Files: homelab-documentation.md (lines 432-449), ~/bin/dns script
  • Impact: Risk of records becoming stale after IP migrations, no automation for new containers
  • Fix approach: Implement dns-services helper script (TODO.md line 27) with API integration for automatic updates

Unimplemented Helper Scripts:

  • Issue: dns-services API integration promised in TODO but not implemented
  • Files: TODO.md (line 27), dns-services/credentials exists but script doesn't
  • Impact: Manual dns.services operations required, cannot automate domain setup
  • Fix approach: Create ~/bin/dns-services wrapper (endpoint documented in TODO)

Ping Capability Missing on 12 Containers:

  • Issue: Unprivileged LXC containers drop cap_net_raw, breaking ping on VMIDs 100, 101, 102, 103, 104, 105, 107, 108, 110, 111, 112, 114, 115, 1000
  • Files: TODO.md (lines 31-33), CLAUDE.md (line 252-255)
  • Impact: Health monitoring fails, network diagnostics broken, Telegram bot status checks incomplete (bot has no ping on home network itself), Uptime Kuma monitors may show false negatives
  • Fix approach: Run setcap cap_net_raw+ep /bin/ping on each container (must be reapplied after iputils-ping updates)

Version Pinning Warnings:

  • Issue: CLAUDE.md section 227-241 warns about hardcoded versions becoming stale
  • Files: homelab-documentation.md (lines 217, 228, 239), ~/bin/updates script shows version checking is implemented but some configs have latest tags
  • Impact: Security patch delays, incompatibilities when manually deploying services
  • Fix approach: Always query GitHub API for latest versions (updates script does this correctly for discovery phase)

Known Bugs

Telegram Bot Inbox Storage Race Condition:

  • Symptoms: Concurrent message writes could corrupt inbox file, messages may be lost
  • Files: telegram/bot.py (lines 39, 200-220 message handling), ~/bin/telegram (lines 73-79 clear command)
  • Trigger: Multiple rapid messages from admin or concurrent bot operations
  • Workaround: Clear inbox frequently and check for corruption; bot currently appends to file without locking
  • Root cause: File-based inbox with no atomic writes or mutex protection

PBS Backup Mount Dependency Not Enforced:

  • Symptoms: PBS services may start before Synology CIFS mount is available, backup path unreachable
  • Files: homelab-documentation.md (lines 372-384), container 106 config
  • Trigger: System reboot when Tailscale connectivity is delayed
  • Workaround: Manual restart of proxmox-backup-proxy and proxmox-backup services
  • Root cause: systemd dependency chain After=mnt-synology.mount doesn't guarantee mount is ready at service start time

DragonflyDB Password in Plain Text in Documentation:

  • Symptoms: Database password visible in compose file and documentation
  • Files: homelab-documentation.md (lines 248-250)
  • Trigger: Anyone reading docs or inspecting git history
  • Workaround: Consider password non-critical if container only accessible on internal network
  • Root cause: Password stored in version control and documentation rather than .env or secrets file

NPM Proxy Host 18 (mh.datalos.dk) Not Configured:

  • Symptoms: Domain not resolving despite DNS record missing and NPM entry (ID 18) mentioned in TODO
  • Files: TODO.md (line 29), homelab-documentation.md (proxy hosts section)
  • Trigger: Accessing mh.datalos.dk from browser
  • Workaround: Must be configured manually via NPM web UI
  • Root cause: Setup referenced in TODO but not completed

Security Considerations

Exposed Credentials in Git History:

  • Risk: Credential files committed (credentials, SSH keys, telegram token examples)
  • Files: All credential files in telegram/, pve/, forgejo/, dns/, dockge/, uptime-kuma/, beszel/, dns-services/ directories (8+ files)
  • Current mitigation: Files are .gitignored in main repo but present in working directory
  • Recommendations: Rotate all credentials listed, audit git log for historical commits, use HashiCorp Vault or pass for credential storage, document secret rotation procedure

Public IP Hardcoded in Documentation:

  • Risk: Home IP 83.89.248.247 exposed in multiple locations
  • Files: homelab-documentation.md (lines 98, 102), CLAUDE.md (line 256)
  • Current mitigation: IP is already public/static, used for whitelist access
  • Recommendations: Document that whitelisting this IP is intentional, no other PII mixed in

Telegram Bot Authorization Model Too Permissive:

  • Risk: First user to message bot becomes admin automatically with no verification
  • Files: telegram/bot.py (lines 86-95)
  • Current mitigation: Bot only responds to authorized user, requires bot discovery
  • Recommendations: Require multi-factor authorization on first start (e.g., PIN from environment variable), implement audit logging of all bot commands

Database Credentials in Environment Variables:

  • Risk: DragonflyDB password passed via Docker command line (visible in docker ps, logs, process listings)
  • Files: homelab-documentation.md (line 248)
  • Current mitigation: Container only accessible on internal vmbr1 network
  • Recommendations: Use Docker secrets or mounted .env files instead of command-line arguments

Synology CIFS Credentials in fstab:

  • Risk: SMB credentials stored in plaintext in fstab file with mode 0644 (world-readable)
  • Files: homelab-documentation.md (line 369)
  • Current mitigation: Mounted on container-only network, requires PBS container access
  • Recommendations: Use credentials file with mode 0600, rotate credentials regularly, monitor file permissions

SSH Keys Included in Documentation:

  • Risk: Public SSH keys hardcoded in CLAUDE.md setup examples
  • Files: CLAUDE.md and homelab-documentation.md SSH key examples
  • Current mitigation: Public keys only (not private), used for container access
  • Recommendations: Rotate these keys if documentation is ever exposed, don't include in public repos

Performance Bottlenecks

Single NVMe Storage (RAID0) Without Local Redundancy:

  • Problem: Core server has 2x1TB NVMe in RAID0 (striped, no redundancy)
  • Files: homelab-documentation.md (lines 17-24)
  • Cause: Cost optimization for Hetzner dedicated server
  • Impact: Single drive failure = total data loss; database corruption risk from RAID0 stripe inconsistency
  • Improvement path: (1) Ensure PBS backups run successfully to Synology, (2) Test backup restore procedure monthly, (3) Plan upgrade path if budget allows (3-way mirror or RAID1)

Backup Dependency on Single Tailscale Gateway:

  • Problem: All PBS backups to Synology go through Tailscale relay (10.5.0.134), single point of failure
  • Files: homelab-documentation.md (lines 317-427)
  • Cause: Synology only accessible via Tailscale network, relay container required
  • Impact: Tailscale relay downtime = backup failure; no local backup option
  • Improvement path: (1) Add second Tailscale relay for redundancy, (2) Explore PBS direct SSH backup mode, (3) Monitor relay container health

DNS Queries All Route Through Single Technitium Container:

  • Problem: All internal DNS (lab.georgsen.dk) goes through container 115, DHCP defaults to this server
  • Files: homelab-documentation.md (lines 309-315), container config
  • Cause: Single container architecture
  • Impact: DNS outage = network unreachable (containers can't resolve any hostnames)
  • Improvement path: (1) Deploy DNS replica on another container, (2) Configure DHCP to use multiple DNS servers, (3) Set upstream DNS fallback

Script Execution via Telegram Bot with Subprocess Timeout:

  • Problem: Bot runs helper scripts with 30-second timeout, commands like PBS backup query can exceed limit
  • Files: telegram/bot.py (lines 60-78, 191)
  • Cause: Helper scripts do remote SSH execution, network latency variable
  • Impact: Commands truncated mid-execution, incomplete status reports, timeouts on slow networks
  • Improvement path: Increase timeout selectively, implement command queuing, cache results for frequently-called commands

Fragile Areas

Installer Shell Script with Unimplemented Sections:

  • Files: pve-homelab-kit/install.sh (495+ lines with TODO comments)
  • Why fragile: Multiple TODO placeholders indicate incomplete implementation; wizard UI done but ~30 implementation TODOs remain
  • Safe modification: (1) Don't merge branches without running through full install, (2) Test each section independently, (3) Add shell set -e error handling
  • Test coverage: Script has no tests, no dry-run mode, no rollback capability

Container Configuration Manual in LXC Config Files:

  • Files: /etc/pve/lxc/*.conf across Proxmox host (not in repo, not version controlled)
  • Why fragile: Critical settings (features, ulimits, AppArmor) outside version control, drift risk after manual fixes
  • Safe modification: Keep backup copies in homelab-documentation.md (already done for PBS), automate via Terraform/Ansible if future containers added
  • Test coverage: Config changes only tested on live container (no staging env)

Helper Scripts with Hardcoded IPs and Paths:

  • Files: ~/bin/updates (lines 16-17, 130), ~/bin/pbs, ~/bin/pve, ~/bin/dns
  • Why fragile: DOCKGE_HOST, PVE_HOST hardcoded; if IPs change during migration, all scripts must be updated manually
  • Safe modification: Extract to config file (e.g., /etc/homelab/config.sh or environment variables)
  • Test coverage: Scripts tested against live infrastructure only

SSH-Based Container Access Without Key Verification:

  • Files: ~/bin/updates (lines 115-131), scripts use -q flag suppressing host key checks
  • Why fragile: ssh -q disables StrictHostKeyChecking, vulnerable to MITM; scripts assume SSH keys are pre-installed
  • Safe modification: Add -o StrictHostKeyChecking=accept-new to verify on first connection, document key distribution procedure
  • Test coverage: SSH connectivity assumed working

Backup Monitoring Without Alerting on Failure:

  • Files: ~/bin/pbs, telegram/bot.py (status command only, no automatic failure alerts)
  • Why fragile: Failed backups only visible if manually checked; no monitoring of backup completion
  • Safe modification: Add systemd timer to check PBS status hourly, send Telegram alert on failure
  • Test coverage: Manual checks only

Scaling Limits

Container IP Space Exhaustion:

  • Current capacity: vmbr1 is /24 (256 IPs, .0-.255), DHCP range .100-.200 (101 IPs available for DHCP), static IPs scattered
  • Limit: After ~150 containers, IP fragmentation becomes difficult to manage; DHCP range conflicts with static allocation
  • Scaling path: (1) Implement TODO IP scheme (VMID=IP), (2) Expand to /23 (512 IPs) if more containers needed, (3) Use vmbr2 (vSwitch) for secondary network

Backup Datastore Single Synology Volume:

  • Current capacity: Synology pbs-backup share unknown size (not documented)
  • Limit: Unknown when share becomes full; no warning system implemented
  • Scaling path: (1) Document share capacity in homelab-documentation.md, (2) Add usage monitoring to beszel or Uptime Kuma, (3) Plan expansion to second NAS

Dockge Stack Limit:

  • Current capacity: Dockge container 101 running ~8-10 stacks visible in documentation
  • Limit: No documented resource constraints; may hit CPU/RAM limits on Hetzner AX52 with more containers
  • Scaling path: (1) Monitor Dockge resource usage via Beszel, (2) Profile Dragonfly memory usage, (3) Plan VM migration for heavy workloads

DNS Query Throughput:

  • Current capacity: Single Technitium container handling all internal DNS
  • Limit: Container CPU/RAM limits unknown; no QPS monitoring
  • Scaling path: (1) Add DNS replica, (2) Monitor query latency, (3) Profile Technitium logs for slow queries

Dependencies at Risk

Technitium DNS (Unmaintained Risk):

  • Risk: TechnitiumSoftware/DnsServer has irregular commit history; last significant release early 2024
  • Impact: Security fixes may be delayed; compatibility with newer Linux kernels unknown
  • Migration plan: (1) Profile current Technitium features used, (2) Evaluate CoreDNS or Dnsmasq alternatives, (3) Plan gradual migration with dual DNS

DragonflyDB as Redis Replacement:

  • Risk: Dragonfly smaller ecosystem than Redis; breaking changes possible in minor updates
  • Impact: Applications expecting Redis behavior may fail; less community support for issues
  • Migration plan: (1) Pin Dragonfly version in compose file (currently latest), (2) Test upgrades in dev environment, (3) Document any API incompatibilities found

Dockge (Single Maintainer Project):

  • Risk: Dockge maintained by one developer (louislam); bus factor high
  • Impact: If maintainer loses interest, fixes and features stop; dependency on their release schedule
  • Migration plan: (1) Use Dockge for UI only, don't depend on it for production orchestration, (2) Keep docker-compose expertise on team, (3) Consider Portainer as fallback alternative

Forgejo (Younger than Gitea):

  • Risk: Forgejo is recent fork of Gitea; database schema changes possible in patch versions
  • Impact: Upgrades may require manual migrations; data loss risk if migration fails
  • Migration plan: (1) Test Forgejo upgrades on backup copy first, (2) Document upgrade procedure, (3) Keep Gitea as fallback if Forgejo breaks

Missing Critical Features

No Automated Health Monitoring/Alerting:

  • Problem: Status checks exist (via Telegram bot, Uptime Kuma) but no automatic alerts when services fail
  • Blocks: Cannot sleep soundly; must manually check status to detect outages
  • Implementation path: (1) Add Uptime Kuma HTTP monitors for all public services, (2) Create Telegram alert webhook, (3) Monitor PBS backup success daily

No Automated Certificate Renewal Verification:

  • Problem: NPM handles Let's Encrypt renewal, but no monitoring for renewal failures
  • Blocks: Certificates could expire silently; discovered during service failures
  • Implementation path: (1) Add Uptime Kuma alert for HTTP 200 on https://* services, (2) Add monthly certificate expiry check, (3) Set up renewal failure alerts

No Disaster Recovery Runbook:

  • Problem: Procedures for rescuing locked-out server (Hetzner Rescue Mode) not documented
  • Blocks: If SSH access lost, cannot recover without external procedures
  • Implementation path: (1) Document Hetzner Rescue Mode recovery steps, (2) Create network reconfiguration backup procedures, (3) Test rescue mode monthly

No Change Log / Audit Trail:

  • Problem: Infrastructure changes not logged; drift from documentation occurs silently
  • Blocks: Unknown who made changes, when, and why; cannot track config evolution
  • Implementation path: (1) Add git commit requirement for all manual changes, (2) Create change notification to Telegram, (3) Weekly drift detection report

No Secrets Management System:

  • Problem: Credentials scattered across plaintext files, git history, and documentation
  • Blocks: Cannot safely share access with team members; no credential rotation capability
  • Implementation path: (1) Deploy HashiCorp Vault or Vaultwarden, (2) Migrate all secrets to vault, (3) Create credential rotation procedures

Test Coverage Gaps

PBS Backup Restore Not Tested:

  • What's not tested: Full restore procedures; assumed to work but never verified
  • Files: homelab-documentation.md (lines 325-392), no restore test documented
  • Risk: If restore needed, may discover issues during actual data loss emergency
  • Priority: HIGH - Add monthly restore test procedure (restore single VM to temporary location, verify data integrity)

Network Failover Scenarios:

  • What's not tested: What happens if Tailscale relay (1000) goes down, if NPM container restarts, if DNS returns SERVFAIL
  • Files: No documented failure scenarios
  • Risk: Unknown recovery time; applications may hang instead of failing gracefully
  • Priority: HIGH - Document and test each service's failure mode

Helper Script Error Handling:

  • What's not tested: Scripts with SSH timeouts, host unreachable, malformed responses
  • Files: ~/bin/updates, ~/bin/pbs, ~/bin/pve (error handling exists but not tested against failures)
  • Risk: Silent failures could go unnoticed; incomplete output returned to caller
  • Priority: MEDIUM - Add error injection tests (mock SSH failures)

Telegram Bot Commands Under Load:

  • What's not tested: Bot response when running concurrent commands, or when helper scripts timeout
  • Files: telegram/bot.py (no load tests, concurrency behavior unknown)
  • Risk: Bot may hang or lose messages under heavy load
  • Priority: MEDIUM - Add load test with 10+ concurrent commands

Container Migration (VMID IP Scheme Change):

  • What's not tested: Migration of 15+ containers to new IP scheme; full rollback procedures
  • Files: TODO.md (line 5-15, planned but not executed)
  • Risk: Single IP misconfiguration could take multiple services offline
  • Priority: HIGH - Create detailed migration runbook with rollback at each step before executing

Concerns audit: 2026-02-04