diff --git a/.planning/phases/35-npx-buildthis-cli/35-RESEARCH.md b/.planning/phases/35-npx-buildthis-cli/35-RESEARCH.md new file mode 100644 index 00000000..9d99cdd5 --- /dev/null +++ b/.planning/phases/35-npx-buildthis-cli/35-RESEARCH.md @@ -0,0 +1,453 @@ +# Phase 35: npx buildthis CLI - Research + +**Researched:** 2026-04-01 +**Domain:** Node.js CLI packaging, npx entrypoints, hardware detection, interactive terminal UX +**Confidence:** HIGH + +--- + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions +None — all implementation choices are at Claude's discretion. + +### Claude's Discretion +All implementation choices are at Claude's discretion. + +### Deferred Ideas (OUT OF SCOPE) +None. + + + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|------------------| +| CLI-01 | User can run `npx buildthis` to bootstrap Nexus from scratch | `buildthis` package with `bin.buildthis` pointing to CLI entry; detects running instance vs fresh install path | +| CLI-02 | CLI bootstrapper detects hardware and walks through same provider tiering as web onboarding | Hardware detection via inline `os` module (no `systeminformation` needed for Apple Silicon; GPU probe optional) + provider tiering display mirrored from web onboarding | + + +--- + +## Summary + +Phase 35 introduces a standalone `npx buildthis` entrypoint — a developer-facing bootstrapper that a user can run on a fresh machine without having cloned the repo. When invoked it follows two distinct paths: if Nexus is already running on the default port (3100) it opens the browser immediately; if not, it walks the user through hardware-aware provider selection (matching the web onboarding Phase 30-32 logic) and then calls the existing `paperclipai onboard --run` flow. + +The `buildthis` npm package is a new, minimal package under `packages/buildthis/` (or as a second `bin` entry on the existing `paperclipai` package — see Architecture Patterns). It bundles its own tiny entry with esbuild, following the same pattern as `cli/`. Hardware detection can be done entirely with the Node.js built-in `os` module for Apple Silicon (no `systeminformation` needed at the CLI layer) and a 3-second `si.graphics()` probe for GPU — or, if Nexus is running, by calling `GET /system/providers` directly. The second approach is simpler and avoids adding `systeminformation` to the `buildthis` package. + +The hardest design question is where `buildthis` lives: as a new `packages/buildthis/` workspace, or as a second `bin` on the existing `cli/package.json`. The new package approach is cleaner for users (`npx buildthis` vs `npx paperclipai buildthis`) and avoids leaking all of `paperclipai`'s heavy dependencies into a lightweight bootstrapper download. + +**Primary recommendation:** Create `packages/buildthis/` as a new, minimal workspace package with `bin: { buildthis: "./dist/index.js" }`, a single `src/index.ts` entry, and a thin esbuild build. Hardware detection uses inline `os` + optional `systeminformation` (already in monorepo) for the local probe path. + +--- + +## Standard Stack + +### Core +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| `@clack/prompts` | `^0.10.0` | Interactive terminal prompts (spinners, select, text, confirm) | Already used across all `cli/` commands; consistent UX | +| `commander` | `^13.1.0` | CLI argument parsing | Already used in `cli/`; mature, zero-dep | +| `picocolors` | `^1.1.1` | Terminal colouring | Already used in `cli/`; smallest colour lib | +| `open` | `^11.0.0` | Open URLs in the default browser | Already used in `@paperclipai/server` for `PAPERCLIP_OPEN_ON_LISTEN`; same version | +| `systeminformation` | `5` | GPU detection (VRAM probe) | Already used in `@paperclipai/server`; pinned to v5 per project decision | + +### Supporting +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| `esbuild` | workspace devDep | Bundle to single `dist/index.js` | Build step, same pattern as `cli/esbuild.config.mjs` | +| `tsx` | `^4.19.2` | Dev mode entrypoint (`pnpm dev`) | Dev only, same pattern as `cli/` | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| New `packages/buildthis/` package | Second `bin` on `cli/package.json` | `cli/` is 1.8 MB unpacked with many deps; `buildthis` should be a lightweight download — separate package wins | +| Inline `os`-only hardware detection | Call running server's `/system/providers` | If server is not yet running, we need local detection; use local probe first, server probe as fallback | +| `systeminformation` for GPU | Skip GPU detection in `buildthis` | GPU detection is part of CLI-02; include it but behind the same 3-second `Promise.race` timeout pattern | + +**Installation (new package):** +```bash +pnpm add @clack/prompts commander picocolors open systeminformation --filter buildthis +``` + +**Version verification (confirmed 2026-04-01):** +- `@clack/prompts`: 1.2.0 (latest) +- `commander`: 14.0.3 (latest; `^13.1.0` in cli — update to `^14.0.0` or pin to `^13.1.0` for consistency) +- `picocolors`: 1.1.1 (latest) +- `open`: 11.0.0 (latest) +- `systeminformation`: 5.31.5 (latest v5) + +--- + +## Architecture Patterns + +### Recommended Project Structure +``` +packages/buildthis/ +├── package.json # name: "buildthis", bin: { buildthis: "./dist/index.js" } +├── tsconfig.json # extends ../../tsconfig.base.json +├── esbuild.config.mjs # mirror of cli/esbuild.config.mjs, single entry +├── src/ +│ ├── index.ts # CLI entry: program setup + default command = bootstrapCommand +│ ├── bootstrap.ts # main logic: detect-running → open, or guide-install → open +│ ├── hardware.ts # inline hardware detection (copied from server/src/services/hardware.ts) +│ └── banner.ts # printBuildthisBanner (mirrors cli/src/utils/banner.ts) +└── dist/ # gitignored, built artifact +``` + +### Pattern 1: npx entrypoint — `bin` field +**What:** npm `bin` field maps a command name to a file. When `npx buildthis` is run, npm downloads the package and executes `./dist/index.js` directly. +**When to use:** Any time you want `npx ` to work without global install. +**Example:** +```json +// packages/buildthis/package.json +{ + "name": "buildthis", + "bin": { "buildthis": "./dist/index.js" }, + "files": ["dist"], + "publishConfig": { "access": "public" } +} +``` +The bundled `dist/index.js` **must** have a `#!/usr/bin/env node` shebang — the esbuild `banner.js` option injects this automatically (see `cli/esbuild.config.mjs`). + +### Pattern 2: Detect running instance before prompting +**What:** Probe `http://127.0.0.1:/api/health` first. Port is read from config if config exists, else try the default 3100. +**When to use:** `buildthis`'s primary happy path — "already running, just open browser." +**Example:** +```typescript +// Source: onboard.ts bootstrapNexusAgents() health-check pattern +async function probeRunningInstance(port: number): Promise { + try { + const res = await fetch(`http://127.0.0.1:${port}/api/health`, { signal: AbortSignal.timeout(2000) }); + return res.ok; + } catch { + return false; + } +} +``` +Health URL: `http://127.0.0.1:/api/health` — confirmed from `server/src/app.ts` (mounted at `/api/health`). + +### Pattern 3: Hardware detection — inline os + systeminformation +**What:** Same logic as `server/src/services/hardware.ts`. Apple Silicon path uses only `os` module (no `systeminformation` call). GPU path uses `si.graphics()` with a 3-second `Promise.race` timeout. +**When to use:** CLI-02 requirement — detect tier before presenting provider options. +**Example:** +```typescript +// Source: server/src/services/hardware.ts (copy-adapted for buildthis) +import os from "node:os"; +import si from "systeminformation"; + +export type HardwareTier = "gpu" | "apple_silicon" | "cpu_only"; + +export async function detectHardware(): Promise<{ tier: HardwareTier; totalGb: number }> { + const totalGb = Math.round(os.totalmem() / (1024 ** 3) * 10) / 10; + const cpuModel = os.cpus()[0]?.model ?? null; + + if (process.platform === "darwin" && cpuModel?.startsWith("Apple")) { + return { tier: "apple_silicon", totalGb }; + } + + try { + const result = await Promise.race([ + si.graphics(), + new Promise((_, reject) => setTimeout(() => reject(new Error("timeout")), 3000)), + ]); + const vramGb = (result.controllers[0]?.vram ?? 0) / 1024; + if (vramGb >= 4) return { tier: "gpu", totalGb }; + } catch { /* fallthrough to cpu_only */ } + + return { tier: "cpu_only", totalGb }; +} +``` + +### Pattern 4: Provider tiering display (CLI-02) +**What:** After hardware detection, present the same tier-appropriate options as the web wizard: +- GPU / Apple Silicon: recommend local Ollama + show top model from catalog +- Any tier: offer Puter (zero-config), Google OAuth, API key +- CPU-only: up-front note that cloud AI is recommended + +**When to use:** Second path (no running instance, fresh install). + +**CLI display pattern (clack/prompts):** +```typescript +import * as p from "@clack/prompts"; + +p.log.info(`Hardware: ${tier} | RAM: ${totalGb} GB`); +const provider = await p.select({ + message: "Choose a starting provider", + options: [ + { value: "puter", label: "Puter -- free, zero-config", hint: "No API key needed" }, + { value: "google", label: "Google -- Gemini free tier", hint: "Sign in with Google" }, + { value: "apikey", label: "API key -- subscription provider", hint: "OpenAI, Anthropic, Groq" }, + ...(tier !== "cpu_only" ? [{ value: "local", label: "Local AI (Ollama)", hint: "Private, offline" }] : []), + { value: "skip", label: "Skip for now" }, + ], +}); +``` + +### Pattern 5: Delegating to `paperclipai onboard --run` +**What:** After provider guidance, `buildthis` hands off to the existing `paperclipai onboard` or `paperclipai run` flow. This avoids duplicating the full install logic. +**Constraint:** `buildthis` cannot `import` from `@paperclipai/*` workspace packages — it is a standalone public package. It must invoke `paperclipai` as a subprocess via `child_process.spawn` (if already installed) or guide the user to install it first. + +**Key decision:** `buildthis` is a *bootstrapper*, not a full CLI. Its job ends when it either opens the browser or tells the user the next command to run. It does NOT embed the full install logic. + +### Pattern 6: Browser open +**What:** Use `open` package to launch the browser. +**Example:** +```typescript +import open from "open"; +await open(`http://127.0.0.1:${port}`); +``` +Server already uses `open` via dynamic import; `buildthis` can use it as a static dependency. + +### Anti-Patterns to Avoid +- **Bundling `@paperclipai/server` into `buildthis`:** Server is 1.8 MB+ and has native deps (embedded postgres). `buildthis` must stay small — it's what `npx` downloads on a fresh machine. +- **Skipping hardware detection on first path:** Even when Nexus is already running, CLI-02 says hardware detection must be part of the flow. If server is running, call `GET /system/providers` instead of local probe — avoids adding `systeminformation` at all. +- **Using `PAPERCLIP_OPEN_ON_LISTEN` env pattern:** That pattern is for the server process. `buildthis` opens the browser itself after confirming the server is listening. +- **Blocking forever on GPU probe:** Must use the 3-second `Promise.race` timeout, same as `server/src/services/hardware.ts`. + +--- + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| Terminal prompts | Custom readline | `@clack/prompts` | Already used project-wide; handles TTY edge cases, cancellation, piped input | +| Terminal colour | ANSI escape codes | `picocolors` | Tree-shakeable, no deps, already project standard | +| Browser open | Platform-specific shell commands | `open@11` | Handles macOS/Linux/Windows correctly; already in server | +| GPU VRAM detection | Custom WMI/system calls | `systeminformation@5` | Handles cross-platform GPU probing; project already uses v5 | +| CLI arg parsing | Manual `process.argv` | `commander` | Already project standard; handles flags, help, errors | + +**Key insight:** `buildthis` can be very thin — 200-300 lines of TypeScript. Every heavy problem is already solved by existing project libraries. + +--- + +## Common Pitfalls + +### Pitfall 1: `buildthis` name already taken on npm +**What goes wrong:** `npm publish` fails with E403 because `buildthis` is registered. +**Why it happens:** `buildthis` (404 on npm as of 2026-04-01 — confirmed) is not registered, but this can change. +**How to avoid:** `npm view buildthis` returned 404 — name is available. Reserve it early by publishing the package. +**Warning signs:** `npm error 403` during publish. + +### Pitfall 2: `#!/usr/bin/env node` shebang missing from dist +**What goes wrong:** `npx buildthis` runs but produces "Permission denied" or "SyntaxError: Unexpected token #". +**Why it happens:** esbuild doesn't add the shebang automatically unless configured. +**How to avoid:** In `esbuild.config.mjs`, set `banner: { js: "#!/usr/bin/env node" }` — same as `cli/esbuild.config.mjs`. Also run `chmod +x dist/index.js` in build script. +**Warning signs:** `permission denied` from npx; file doesn't start with `#!/usr/bin/env node`. + +### Pitfall 3: `AbortSignal.timeout` not available on old Node.js +**What goes wrong:** `AbortSignal.timeout(2000)` throws `TypeError: AbortSignal.timeout is not a function` on Node 16. +**Why it happens:** `AbortSignal.timeout` was added in Node 17.3. +**How to avoid:** Target Node 20 in esbuild config (`target: "node20"`). Project already uses `node20` in `cli/esbuild.config.mjs`. Add `"engines": { "node": ">=20" }` to `package.json`. +**Warning signs:** Errors on Node 16/18. + +### Pitfall 4: `systeminformation` native bindings in esbuild bundle +**What goes wrong:** `systeminformation` includes platform-specific binaries that esbuild can't bundle; build succeeds but crashes at runtime. +**Why it happens:** Some `systeminformation` functions use optional native bindings. +**How to avoid:** Mark `systeminformation` as `external` in esbuild config (same as how `cli/esbuild.config.mjs` handles all npm deps). It will be a regular `node_modules` dependency, not bundled. +**Warning signs:** `Cannot find module` errors for `.node` files at runtime. + +### Pitfall 5: Health URL path confusion +**What goes wrong:** Probing the wrong URL — `http://127.0.0.1:3100/health` instead of `/api/health`. +**Why it happens:** The server mounts health at `/api/health` (inside the `api` router at `app.ts:237`) not `/health`. +**How to avoid:** Use `http://127.0.0.1:${port}/api/health` — confirmed from `server/src/app.ts` line 237 + 138. +**Warning signs:** 404 on health probe even when server is running. + +### Pitfall 6: Non-TTY environments (CI, piped input) +**What goes wrong:** `@clack/prompts` hangs or crashes when stdin is not a TTY. +**Why it happens:** Interactive prompts require a TTY. +**How to avoid:** Check `process.stdin.isTTY && process.stdout.isTTY` before entering interactive mode (same guard used in `onboard.ts:395`). In non-TTY, print instructions and exit 0. +**Warning signs:** `buildthis` hanging in CI. + +--- + +## Code Examples + +### Detect running instance +```typescript +// Source: adapted from cli/src/commands/onboard.ts bootstrapNexusAgents() +async function probeRunningInstance(port: number): Promise { + try { + const res = await fetch(`http://127.0.0.1:${port}/api/health`, { + signal: AbortSignal.timeout(2000), + }); + return res.ok; + } catch { + return false; + } +} +``` + +### Read port from existing config (if present) +```typescript +// Source: pattern from cli/src/config/store.ts + cli/src/config/schema.ts +import { configExists, readConfig } from "./config/store.js"; + +function resolveNexusPort(): number { + const DEFAULT_PORT = 3100; + try { + if (configExists()) { + const config = readConfig(); + return config?.server?.port ?? DEFAULT_PORT; + } + } catch { /* config parse error — use default */ } + return DEFAULT_PORT; +} +``` + +Note: `buildthis` cannot import from workspace packages directly. It must either replicate the config-reading logic inline or accept that it only checks port 3100. The simplest approach: probe 3100 only. Config path resolution is an advanced feature for a v2. + +### Hardware detection (server GET /system/providers fallback) +```typescript +// If server is running, use its hardware endpoint instead of local probe +async function fetchHardwareFromServer(port: number): Promise { + try { + const res = await fetch(`http://127.0.0.1:${port}/system/providers`, { + signal: AbortSignal.timeout(3000), + }); + if (res.ok) return res.json() as Promise; + } catch { /* server not running */ } + return null; +} +``` + +### esbuild config for buildthis +```javascript +// Source: modelled on cli/esbuild.config.mjs +export default { + entryPoints: ["src/index.ts"], + bundle: true, + platform: "node", + target: "node20", + format: "esm", + outfile: "dist/index.js", + banner: { js: "#!/usr/bin/env node" }, + external: [ + "systeminformation", // native bindings — must stay external + "open", // uses dynamic import internally + ], + treeShaking: true, + sourcemap: true, +}; +``` + +--- + +## Runtime State Inventory + +> Skipped — this is a greenfield package creation phase. No rename, refactor, or migration involved. + +--- + +## Environment Availability + +| Dependency | Required By | Available | Version | Fallback | +|------------|------------|-----------|---------|----------| +| Node.js 20+ | Build + runtime | ✓ | Linux 6.17.4 / node present | — | +| pnpm | Workspace management | ✓ | workspace in use | — | +| npm registry | `npx buildthis` resolution | ✓ (confirmed `paperclipai` 2026.325.0 live) | — | publish step is manual | +| `systeminformation` v5 | GPU detection | ✓ (in server/node_modules) | 5.31.5 | Apple Silicon + cpu_only paths work without it | +| `open` v11 | Browser open | ✓ (in server/node_modules) | 11.0.0 | Print URL to console as fallback | +| `@clack/prompts` | Interactive prompts | ✓ (in cli/node_modules) | 1.2.0 | — | + +**Missing dependencies with no fallback:** None. + +**Missing dependencies with fallback:** None — all required packages already exist in the monorepo. + +--- + +## Validation Architecture + +### Test Framework +| Property | Value | +|----------|-------| +| Framework | vitest 2.x (workspace project) | +| Config file | `packages/buildthis/vitest.config.ts` (Wave 0 gap — doesn't exist yet) | +| Quick run command | `npx vitest run packages/buildthis/src/__tests__/` | +| Full suite command | `npx vitest run` | + +### Phase Requirements → Test Map +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| CLI-01 | `probeRunningInstance` returns true when server responds 200 | unit | `npx vitest run packages/buildthis/src/__tests__/bootstrap.test.ts` | ❌ Wave 0 | +| CLI-01 | `probeRunningInstance` returns false when port closed / timeout | unit | `npx vitest run packages/buildthis/src/__tests__/bootstrap.test.ts` | ❌ Wave 0 | +| CLI-02 | `detectHardware` returns `apple_silicon` on darwin + Apple CPU | unit | `npx vitest run packages/buildthis/src/__tests__/hardware.test.ts` | ❌ Wave 0 | +| CLI-02 | `detectHardware` returns `cpu_only` when GPU probe times out | unit | `npx vitest run packages/buildthis/src/__tests__/hardware.test.ts` | ❌ Wave 0 | +| CLI-02 | Provider options include local AI only for non-cpu_only tiers | unit | `npx vitest run packages/buildthis/src/__tests__/bootstrap.test.ts` | ❌ Wave 0 | + +### Sampling Rate +- **Per task commit:** `npx vitest run packages/buildthis/src/__tests__/` +- **Per wave merge:** `npx vitest run` +- **Phase gate:** Full suite green before `/gsd:verify-work` + +### Wave 0 Gaps +- [ ] `packages/buildthis/vitest.config.ts` — vitest project config +- [ ] `packages/buildthis/src/__tests__/bootstrap.test.ts` — covers CLI-01 (probe, open, non-TTY guard) +- [ ] `packages/buildthis/src/__tests__/hardware.test.ts` — covers CLI-02 (tier detection by platform/CPU/GPU) + +--- + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| Single binary with all logic | Thin bootstrapper + delegates to heavy CLI | Phase 35 | `buildthis` stays small (~200KB bundle) | +| Hardware detection only in server | Hardware detection available in CLI layer | Phase 35 | No server needed for pre-install guidance | + +**Deprecated/outdated:** +- None relevant to this phase. + +--- + +## Open Questions + +1. **Should `buildthis` invoke `paperclipai` as a subprocess, or just print instructions?** + - What we know: `buildthis` cannot bundle `@paperclipai/*` workspace packages; subprocess invoke requires `paperclipai` to be installed globally or via docker. + - What's unclear: Is `paperclipai` expected to be installed alongside `buildthis`, or is `buildthis` truly a fresh-machine bootstrapper? + - Recommendation: Print clear next-step instructions (`npx paperclipai@latest onboard` or `npm install -g paperclipai && paperclipai run`) rather than spawning a subprocess. This is simpler and more reliable. + +2. **Should the `buildthis` package version track the `paperclipai` version?** + - What we know: `paperclipai` uses date-based versioning (2026.MDD.P). `buildthis` is a different package. + - What's unclear: Whether both should be published together in the same release script. + - Recommendation: Start with independent versioning (0.1.0); add to the release script in a follow-up. The release script in `scripts/release.sh` only handles `cli/`. + +3. **Where does `buildthis` live in the workspace — `packages/buildthis/` or `cli-bootstrap/`?** + - What we know: `pnpm-workspace.yaml` includes `packages/*`. + - What's unclear: Naming convention preference. + - Recommendation: `packages/buildthis/` — consistent with workspace layout; `packages/*` is already in workspace glob. + +--- + +## Sources + +### Primary (HIGH confidence) +- `cli/src/index.ts` — Commander registration, `onboard` and `run` command wiring +- `cli/src/commands/onboard.ts` — Full interactive onboard flow, health-check pattern, `bootstrapNexusAgents` +- `cli/src/commands/run.ts` — `runCommand` implementation, server start, `PAPERCLIP_OPEN_ON_LISTEN` +- `cli/esbuild.config.mjs` — esbuild bundle strategy, shebang pattern, external deps +- `cli/package.json` — Exact dependency versions, build/publish setup +- `server/src/services/hardware.ts` — Hardware detection logic (Apple Silicon path, GPU probe, 3-second timeout) +- `server/src/app.ts` — Health route mounted at `/api/health` (line 237 + 138 confirmation) +- `server/src/index.ts` — `PAPERCLIP_OPEN_ON_LISTEN` env var + `open` package usage +- `pnpm-workspace.yaml` — Workspace package globs +- `scripts/build-npm.sh` — CLI build pipeline steps + +### Secondary (MEDIUM confidence) +- npm registry check: `buildthis` package name — confirmed 404 (available) as of 2026-04-01 +- npm registry check: `paperclipai` — confirmed published at 2026.325.0 + +### Tertiary (LOW confidence) +- None. + +--- + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH — all libraries already in use in project; versions confirmed via `npm view` +- Architecture: HIGH — based on direct codebase inspection; patterns copied from existing `cli/` implementation +- Pitfalls: HIGH — derived from existing code patterns and confirmed npm registry state + +**Research date:** 2026-04-01 +**Valid until:** 2026-05-01 (stable domain; npm package availability could change sooner)