nexus/.planning/codebase/ARCHITECTURE.md

# Architecture

**Analysis Date:** 2026-03-30
**Codebase:** Paperclip (nexus repo at `/Volumes/UsbNvme/repos/nexus/`)

---

## Pattern Overview

**Overall:** Multi-package TypeScript monorepo. Server-rendered React SPA (Vite) + Express API server + CLI tool + adapter plugin system.

**Key Characteristics:**
- pnpm workspaces monorepo with five top-level workspace members: `server`, `ui`, `cli`, `packages/*`, `plugins/*`
- Express REST API with actor-based auth middleware (board users, agents, instance admin)
- React SPA using React Router v6 with company-prefixed URL scheme (e.g., `/PAP/issues`)
- Agent execution driven by a heartbeat lifecycle: wakeup requests → heartbeat runs → adapter subprocess calls
- Plugin system with isolated worker processes and structured host/worker RPC

---

## Monorepo Structure

```
nexus/
├── server/              # Express API + business logic
│   ├── src/
│   │   ├── routes/      # Express route handlers
│   │   ├── services/    # Business logic layer
│   │   ├── adapters/    # Agent execution adapter registry
│   │   ├── auth/        # BetterAuth integration
│   │   ├── middleware/  # auth, logging, validation, error handling
│   │   ├── realtime/    # WebSocket live-events bridge
│   │   ├── secrets/     # Secret provider abstraction
│   │   ├── storage/     # S3/local file storage abstraction
│   │   └── app.ts       # Express app factory
├── ui/                  # React SPA (Vite)
│   └── src/
│       ├── pages/       # Top-level page components
│       ├── components/  # Shared UI components
│       ├── context/     # React contexts (company, dialog, live updates)
│       ├── api/         # API client modules (one per domain)
│       ├── hooks/       # Custom React hooks
│       └── App.tsx      # Router + route tree
├── cli/                 # paperclipai CLI (Commander.js)
│   └── src/
│       ├── commands/    # CLI command implementations
│       ├── config/      # Config file read/write
│       └── prompts/     # @clack/prompts interactive wizards
├── packages/
│   ├── db/              # Drizzle ORM schema + migrations + DB client
│   ├── shared/          # Shared types, validators (Zod), constants
│   ├── adapter-utils/   # Shared adapter type contracts + session utils
│   └── adapters/        # Adapter packages (one per AI tool)
│       ├── claude-local/
│       ├── codex-local/
│       ├── cursor-local/
│       ├── gemini-local/
│       ├── opencode-local/
│       ├── pi-local/
│       └── openclaw-gateway/
└── plugins/
    ├── sdk/             # @paperclipai/plugin-sdk (worker-side SDK)
    ├── create-paperclip-plugin/  # Plugin scaffolding tool
    └── examples/        # Example plugins
```

---

## Layers

**Database Layer:**
- Purpose: Schema definitions, migrations, Drizzle ORM client
- Location: `packages/db/src/`
- Contains: Drizzle table definitions in `schema/`, migration SQL in `migrations/`, `client.ts` (embedded Postgres or external URL)
- Depends on: PostgreSQL (embedded via `embedded-postgres` or external)
- Used by: Server services exclusively

**Services Layer:**
- Purpose: All business logic; one service function per domain
- Location: `server/src/services/`
- Contains: `agentService`, `companyService`, `heartbeatService`, `issueService`, `pluginLifecycleManager`, etc.
- Pattern: Each service is a factory function taking `db: Db` and returning an object of async methods. No class instances.
- Depends on: `@paperclipai/db` schema tables, other services
- Used by: Route handlers

**Route Layer:**
- Purpose: HTTP request parsing, auth assertions, response shaping
- Location: `server/src/routes/`
- Contains: One file per resource domain (companies, agents, issues, projects, routines, plugins, etc.)
- Pattern: Each route file exports a factory `function fooRoutes(db: Db): Router`. Routes call service methods, assert auth via `assertBoard` / `assertCompanyAccess` helpers.
- Depends on: Services layer, middleware

**Middleware Layer:**
- Purpose: Cross-cutting request processing
- Location: `server/src/middleware/`
- Files:
  - `auth.ts` — `actorMiddleware`: resolves `req.actor` as board user, agent, or none
  - `board-mutation-guard.ts` — blocks unsafe mutations on read-only mounts
  - `private-hostname-guard.ts` — hostname allowlist enforcement
  - `validate.ts` — Zod schema validation helper
  - `logger.ts` — pino logger
  - `error-handler.ts` — global Express error handler

**Adapter Layer:**
- Purpose: Execute agents via different AI tools (Claude, Codex, Cursor, etc.)
- Location: `server/src/adapters/` (registry + process/http sub-adapters), `packages/adapters/` (per-tool implementations)
- Pattern: Each adapter implements `ServerAdapterModule` from `@paperclipai/adapter-utils`. The registry (`registry.ts`) maps adapter type strings to module instances. `heartbeatService` calls `getServerAdapter(type).execute(...)` to run agents.

**UI Layer:**
- Purpose: React SPA for the board (human operators)
- Location: `ui/src/`
- API communication: `ui/src/api/client.ts` — simple `fetch` wrapper. All domain API modules (e.g., `ui/src/api/agents.ts`) use this `api` object. No external HTTP library.
- State: TanStack Query (`@tanstack/react-query`) for server state. React Context for cross-component state (company selection, dialogs, live updates, theme).
- Real-time: SSE from `/api/realtime` consumed by `LiveUpdatesProvider` in `ui/src/context/LiveUpdatesProvider.tsx`.

**CLI Layer:**
- Purpose: Operator tooling — first-run setup, diagnostics, server run, one-off commands
- Location: `cli/src/`
- Framework: Commander.js with `@clack/prompts` for interactive steps

---

## Authentication Model

Two principal types flow through `actorMiddleware` (`server/src/middleware/auth.ts`):

**Board (human):**
- `local_trusted` deployment: automatically granted `isInstanceAdmin: true`, no auth needed
- `authenticated` deployment: BetterAuth session cookie OR bearer API key (from `board_api_keys` table)
- `req.actor.type === "board"`, carries `userId`, `companyIds[]`, `isInstanceAdmin`

**Agent:**
- Bearer token: either a hashed API key from `agent_api_keys` table, or a short-lived JWT (`createLocalAgentJwt`)
- `req.actor.type === "agent"`, carries `agentId`, `companyId`
- Agents cannot access other companies; CEO agents get additional mutation rights (branding, portability)

Route authorization helpers live in `server/src/routes/authz.ts`:
- `assertBoard(req)` — throws 403 if not board
- `assertCompanyAccess(req, companyId)` — throws 403 if actor cannot access this company
- `assertInstanceAdmin(req)` — throws 403 if not instance admin

---

## Company Model

Companies are the top-level tenant container. They map to what will become "projects" in Nexus.

**Schema:** `packages/db/src/schema/companies.ts`
```
companies
  id uuid (PK)
  name text
  description text
  status text  (active | paused | archived)
  issuePrefix text UNIQUE  // e.g. "PAP" — used in URLs and identifiers
  issueCounter integer     // auto-increment for issue numbers
  budgetMonthlyCents integer
  requireBoardApprovalForNewAgents boolean
  brandColor text
```

**Company creation flow (`POST /api/companies`):**
1. Board asserts instance admin
2. `companyService.create(data)` inserts company
3. `accessService.ensureMembership(...)` adds creator as owner in `company_memberships`
4. If budget set, `budgetService.upsertPolicy(...)` creates budget policy
5. Activity logged

**Company service:** `server/src/services/companies.ts` — `companyService(db)` factory returning `list`, `getById`, `create`, `update`, `archive`, `remove`, `stats`.

**Company portability:** `server/src/services/company-portability.ts` — import/export bundles; CEO agents can run imports/exports on their own company.

**Company members:** `packages/db/src/schema/company_memberships.ts` — `principalType` (user|agent), `principalId`, `companyId`, `role` (owner|member), `status` (active|invited|etc.)

---

## Agent Model

Agents are AI workers belonging to a company. CEO is a special role with elevated permissions.

**Schema:** `packages/db/src/schema/agents.ts`
```
agents
  id uuid (PK)
  companyId uuid (FK companies)
  name text
  role text         // "ceo" | "general" | custom
  title text
  reportsTo uuid    // self-referential FK for org chart
  status text       // idle | running | paused | terminated | pending_approval
  adapterType text  // "claude_local" | "codex_local" | "cursor" | etc.
  adapterConfig jsonb   // adapter-specific config (model, instructionsFilePath, etc.)
  runtimeConfig jsonb   // session compaction policy, max concurrent runs
  budgetMonthlyCents integer
  permissions jsonb     // canCreateAgents, etc.
  lastHeartbeatAt timestamp
```

**Agent service:** `server/src/services/agents.ts` — `agentService(db)` factory. Key methods:
- `create`, `update`, `pause`, `resume`, `terminate`, `remove`
- `orgForCompany(companyId)` — builds hierarchical org tree from `reportsTo` links
- `getChainOfCommand(agentId)` — walks up `reportsTo` chain
- `createApiKey`, `listKeys`, `revokeKey`
- `rollbackConfigRevision` — restores an agent config from `agent_config_revisions`

**CEO agents:** Identified by `role === "ceo"`. Only CEO agents can:
- Update company branding (`PATCH /api/companies/:id/branding`)
- Manage company portability (import/export)
- The default first task in onboarding is assigned to the CEO agent

---

## Agent Heartbeat / Task Lifecycle

This is the core execution engine. An "agent run" is called a **heartbeat run**.

**Flow:**

```
1. Wake request queued
   → agent_wakeup_requests (status: queued)
   → agentWakeupRequest.source = "timer" | "assignment" | "on_demand" | "automation"

2. heartbeatService.runHeartbeat(agentId, options)
   → acquires per-agent start lock (Map<agentId, Promise>)
   → checks concurrent run limits (runtimeConfig.maxConcurrentRuns, default 1)
   → resolves execution workspace (project repo, task session, or agent home dir)
   → resolves session state (previous session params from agent_task_sessions)
   → checks session compaction (rotate if maxSessionRuns / maxRawInputTokens exceeded)
   → inserts heartbeat_runs row (status: queued → running)

3. getServerAdapter(adapterType).execute(context, meta)
   → adapter launches subprocess or HTTP call
   → streams stdout/stderr chunks back
   → heartbeatService writes live log chunks to run log store

4. On completion:
   → updates heartbeat_runs (status: done | error | cancelled)
   → persists session state to agent_task_sessions
   → writes cost event to cost_events
   → publishes live event to EventEmitter for SSE clients
   → updates issue status if issue was being worked

5. Wakeup request marked finished
```

**Key tables:**
- `heartbeat_runs` — one row per execution (`packages/db/src/schema/heartbeat_runs.ts`)
- `agent_wakeup_requests` — queued wakeup requests, coalesced by idempotency key
- `agent_task_sessions` — persisted session params per (agent, adapterType, taskKey)
- `agent_runtime_state` — current session ID for an agent

**heartbeatService:** `server/src/services/heartbeat.ts` (~1400 lines). The central orchestrator. Dependencies: adapter registry, workspace services, session codec, budget service, issue service, cost service.

---

## Adapter System

Adapters wrap specific AI tools (Claude Code, OpenAI Codex, Cursor, Gemini, etc.).

**Contract:** `ServerAdapterModule` from `packages/adapter-utils/`. Key methods:
- `execute(context, meta): Promise<AdapterExecutionResult>` — run the agent
- `testEnvironment(config): Promise<AdapterEnvironmentTestResult>` — check prerequisites
- `listSkills/syncSkills` — skill management
- `sessionCodec` — serialize/deserialize session state
- `onHireApproved` (optional) — notify adapter when a hired agent is approved

**Registry:** `server/src/adapters/registry.ts` — maps type string to `ServerAdapterModule`. Falls back to `processAdapter` for unknown types.

**Registered adapter types:**
- `claude_local` — Claude Code CLI subprocess (`packages/adapters/claude-local/`)
- `codex_local` — OpenAI Codex (`packages/adapters/codex-local/`)
- `opencode_local` — OpenCode (`packages/adapters/opencode-local/`)
- `pi_local` — Pi adapter (`packages/adapters/pi-local/`)
- `cursor` — Cursor IDE (`packages/adapters/cursor-local/`)
- `gemini_local` — Gemini CLI (`packages/adapters/gemini-local/`)
- `openclaw_gateway` — HTTP gateway adapter (`packages/adapters/openclaw-gateway/`)
- `process` — generic subprocess adapter (`server/src/adapters/process/`)
- `http` — generic HTTP adapter (`server/src/adapters/http/`)
- `hermes_local` — third-party Hermes adapter

**Process adapter:** `server/src/adapters/process/` — launches a configured command as a child process, streams stdout/stderr, parses structured JSON event chunks from the process.

---

## Plugin System

Plugins extend Paperclip with custom tools, UI slots, scheduled jobs, and webhooks.

**Architecture:**
- Each plugin runs as a separate worker process (isolated via `pluginWorkerManager`)
- Host (server) communicates with worker via JSON-RPC over stdin/stdout
- Host provides `PluginContext` services to the worker via the SDK

**Key services:**
- `plugin-lifecycle.ts` — state machine: `installed → ready → disabled/error/upgrade_pending → uninstalled`. Starting/stopping worker processes on transitions.
- `plugin-loader.ts` — loads plugin packages from disk, validates manifests
- `plugin-worker-manager.ts` — spawns/manages worker processes
- `plugin-job-coordinator.ts` + `plugin-job-scheduler.ts` — scheduled job execution
- `plugin-tool-dispatcher.ts` — routes tool invocations from agents to plugin workers
- `plugin-event-bus.ts` — dispatches Paperclip events (issue.created, agent.run.completed, etc.) to plugin workers
- `plugin-host-services.ts` — host-side service implementations exposed to workers via SDK
- `plugin-registry.ts` — DB CRUD for plugin records

**Worker SDK:** `packages/plugins/sdk/src/` — `@paperclipai/plugin-sdk`. Plugin authors import this to define tools, jobs, webhooks, UI slots, and interact with Paperclip state.

**Plugin manifest:** `PaperclipPluginManifestV1` from `@paperclipai/shared` — declares capabilities, tools, jobs, webhooks, UI slots.

**DB tables:** `plugins`, `plugin_config`, `plugin_state`, `plugin_jobs`, `plugin_logs`, `plugin_entities`, `plugin_webhooks`, `plugin_company_settings`

---

## Issue / Task Model

Issues are work items assigned to agents or users.

**Schema:** `packages/db/src/schema/issues.ts` — key fields:
```
issues
  id, companyId, projectId, goalId, parentId (self-ref)
  title, description, status (backlog|todo|in_progress|in_review|blocked|done|cancelled)
  priority (low|medium|high|urgent)
  assigneeAgentId, assigneeUserId
  checkoutRunId, executionRunId  // FK to heartbeat_runs
  executionWorkspaceId
  issueNumber, identifier (e.g. "PAP-42")
  originKind, originId  // "manual" | "routine_execution" | "heartbeat"
  requestDepth  // for sub-issues created by agents
```

**Identifier format:** `{issuePrefix}-{issueNumber}` — e.g., `PAP-42`. The prefix is unique per company.

**Sub-issues:** `parentId` self-reference allows agent-created sub-issues. `requestDepth` tracks nesting.

**Execution workspace:** Each issue can have an `executionWorkspaceId` pointing to a specific git worktree or container workspace. Managed by `execution-workspace-policy.ts`.

---

## Project Model

Projects group issues and have an optional lead agent and git repo.

**Schema:** `packages/db/src/schema/projects.ts`
```
projects
  id, companyId, goalId
  name, description, status (backlog|active|paused|done|archived)
  leadAgentId
  targetDate, color
  executionWorkspacePolicy jsonb  // repo URL, worktree settings
```

Projects will become the renamed entity in Nexus (replacing "companies" as the primary project container).

---

## Goal Model

Goals are objectives that group projects and issues.

**Schema:** `packages/db/src/schema/goals.ts` — `id, companyId, title, description, status, parentId` (hierarchical)

---

## Onboarding Flow

### CLI Onboarding (`paperclipai onboard`)

Interactive wizard in `cli/src/commands/onboard.ts` driven by `@clack/prompts`:

1. **Mode selection** — "quickstart" (embedded Postgres, sane defaults) or "advanced" (custom DB, S3, auth)
2. **Database** — embedded Postgres (auto-configured) or external `DATABASE_URL`
3. **LLM** — configure default LLM provider/API key
4. **Server** — port, host, deployment mode (`local_trusted` vs `authenticated`), public URL
5. **Storage** — local filesystem or S3
6. **Secrets** — local encrypted file or external secret manager
7. Writes config to `~/.paperclip/config.yaml` (or `--config` path)
8. Optionally runs `auth bootstrap-ceo` to generate first admin invite

### UI Onboarding Wizard (`OnboardingWizard.tsx`)

4-step dialog at `ui/src/components/OnboardingWizard.tsx`:

- **Step 1 — Company**: enter company name + high-level goal. Creates company via `POST /api/companies`, creates a Goal record.
- **Step 2 — Agent**: name the CEO agent, select adapter type (claude_local, codex_local, cursor, etc.), configure model. Tests adapter environment. Creates agent via `POST /api/companies/:id/agents`.
- **Step 3 — First Task**: pre-filled task description (`"Hire a founding engineer, write a hiring plan, break the roadmap into concrete tasks"`). Creates a project + issue assigned to the CEO agent.
- **Step 4 — Launch**: confirms entities created, navigates to issue detail. Shows run transcript.

**Trigger:** `openOnboarding()` from `DialogContext`. Also triggered by route `/onboarding` or by navigating to company-prefixed routes when no companies exist.

**Default task:** The CEO agent's first issue is "Hire your first engineer and create a hiring plan" with the task description guiding it to hire a founding engineer and begin delegating work.

---

## Real-time Updates

**Server:** `server/src/services/live-events.ts` — in-process `EventEmitter`. Services call `publishLiveEvent({ companyId, type, payload })` after mutations. SSE endpoint in `server/src/realtime/live-events-ws.ts` streams events per company.

**Client:** `ui/src/context/LiveUpdatesProvider.tsx` — subscribes to SSE, calls `queryClient.invalidateQueries(...)` to refresh TanStack Query caches on relevant events. Also shows toast notifications for agent run completions and issue status changes.

---

## CLI Commands

Entry point: `cli/src/index.ts` — Commander.js program named `paperclipai`.

**Setup commands:**
- `onboard` — interactive first-run setup wizard
- `run` — onboard + doctor + start server
- `configure` — update config sections (llm, database, logging, server, storage, secrets)
- `doctor` — diagnostic checks + optional auto-repair
- `env` — print deployment environment variables
- `allowed-hostname` — add hostname to allowlist
- `db:backup` — one-off database backup

**Runtime commands:**
- `heartbeat run --agent-id <id>` — trigger one agent heartbeat and stream logs

**Client commands (interact with a running Paperclip instance):**
- `company list|create|get` — CRUD on companies
- `issue list|create|get|update` — CRUD on issues
- `agent list|create|get|update` — CRUD on agents
- `approval list|get|approve|reject` — manage approvals
- `activity list` — view activity log
- `dashboard` — display dashboard stats
- `plugin list|install|enable|disable|uninstall` — plugin management
- `auth login|logout|whoami|bootstrap-ceo` — auth operations
- `worktree ...` — git worktree management for execution workspaces

Config/context for client commands loaded from `~/.paperclip/config.yaml` or `--config` flag.

---

## Routine Model

Routines are scheduled triggers that create issues on a cron-like cadence.

**Schema:** `packages/db/src/schema/routines.ts`

Routines fire at configured intervals, create issues with the configured title/description, and assign them to the specified agent. `server/src/services/cron.ts` drives the scheduling.

---

## Execution Workspace Model

Execution workspaces provide isolated git checkouts per project or issue.

**Schema:** `packages/db/src/schema/execution_workspaces.ts`, `project_workspaces.ts`

Managed by `server/src/services/execution-workspace-policy.ts` and `workspace-runtime.ts`. On heartbeat start, `heartbeatService` calls `realizeExecutionWorkspace(...)` to ensure the workspace directory + git clone exist before passing `cwd` to the adapter.

---

## Key Data Model Relationships

```
Instance
  └── companies (1:N)
        ├── agents (1:N) — role: ceo | general | custom
        │     └── reportsTo (self-ref tree)
        ├── projects (1:N)
        │     └── issues (1:N)
        ├── goals (1:N, hierarchical)
        ├── routines (1:N)
        ├── heartbeat_runs (1:N, via agents)
        │     └── heartbeat_run_events (1:N)
        ├── agent_wakeup_requests (1:N)
        ├── agent_task_sessions (1:N)
        ├── approvals (1:N)
        ├── cost_events (1:N)
        ├── company_secrets (1:N)
        └── plugins (M:N via plugin_company_settings)
```

---

## Entry Points

**Server:**
- `server/src/index.ts` — boots Postgres, runs migrations, calls `createApp(db, opts)`, starts HTTP listener
- `server/src/app.ts` — `createApp(db, opts)` — wires Express middleware, mounts all routes, initializes plugin system, returns `app`

**UI:**
- `ui/src/main.tsx` — React root, wraps `<App />` in TanStack Query provider, `CompanyProvider`, etc.
- `ui/src/App.tsx` — top-level router with `CloudAccessGate`, company-prefixed routes, `<OnboardingWizard />`

**CLI:**
- `cli/src/index.ts` — Commander program, registers all commands, calls `program.parseAsync()`

---

## Error Handling

**Server:** `server/src/errors.ts` exports `notFound`, `conflict`, `forbidden`, `unprocessable` helpers that throw typed errors. `server/src/middleware/error-handler.ts` catches them and returns structured JSON: `{ error: string }`.

**UI:** `ui/src/api/client.ts` — `ApiError` class with `status` and `body`. Components check `error instanceof ApiError` and display `error.message`.

---

## Deployment Modes

Controlled by `PAPERCLIP_DEPLOYMENT_MODE`:
- `local_trusted` — single-user local, no auth required. All requests automatically get `isInstanceAdmin: true`.
- `authenticated` — multi-user, requires BetterAuth session or API key. Board API keys stored in `board_api_keys`.

`PAPERCLIP_DEPLOYMENT_EXPOSURE`:
- `public` — open to external traffic
- `private` — private-hostname-guard enforces allowlist

---

*Architecture analysis: 2026-03-30*