mikkel/nexus
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
nexus/.planning/phases/45-content-as-skills/45-RESEARCH.md

22 KiB
Raw Blame History

Phase 45: Content as Skills - Research

Researched: 2026-04-04 Domain: Nexus skill registry — local SKILL.md authoring, source seeding, group membership, generalist agent preloading Confidence: HIGH

Summary

Phase 45 bridges the content generation work of Phases 41-44 with the skill registry system already in place. Every content type (diagram, icon, theme, wallpaper, social, convert, pdf-document, brand-kit, presentation) must become an installable Markdown skill, the built-in "Creative" group must contain all of them, and the Generalist agent must ship pre-loaded with that group.

The skill registry infrastructure is complete. The registry stores skills in a libSQL database under ~/.paperclip/instances/default/skills/registry.db. Skills are fetched from remote GitHub sources (anthropic-marketplace and github-tree types) or from native agent runtimes (Hermes). There is currently no local-filesystem source type. Phase 45 must add one, seed it on server startup, and wire the Creative group to the new skills.

The generalist agent preloading mechanism already exists: index.ts seeds pendingSkillGroups: ["Creative"] in agent metadata at creation time, and a fire-and-forget startup reconciler assigns the group (using assignGroup) to every agent that has that metadata flag. Phase 45 needs the Creative group to have members before the reconciler runs — which means the skill SKILL.md files and their registry entries must be in place before any agent is created.

Primary recommendation: Add a "local-nexus-content" source type to the skill registry fetcher. On server startup, register nine content SKILL.md files from server/src/skills/content/ into the registry DB and add all of them to the builtin/creative group.

<user_constraints>

User Constraints (from CONTEXT.md)

Locked Decisions

All implementation choices are at Claude's discretion — discuss phase was skipped per user setting.

Claude's Discretion

All implementation choices are at Claude's discretion. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions.

Deferred Ideas (OUT OF SCOPE)

None. </user_constraints>

<phase_requirements>

Phase Requirements

ID Description Research Support
SKILL-01 Each content type is implemented as an installable Nexus skill Nine SKILL.md files authored; registered in skill registry DB via local-nexus-content source type; installable through existing svc.install() path
SKILL-02 Generalist agent is pre-loaded with a "Creative" skill group builtin/creative group already exists in DB seed; group members populated on startup; pendingSkillGroups reconciler already assigns the group to Generalist agents
SKILL-03 Users can add or remove content type skills through the Skill Aggregator Content skills appear in the existing SkillBrowser UI once registered; install/uninstall routes already work for any registered skill
</phase_requirements>

Standard Stack

Core

Library Version Purpose Why Standard
libSQL / drizzle-orm Already installed Skill registry persistence All skill data lives in this SQLite DB — no new dep needed
node:fs/promises Node built-in Read SKILL.md files from disk Used by existing skill cache logic
vitest Already installed Unit tests Project standard in server/

Supporting

No new packages required. The full content generation stack (jobType dispatch, renderers, SSE) is already in place from Phases 40-44.

Installation: None required.

Architecture Patterns

Recommended Project Structure

server/src/
├── skills/
│   └── content/
│       ├── diagram.SKILL.md
│       ├── icon-set.SKILL.md
│       ├── theme-palette.SKILL.md
│       ├── wallpaper.SKILL.md
│       ├── social-post.SKILL.md
│       ├── convert.SKILL.md
│       ├── pdf-document.SKILL.md
│       ├── brand-kit.SKILL.md
│       └── presentation.SKILL.md
├── services/
│   └── skill-registry-fetcher.ts   ← add local-nexus-content source type
│   └── skill-registry-db.ts        ← seed Creative group members on init
└── index.ts                         ← call seedContentSkills on startup

Pattern 1: Local-Filesystem Source Type

What: A new "local-nexus-content" entry in SkillSourceConfig union and a fetchLocalNexusContent() function that reads SKILL.md files from server/src/skills/content/ and upserts them into the registry DB.

When to use: Server startup — called once after getSkillRegistryDb() initialises.

Example:

// In skill-registry-fetcher.ts — extend the union type
export type SkillSourceConfig =
  | { id: string; type: "anthropic-marketplace"; owner: string; repo: string; ref: string; label: string }
  | { id: string; type: "github-tree"; owner: string; repo: string; ref: string; label: string }
  | { id: string; type: "local-nexus-content"; dir: string; label: string };

// New handler called by fetchAllSources when type === "local-nexus-content"
async function fetchLocalNexusContent(
  source: Extract<SkillSourceConfig, { type: "local-nexus-content" }>,
  db: SkillRegistryDb,
): Promise<number> {
  const entries = await readdir(source.dir, { withFileTypes: true });
  let fetched = 0;
  for (const entry of entries) {
    if (!entry.name.endsWith(".SKILL.md")) continue;
    const slug = entry.name.replace(/\.SKILL\.md$/, "");
    const skillId = `${source.id}/${slug}`;
    const skillMdContent = await readFile(path.join(source.dir, entry.name), "utf-8");
    const { name, description } = parseSkillFrontmatter(skillMdContent);
    // Use a content hash as the version SHA (deterministic, no network needed)
    const sha = crypto.createHash("sha1").update(skillMdContent).digest("hex");
    await upsertSkill(db, { skillId, sourceId: source.id, name: name ?? slug, description, sourceUrl: "" });
    await cacheSkillVersion(db, { skillId, sha, skillMdContent, skillMdUrl: "" });
    await upsertCommunityRatingsStub(db, skillId, source.id);
    fetched++;
  }
  return fetched;
}

BUILT_IN_SOURCES addition:

export const BUILT_IN_SOURCES: SkillSourceConfig[] = [
  // … existing remote sources …
  {
    id: "nexus-content",
    type: "local-nexus-content",
    dir: path.join(path.dirname(fileURLToPath(import.meta.url)), "..", "skills", "content"),
    label: "Nexus Content Tools",
  },
];

Pattern 2: Creative Group Membership Seeding

What: After the local skills are registered, add them as members of the builtin/creative group. This should happen inside getSkillRegistryDb() (or a startup helper called right after) so the group has members before the pendingSkillGroups reconciler runs.

When to use: Server startup, idempotent (uses INSERT OR IGNORE).

// In skill-registry-db.ts — extend seedBuiltinGroups or add seedCreativeGroupMembers
const NEXUS_CONTENT_SKILL_IDS = [
  "nexus-content/diagram",
  "nexus-content/icon-set",
  "nexus-content/theme-palette",
  "nexus-content/wallpaper",
  "nexus-content/social-post",
  "nexus-content/convert",
  "nexus-content/pdf-document",
  "nexus-content/brand-kit",
  "nexus-content/presentation",
] as const;

async function seedCreativeGroupMembers(client: LibSQLClient): Promise<void> {
  const now = Date.now();
  for (const skillId of NEXUS_CONTENT_SKILL_IDS) {
    await client.execute({
      sql: `INSERT OR IGNORE INTO skill_group_members (group_id, skill_id, added_at) VALUES (?, ?, ?)`,
      args: ["builtin/creative", skillId, now],
    });
  }
}

Pattern 3: SKILL.md Authoring Convention

Each content skill SKILL.md must follow the same frontmatter convention as existing skills:

---
name: diagram
description: >
  Generate diagrams from a natural language description. Produces Mermaid
  syntax rendered to SVG/PNG. Supports architecture, flowchart, ERD,
  sequence, and mind-map types. Use when a user asks to visualise a
  system, process, or data structure as a diagram.
---

# Diagram Generation

Generates diagrams from a text description via the Nexus content API.

## Usage

Submit a `POST /api/content-jobs` with:
- `jobType: "diagram"`
- `input.description` — natural language description of the diagram
- `input.type` (optional) — `"flowchart"`, `"architecture"`, `"erd"`, `"sequence"`, or `"mindmap"`

The job returns a 202 with a `jobId`. Poll `GET /api/content-jobs/:jobId` or subscribe
to SSE `content_job.done` events to retrieve the resulting asset URL.

## Output

Returns an SVG file (also available as PNG via the export button in the UI).

Startup Wiring

In server/src/index.ts, the existing startup block already calls getSkillRegistryDb() as fire-and-forget. Extend that block to also call fetchAllSources() with BUILT_IN_SOURCES immediately after DB init so content skills are seeded on every cold start:

// [nexus] Initialize skill registry and seed content skills
void (async () => {
  try {
    const { getSkillRegistryDb } = await import("./services/skill-registry-db.js");
    await getSkillRegistryDb();          // creates tables + builtin groups
    const { skillRegistryService } = await import("./services/skill-registry.js");
    await skillRegistryService().fetchAll();  // seeds local-nexus-content skills
    logger.info("skill registry database initialized and content skills seeded");
  } catch (err) {
    logger.error({ err }, "skill registry init failed");
  }
})();

Anti-Patterns to Avoid

  • Seeding Creative group members before the skills table rows exist: skill_group_members.skill_id has no FK constraint, so INSERT succeeds even if the skill row is absent — but resolveEffectiveSkills then returns orphaned IDs. Always seed skill rows first, group members second.
  • Calling fetchAll() before getSkillRegistryDb(): DB tables must exist before the local fetcher tries to upsert.
  • Using mutable remote SHA for local file versioning: Local files don't have a git SHA. Use a SHA-1 of the file content instead — deterministic and cheap.
  • Blocking HTTP on fetchAll() at startup: Keep as fire-and-forget; the external github-tree fetches can be slow. Content skills (local path) are synchronous but the remote sources may timeout.

Don't Hand-Roll

Problem Don't Build Use Instead Why
Skill install/uninstall Custom file copy logic skillRegistryService().install() / uninstall() Already handles version tracking, cache dirs, agentSkills table
Group assignment Direct INSERT into agentSkillGroups skillGroupService().assignGroup() Handles effective skill resolution, per-skill install, dedup
Skill Browser display New UI component Existing SkillBrowser page at /skills Already renders all registered skills with install/remove actions
Group membership seeding New service INSERT OR IGNORE INTO skill_group_members on DB init Simpler than adding a new route; idempotent

Common Pitfalls

Pitfall 1: Creative group populated but skills not yet in registry

What goes wrong: pendingSkillGroups reconciler runs at startup and calls assignGroup("builtin/creative", agentId, skillsDir). resolveEffectiveSkills returns the 9 skill IDs, but svc.install() fails with "Skill not found" because local skills haven't been fetched yet. Why it happens: The skill-registry init and the pendingSkillGroups reconciler are both fire-and-forget; their order is non-deterministic. How to avoid: Seed creative group members only after fetchAll() completes (not at DB init time). Alternatively, sequence the two startup blocks: skill init → fetchAll → then let the reconciler run. Warning signs: skipped array non-empty in assignGroup result; Generalist agent has group assigned but no SKILL.md files on disk.

Pitfall 2: Local source dir resolution broken in production

What goes wrong: path.join(path.dirname(fileURLToPath(import.meta.url)), "..", "skills", "content") resolves to server/src/skills/content in dev but points to a non-existent path in a compiled/packaged build. Why it happens: import.meta.url refers to the compiled output location. How to avoid: In tsconfig.json, include server/src/skills/content/*.SKILL.md in the assets copied to dist, OR read the source dir via process.cwd() with a known repo-relative path. Check that the resolved dir exists before iterating; log a clear error if absent.

Pitfall 3: SKILL.md frontmatter parse returns undefined name

What goes wrong: parseSkillFrontmatter fails silently on a multi-line YAML description block (description: >\n line1\n line2). The name field parses correctly but description is truncated to > literal. Why it happens: parseSkillFrontmatter uses a single-line regex (^description:\s*(.+)$) — it does not handle YAML block scalars. How to avoid: For the name field (used for registry display), keep it on a single line. For description use a single-line string or expand the frontmatter parser to handle folded scalars. Alternatively, accept the > literal as description — it won't break anything.

Pitfall 4: Agent SKILL.md dir wrong for Generalist agent

What goes wrong: assignGroup copies skills to agentSkillsDir but the Generalist agent's adapter is claude_local — the skills dir is resolved by resolveAdapterSkillConfig("claude_local").skillDir. If the agent's workspace does not exist yet (new onboard), cp fails. Why it happens: The workspace dir is created lazily when the agent first runs. How to avoid: In assignGroup, mkdir(agentSkillsDir, { recursive: true }) before attempting the copy. Inspect the existing code path — if this mkdir is already there, no action needed.

Pitfall 5: Duplicate registration on restarts

What goes wrong: Every server restart calls fetchAll(), which re-registers all 9 local skills. Without idempotency guards, this creates duplicate skill_versions and skill_files rows. Why it happens: The content hash SHA is deterministic, so versionExists() returns true and cacheSkillVersion is skipped. But upsertSkill uses onConflictDoUpdate — the name/description are updated but no duplicate is created. How to avoid: The existing versionExists() check handles this. Confirm the content-hash versionId format matches: ${skillId}@${sha}.

Code Examples

Registering skills from disk (local-nexus-content handler)

// Source: server/src/services/skill-registry-fetcher.ts (new function)
import { readdir, readFile } from "node:fs/promises";
import crypto from "node:crypto";

async function fetchLocalNexusContent(
  source: Extract<SkillSourceConfig, { type: "local-nexus-content" }>,
  db: SkillRegistryDb,
): Promise<number> {
  let entries: Awaited<ReturnType<typeof readdir>>;
  try {
    entries = await readdir(source.dir, { withFileTypes: true });
  } catch {
    // Directory doesn't exist (e.g. missing after a build step) — log and skip
    return 0;
  }
  let fetched = 0;
  for (const entry of entries) {
    if (!entry.isFile() || !entry.name.endsWith(".SKILL.md")) continue;
    const slug = entry.name.replace(/\.SKILL\.md$/, "");
    const skillId = `${source.id}/${slug}`;
    const content = await readFile(path.join(source.dir, entry.name), "utf-8");
    const sha = crypto.createHash("sha1").update(content).digest("hex");
    const { name, description } = parseSkillFrontmatter(content);
    await upsertSkill(db, {
      skillId, sourceId: source.id,
      name: name ?? slug, description,
      sourceUrl: "",
    });
    await cacheSkillVersion(db, { skillId, sha, skillMdContent: content, skillMdUrl: "" });
    await upsertCommunityRatingsStub(db, skillId, source.id);
    fetched++;
  }
  return fetched;
}

Seeding Creative group membership after fetchAll

// In skill-registry-db.ts — called after seedBuiltinGroups
async function seedCreativeGroupMembers(client: LibSQLClient): Promise<void> {
  const now = Date.now();
  const SKILLS = [
    "nexus-content/diagram", "nexus-content/icon-set", "nexus-content/theme-palette",
    "nexus-content/wallpaper", "nexus-content/social-post", "nexus-content/convert",
    "nexus-content/pdf-document", "nexus-content/brand-kit", "nexus-content/presentation",
  ];
  for (const skillId of SKILLS) {
    await client.execute({
      sql: `INSERT OR IGNORE INTO skill_group_members (group_id, skill_id, added_at) VALUES (?, ?, ?)`,
      args: ["builtin/creative", skillId, now],
    });
  }
}

Ensuring Creative is assigned to a new Generalist agent

// The existing path in index.ts (already present, no change needed):
await agentSvc.create(company.id, {
  name: "Generalist",
  role: "general",
  adapterType: "claude_local",
  adapterConfig: {},
  runtimeConfig: {},
  metadata: { pendingSkillGroups: ["Creative"], backfilled: true },
});
// The reconciler at server startup calls svc.assignGroup("builtin/creative", agentId, skillsDir)
// which installs all Creative group members to the agent's .claude/skills/ directory.

State of the Art

Old Approach Current Approach When Changed Impact
Skills only from remote GitHub repos Skills can come from local filesystem too (new local-nexus-content type) Phase 45 Nexus-specific content skills don't need a public GitHub repo
Creative group is empty (seed only creates the group row) Creative group has 9 content skill members after server init Phase 45 Generalist agents automatically get all content tools on first run

Open Questions

  1. Should cacheSkillVersion write files to a tmpdir for local skills?

    • What we know: cacheSkillVersion writes SKILL.md to ~/.paperclip/instances/default/skills/cache/nexus-content/<slug>/<sha>/SKILL.md, then svc.install() copies from that cache dir to the agent's skills dir.
    • What's unclear: For local skills, caching to disk before installing is redundant — the source file is already on disk.
    • Recommendation: Keep the cache write for consistency (install always reads from cache); the file is tiny. If performance matters, skip caching and have install copy the source file directly.

  2. Where exactly is the server/src/skills/content/ dir resolved in compiled builds?

    • What we know: The server compiles TypeScript but does not currently bundle assets — source files are referenced at runtime via import.meta.url.
    • What's unclear: Whether .SKILL.md files are included in the dist output.
    • Recommendation: Add server/src/skills/content/ to the list of directories copied during build (check the server's build script), or use process.env.SKILL_CONTENT_DIR as an override for production.

  3. Should the Skill Aggregator UI show a "Creative" filter tab?

    • What we know: SkillBrowser.tsx filters by adapter compatibility; the "Groups" tab shows group membership via skillGroupsApi. Content skills installed via the registry appear in the installed tab.
    • What's unclear: Whether the browse experience needs a dedicated "Creative" section or if the existing group-based filtering is sufficient.
    • Recommendation: No UI changes needed for SKILL-03. The existing SkillBrowser already supports install/remove for any registered skill. Adding a dedicated Creative tab is a future enhancement.

Environment Availability

Step 2.6: SKIPPED (no new external dependencies — local filesystem only)

Validation Architecture

Test Framework

Property Value
Framework Vitest (node environment)
Config file server/vitest.config.ts
Quick run command cd /opt/nexus/server && npx vitest run --reporter=verbose src/__tests__/skill-registry-content-skills.test.ts
Full suite command cd /opt/nexus/server && npx vitest run --reporter=verbose

Phase Requirements → Test Map

Req ID Behavior Test Type Automated Command File Exists?
SKILL-01 9 SKILL.md files are seeded into registry DB via local-nexus-content source unit npx vitest run src/__tests__/skill-registry-content-skills.test.ts -t "seeds content skills" ❌ Wave 0
SKILL-01 Each skill is installable to an agent skills dir unit npx vitest run src/__tests__/skill-registry-content-skills.test.ts -t "installs content skill" ❌ Wave 0
SKILL-02 Creative group has 9 members after DB init unit npx vitest run src/__tests__/skill-registry-content-skills.test.ts -t "Creative group members" ❌ Wave 0
SKILL-03 Content skills appear in registry list unit npx vitest run src/__tests__/skill-registry-content-skills.test.ts -t "list includes content skills" ❌ Wave 0

Sampling Rate

  • Per task commit: cd /opt/nexus/server && npx vitest run src/__tests__/skill-registry-content-skills.test.ts
  • Per wave merge: cd /opt/nexus/server && npx vitest run
  • Phase gate: Full suite green before /gsd:verify-work

Wave 0 Gaps

  • server/src/__tests__/skill-registry-content-skills.test.ts — covers SKILL-01, SKILL-02, SKILL-03
  • server/src/skills/content/*.SKILL.md — 9 skill files (authored, not test infra)

Sources

Primary (HIGH confidence)

  • Direct source code inspection — server/src/services/skill-registry-fetcher.ts, skill-registry-db.ts, skill-registry-groups.ts, skill-registry.ts
  • Direct source code inspection — server/src/index.ts lines 256-279 (ensureGeneralistAgents), 628-658 (pendingSkillGroups reconciler)
  • Direct source code inspection — server/src/adapters/registry.ts, server/src/routes/skill-registry.ts, server/src/routes/skill-registry-groups.ts

Secondary (MEDIUM confidence)

  • Existing test patterns from server/src/__tests__/skill-registry-install.test.ts and hermes-dual-source.test.ts — confirms vitest + real temp dirs as test pattern

Metadata

Confidence breakdown:

  • Standard stack: HIGH — all libraries already installed; no new deps
  • Architecture: HIGH — local source type follows identical pattern to existing github-tree handler
  • Pitfalls: HIGH — race conditions and dir resolution issues verified against actual startup code

Research date: 2026-04-04 Valid until: 2026-05-04 (stable codebase)