nexus/doc/plans/workspace-strategy-and-git-worktrees.md

# Workspace Strategy and Git Worktrees

## Context

`PAP-447` asks how Paperclip should support worktree-driven coding workflows for local coding agents without turning that into a universal product requirement.

The motivating use case is strong:

- when an issue starts, a local coding agent may want its own isolated checkout
- the agent may need a dedicated branch and a predictable path to push later
- the agent may need to start one or more long-lived workspace runtime services, discover reachable ports or URLs, and report them back into the issue
- the workflow should reuse the same Paperclip instance and embedded database instead of creating a blank environment
- local agent auth should remain low-friction

At the same time, we do not want to hard-code "every agent uses git worktrees" into Paperclip:

- some operators use Paperclip to manage Paperclip and want worktrees heavily
- other operators will not want worktrees at all
- not every adapter runs in a local git repository
- not every adapter runs on the same machine as Paperclip
- Claude and Codex expose different built-in affordances, so Paperclip should not overfit to one tool

## Core Product Decision

Paperclip should model **execution workspaces**, not **worktrees**.

More specifically:

- the durable anchor is the **project workspace** or repo checkout
- an issue may derive a temporary **execution workspace** from that project workspace
- one implementation of an execution workspace is a **git worktree**
- adapters decide whether and how to use that derived workspace

This keeps the abstraction portable:

- `project workspace` is the repo/project-level concept
- `execution workspace` is the runtime checkout/cwd for a run
- `git worktree` is one strategy for creating that execution workspace
- `workspace runtime services` are long-lived processes or previews attached to that workspace

This also keeps the abstraction valid for non-local adapters:

- local adapters may receive a real filesystem cwd produced by Paperclip
- remote or cloud adapters may receive the same execution intent in structured form and realize it inside their own environment
- Paperclip should not assume that every adapter can see or use a host filesystem path directly

## Answer to the Main Framing Questions

### Are worktrees for agents or for repos/projects?

They should be treated as **repo/project-scoped infrastructure**, not agent identity.

The stable object is the project workspace. Agents come and go, ownership changes, and the same issue may be reassigned. A git worktree is a derived checkout of a repo workspace for a specific task or issue. The agent uses it, but should not own the abstraction.

If Paperclip makes worktrees agent-first, it will blur:

- agent home directories
- project repo roots
- issue-specific branches/checkouts

That makes reuse, reassignment, cleanup, and UI visibility harder.

### How do we preserve optionality?

By making execution workspace strategy **opt-in at the adapter/config layer**, not a global invariant.

Defaults should remain:

- existing project workspace resolution
- existing task-session resume
- existing agent-home fallback

Then local coding agents can opt into a strategy like `git_worktree`.

### How do we make this portable and adapter-appropriate?

By splitting responsibilities:

- Paperclip core resolves and records execution workspace state
- a shared local runtime helper can implement git-based checkout strategies
- each adapter launches its tool inside the resolved cwd using adapter-specific flags

This avoids forcing a Claude-shaped or Codex-shaped model onto all adapters.

It also avoids forcing a host-filesystem model onto cloud agents. A cloud adapter may interpret the same requested strategy as:

- create a fresh sandbox checkout from repo + ref
- create an isolated branch/workspace inside the provider's remote environment
- ignore local-only fields like host cwd while still honoring branch/ref/isolation intent

## What the Current Code Already Supports

Paperclip already has the right foundation for a project-first model.

### Project workspace is already first-class

- `project_workspaces` already exists in `packages/db/src/schema/project_workspaces.ts`
- the shared `ProjectWorkspace` type already includes `cwd`, `repoUrl`, and `repoRef` in `packages/shared/src/types/project.ts`
- docs already state that agents use the project's primary workspace for project-scoped tasks in `docs/api/goals-and-projects.md`

### Heartbeat already resolves workspace in the right order

Current run resolution already prefers:

1. project workspace
2. prior task session cwd
3. agent-home fallback

See `server/src/services/heartbeat.ts`.

### Session resume is already cwd-aware

Both local coding adapters treat session continuity as cwd-bound:

- Codex: `packages/adapters/codex-local/src/server/execute.ts`
- Claude: `packages/adapters/claude-local/src/server/execute.ts`

That means the clean insertion point is before adapter execution: resolve the final execution cwd first, then let the adapter run normally.

### Server-spawned local auth already exists

For server-spawned local adapters, Paperclip already injects a short-lived local JWT:

- JWT creation: `server/src/services/heartbeat.ts`
- adapter env injection:
  - `packages/adapters/codex-local/src/server/execute.ts`
  - `packages/adapters/claude-local/src/server/execute.ts`

The manual-local bootstrap path is still weaker in authenticated mode, but that is a related auth ergonomics problem, not a reason to make worktrees a core invariant.

## Tooling Observations from Vendor Docs

The linked tool docs support a project-first, adapter-specific launch model.

### Codex

- Codex app has a native worktree concept for parallel tasks in git repos
- Codex CLI documents running in a chosen working directory and resuming sessions from the current working directory
- Codex CLI does not present a single first-class portable CLI worktree abstraction that Paperclip should mirror directly

Implication:

- for `codex_local`, Paperclip should usually create/select the checkout itself and then launch Codex inside that cwd

### Claude

- Claude documents explicit git worktree workflows for parallel sessions
- Claude CLI supports `--worktree` / `-w`
- Claude sessions also remain tied to directory context

Implication:

- `claude_local` can optionally use native `--worktree`
- but Paperclip should still treat that as an adapter optimization, not the canonical cross-adapter model

## Local vs Remote Adapters

This plan must explicitly account for the fact that many adapters are not local.

Examples:

- local CLI adapters such as `codex_local` and `claude_local`
- cloud-hosted coding agents such as Cursor cloud agents
- future hosted Codex or Claude agent modes
- custom sandbox adapters built on E2B, Cloudflare, or similar environments

These adapters do not all share the same capabilities:

- some can use host git worktrees directly
- some can clone a repo and create branches remotely
- some may expose a virtual workspace concept with no direct git worktree equivalent
- some may not allow persistent filesystem state at all

Because of that, Paperclip should separate:

- **execution workspace intent**: what isolation/branch/repo behavior we want
- **adapter realization**: how a specific adapter implements that behavior

### Execution workspace intent

Paperclip should be able to express intentions such as:

- use the project's primary workspace directly
- create an isolated issue-scoped checkout
- base work on a given repo ref
- derive a branch name from the issue
- expose one or more reachable preview or service URLs if runtime services are started

### Adapter realization

Adapters should be free to map that intent into their own environment:

- local adapter: create a host git worktree and run in that cwd
- cloud sandbox adapter: clone repo into a sandbox, create a branch there, and return sandbox metadata
- hosted remote coding agent: call provider APIs that create a remote workspace/thread bound to the requested branch/ref

The important constraint is that the adapter reports back the realized execution workspace metadata in a normalized shape, even if the underlying implementation is not a git worktree.

## Proposed Model

Use three layers:

1. `project workspace`
2. `execution workspace`
3. `workspace runtime services`
4. `adapter session`

### 1. Project workspace

Long-lived repo anchor.

Examples:

- `./paperclip`
- repo URL and base ref
- primary checkout for a project

### 2. Execution workspace

Derived runtime checkout for a specific issue/run.

Examples:

- direct use of the project primary workspace
- git worktree derived from the project workspace
- remote sandbox checkout derived from repo URL + ref
- custom checkout produced by an adapter-specific script

### 3. Adapter session

Long-lived or semi-long-lived processes associated with a workspace.

Examples:

- local web server
- background worker
- sandbox preview URL
- test watcher
- tunnel process

These are not specific to Paperclip. They are a common property of working in a dev workspace, whether local or remote.

### 4. Adapter session

Claude/Codex conversation continuity and runtime state, which remains cwd-aware and should follow the execution workspace rather than define it.

## Recommended Configuration Surface

Introduce a generic execution workspace strategy in adapter config.

Example shape:

```json
{
  "workspaceStrategy": {
    "type": "project_primary"
  }
}
```

Or:

```json
{
  "workspaceStrategy": {
    "type": "git_worktree",
    "baseRef": "origin/main",
    "branchTemplate": "{{issue.identifier}}-{{slug}}",
    "worktreeParentDir": ".paperclip/instances/default/worktrees/projects/{{project.id}}",
    "cleanupPolicy": "on_merged",
    "startDevServer": true,
    "devServerCommand": "pnpm dev",
    "devServerReadyUrlTemplate": "http://127.0.0.1:{{port}}/api/health"
  }
}
```

Remote adapters may instead use shapes like:

```json
{
  "workspaceStrategy": {
    "type": "isolated_checkout",
    "provider": "adapter_managed",
    "baseRef": "origin/main",
    "branchTemplate": "{{issue.identifier}}-{{slug}}"
  }
}
```

The important point is that `git_worktree` is a strategy value for adapters that can use it, not the universal contract.

### Workspace runtime services

Do not model this as a Paperclip-specific `devServer` flag.

Instead, model it as a generic list of workspace-attached runtime services.

Example shape:

```json
{
  "workspaceRuntime": {
    "services": [
      {
        "name": "web",
        "description": "Primary app server for this workspace",
        "command": "pnpm dev",
        "cwd": ".",
        "env": {
          "DATABASE_URL": "${workspace.env.DATABASE_URL}"
        },
        "port": {
          "type": "auto"
        },
        "readiness": {
          "type": "http",
          "urlTemplate": "http://127.0.0.1:${port}/api/health"
        },
        "expose": {
          "type": "url",
          "urlTemplate": "http://127.0.0.1:${port}"
        },
        "reuseScope": "project_workspace",
        "lifecycle": "shared",
        "stopPolicy": {
          "type": "idle_timeout",
          "idleSeconds": 1800
        }
      }
    ]
  }
}
```

This contract is intentionally generic:

- `command` can start any workspace-attached process, not just a web server
- database reuse is handled through env/config injection, not a product-specific special case
- local and remote adapters can realize the same service intent differently

### Service intent vs service realization

Paperclip should distinguish between:

- **service intent**: what kind of companion runtime the workspace wants
- **service realization**: how a local or remote adapter actually starts and exposes it

Examples:

- local adapter:
  - starts `pnpm dev`
  - allocates a free host port
  - health-checks a localhost URL
  - reports `{ pid, port, url }`
- cloud sandbox adapter:
  - starts a preview process inside the sandbox
  - receives a provider preview URL
  - reports `{ sandboxId, previewUrl }`
- hosted remote coding agent:
  - may ask the provider to create a preview environment
  - reports provider-native workspace/service metadata

Paperclip should normalize the reported metadata without requiring every adapter to look like a host-local process.

Keep issue-level overrides possible through the existing `assigneeAdapterOverrides` shape in `packages/shared/src/types/issue.ts`.

## Responsibilities by Layer

### Paperclip Core

Paperclip core should:

- resolve the base project workspace for the issue
- resolve or request an execution workspace
- resolve or request workspace runtime services when configured
- inject execution workspace metadata into run context
- persist enough metadata for board visibility and cleanup
- manage lifecycle hooks around run start/finish where needed

Paperclip core should not:

- require worktrees for all agents
- assume every adapter is local and git-backed
- assume every runtime service is a localhost process with a PID
- encode tool-specific worktree prompts as core product behavior

### Shared Local Runtime Helper

A shared server-side helper should handle local git mechanics:

- validate repo root
- create/select branch
- create/select git worktree
- allocate a free port
- optionally start and track a dev server
- return `{ cwd, branchName, url }`

This helper can be reused by:

- `codex_local`
- `claude_local`
- future local adapters like Cursor/OpenCode equivalents

This helper is intentionally for local adapters only. Remote adapters should not be forced through a host-local git helper.

### Shared Runtime Service Manager

In addition to the local git helper, Paperclip should define a generic runtime service manager contract.

Its job is to:

- decide whether a configured service should be reused or started fresh
- allocate local ports when needed
- start and monitor local processes when the adapter/runtime realization is host-local
- record normalized service metadata for remote realizations
- run readiness checks
- surface service URLs and state to the board
- apply shutdown policy

This manager should not be hard-coded to "dev servers". It should work for any long-lived workspace companion process.

### Adapter

The adapter should:

- accept the resolved execution cwd
- or accept structured execution workspace intent when no host cwd is available
- accept structured workspace runtime service intent when service orchestration is delegated to the adapter
- launch its tool with adapter-specific flags
- keep its own session continuity semantics

For example:

- `codex_local`: run inside cwd, likely with `--cd` or process cwd
- `claude_local`: run inside cwd, optionally use `--worktree` when it helps
- remote sandbox adapter: create its own isolated workspace from repo/ref/branch intent and report the realized remote workspace metadata back to Paperclip

For runtime services:

- local adapter or shared host manager: start the local process and return host-local metadata
- remote adapter: create or reuse the remote preview/service and return normalized remote metadata

## Minimal Data Model Additions

Do not create a fully first-class `worktrees` table yet.

Start smaller by recording derived execution workspace metadata on runs, issues, or both.

Suggested fields to introduce:

- `executionWorkspaceStrategy`
- `executionWorkspaceCwd`
- `executionBranchName`
- `executionWorkspaceStatus`
- `executionServiceRefs`
- `executionCleanupStatus`

These can live first on `heartbeat_runs.context_snapshot` or adjacent run metadata, with an optional later move into a dedicated table if the UI and cleanup workflows justify it.

For runtime services specifically, Paperclip should eventually track normalized fields such as:

- `serviceName`
- `serviceKind`
- `scopeType`
- `scopeId`
- `status`
- `command`
- `cwd`
- `envFingerprint`
- `port`
- `url`
- `provider`
- `providerRef`
- `startedByRunId`
- `ownerAgentId`
- `lastUsedAt`
- `stopPolicy`
- `healthStatus`

The first implementation can keep this in run metadata if needed, but the long-term shape is a generic runtime service registry rather than one-off server URL fields.

## Concrete Implementation Plan

## Phase 1: Define Shared Contracts

1. Introduce a shared execution workspace strategy contract in `packages/shared`.
2. Add adapter-config schema support for:
   - `workspaceStrategy.type`
   - `baseRef`
   - `branchTemplate`
   - `worktreeParentDir`
   - `cleanupPolicy`
   - optional workspace runtime service settings
3. Keep the existing `useProjectWorkspace` flag working as a lower-level compatibility control.
4. Distinguish local realization fields from generic intent fields so remote adapters are not forced to consume host cwd values.
5. Define a generic `workspaceRuntime.services[]` contract with:
   - service name
   - command or provider-managed intent
   - env overrides
   - readiness checks
   - exposure metadata
   - reuse scope
   - lifecycle
   - stop policy

Acceptance:

- adapter config can express `project_primary` and `git_worktree`
- config remains optional and backwards-compatible
- runtime services are expressed generically, not as Paperclip-only dev-server flags

## Phase 2: Resolve Execution Workspace in Heartbeat

1. Extend heartbeat workspace resolution so it can return a richer execution workspace result.
2. Keep current fallback order, but distinguish:
   - base project workspace
   - derived execution workspace
3. Inject resolved execution workspace details into `context.paperclipWorkspace` for local adapters and into a generic execution-workspace intent payload for adapters that need structured remote realization.
4. Resolve configured runtime service intent alongside the execution workspace so the adapter or host manager receives a complete workspace runtime contract.

Primary touchpoints:

- `server/src/services/heartbeat.ts`

Acceptance:

- runs still work unchanged when no strategy is configured
- the resolved context clearly indicates which strategy produced the cwd

## Phase 3: Add Shared Local Git Workspace Helper

1. Create a server-side helper module for local repo checkout strategies.
2. Implement `git_worktree` strategy:
   - validate git repo at base workspace cwd
   - derive branch name from issue
   - create or reuse a worktree path
   - detect collisions cleanly
3. Return structured metadata:
   - final cwd
   - branch name
   - worktree path
   - repo root

Acceptance:

- helper is reusable outside a single adapter
- worktree creation is deterministic for a given issue/config
- remote adapters remain unaffected by this helper

## Phase 4: Optional Dev Server Lifecycle

Rename this phase conceptually to **workspace runtime service lifecycle**.

1. Add optional runtime service startup on execution workspace creation.
2. Support both:
   - host-managed local services
   - adapter-managed remote services
3. For local services:
   - allocate a free port before launch when required
   - start the configured command in the correct cwd
   - run readiness checks
   - register the realized metadata
4. For remote services:
   - let the adapter return normalized service metadata after provisioning
   - do not assume PID or localhost access
5. Post or update issue-visible metadata with the service URLs and labels.

Acceptance:

- runtime service startup remains opt-in
- failures produce actionable run logs and issue comments
- same embedded DB / Paperclip instance can be reused through env/config injection when appropriate
- remote service realizations are represented without pretending to be local processes

## Phase 5: Runtime Service Reuse, Tracking, and Shutdown

1. Introduce a generic runtime service registry.
2. Each service should be tracked with:
   - `scopeType`: `project_workspace | execution_workspace | run | agent`
   - `scopeId`
   - `serviceName`
   - `status`
   - `command` or provider metadata
   - `cwd` if local
   - `envFingerprint`
   - `port`
   - `url`
   - `provider` / `providerRef`
   - `ownerAgentId`
   - `startedByRunId`
   - `lastUsedAt`
   - `stopPolicy`
3. Introduce a deterministic `reuseKey`, for example:
   - `projectWorkspaceId + serviceName + envFingerprint`
4. Reuse policy:
   - if a healthy service with the same reuse key exists, attach to it
   - otherwise start a new service
5. Distinguish lifecycle classes:
   - `shared`: reusable across runs, usually scoped to `project_workspace`
   - `ephemeral`: tied to `execution_workspace` or `run`
6. Shutdown policy:
   - `run` scope: stop when run ends
   - `execution_workspace` scope: stop when workspace is cleaned up
   - `project_workspace` scope: stop on idle timeout, explicit stop, or workspace removal
   - `agent` scope: stop when ownership is transferred or agent policy requires it
7. Health policy:
   - readiness check at startup
   - periodic or on-demand liveness checks
   - mark unhealthy before killing when possible

Acceptance:

- Paperclip can decide whether to reuse or start a fresh service deterministically
- local and remote services share a normalized tracking model
- shutdown is policy-driven instead of implicit
- board can understand why a service was kept, reused, or stopped

## Phase 6: Adapter Integration

1. Update `codex_local` to consume resolved execution workspace cwd.
2. Update `claude_local` to consume resolved execution workspace cwd.
3. Define a normalized adapter contract for remote adapters that receive execution workspace intent instead of a host-local cwd.
4. Optionally allow Claude-specific optimization paths using native `--worktree`, but keep the shared server-side checkout strategy as canonical for local adapters.
5. Define how adapters return runtime service realizations:
   - local host-managed service reference
   - remote provider-managed service reference

Acceptance:

- adapter behavior remains unchanged when strategy is absent
- session resume remains cwd-safe
- no adapter is forced into git behavior
- remote adapters can implement equivalent isolation without pretending to be local worktrees
- adapters can report service URLs and lifecycle metadata in a normalized shape

## Phase 7: Visibility and Issue Comments

1. Expose execution workspace metadata in run details and optionally issue detail UI:
   - strategy
   - cwd
   - branch
   - runtime service refs
2. Expose runtime services with:
   - service name
   - status
   - URL
   - scope
   - owner
   - health
3. Add standard issue comment output when a worktree-backed or remotely isolated run starts:
   - branch
   - worktree path
   - service URLs if present

Acceptance:

- board can see where the agent is working
- board can see what runtime services exist for that workspace
- issue thread becomes the handoff surface for branch names and reachable URLs

## Phase 8: Cleanup Policies

1. Implement cleanup policies:
   - `manual`
   - `on_done`
   - `on_merged`
2. For worktree cleanup:
   - stop tracked runtime services if owned by the workspace lifecycle
   - remove worktree
   - optionally delete local branch after merge
3. Start with conservative defaults:
   - do not auto-delete anything unless explicitly configured

Acceptance:

- cleanup is safe and reversible by default
- merge-based cleanup can be introduced after basic lifecycle is stable

## Phase 9: Auth Ergonomics Follow-Up

This is related, but should be tracked separately from the workspace strategy work.

Needed improvement:

- make manual local agent bootstrap in authenticated/private mode easier, so operators can become `codexcoder` or `claudecoder` locally without depending on an already-established browser-auth CLI context

This should likely take the form of a local operator bootstrap flow, not a weakening of runtime auth boundaries.

## Rollout Strategy

1. Ship the shared config contract and no-op-compatible heartbeat changes first.
2. Pilot with `codexcoder` and `claudecoder` only.
3. Test against Paperclip-on-Paperclip workflows first.
4. Keep `project_primary` as the default for all existing agents.
5. Add UI exposure and cleanup only after the core runtime path is stable.

## Acceptance Criteria

1. Worktree behavior is optional, not a global requirement.
2. Project workspaces remain the canonical repo anchor.
3. Local coding agents can opt into isolated issue-scoped execution workspaces.
4. The same model works for both `codex_local` and `claude_local` without forcing a tool-specific abstraction into core.
5. Remote adapters can consume the same execution workspace intent without requiring host-local filesystem access.
6. Session continuity remains correct because each adapter resumes relative to its realized execution workspace.
7. Workspace runtime services are modeled generically, not as Paperclip-specific dev-server toggles.
8. Board users can see branch/path/URL information for worktree-backed or remotely isolated runs.
9. Service reuse and shutdown are deterministic and policy-driven.
10. Cleanup is conservative by default.

## Recommended Initial Scope

To keep this tractable, the first implementation should:

- support only local coding adapters
- support only `project_primary` and `git_worktree`
- avoid a new dedicated database table for worktrees
- start with a single host-managed runtime service implementation path
- postpone merge-driven cleanup automation until after basic start/run/visibility is proven

That is enough to validate the local product shape without prematurely freezing the wrong abstraction.

Follow-up expansion after that validation:

- define the remote adapter contract for adapter-managed isolated checkouts
- add one cloud/sandbox adapter implementation path
- normalize realized metadata so local and remote execution workspaces appear similarly in the UI
- expand the runtime service registry from local host-managed services to remote adapter-managed services