From 76fb57877d71d47477aabc2106c716eae3fd0f25 Mon Sep 17 00:00:00 2001 From: Mikkel Georgsen Date: Wed, 4 Feb 2026 19:19:07 +0000 Subject: [PATCH] docs(02-01): complete persistent subprocess + telegram utils plan Tasks completed: 2/2 - Refactor ClaudeSubprocess to persistent process with stream-json I/O - Create telegram_utils.py with message formatting and typing indicator SUMMARY: .planning/phases/02-telegram-integration/02-01-SUMMARY.md --- .planning/ROADMAP.md | 4 +- .planning/STATE.md | 45 +++--- .../02-telegram-integration/02-01-SUMMARY.md | 148 ++++++++++++++++++ 3 files changed, 171 insertions(+), 26 deletions(-) create mode 100644 .planning/phases/02-telegram-integration/02-01-SUMMARY.md diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index a9a7462..4362344 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -50,7 +50,7 @@ Plans: **Plans:** 2 plans Plans: -- [ ] 02-01-PLAN.md -- Persistent subprocess engine + message formatting utilities +- [x] 02-01-PLAN.md -- Persistent subprocess engine + message formatting utilities - [ ] 02-02-PLAN.md -- Bot integration with batching, file handling, and systemd service ### Phase 3: Lifecycle Management @@ -89,6 +89,6 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 | Phase | Plans Complete | Status | Completed | |-------|----------------|--------|-----------| | 1. Session & Process Foundation | 3/3 | Complete | 2026-02-04 | -| 2. Telegram Integration | 0/2 | Not started | - | +| 2. Telegram Integration | 1/2 | In progress | - | | 3. Lifecycle Management | 0/TBD | Not started | - | | 4. Output Modes | 0/TBD | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index 53b80be..1f0ecf1 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -5,33 +5,34 @@ See: .planning/PROJECT.md (updated 2026-02-04) **Core value:** Frictionless conversation with Claude Code from anywhere via Telegram — no SSH, no manual inbox checking, just message and get a response. -**Current focus:** Phase 1 (Session & Process Foundation) +**Current focus:** Phase 2 (Telegram Integration) ## Current Position -Phase: 1 of 4 (Session & Process Foundation) -Plan: 01-03 complete (3 of 3 plans completed) -Status: Phase 1 complete -Last activity: 2026-02-04 — Completed 01-03-PLAN.md (Bot command integration) +Phase: 2 of 4 (Telegram Integration) +Plan: 02-01 complete (1 of 2 plans completed) +Status: In progress +Last activity: 2026-02-04 — Completed 02-01-PLAN.md (Persistent subprocess + telegram utils) -Progress: [███░░░░░░░░░░░░] 25% +Progress: [████░░░░░░░░░░░] 33% ## Performance Metrics **Velocity:** -- Total plans completed: 3 -- Average duration: 10 min -- Total execution time: 0.50 hours +- Total plans completed: 4 +- Average duration: 8 min +- Total execution time: 0.53 hours **By Phase:** | Phase | Plans | Total | Avg/Plan | |-------|-------|-------|----------| | 1 | 3 | 27min | 9min | +| 2 | 1 | 5min | 5min | **Recent Trend:** -- Last 3 plans: 01-01 (3min), 01-02 (9min), 01-03 (15min) -- Trend: Increasing complexity (more orchestrator fixes) +- Last 3 plans: 01-02 (9min), 01-03 (15min), 02-01 (5min) +- Trend: Improved velocity (yolo mode + better planning) *Updated after each plan completion* @@ -53,6 +54,9 @@ Recent decisions affecting current work: - Callback architecture: Decouple subprocess from session management via on_output/on_error/on_complete/on_status (01-02) - Sibling imports over package imports: Avoids shadowing pip telegram package (01-03) - Archive sessions with tar+pigz: Compression + cleanup to sessions_archive/ (01-03) +- Persistent subprocess instead of fresh per turn: Eliminates ~1s spawn overhead, maintains context (02-01) +- Split messages at 4000 chars (not 4096): Leaves room for MarkdownV2 escape expansion (02-01) +- Never split inside code blocks: Track in_code_block state, only split when safe (02-01) ### Pending Todos @@ -64,21 +68,14 @@ None yet. - ~~Claude Code CLI --resume behavior with pipes vs PTY unknown~~ — RESOLVED: Research confirms pipes + stream-json is correct approach - ~~Output format for tool calls not documented~~ — RESOLVED: stream-json format documented and implemented -**Phase 2 (Persistent Processes) — BLOCKER:** -- **Non-persistent process model:** Current implementation spawns fresh `claude -p` per turn (~1s overhead each) - - Design goal was persistent processes that suspend when switching sessions - - This gap affects response latency and user experience - - Must be addressed before Phase 3 integration - - Requires refactoring ClaudeSubprocess to lifecycle-manage process instances - -**Phase 3 (Telegram Integration):** +**Phase 2 (Telegram Integration) — IN PROGRESS:** +- ~~Non-persistent process model (spawned fresh per turn)~~ — RESOLVED: Refactored to persistent subprocess with stream-json stdin (02-01) - Message batching strategy needs validation against actual Claude output patterns -- Optimal chunk split points require experimentation -- Streaming responses / typing indicators needed for UX during long API waits +- File upload flow (save to session dir, notify Claude) needs end-to-end testing ## Session Continuity -Last session: 2026-02-04T18:00:00Z -Stopped at: Completed 01-03-PLAN.md (Bot command integration) +Last session: 2026-02-04T19:17:24Z +Stopped at: Completed 02-01-PLAN.md (Persistent subprocess + telegram utils) Resume file: None -Next: Phase 2 ready (persistent process model needed) +Next: 02-02-PLAN.md (Bot integration with batching, file handling, systemd service) diff --git a/.planning/phases/02-telegram-integration/02-01-SUMMARY.md b/.planning/phases/02-telegram-integration/02-01-SUMMARY.md new file mode 100644 index 0000000..0fe2963 --- /dev/null +++ b/.planning/phases/02-telegram-integration/02-01-SUMMARY.md @@ -0,0 +1,148 @@ +--- +phase: 02-telegram-integration +plan: 01 +subsystem: infra +tags: [asyncio, subprocess, python, claude-code-cli, stream-json, telegram, markdown, typing-indicator] + +# Dependency graph +requires: + - phase: 01-session-process-foundation + provides: Session management and subprocess foundations +provides: + - Persistent Claude Code subprocess with stream-json I/O (eliminates ~1s spawn overhead per turn) + - Tool call progress notifications via on_tool_use callback + - Smart message splitting respecting MarkdownV2 code block boundaries + - MarkdownV2 escaping with code block awareness + - Typing indicator loop for long operations +affects: [02-02, bot-integration, message-handling] + +# Tech tracking +tech-stack: + added: + - telegram.constants.ChatAction (typing indicators) + patterns: + - "Persistent subprocess with NDJSON stdin streaming" + - "Stream-json event routing including tool_use events" + - "Smart message splitting at code block boundaries" + - "Context-sensitive MarkdownV2 escaping (preserves code blocks)" + - "Typing indicator loop with asyncio.Event cancellation" + +key-files: + created: + - telegram/telegram_utils.py + modified: + - telegram/claude_subprocess.py + +key-decisions: + - "Use persistent subprocess instead of fresh process per turn (eliminates ~1s overhead)" + - "Write NDJSON to stdin with drain() to prevent pipe buffer deadlock" + - "Persistent readers run for process lifetime, result events mark not busy but don't exit loop" + - "Parse content_block_start events for tool_use to enable progress notifications" + - "Split messages at 4000 chars (not 4096) to leave room for escape character expansion" + - "Never split inside code blocks (track in_code_block state while iterating lines)" + - "Escape MarkdownV2 special chars only outside code blocks (regex pattern to identify code regions)" + +patterns-established: + - "Pattern 1: Persistent stdin streaming - start() spawns process, send_message() writes NDJSON, readers run until process dies" + - "Pattern 2: Tool use notifications - Parse content_block_start with type=tool_use, extract name and input dict, pass to callback" + - "Pattern 3: Smart message splitting - Track code block state, split only when NOT in code block and would exceed limit" + - "Pattern 4: Code-aware escaping - Split by code regions (regex), escape only non-code text, rejoin" + - "Pattern 5: Typing loop - Re-send every 4s via asyncio.wait_for timeout until Event is set" + +# Metrics +duration: 5min +completed: 2026-02-04 +--- + +# Phase 2 Plan 1: Persistent Subprocess + Telegram Utils Summary + +**Persistent Claude Code subprocess with stream-json I/O and Telegram message formatting utilities for bot integration** + +## Performance + +- **Duration:** 5 minutes +- **Started:** 2026-02-04T19:12:28Z +- **Completed:** 2026-02-04T19:17:24Z +- **Tasks:** 2 +- **Files created:** 1 +- **Files modified:** 1 + +## Accomplishments + +- Refactored ClaudeSubprocess from fresh-process-per-turn to persistent process model +- Eliminated ~1s spawn overhead per message turn +- Added stream-json stdin I/O for NDJSON message delivery to persistent process +- Implemented tool_use event parsing and on_tool_use callback for progress notifications +- Created telegram_utils.py with smart message splitting, MarkdownV2 escaping, and typing indicator loop +- Smart splitting respects code block boundaries (never splits inside triple-backtick blocks) +- MarkdownV2 escaping preserves code blocks while escaping 17 special characters outside code +- Typing indicator loop re-sends every 4 seconds with clean asyncio.Event cancellation + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Refactor ClaudeSubprocess to persistent process** - `6a115a4` (refactor) +2. **Task 2: Create telegram_utils.py** - `6b624d7` (feat) + +## Files Created/Modified + +**Created:** +- `telegram/telegram_utils.py` - Message formatting utilities: split_message_smart (code-block-aware splitting), escape_markdown_v2 (context-sensitive escaping), typing_indicator_loop (4s re-send pattern) + +**Modified:** +- `telegram/claude_subprocess.py` - Refactored to persistent subprocess with --input-format stream-json stdin, persistent stdout/stderr readers, tool_use event parsing, NDJSON message delivery via stdin.write + drain pattern + +## Decisions Made + +1. **Persistent process architecture**: Spawn Claude Code once with stream-json I/O, write NDJSON to stdin for each turn. Eliminates ~1s spawn overhead per message and maintains conversation context across turns. + +2. **Tool use callback design**: Extract tool name and raw input dict from content_block_start events, pass directly to callback. Bot layer (Plan 02) will format human-readable progress messages. Keeps subprocess layer format-agnostic. + +3. **Smart splitting strategy**: Track code block state (in/out of triple backticks), split only when NOT in code block and would exceed 4000 char limit. If code block itself exceeds limit, include it whole (Telegram handles overflow). + +4. **Escape only outside code**: Use regex to identify code regions (```...``` and `...`), escape MarkdownV2 special chars only in non-code text. Prevents breaking code syntax while properly escaping user-facing text. + +5. **4000 char split threshold**: Use 4000 instead of Telegram's 4096 limit to leave room for MarkdownV2 escape character expansion (each special char becomes 2 chars with backslash). + +6. **Typing loop cancellation pattern**: Use asyncio.wait_for with 4s timeout on Event.wait(). Clean cancellation when Event is set, automatic re-send on timeout. Pattern from PTB community best practices. + +## Deviations from Plan + +None - plan executed exactly as written. + +## Issues Encountered + +**Issue 1: ChatAction import path** +- **Problem:** Initial import `from telegram import ChatAction` failed with ImportError +- **Root cause:** python-telegram-bot v22+ moved ChatAction to telegram.constants +- **Resolution:** Changed to `from telegram.constants import ChatAction` +- **Impact:** Minor, caught during testing, fixed immediately + +## User Setup Required + +None - no external service configuration required. + +## Next Phase Readiness + +**Ready for Plan 02 (Bot Integration):** +- ClaudeSubprocess provides persistent process with tool call notifications +- telegram_utils provides all message formatting needed for bot layer +- Smart splitting and escaping handle Claude's output for Telegram delivery +- Typing indicator loop ready for integration with message handlers +- stdin.write + drain pattern ensures no pipe deadlock + +**Integration points for Plan 02:** +- Call `subprocess.start()` once per session +- Pass `on_tool_use` callback to format and send progress messages +- Use `split_message_smart()` on Claude output before sending to Telegram +- Use `escape_markdown_v2()` if sending with MarkdownV2 parse mode +- Wrap `subprocess.send_message()` with `typing_indicator_loop()` background task +- Handle file uploads by saving to session directory +- Implement message batching with debounce timer + +**No blockers or concerns.** + +--- +*Phase: 02-telegram-integration* +*Completed: 2026-02-04*