From 1103da339cb5b40a6bb758fbd725e98118186125 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 24 Feb 2026 16:31:47 +0000 Subject: [PATCH] feat: add full creative studio + DevOps tools (Pixel, Lyra, Reel personas) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds 3 new personas (Pixel, Lyra, Reel) and 5 new tool modules: - Git/DevOps tools (GitPython): clone, status, diff, log, blame, branch, add, commit, push, pull, stash — wired to Forge and Helm personas - Image generation (FLUX via diffusers): text-to-image, storyboards, variations — Pixel persona - Music generation (ACE-Step 1.5): full songs with vocals+instrumentals, instrumental tracks, vocal-only tracks — Lyra persona - Video generation (Wan 2.1 via diffusers): text-to-video, image-to-video clips — Reel persona - Creative Director pipeline: multi-step orchestration that chains storyboard → music → video → assembly into 3+ minute final videos - Video assembler (MoviePy + FFmpeg): stitch clips, overlay audio, title cards, subtitles, final export Also includes: - Spark Intelligence tool-level + creative pipeline event capture - Creative Studio dashboard page (/creative/ui) with 4 tabs - Config settings for all new models and output directories - pyproject.toml creative optional extra for GPU dependencies - 107 new tests covering all modules (624 total, all passing) https://claude.ai/code/session_01KJm6jQkNi3aA3yoQJn636c --- PLAN.md | 478 ++++++++++++++++++++++++++ pyproject.toml | 14 + src/config.py | 22 ++ src/creative/__init__.py | 1 + src/creative/assembler.py | 300 ++++++++++++++++ src/creative/director.py | 378 ++++++++++++++++++++ src/dashboard/app.py | 2 + src/dashboard/routes/creative.py | 87 +++++ src/dashboard/templates/base.html | 1 + src/dashboard/templates/creative.html | 198 +++++++++++ src/spark/engine.py | 67 ++++ src/swarm/personas.py | 58 +++- src/swarm/tool_executor.py | 40 ++- src/timmy/tools.py | 107 +++++- src/tools/__init__.py | 1 + src/tools/git_tools.py | 281 +++++++++++++++ src/tools/image_tools.py | 171 +++++++++ src/tools/music_tools.py | 210 +++++++++++ src/tools/video_tools.py | 206 +++++++++++ tests/test_assembler.py | 69 ++++ tests/test_creative_director.py | 190 ++++++++++ tests/test_creative_route.py | 61 ++++ tests/test_dashboard_routes.py | 4 +- tests/test_git_tools.py | 183 ++++++++++ tests/test_image_tools.py | 120 +++++++ tests/test_music_tools.py | 124 +++++++ tests/test_spark_tools_creative.py | 110 ++++++ tests/test_swarm_personas.py | 10 +- tests/test_video_tools.py | 93 +++++ 29 files changed, 3573 insertions(+), 13 deletions(-) create mode 100644 PLAN.md create mode 100644 src/creative/__init__.py create mode 100644 src/creative/assembler.py create mode 100644 src/creative/director.py create mode 100644 src/dashboard/routes/creative.py create mode 100644 src/dashboard/templates/creative.html create mode 100644 src/tools/__init__.py create mode 100644 src/tools/git_tools.py create mode 100644 src/tools/image_tools.py create mode 100644 src/tools/music_tools.py create mode 100644 src/tools/video_tools.py create mode 100644 tests/test_assembler.py create mode 100644 tests/test_creative_director.py create mode 100644 tests/test_creative_route.py create mode 100644 tests/test_git_tools.py create mode 100644 tests/test_image_tools.py create mode 100644 tests/test_music_tools.py create mode 100644 tests/test_spark_tools_creative.py create mode 100644 tests/test_video_tools.py diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..a54ab1b --- /dev/null +++ b/PLAN.md @@ -0,0 +1,478 @@ +# Plan: Full Creative & DevOps Capabilities for Timmy + +## Overview + +Add five major capability domains to Timmy's agent system, turning it into a +sovereign creative studio and full-stack DevOps operator. All tools are +open-source, self-hosted, and GPU-accelerated where needed. + +--- + +## Phase 1: Git & DevOps Tools (Forge + Helm personas) + +**Goal:** Timmy can observe local/remote repos, read code, create branches, +stage changes, commit, diff, log, and manage PRs — all through the swarm +task system with Spark event capture. + +### New module: `src/tools/git_tools.py` + +Tools to add (using **GitPython** — BSD-3, `pip install GitPython`): + +| Tool | Function | Persona Access | +|---|---|---| +| `git_clone` | Clone a remote repo to local path | Forge, Helm | +| `git_status` | Show working tree status | Forge, Helm, Timmy | +| `git_diff` | Show staged/unstaged diffs | Forge, Helm, Timmy | +| `git_log` | Show recent commit history | Forge, Helm, Echo, Timmy | +| `git_branch` | List/create/switch branches | Forge, Helm | +| `git_add` | Stage files for commit | Forge, Helm | +| `git_commit` | Create a commit with message | Forge, Helm | +| `git_push` | Push to remote | Forge, Helm | +| `git_pull` | Pull from remote | Forge, Helm | +| `git_blame` | Show line-by-line authorship | Forge, Echo | +| `git_stash` | Stash/pop changes | Forge, Helm | + +### Changes to existing files + +- **`src/timmy/tools.py`** — Add `create_git_tools()` factory, wire into + `PERSONA_TOOLKITS` for Forge and Helm +- **`src/swarm/tool_executor.py`** — Enhance `_infer_tools_needed()` with + git keywords (commit, branch, push, pull, diff, clone, merge) +- **`src/config.py`** — Add `git_default_repo_dir: str = "~/repos"` setting +- **`src/spark/engine.py`** — Add `on_tool_executed()` method to capture + individual tool invocations (not just task-level events) +- **`src/swarm/personas.py`** — Add git-related keywords to Forge and Helm + preferred_keywords + +### New dependency + +```toml +# pyproject.toml +dependencies = [ + ..., + "GitPython>=3.1.40", +] +``` + +### Dashboard + +- **`/tools`** page updated to show git tools in the catalog +- Git tool usage stats visible per agent + +### Tests + +- `tests/test_git_tools.py` — test all git tool functions against tmp repos +- Mock GitPython's `Repo` class for unit tests + +--- + +## Phase 2: Image Generation (new "Pixel" persona) + +**Goal:** Generate storyboard frames and standalone images from text prompts +using FLUX.2 Klein 4B locally. + +### New persona: Pixel — Visual Architect + +```python +"pixel": { + "id": "pixel", + "name": "Pixel", + "role": "Visual Architect", + "description": "Image generation, storyboard frames, and visual design.", + "capabilities": "image-generation,storyboard,design", + "rate_sats": 80, + "bid_base": 60, + "bid_jitter": 20, + "preferred_keywords": [ + "image", "picture", "photo", "draw", "illustration", + "storyboard", "frame", "visual", "design", "generate", + "portrait", "landscape", "scene", "artwork", + ], +} +``` + +### New module: `src/tools/image_tools.py` + +Tools (using **diffusers** + **FLUX.2 Klein 4B** — Apache 2.0): + +| Tool | Function | +|---|---| +| `generate_image` | Text-to-image generation (returns file path) | +| `generate_storyboard` | Generate N frames from scene descriptions | +| `image_variations` | Generate variations of an existing image | + +### Architecture + +``` +generate_image(prompt, width=1024, height=1024, steps=4) + → loads FLUX.2 Klein via diffusers FluxPipeline + → saves to data/images/{uuid}.png + → returns path + metadata +``` + +- Model loaded lazily on first use, kept in memory for subsequent calls +- Falls back to CPU generation (slower) if no GPU +- Output saved to `data/images/` with metadata JSON sidecar + +### New dependency (optional extra) + +```toml +[project.optional-dependencies] +creative = [ + "diffusers>=0.30.0", + "transformers>=4.40.0", + "accelerate>=0.30.0", + "torch>=2.2.0", + "safetensors>=0.4.0", +] +``` + +### Config + +```python +# config.py additions +flux_model_id: str = "black-forest-labs/FLUX.2-klein-4b" +image_output_dir: str = "data/images" +image_default_steps: int = 4 +``` + +### Dashboard + +- `/creative/ui` — new Creative Studio page (image gallery + generation form) +- HTMX-powered: submit prompt, poll for result, display inline +- Gallery view of all generated images with metadata + +### Tests + +- `tests/test_image_tools.py` — mock diffusers pipeline, test prompt handling, + file output, storyboard generation + +--- + +## Phase 3: Music Generation (new "Lyra" persona) + +**Goal:** Generate full songs with vocals, instrumentals, and lyrics using +ACE-Step 1.5 locally. + +### New persona: Lyra — Sound Weaver + +```python +"lyra": { + "id": "lyra", + "name": "Lyra", + "role": "Sound Weaver", + "description": "Music and song generation with vocals, instrumentals, and lyrics.", + "capabilities": "music-generation,vocals,composition", + "rate_sats": 90, + "bid_base": 70, + "bid_jitter": 20, + "preferred_keywords": [ + "music", "song", "sing", "vocal", "instrumental", + "melody", "beat", "track", "compose", "lyrics", + "audio", "sound", "album", "remix", + ], +} +``` + +### New module: `src/tools/music_tools.py` + +Tools (using **ACE-Step 1.5** — Apache 2.0, `pip install ace-step`): + +| Tool | Function | +|---|---| +| `generate_song` | Text/lyrics → full song (vocals + instrumentals) | +| `generate_instrumental` | Text prompt → instrumental track | +| `generate_vocals` | Lyrics + style → vocal track | +| `list_genres` | Return supported genre/style tags | + +### Architecture + +``` +generate_song(lyrics, genre="pop", duration=120, language="en") + → loads ACE-Step model (lazy, cached) + → generates audio + → saves to data/music/{uuid}.wav + → returns path + metadata (duration, genre, etc.) +``` + +- Model loaded lazily, ~4GB VRAM minimum +- Output saved to `data/music/` with metadata sidecar +- Supports 19 languages, genre tags, tempo control + +### New dependency (optional extra, extends `creative`) + +```toml +[project.optional-dependencies] +creative = [ + ..., + "ace-step>=1.5.0", +] +``` + +### Config + +```python +music_output_dir: str = "data/music" +ace_step_model: str = "ace-step/ACE-Step-v1.5" +``` + +### Dashboard + +- `/creative/ui` expanded with Music tab +- Audio player widget (HTML5 `