# Open-Source Text-to-Music-Video Pipeline Research

## Executive Summary

**The complete text-to-music-video pipeline does NOT exist as a single open-source tool.** The landscape consists of powerful individual components that must be manually stitched together. This is the gap our Video Forge can fill.

---

## 1. EXISTING OPEN-SOURCE PIPELINES

### Complete (but crude) Pipelines

| Project | Stars | Description | Status |
|---------|-------|-------------|--------|
| **MusicVideoMaker** | 3 | Stable Diffusion pipeline for music videos from lyrics. Uses Excel spreadsheet for lyrics+timing, generates key frames, smooths between them. | Proof-of-concept, Jupyter notebook, not production-ready |
| **DuckTapeVideos** | 0 | Node-based AI pipeline for beat-synced music videos from lyrics | Minimal, early stage |
| **song-video-gen** | 0 | Stable Diffusion lyrics-based generative AI pipeline | Fork/copy of above |
| **TikTok-Lyric-Video-Pipeline** | 1 | Automated Python pipeline for TikTok lyric videos (10-15/day) | Focused on lyric overlay, not generative visuals |

**Verdict: Nothing production-ready exists as a complete pipeline.**

---

## 2. INDIVIDUAL COMPONENTS (What's Already Free)

### A. Music Generation (Suno Alternatives)

| Project | Stars | License | Self-Hostable | Quality |
|---------|-------|---------|---------------|---------|
| **YuE** | 6,144 | Apache-2.0 | ✅ Yes | Full-song generation with vocals, Suno-level quality |
| **HeartMuLa** | 4,037 | Apache-2.0 | ✅ Yes | Most powerful open-source music model (2026), multilingual |
| **ACE-Step 1.5 + UI** | 970 | MIT | ✅ Yes | Professional Spotify-like UI, full song gen, 4+ min with vocals |
| **Facebook MusicGen** | ~45k downloads | MIT | ✅ Yes | Good quality, melody conditioning, well-documented |
| **Riffusion** | ~6k stars | Apache-2.0 | ✅ Yes | Spectrogram-based, unique approach |

**Status: Suno is effectively "given away" for free. YuE and HeartMuLa are production-ready.**

### B. Image Generation (Per-Scene/Beat)

| Project | Downloads/Stars | License | Notes |
|---------|-----------------|---------|-------|
| **Stable Diffusion XL** | 1.9M downloads | CreativeML | Best quality, huge ecosystem |
| **Stable Diffusion 1.5** | 1.6M downloads | CreativeML | Fast, lightweight |
| **FLUX** | Emerging | Apache-2.0 | Newest, excellent quality |
| **ComfyUI** | 60k+ stars | GPL-3.0 | Node-based pipeline editor, massive plugin ecosystem |

**Status: Image generation is completely "given away." SD XL + ComfyUI is production-grade.**

### C. Text-to-Video Generation

| Project | Stars | License | Capabilities |
|---------|-------|---------|--------------|
| **Wan2.1** | 15,815 | Apache-2.0 | State-of-the-art, text-to-video and image-to-video |
| **CogVideoX** | 12,634 | Apache-2.0 | Text and image to video, good quality |
| **HunyuanVideo** | 11,965 | Custom | Tencent's framework, high quality |
| **Stable Video Diffusion** | 3k+ likes | Stability AI | Image-to-video, good for short clips |
| **LTX-Video** | Growing | Apache-2.0 | Fast inference, good quality |

**Status: Text-to-video is rapidly being "given away." Wan2.1 is production-ready for short clips (4-6 seconds).**

### D. Video Composition & Assembly

| Project | Stars | License | Use Case |
|---------|-------|---------|----------|
| **Remotion** | 43,261 | Custom (SSPL) | Programmatic video with React, production-grade |
| **MoviePy** | 12k+ stars | MIT | Python video editing, widely used |
| **Mosaico** | 16 | MIT | Python video composition with AI integration |
| **FFmpeg** | N/A | LGPL/GPL | The universal video tool |

**Status: Video composition tools are mature and free. Remotion is production-grade.**

### E. Lyrics/Text Processing

| Component | Status | Notes |
|-----------|--------|-------|
| **Lyrics-to-scene segmentation** | ❌ Missing | No good open-source tool for breaking lyrics into visual scenes |
| **Beat detection** | ✅ Exists | Librosa, madmom, aubio - all free and mature |
| **Text-to-prompt generation** | ✅ Exists | LLMs (Ollama, local models) can do this |
| **LRC/SRT parsing** | ✅ Exists | Many libraries available |

---

## 3. WHAT'S BEEN "GIVEN AWAY" FOR FREE

### Fully Solved (Production-Ready, Self-Hostable)
- ✅ **Music generation**: YuE, HeartMuLa, ACE-Step match Suno quality
- ✅ **Image generation**: SD XL, FLUX - commercial quality
- ✅ **Video composition**: FFmpeg, MoviePy, Remotion
- ✅ **Beat/audio analysis**: Librosa, madmom
- ✅ **Text-to-video (short clips)**: Wan2.1, CogVideoX
- ✅ **TTS/voice**: XTTS-v2, Kokoro, Bark

### Partially Solved
- ⚠️ **Image-to-video**: Good for 4-6 second clips, struggles with longer sequences
- ⚠️ **Style consistency**: LoRAs and ControlNet help, but not perfect across scenes
- ⚠️ **Prompt engineering**: LLMs can help, but no dedicated lyrics-to-visual-prompt tool

---

## 4. WHERE THE REAL GAPS ARE

### Critical Gaps (Our Opportunity)

1. **Unified Pipeline Orchestration**
   - NO tool chains: lyrics → music → scene segmentation → image prompts → video composition
   - Everything requires manual stitching
   - Our Video Forge can be THE glue layer

2. **Lyrics-to-Visual-Scene Segmentation**
   - No tool analyzes lyrics and breaks them into visual beats/scenes
   - MusicVideoMaker uses manual Excel entry - absurd
   - Opportunity: LLM-powered scene segmentation with beat alignment

3. **Temporal Coherence Across Scenes**
   - Short clips (4-6s) work fine, but maintaining visual coherence across a 3-4 minute video is unsolved
   - Character consistency, color palette continuity, style drift
   - Opportunity: Style anchoring + scene-to-scene conditioning

4. **Beat-Synchronized Visual Transitions**
   - No tool automatically syncs visual cuts to musical beats
   - Manual timing is required everywhere
   - Opportunity: Beat detection → transition scheduling → FFmpeg composition

5. **Long-Form Video Generation**
   - Text-to-video models max out at 4-6 seconds
   - Stitching clips with consistent style/characters is manual
   - Opportunity: Automated clip chaining with style transfer

6. **One-Click "Lyrics In, Video Out"**
   - The dream pipeline doesn't exist
   - Current workflows require 5+ separate tools
   - Opportunity: Single command/endpoint that does everything

### Technical Debt in Existing Tools

- **YuE/HeartMuLa**: No video awareness - just audio generation
- **Wan2.1/CogVideoX**: No lyrics/text awareness - just prompt-to-video
- **ComfyUI**: Great for images, weak for video composition
- **Remotion**: Great for composition, no AI generation built-in

---

## 5. RECOMMENDED ARCHITECTURE FOR VIDEO FORGE

Based on this research, the optimal Video Forge pipeline:

```
[Lyrics/Poem Text]
        ↓
[LLM Scene Segmenter] → Beat-aligned scene descriptions + visual prompts
        ↓
[HeartMuLa/YuE] → Music audio (.wav)
        ↓
[Beat Detector (librosa)] → Beat timestamps + energy curve
        ↓
[SD XL / FLUX] → Scene images (one per beat/section)
        ↓
[Wan2.1 img2vid] → Short video clips per scene (4-6s each)
        ↓
[FFmpeg + Beat Sync] → Transitions aligned to beats
        ↓
[Final Music Video (.mp4)]
```

### Key Design Decisions

1. **Music**: HeartMuLa (best quality, multilingual, Apache-2.0)
2. **Images**: SD XL via ComfyUI (most mature ecosystem)
3. **Video clips**: Wan2.1 for img2vid (state-of-the-art)
4. **Composition**: FFmpeg (universal, battle-tested)
5. **Orchestration**: Python pipeline with config file
6. **Scene segmentation**: Local LLM (Ollama + Llama 3 or similar)

### What We Build vs. What We Use

| Component | Build or Use | Reasoning |
|-----------|--------------|-----------|
| Lyrics → Scenes | **BUILD** | No good tool exists, core differentiator |
| Music generation | **USE** HeartMuLa/YuE | Already excellent, Apache-2.0 |
| Image generation | **USE** SD XL | Mature, huge ecosystem |
| Beat detection | **USE** librosa | Mature, reliable |
| Video clips | **USE** Wan2.1 | Best quality, Apache-2.0 |
| Video composition | **BUILD** (ffmpeg wrapper) | Need beat-sync logic |
| Pipeline orchestration | **BUILD** | The main value-add |

---

## 6. COMPETITIVE LANDSCAPE SUMMARY

### Commercial (Not Self-Hostable)
- **Suno**: Music only, no video
- **Runway**: Video only, expensive
- **Pika**: Short clips only
- **Kaiber**: Closest to music video, but closed/subscription
- **Synthesia**: Avatar-based, not generative art

### Open-Source Gaps That Matter
1. Nobody has built the orchestration layer
2. Nobody has solved lyrics-to-visual-scene well
3. Nobody has beat-synced visual transitions automated
4. Nobody maintains temporal coherence across minutes

**Our Video Forge fills the most important gap: the glue that makes individual AI components work together to produce a complete music video from text.**

---

*Research conducted: April 14, 2026*
*Sources: GitHub API, HuggingFace API, project READMEs*
